[MoebiusX]

The repo and folder crawler now learns metrics from prebuilt exporters and from an OTel collector's scrape jobs. The motivating case: a service runs IBM's mq-metric-samples exporter (its repo is a Dockerfile plus config.yaml), and an OTel collector scrapes a fleet of those exporters and ships to Kafka. Before this change, crawling those repos yielded 0 metrics and 0 scrape jobs.

What changes

Metrics from a prebuilt exporter (tools/lib/exporter-metrics.mjs, new and listed), strongest first:

Source	Files	origin_kind	Confidence
metrics manifest	metrics-manifest.json, *.metrics-manifest.json	exporter-manifest	declared
saved exposition	*.prom, *.metrics (never under a test, fixture or mock folder)	exporter-snapshot	declared
exporter config × catalogue	YAML whose top-level keys match a catalogue entry	exporter-config	inferred (candidate)

• Ranking: metricOriginRank puts the manifest first and the inferred config last. The relative order of the existing origin kinds is unchanged.
• Snapshots: label names are kept, label values never. A family is named as Prometheus stores it (an OpenMetrics jobs counter is jobs_total).
• Catalogue: the family catalogue is data, not engine code (tools/lib/exporter-catalogue.mjs, with a small built-in IBM MQ entry). A downstream passes a larger one with packc x-ray --exporter-catalogue <file.json> or crawlFiles(files, { exporterCatalogue }). Its entries replace or extend the built-in ones. The format is in docs/DOWNSTREAM.md §18.
• Annotation: crawler.discovered.exporters lists each exporter and its sources.

An OTel collector's scrape jobs:

• The prometheus receivers' config.scrape_configs are read, with or without service.pipelines. They are found in a config file, a Helm ConfigMap, or a chart's values*.yaml (top level, config:, or an umbrella subchart's <key>.config: unless that subchart is disabled).
• Each job's crawler.discovered.scrape_job_origins entry adds via, receiver, target_count, and the exporters its samples leave by: a kafka exporter's topic, or any other exporter's endpoint without userinfo, query or fragment.
• The collector's metrics exporter (e.g. kafka) replaces the prometheusremotewrite scaffold in spec.pipelines.exporters.metrics, which keeps its { kind } shape.
• A pipeline-less config's exporter is only a guess. It is settled after the walk, so a pipeline declared in another --config file wins.

Environment scope: a manifest and a snapshot are read in every environment. A file that the scope excludes and that carries scrape jobs or an exporter config is named in a warning, together with the way to read it (--env all).

packc emit --exporter <url> (tools/emit-manifest.mjs) scrapes a /metrics endpoint once and writes metrics-manifest.json; --from <file.prom> reads a saved one instead.

• A redirect, a URL with credentials, a non-200 status, a body that is cut off, and an empty response are each refused with a way out that works when followed literally.
• No credential or query value is ever echoed.

GitHub crawl: fetches manifests with the files it already fetched. Saved expositions, config.yaml files and a chart's files/*.yaml only fill whatever the 200-file budget leaves over.

Declared default changes

• SCAN_EXT adds .prom and .metrics.
• Collector scrape jobs are read.
• A collector config with no service block is classified as a collector.
• A collector config inside a values file is walked.
• A null pipeline is skipped instead of failing the whole file (the opentelemetry-collector chart uses null to drop a pipeline).
• The environment-scope warning is new.

Every input with no exporter or collector-scrape evidence produces byte-identical output: all goldens (crawl, compile, diff, snapshot, audit report, board) are unchanged.

Review

Four adversarial verification rounds ran against this branch, and every reproduced finding is fixed and pinned in tools/test-crawl-exporters.mjs.

• Findings included credential leaks through safeSourceUrl, regressions in existing output, and a malformed values file losing its images.
• The last round generated 81M safeSourceUrl inputs, ran 97k crawls and 29k packc emit runs, and found 0 leaks and 0 throws.
• A byte-identity diff against develop over 220 captured inputs differs only where there is exporter or collector-scrape evidence.

Checks (local, at the head)

• npm run lint: 0 errors, 153 warnings, no new file+rule+message key.
• npm test: passes on Node 22.22, on Node 22.16, and on the floor run (no Playwright).
• Goldens: test:golden, :compile, :diff, :snapshot, :audit-report, and the golden board.
• vendor-manifest:check (in sync at every commit), sync-spec --check, test-doc-test-totals, test-platform.
• Prestore strict and backend-live strict.

🤖 Generated with Claude Code

https://claude.ai/code/session_01UhPGnqVJ24goSVtbKpdkLJ

---

Generated by Claude Code