crawler: learn metrics from prebuilt exporters and OTel collector scrape jobs

README.md

@@ -1289,9 +1289,15 @@ The crawler reads source files such as:
 - Grafana unified-alerting provisioning YAML and Loki rule files
 - Grafana dashboard JSON
 - Alertmanager config
-- OTel Collector config
+- OTel Collector config, its prometheus receivers' scrape jobs included (with
+  or without a `service:` block)
 - Helm and Kubernetes manifests
 - Docker Compose files
+- A prebuilt exporter's metrics: its `metrics-manifest.json`, a saved
+  `/metrics` (`.prom`, `.metrics`), or its config read against an exporter
+  catalogue (`--exporter-catalogue <file.json>` adds or replaces entries).
+  `packc emit --exporter <url>` writes the manifest from a live endpoint
+  ([`docs/DOWNSTREAM.md`](docs/DOWNSTREAM.md) §18)
 
 It emits a canonical v1.4 pack plus crawler annotations describing what was
 scanned and what was inferred. A fresh crawl validates (exit 0) and needs no

VENDOR-MANIFEST.json

@@ -378,8 +378,8 @@
       "changedSinceRelease": false
     },
     "tools/lib/crawler.mjs": {
-      "sha256": "6705a45caf5253caf795b0c23b5d1ea8cdbe75d8f91b4d748fec20911265c16f",
-      "bytes": 110301,
+      "sha256": "d0387b62b132f273fc622aaa7ca0411c3e90dc907db4bd07e4a98c89f552455e",
+      "bytes": 133296,
       "exports": [
         "SCAN_EXT",
         "SCAN_IGNORE_DIRS",
@@ -395,6 +395,7 @@
       "imports": [
         "tools/lib/alert-routes.mjs",
         "tools/lib/backend-products.mjs",
+        "tools/lib/exporter-metrics.mjs",
         "tools/lib/l2x.mjs",
         "tools/lib/mini-yaml.mjs",
         "tools/lib/promql.mjs",
@@ -510,6 +511,42 @@
       "npm": [],
       "changedSinceRelease": false
     },
+    "tools/lib/exporter-catalogue.mjs": {
+      "sha256": "594c0bb91e68a1a87a598ff57532d388e40f17a0133de6eec64db8e06c82cb9a",
+      "bytes": 5963,
+      "exports": [
+        "EXPORTER_CATALOGUE"
+      ],
+      "imports": [],
+      "npm": [],
+      "changedSinceRelease": true,
+      "releasedExports": null
+    },
+    "tools/lib/exporter-metrics.mjs": {
+      "sha256": "57be2804bd81a9c9c4666839540d0c22d00486f292a85d54b852226099a88471",
+      "bytes": 20774,
+      "exports": [
+        "EXPOSITION_TYPES",
+        "asResolvedCatalogue",
+        "inferExporterFamilies",
+        "isExpositionSnapshotPath",
+        "isMetricsManifestPath",
+        "isTestOrFixturePath",
+        "manifestFromExposition",
+        "matchExporterConfig",
+        "mayBeExporterConfig",
+        "parseExposition",
+        "readMetricsManifest",
+        "resolveExporterCatalogue",
+        "safeSourceUrl"
+      ],
+      "imports": [
+        "tools/lib/exporter-catalogue.mjs"
+      ],
+      "npm": [],
+      "changedSinceRelease": true,
+      "releasedExports": null
+    },
     "tools/lib/good-when.mjs": {
       "sha256": "f89b81ef3c6afc30a60453deee4cc659503ec16f720351fcb61991766337e474",
       "bytes": 2557,

docs/CHANGELOG.md

@@ -2,6 +2,18 @@
 
 ## Unreleased
 
+### Follow-up `exporter-metrics`: the crawler learns metrics from prebuilt exporters and from an OTel collector's scrape jobs
+- **A metrics manifest is an artefact** (`tools/lib/exporter-metrics.mjs`, new and listed: `readMetricsManifest`, `manifestFromExposition`, `parseExposition`, `safeSourceUrl`, `resolveExporterCatalogue`, `asResolvedCatalogue`, `matchExporterConfig`, `mayBeExporterConfig`, `inferExporterFamilies`, `isMetricsManifestPath`, `isExpositionSnapshotPath`, `isTestOrFixturePath`, `EXPOSITION_TYPES`). `metrics-manifest.json` and `*.metrics-manifest.json` are the kind `metrics-manifest`. Each entry is a metric with `origin_kind: exporter-manifest`, `confidence: declared`, and the exporter as its service. A file that is not a manifest (none of its text quoted: the JSON parser's message would quote its start), or entries without a name and an exposition type, are named in a warning. A manifest or exporter config saved with a leading BOM is read.
+- **A saved exposition is read** (**a declared default change**: `SCAN_EXT` adds `.prom` and `.metrics`, so the CLI walk, the studio's folder picker and a journey's crawl read them; `server/github-crawl.mjs` fetches a manifest with the files it fetched before, and a saved exposition, any `config.yaml` and a chart's `files/*.yaml` only after them (`isExporterCandidateFile`, `selectCrawlerFiles`): a monorepo's many `config.yaml` files never use up the 200-file budget its rule, Alertmanager or collector files had). The kind is `exposition-snapshot`: `# HELP` / `# TYPE` and the label names of the samples, label values never kept, `origin_kind: exporter-snapshot`. A family is named as Prometheus stores it: an OpenMetrics counter `jobs` is `jobs_total`, an info `build` is `build_info`. It is never read from a folder named by a test, spec, fixture, mock or test-data word, alone or joined to others by `-` / `_` (`test/`, `__fixtures__/`, `testFixtures/`, `__mocks__/`, `e2e/`, `integration-tests/`, `integrationTest/`, `unit_tests/`, `unittests/`, `testing/`, `test-data/`, `__snapshots__/`, …), a camelCase test source set (`componentTest/`, `smokeTest/`, `testUtils/`) or Maven's `src/it/`, nor from a `*.test.*`, `*.spec.*`, `*_test.*` or `*_spec.*` file (`isTestOrFixturePath`).
+- **An exporter's config is read against a family catalogue** (`tools/lib/exporter-catalogue.mjs`, new and listed: `EXPORTER_CATALOGUE`, data only, a small IBM mq-metric-samples entry). The kind is `exporter-config`, recognised by the top-level keys an entry names (`global`, `connection`, `objects`, `prometheus`) before the filename heuristics. It emits the families the config's gates (`useObjectStatus`, `usePublications`) and scopes (`objects.queues`, `objects.channels`) switch on, under its `prometheus.namespace`, as candidates: `origin_kind: exporter-config`, `confidence: inferred` (curation keeps it). A downstream's catalogue (`packc x-ray --exporter-catalogue <file.json>`, `crawlFiles(files, { exporterCatalogue })`, `detectArtefactKind(path, text, { exporterCatalogue })`) replaces a built-in entry with its id and adds the rest, in either form (`{ exporters: [...] }` or the array). A malformed entry, or a family gated or scoped on a name its entry does not define (`toString` included), is dropped, with a crawl warning from `crawlFiles` and the CLI (`asResolvedCatalogue` resolves a catalogue once). `docs/DOWNSTREAM.md` §18 has the format.
+- **The strongest origin speaks for a metric**: `metricOriginRank` reads exporter-manifest, exporter-snapshot, source code, source fragment, recording-rule output, PromQL reference, then exporter-config. The relative order of the existing kinds is unchanged.
+- **An OTel collector's scrape jobs are read** (**a declared default change**). A collector config's prometheus receivers (`prometheus`, `prometheus/<name>`) give their `config.scrape_configs` jobs, with or without `service.pipelines`. A config with no service block is classified by its receivers mapping (with scrape jobs or exporters), as a file, a Helm ConfigMap value or a whole template. A chart's `values*.yaml` that carries a collector config, at its top level, under `config:` (the opentelemetry-collector chart's idiom) or under a subchart's `<key>.config:` (an umbrella chart, unless the subchart is disabled: its `enabled` toggles merged across the in-scope values files, an overlay winning as with `helm -f`, so an overlay can disable or re-enable it), is walked as one too, receivers, processors, pipelines and all, before its images (**a declared default change** for such a values file). A collector config there that the walk cannot read is warned about and never costs the file its images. A pipeline set to `null` (the chart's way to drop one) is skipped, in a values file and in a plain collector config (**a declared default change**: such a config was `Failed to parse` and read nothing). A collector job's `crawler.discovered.scrape_job_origins` entry adds `via`, `receiver`, `target_count` and `exporters` (`{ name, kind, topic | endpoint }`, an endpoint without userinfo, query or fragment). A job from a standalone `prometheus.yml` keeps its four fields. With no pipelines, the metrics-capable exporter the config names (kafka, prometheusremotewrite, otlp, …; the first, with a warning, if several) is a guess, settled after every file is walked. A scrape job leaves by the exporters of its own file's metrics pipelines that list its receiver, else by those of the declared metrics pipelines in any other file that do (a config split across `--config` files), else by the guess. A collector job that cannot be read is warned about and skipped; its siblings and the rest of the config are read. The guess sets `spec.pipelines.exporters.metrics` (still `{ kind }`) in place of the `prometheusremotewrite` scaffold only when nothing else set it: no declared pipeline and no metrics backend image (those two fill the metrics exporter from whichever file the walk reads first, as before; a guess comes after both). The config's scrape receivers become `pipelines.receivers`, once. A first-of-several guess that stands is warned about.
+- **`crawler.discovered.exporters`** (+ `_count`, `summary.discovered.exporters`) lists each exporter the crawl learned from and its sources (`{ origin_kind, file, metrics, version?, source?, generated_at?, namespace?, gates?, scopes? }`). It appears only when there is one.
+- **An environment scope says which collection evidence it leaves out** (**a declared default change**: a warning, emitted only when such a file is excluded). A manifest and a snapshot are read in every environment. A file the scope excludes that carries scrape jobs (a `scrape_configs` file, a collector's prometheus receiver, a Helm template that carries either, a values file that carries a collector config) or an exporter config is named in a warning, with the way to read it: an environment that names no deployment profile, e.g. `--env all`.
+- **`packc emit --exporter <url>`** (`tools/emit-manifest.mjs`, `tools/cli.mjs`) scrapes an exporter's /metrics once and writes `metrics-manifest.json` (`--name`, `--version`, `--out <file>|-`, `--generated-at` / `--no-timestamp`, `--timeout`). `--from <file.prom>` reads a saved exposition. A redirect, a URL with credentials, a status other than 200, and a response with no family are refused, each naming its way out.
+- No golden changed: the golden crawl, compile, diff, snapshot and audit-report outputs are byte-identical, and a crawl with no exporter evidence and no collector scrape job carries none of the new keys. `tools/test-crawl-exporters.mjs` is new: the fixture repository `tools/fixtures/crawl/exporters/` (an mq-metric-samples config, its manifest, and a chart collector with a `qmgrs` job and a kafka exporter) yields the manifest's metrics, the job and the kafka exporter; the manifest outranks the config. `tools/test-platform.mjs` P4 pins the one new `.pathname` use, the fake exporter's request path (a URL path).
+- **Nothing secret is recorded or echoed.** `safeSourceUrl` is textual, never a URL parse, because a password may hold `/`, `?` or `@`, which end a parsed authority early and leave the secret in the "host". It drops tabs and line breaks (a URL parser drops them, so the value would still send its query), then everything before the last `@`, then everything from the first `?` or `#` to the end, and keeps a leading `scheme://` or `//`. A value with a `?` or `#` before its last `@` is recorded as nothing: the `@` may be in a query (`?user=ops@corp&api_key=…`) or a `?` in a password, and text cannot tell which. An endpoint or source that leaves nothing past its scheme is left out. A value whose path holds an `@` keeps only what follows it, losing the host rather than risking a credential. `packc emit` echoes an `--exporter` it cannot parse only in that form. It reads `--flag=value` as well as `--flag value`, and never echoes an unknown argument, a `--from`, `--out` or `--generated-at` value that looks like a URL (`://`, `@`, `?`, or `host:port`). `--from` takes a file (a file whose name holds `@` is read); a URL-like value that is no file is refused. An `--out` it cannot write is refused, not left as a stack trace. A response body that ends early or stalls is refused like a failed scrape, not left as a stack trace. A redirect is named by its target with the query as the keys of its `key=value` parts only (`?token=…`; a bare `?<value>` is named as "a query part"), and a `Location` that is not a URL is named as such. Neither one is followed.
+
 ### Follow-up `windows-ci-races`: races the Windows leg lost under load, and the same races beside them
 - **`tools/test-bundle-compare.mjs` C6: a multipart request to the child server leaves no tail unread** (a test fix; no runtime, `tools/lib`, golden or output change). C6 failed the `windows (node 22)` job of run 38737724941 (`not ok 810 - C6 retrofeed in the shim`, `fetch failed` at the `serverFetch` call in `same()`) on the tree that had passed at 7950b03. Its `multipart/form-data is not parsed` case POSTs a `FormData`, which fetch streams in parts: the headers and the first part in one write, the closing boundary in a later one. The server parses no multipart, so retrofeed answers its 404 from the headers alone, and with `connection: close` (which `serverFetch` sets since the C3 keep-alive fix) Node's server closes the socket right after the answer, the closing boundary unread: its TCP stack sends an RST behind the 404. Linux still hands the client the 404; Windows discards bytes the client had not read when the RST lands (`WSAECONNRESET`), and fetch, with no status line yet, fails. `serverFetch`, in this file and in `tools/test-bundle-compare-pairs.mjs` (the same text, C3 pins it), now builds the request through `new Request()` and hands fetch its body as one `ArrayBuffer`, so the body leaves in the headers' write and the server reads all of it before it closes; the bytes and the Content-Type, boundary included, are the ones fetch would send, the shim still receives the original `FormData`, and a GET sends no body. On Linux (node 22.22.2), 40 multipart requests to a real child server made the server's kernel send 26–29 RSTs before and 0 after; through a relay that discards unread data on an RST as Winsock does, `TypeError: fetch failed` (cause `ECONNRESET`) came back 1–5 times in 40 before and 0 after, and C6's other request kinds 0 either way. Under four busy loops, ten runs of the 40: 6–16 RSTs and 3–10 failures in each before, 0 and 0 in each after. C3 now also pins it: a `FormData` goes to fetch as one buffer, with the bytes fetch serializes it to, `connection: close`, and a GET with a null body — an assertion in an existing test (it fails against the previous `serverFetch`), so Suite total 1519 → 1519.
 - **`server/test-store.mjs` Concurrency: a parent waits out the lock-holding child for as long as the child takes to commit** (a test fix; no runtime, `tools/lib`, golden or output change — the store's `BUSY_TIMEOUT_MS`, 5000 ms, is unchanged and still pinned on `openStore`). `Concurrency: a migration waits out a child holding BEGIN IMMEDIATE, then applies` failed the `windows (node 22)` job of run 38737678276 with `database is locked` (9.9 s; the same run's write sibling waited 2355 ms of a 700 ms hold). Each of the three Concurrency tests races two clocks: the parent's busy_timeout, 5000 ms from the child's "locked", against the child's hold of 700 ms plus its COMMIT plus however long a loaded runner keeps it off the CPU; the migration's child pays the most, since `closeStore` removed the `-wal` and `openRaw` leaves `synchronous` at FULL, so its commit syncs a new WAL header and then the commit inside the window. When the child's window outlasts the parent's budget, SQLite's busy handler gives up and `BEGIN IMMEDIATE` fails. The lock holder now sets `synchronous=OFF` (its durability is not under test, its lock is: no fsync between "locked" and its exit), and every parent that waits one out — the write, the migration and the repository write — waits up to `LOCK_WAIT_LIVENESS_MS`, 30 s, a liveness bound nothing asserts (the tests assert only that the wait was at least 300 ms, and what was written); a successful wait still ends at the child's COMMIT, about 700 ms. Reproduced on Linux (node 22.22.2) by delaying every `fsync` 1.8 s (`strace -f -e inject=fsync:delay_exit=1800000`) over the three tests, ten runs each: before, the write and the repository write waited 4.2–4.3 s and passed and the migration failed with `database is locked` in every run, as on CI; after, all three passed in every run (the write and the repository write waiting 0.6–0.7 s; the migration's 4.3 s is its own commit's syncs, outside the lock wait). And with the child's hold raised to 6 s, a window past the 5000 ms budget as a stalled child's would be, all three failed with `database is locked` before and pass after. No test added or removed: the suite total stays 1519.