From 634c23e3728645ba36ff3e2b3a30f8f8661b82d8 Mon Sep 17 00:00:00 2001 From: mintaka Date: Sun, 11 Oct 2026 18:30:30 -0400 Subject: [PATCH 1/8] docs(design): route agent OTLP through the bundled collector (RIG-4611) Amendment to the observability architecture: the collector gains an otlphttp upstream exporter, agents reach it over a pinned pasta loopback forward, and devenv runs the same rendered config. Spec-impact: none. Refs RIG-4611 Co-authored-by: Matt Wilkinson --- .../compass-agent-collector-routing/design.md | 251 ++++++++++++++++++ 1 file changed, 251 insertions(+) create mode 100644 docs/designs/observability/compass-agent-collector-routing/design.md diff --git a/docs/designs/observability/compass-agent-collector-routing/design.md b/docs/designs/observability/compass-agent-collector-routing/design.md new file mode 100644 index 000000000..bd21e72ca --- /dev/null +++ b/docs/designs/observability/compass-agent-collector-routing/design.md @@ -0,0 +1,251 @@ +# Agent containers export through the bundled collector + +Tracker: RIG-4611 + +Amendment record of the observability-architecture design +([`../compass-observability-architecture/design.md`](../compass-observability-architecture/design.md), +§D3 and §T4). The parent stays frozen. + +Ledger-impact: none. This record adds no `DL-NNN` decision. The PR that lands +it carries this declaration in its body for the design-ledger gate. + +## Problem / Intent + +Matt ruled on 2026-10-11 (RIG-5144) that agent containers ship OTLP to the +bundled collector, which forwards to an operator upstream. Parent T4 promises +"the Runner-materialized env pointing agent emission at the bundled +collector" but picks no route. This amendment decides the upstream export +(C-a), the agent route (C-b) and the dev wiring (C-c). + +## Global Constraints + +- D3 holds. With no export endpoint, the collector config is today's `nop` + config, byte for byte. With one, nothing touches disk: no `file_storage`, no + `sending_queue.storage`. +- No secret in a podman argv or the 0644 collector config. URL userinfo is a + secret. +- The runner is co-located with the collector. `--otel-collector-port` is + refused unless the runner's `--server` host is loopback (OQ1). +- A pasta forward needs Linux and rootless podman 4.6 or newer, the first + release whose `podman info` reports pasta. On macOS, podman runs containers + in a VM, so a forward would reach the VM's loopback, not the Mac's. +- The collector image stays `DefaultCollectorImage` (otelcol-contrib 0.140.1), + so the exporter key is `otlphttp`. +- `baseRuleset` does not change. +- Describe an operator's receiver generically, e.g. Grafana Alloy. + +## Approach + +**C-a, upstream export.** `collectorConfigYAML` in +`go/internal/stack/collector_container.go` hard-codes `nop: {}`. A new +`--otel-export-endpoint` swaps it for one `otlphttp` exporter, behind a +`memory_limiter` and a 4 MiB request cap, because untrusted agents now reach +the collector. The queue stays in memory. Optional headers come from a 0600 +YAML map, copied into the state dir so `:Z` relabels the copy, not the +operator's file. The image runs as uid 10001, so the mount needs +`--userns=keep-id:uid=10001,gid=10001`; without it the smoke failed with +`permission denied`. A host-loopback upstream is outside the collector's +netns, so the adapter forwards it with pasta and rewrites the URL host, +keeping an `https` name through `tls.server_name_override`. Both collector +versions validate this config. + +**C-b, agent route: a pinned pasta forward.** The agent container gets +`--network pasta:-T,14318:,-U,none` and +`OTEL_EXPORTER_OTLP_ENDPOINT=http://127.0.0.1:14318`. passt(1) defines `-T` as +"TCP port forwarding from target namespace to init namespace", default +**auto**. podman-run(1) says "`-T none` and `-U none` are given" as podman's +defaults. Compass pins both, so `containers.conf` cannot widen them. The +packet leaves on `lo`, which `baseRuleset` in `go/internal/runtime/egress.go` +already accepts (`output oif lo accept`), so no port-scoped `EgressPolicy` and +no CB2. Port 14318 avoids a devenv 4318 inside the container. + +Smoke (podman 5.8.6, armed ruleset, empty allowlist): the forward answered, +and a span reached the upstream with the header. `169.254.1.2`, unforwarded +loopback ports, a host-loopback UDP listener and a TCP port bound after start +all stayed shut. Controls: `-U,auto` leaked the UDP datagram, and `-T,auto` +opened the late port. + +The host backend dials the host port directly. Other backends cannot forward, +so the runner logs that and starts agents without an emitter. + +**C-c, dev wiring.** `devenv.nix` runs no collector today. CC1 runs nixpkgs +`otelcol-contrib` on a config from the bundled container's renderer. + +## Alternatives considered + +- **`host.containers.internal` (169.254.1.2).** Allowlists match hosts, not + ports, so allowing it opens every non-loopback host port. +- **A shared podman network.** It gives agents L3 reach to each other. +- **Podman secrets for the headers.** They add secret state to `up` and + `down`. +- **Host networking for the collector.** Receivers would bind 127.0.0.1, + breaking the byte-for-byte `nop` config. +- **Hand-written Nix configs.** They restate D3 outside the tested renderer. +- **A runner relay on the agent socket.** See OQ2. + +## Plan + +Order: CA1, then CB1, then CC1. + +### CA1 — Collector upstream export + +Files: `go/internal/stack/{config,collector_container}.go`, +`go/internal/stack/adapters/collector_container.go`, +`go/internal/runtime/podman.go`, `go/cmd/compass-stack/main.go`. + +Interfaces: + +```go +// stack +Config.CollectorExportEndpoint, Config.CollectorExportHeadersFile string +CollectorContainerSpec.ExportHeadersFile string // "" = no mount, no keep-id +CollectorContainerSpec.UpstreamForwardPort uint16 // 0 = none +type CollectorYAMLParams struct { + GRPC, HTTP, Health string // "" = 0.0.0.0: + Export string // "" = nop (D3) + TLSServerName string + HeadersPath string // "" = no headers key +} +func CollectorConfigYAML(p CollectorYAMLParams) string // replaces collectorConfigYAML +const CollectorHeadersMountPath = "/run/compass/otlp-headers.yaml" +const CollectorContainerUpstreamPort = "14319" + +// runtime: Linux, podman >= 4.6, Host.Security.Rootless, Host.Pasta.Executable +func (p *PodmanCLI) VerifyPastaSupport(ctx context.Context) error +``` + +- Flags `--otel-export-endpoint` and `--otel-export-headers-file` fall back to + `COMPASS_OTEL_EXPORT_*` env. +- `Validate` rejects a non-http(s) URL, userinfo, headers without an endpoint, + an endpoint with `ExternalOTLPEndpoint`, and a loopback upstream that is off + Linux or not `127.0.0.1`/`localhost`. +- A loopback upstream renders `127.0.0.1:14319`, with `TLSServerName` set for + https. `Start` runs `VerifyPastaSupport`, adds + `--network pasta:-T,14319:,-U,none`, and copies the headers at 0600 as + `ensureGatewayToken` does, mounted `:ro,Z`. + +Red, then green: + +- `TestCollectorConfigYAMLExportsUpstream` is red today: `otlphttp` and + `memory_limiter` in all three pipelines, `max_request_body_size`, no `nop`, + no `file_storage`. `TestCollectorConfigYAMLRealizesD3Posture` stays green. +- `TestCollectorContainerSpecForwardsLoopbackUpstream` (http and https + rewrites; remote kept), `TestCollectorRunArgsExportContract`, + `TestResolveConfigCollectorExportFlags` and `TestVerifyPastaSupport`. +- Smoke: `compass-stack up` delivers a span and header to a local receiver. + +### CB1 — Route emitters to the collector + +Files: `go/internal/runtime/{podman,agent}.go`, +`go/internal/runner/{spec,host,agent_exec}.go`, +`go/cmd/compass-runner/main.go`, `go/internal/stack/spec.go`. + +Interfaces: + +```go +// runtime +type LoopbackForward struct{ Port, HostPort uint16 } // zero = none +WorkloadSpec.LoopbackForward, AgentSpec.LoopbackForward LoopbackForward + +// runner +const AgentCollectorPort = 14318 +type CollectorRoute struct { + Forward runtime.LoopbackForward + Endpoint string // "" = agents get no emitter +} +func ResolveCollectorRoute(ctx context.Context, engine runtime.WorkloadRuntime, hostPort uint16, server *url.URL) (CollectorRoute, error) +SpecDefaults.LoopbackForward runtime.LoopbackForward +AgentHostConfig.OTLPEndpoint, AgentEnv.OTLPEndpoint string +``` + +- `createArgs` renders a forward as `--network pasta:-T,:,-U,none`; + the zero value leaves the argv unchanged. +- compass-runner gains `--otel-collector-port` (`COMPASS_OTEL_COLLECTOR_PORT`; + 0 is off). +- `ResolveCollectorRoute` errors on a non-loopback `server`. The host backend + gets a direct endpoint, and podman a forward when `VerifyPastaSupport` + passes. Any other case returns an empty route, which main logs. +- `runnerSpec` passes the flag only when exporting and not `cfg.microVM()`, + like `--image`. `serverSpec` and `runnerSpec` then set + `OTEL_EXPORTER_OTLP_ENDPOINT`. + +Red, then green: + +- `TestCreateArgsRendersLoopbackForward`, + `TestExecSpecExportsOTLPEndpointOnlyWhenConfigured` and + `TestBuildSpecMapsLoopbackForward`. +- `TestResolveCollectorRoute`: port 0, remote server (error), host, podman, + failed pasta check and microVM (empty, no error). +- `TestRunnerSpecForwardsCollectorPortOnlyWhenExporting` (omitted on microVM) + and its `serverSpec` twin. +- `TestPastaForwardReachesOnlyCollectorPort`, podman-gated like + `TestKeepIDRemapMapsHostUIDToSpecUID`: an armed agent reaches the forward, + not a host-loopback UDP listener or a TCP port bound after start. It goes red + with `-U,auto` or `-T,auto`. + +### CC1 — devenv runs the collector + +Files: `devenv.nix`, `go/cmd/compass-stack/main.go`. + +Interfaces: `compass-stack collector-config --grpc ADDR --http ADDR --health +ADDR [--otel-export-endpoint URL] [--otel-export-headers-file PATH]` prints +`CollectorConfigYAML` with the host headers path. `usageError` names five +subcommands. + +- `processes.otel-collector` renders into the state dir, then execs + `${pkgs.opentelemetry-collector-contrib}/bin/otelcol-contrib` on allocated + loopback ports. The endpoint and headers come from the gitignored + `devenv.local.nix`. +- `compass-server` and `compass-runner` start `after` it. With an endpoint, + both emit to it, and the runner gets `--otel-collector-port`. + +Red, then green: today `devenv up` with an endpoint delivers nothing to a +local receiver; afterwards an agent span arrives. `TestRunCollectorConfig` +pins the renderer's bytes. + +## Tasks + +- [ ] CA1 — Collector upstream export, limits, headers and pasta preflight. +- [ ] CB1 — Pinned pasta forward for agents; server and runner emit. +- [ ] CC1 — devenv runs the collector from the shared renderer. + +## Open Questions + +1. **Where the collector runs (load-bearing).** Runners can be remote, but the + forward reaches only the runner host's loopback. (a) Refuse the flag unless + `--server` is loopback (compass-stack, devenv). (b) A collector per runner + host forwards to the server's; the flag becomes an endpoint. (c) The runner + relays agent OTLP to a remote collector (OQ2 B). **Recommendation:** (a) + now; (b) when a remote runner needs agent telemetry. +2. **Pasta or a runner relay (load-bearing).** A: ship pasta now as a named + stepping stone; the microVM follow-up uses a relay over the agent socket, + which guestd already forwards (`gatewayPortKey` in + `go/internal/guestd/cmdline.go`). B: build the relay now, overwriting + `compass.session.id` from the gateway binding + (`g.sessions.Session(g.containerName)` in + `go/internal/runner/gateway/publish.go`). B stops forgery on every tier, but + the agent exporter needs a unix-socket transport. Either relay reverses the + frozen ruling in `docs/designs/repo/compass-agent-effect-otel/design.md` + Open Questions 1: OTLP goes to the collector, "never the socket/vsock + gateway". **Recommendation:** A. Pasta is smoke-proven and RIG-4611 needs + podman data now. If the reversal is approved, the relay replaces pasta on + every tier. +3. **Emission gate.** Emitters are wired only with an export endpoint. A + `$HOME/.compass/env` value still wins (`main` in + `packages/compass-agent/src/cli.ts`: "File wins per key it defines"), and + tool subprocesses inherit the endpoint. **Recommendation:** keep the gate; + check the env file when verifying RIG-4611. +4. **Untrusted emitters.** The agent is the untrusted principal (`egress.go`) + and gains a new reach. Every agent appears as `127.0.0.1`, so a forged + `compass.session.id` can poison the D7 join. Billing reads the D5 event log. + **Recommendation:** accept for single-tenant use; multi-tenant waits for + OQ2 B. +5. **Unsupported backends.** MicroVM, apple-container and macOS podman get no + agent emitter. **Recommendation:** accept until OQ2 resolves. +6. **Port collision.** An operator receiver on 4317/4318 collides with the + bundled publish. **Recommendation:** follow-up compass-stack + `--collector-*-port` flags. +7. **`--otel-external`.** No child receives that endpoint today. + **Recommendation:** file a follow-up. +8. **Version skew.** devenv runs 0.155.0, the image 0.140.1. + **Recommendation:** accept; both validate the rendered config. From dc5c34369ae52a3ed42107713a6737d719413cbf Mon Sep 17 00:00:00 2001 From: mintaka Date: Sun, 11 Oct 2026 19:35:38 -0400 Subject: [PATCH 2/8] docs(design): fold collector-routing review round 1 (RIG-4611) pasta_options canary, agent endpoint precedence via COMPASS_OTLP_ENDPOINT, host env scrub, preflight seam, endpoint query rejection, header rotation, --otel-external reconciled with D3, and settled OQs moved to constraints. Spec-impact: none. Refs RIG-4611 Co-authored-by: Matt Wilkinson --- .../compass-agent-collector-routing/design.md | 261 +++++++++--------- 1 file changed, 137 insertions(+), 124 deletions(-) diff --git a/docs/designs/observability/compass-agent-collector-routing/design.md b/docs/designs/observability/compass-agent-collector-routing/design.md index bd21e72ca..9a0e635d8 100644 --- a/docs/designs/observability/compass-agent-collector-routing/design.md +++ b/docs/designs/observability/compass-agent-collector-routing/design.md @@ -20,67 +20,72 @@ collector" but picks no route. This amendment decides the upstream export ## Global Constraints - D3 holds. With no export endpoint, the collector config is today's `nop` - config, byte for byte. With one, nothing touches disk: no `file_storage`, no - `sending_queue.storage`. -- No secret in a podman argv or the 0644 collector config. URL userinfo is a - secret. -- The runner is co-located with the collector. `--otel-collector-port` is + config, byte for byte. With one, nothing touches disk. +- No secret in a podman argv or the 0644 collector config. Endpoints carry no + userinfo, query or fragment; credentials go in the 0600 headers file. +- The runner is co-located with the collector: `--otel-collector-port` is refused unless the runner's `--server` host is loopback (OQ1). -- A pasta forward needs Linux and rootless podman 4.6 or newer, the first - release whose `podman info` reports pasta. On macOS, podman runs containers - in a VM, so a forward would reach the VM's loopback, not the Mac's. -- The collector image stays `DefaultCollectorImage` (otelcol-contrib 0.140.1), - so the exporter key is `otlphttp`. -- `baseRuleset` does not change. -- Describe an operator's receiver generically, e.g. Grafana Alloy. +- Server and runner emit only with an export endpoint or `--otel-external`. + A runner route beats any agent env-file endpoint. +- A pasta forward needs Linux and rootless podman 4.6 or newer (the first + `podman info` reporting pasta). MicroVM, apple-container and macOS podman + start agents without an emitter and log it. +- The collector image stays `DefaultCollectorImage` (0.140.0), so the exporter + key is `otlphttp`. devenv's collector is 0.155.0; both validate the config. +- `baseRuleset` does not change. Describe an operator's receiver generically, + e.g. Grafana Alloy. ## Approach **C-a, upstream export.** `collectorConfigYAML` in `go/internal/stack/collector_container.go` hard-codes `nop: {}`. A new -`--otel-export-endpoint` swaps it for one `otlphttp` exporter, behind a +`--otel-export-endpoint` swaps it for one `otlphttp` exporter behind a `memory_limiter` and a 4 MiB request cap, because untrusted agents now reach -the collector. The queue stays in memory. Optional headers come from a 0600 -YAML map, copied into the state dir so `:Z` relabels the copy, not the -operator's file. The image runs as uid 10001, so the mount needs -`--userns=keep-id:uid=10001,gid=10001`; without it the smoke failed with -`permission denied`. A host-loopback upstream is outside the collector's -netns, so the adapter forwards it with pasta and rewrites the URL host, -keeping an `https` name through `tls.server_name_override`. Both collector -versions validate this config. +the collector. The queue stays in memory. Each start replaces a 0600 copy of +the optional headers file, mounted with `--userns=keep-id:uid=10001,gid=10001`. +A host-loopback upstream is reached through a pasta forward. **C-b, agent route: a pinned pasta forward.** The agent container gets -`--network pasta:-T,14318:,-U,none` and -`OTEL_EXPORTER_OTLP_ENDPOINT=http://127.0.0.1:14318`. passt(1) defines `-T` as -"TCP port forwarding from target namespace to init namespace", default -**auto**. podman-run(1) says "`-T none` and `-U none` are given" as podman's -defaults. Compass pins both, so `containers.conf` cannot widen them. The -packet leaves on `lo`, which `baseRuleset` in `go/internal/runtime/egress.go` -already accepts (`output oif lo accept`), so no port-scoped `EgressPolicy` and -no CB2. Port 14318 avoids a devenv 4318 inside the container. - -Smoke (podman 5.8.6, armed ruleset, empty allowlist): the forward answered, -and a span reached the upstream with the header. `169.254.1.2`, unforwarded -loopback ports, a host-loopback UDP listener and a TCP port bound after start -all stayed shut. Controls: `-U,auto` leaked the UDP datagram, and `-T,auto` -opened the late port. - -The host backend dials the host port directly. Other backends cannot forward, -so the runner logs that and starts agents without an emitter. - -**C-c, dev wiring.** `devenv.nix` runs no collector today. CC1 runs nixpkgs -`otelcol-contrib` on a config from the bundled container's renderer. +`--network pasta:-T,14318:,-U,none`. passt(1) +defines `-T` as "TCP port forwarding from target namespace to init +namespace", default **auto**. The packet leaves on `lo`, which `baseRuleset` +(`go/internal/runtime/egress.go`) accepts. + +`containers.conf` `pasta_options` are appended to that string: in the smoke, +`-T 24460` opened that host port to an armed agent, and `podman info` does not +show it. So the runner runs a canary, `pasta:-T,none,-U,none`, which must +start and hold no sockets. A configured `-T` or `-U` makes pasta fail ("-T +none conflicts with previous options"). + +The agent merges `$HOME/.compass/env` after its exec env, and "File wins per +key it defines" (`main` in `packages/compass-agent/src/cli.ts`). The file +cannot set `COMPASS_*` keys (`isReservedEnvKey`), so the route travels as +`COMPASS_OTLP_ENDPOINT`. After the merge the agent copies it to +`OTEL_EXPORTER_OTLP_ENDPOINT` and drops per-signal endpoint keys. + +Smoke (podman 5.8.6, armed, empty allowlist): only the forward opened, and a +span reached the upstream. `-U,auto` and `-T,auto` controls leaked. + +The host backend dials the host port directly; its children inherit +`os.Environ()` (`envSlice` in `go/internal/runtime/host_backend.go`). + +**`--otel-external`.** D3 points Compass surfaces at this endpoint, but no +child receives it. CB1 passes it to server and runner, amending D3's input to +an http(s) URL, because the Go emitters use `otlptracehttp` +(`go/internal/otel/provider.go`). Agents stay as shipped. + +**C-c, dev wiring.** CC1 runs nixpkgs `otelcol-contrib` on a config from the +bundled container's renderer. ## Alternatives considered -- **`host.containers.internal` (169.254.1.2).** Allowlists match hosts, not - ports, so allowing it opens every non-loopback host port. -- **A shared podman network.** It gives agents L3 reach to each other. -- **Podman secrets for the headers.** They add secret state to `up` and - `down`. -- **Host networking for the collector.** Receivers would bind 127.0.0.1, - breaking the byte-for-byte `nop` config. -- **Hand-written Nix configs.** They restate D3 outside the tested renderer. +- **`host.containers.internal`.** Allowlists match hosts, not ports. +- **A shared podman network.** Agents get L3 reach to each other. +- **Podman secrets, or host networking.** Secret state; breaks the `nop` + bytes. +- **Parsing `containers.conf`.** Several sources; the canary reads podman's + merge. +- **The route in `$HOME/.compass/env`.** That is the secrets channel. - **A runner relay on the agent socket.** See OQ2. ## Plan @@ -89,7 +94,7 @@ Order: CA1, then CB1, then CC1. ### CA1 — Collector upstream export -Files: `go/internal/stack/{config,collector_container}.go`, +Files: `go/internal/stack/{config,deps,stack,collector_container}.go`, `go/internal/stack/adapters/collector_container.go`, `go/internal/runtime/podman.go`, `go/cmd/compass-stack/main.go`. @@ -107,6 +112,8 @@ type CollectorYAMLParams struct { HeadersPath string // "" = no headers key } func CollectorConfigYAML(p CollectorYAMLParams) string // replaces collectorConfigYAML +type PastaVerifier interface{ VerifyPastaSupport(ctx context.Context) error } +Deps.PastaVerifier PastaVerifier // buildDeps wires runtime.NewPodmanCLI() const CollectorHeadersMountPath = "/run/compass/otlp-headers.yaml" const CollectorContainerUpstreamPort = "14319" @@ -116,29 +123,36 @@ func (p *PodmanCLI) VerifyPastaSupport(ctx context.Context) error - Flags `--otel-export-endpoint` and `--otel-export-headers-file` fall back to `COMPASS_OTEL_EXPORT_*` env. -- `Validate` rejects a non-http(s) URL, userinfo, headers without an endpoint, - an endpoint with `ExternalOTLPEndpoint`, and a loopback upstream that is off - Linux or not `127.0.0.1`/`localhost`. -- A loopback upstream renders `127.0.0.1:14319`, with `TLSServerName` set for - https. `Start` runs `VerifyPastaSupport`, adds - `--network pasta:-T,14319:,-U,none`, and copies the headers at 0600 as - `ensureGatewayToken` does, mounted `:ro,Z`. +- `Validate` rejects a non-http(s) URL, userinfo, query, fragment, headers + without an endpoint, an endpoint with `ExternalOTLPEndpoint`, and a loopback + upstream off Linux or not `127.0.0.1`/`localhost`. +- A loopback upstream renders `127.0.0.1:14319` (`TLSServerName` for https) + and adds `--network pasta:-T,14319:,-U,none` after `startCollector` + calls `Deps.PastaVerifier`. +- `Start` renames a 0600 temp copy of the headers over the old one; unlike + `ensureGatewayToken`, it never keeps an existing file. Red, then green: - `TestCollectorConfigYAMLExportsUpstream` is red today: `otlphttp` and `memory_limiter` in all three pipelines, `max_request_body_size`, no `nop`, - no `file_storage`. `TestCollectorConfigYAMLRealizesD3Posture` stays green. -- `TestCollectorContainerSpecForwardsLoopbackUpstream` (http and https - rewrites; remote kept), `TestCollectorRunArgsExportContract`, - `TestResolveConfigCollectorExportFlags` and `TestVerifyPastaSupport`. + no `file_storage`. +- `TestCollectorConfigYAMLRealizesD3Posture` compares + `CollectorConfigYAML(CollectorYAMLParams{})` with today's exact bytes. +- `TestCollectorContainerSpecForwardsLoopbackUpstream`, + `TestResolveConfigCollectorExportFlags`, `TestVerifyPastaSupport`, and + `TestStartCollectorRefusesForwardWithoutPasta` (a failing verifier stops + `Up`). +- `TestCollectorStartReplacesHeadersCopy` (0600, rotation, source untouched) + and `TestCollectorRunArgsExportContract` (no header value in argv or config). - Smoke: `compass-stack up` delivers a span and header to a local receiver. ### CB1 — Route emitters to the collector -Files: `go/internal/runtime/{podman,agent}.go`, +Files: `go/internal/runtime/{podman,agent,host_backend}.go`, `go/internal/runner/{spec,host,agent_exec}.go`, -`go/cmd/compass-runner/main.go`, `go/internal/stack/spec.go`. +`go/cmd/compass-runner/main.go`, `go/internal/stack/{config,spec}.go`, +`packages/compass-agent/src/cli.ts`. Interfaces: @@ -146,6 +160,8 @@ Interfaces: // runtime type LoopbackForward struct{ Port, HostPort uint16 } // zero = none WorkloadSpec.LoopbackForward, AgentSpec.LoopbackForward LoopbackForward +// canary: --network pasta:-T,none,-U,none; error on failure or any socket +func (p *PodmanCLI) VerifyNoPastaForwards(ctx context.Context, image string) error // runner const AgentCollectorPort = 14318 @@ -153,35 +169,44 @@ type CollectorRoute struct { Forward runtime.LoopbackForward Endpoint string // "" = agents get no emitter } -func ResolveCollectorRoute(ctx context.Context, engine runtime.WorkloadRuntime, hostPort uint16, server *url.URL) (CollectorRoute, error) +func ResolveCollectorRoute(ctx context.Context, engine runtime.WorkloadRuntime, hostPort uint16, server *url.URL, image string) (CollectorRoute, error) SpecDefaults.LoopbackForward runtime.LoopbackForward -AgentHostConfig.OTLPEndpoint, AgentEnv.OTLPEndpoint string +AgentHostConfig.OTLPEndpoint, AgentEnv.OTLPEndpoint string // execSpec sets COMPASS_OTLP_ENDPOINT +``` + +```ts +// cli.ts: called in main right after the env-file merge +export function applyCollectorRoute(env: Record): void ``` -- `createArgs` renders a forward as `--network pasta:-T,:,-U,none`; - the zero value leaves the argv unchanged. +- `createArgs` renders `--network pasta:-T,:,-U,none`; the + zero value leaves the argv unchanged. - compass-runner gains `--otel-collector-port` (`COMPASS_OTEL_COLLECTOR_PORT`; 0 is off). -- `ResolveCollectorRoute` errors on a non-loopback `server`. The host backend - gets a direct endpoint, and podman a forward when `VerifyPastaSupport` - passes. Any other case returns an empty route, which main logs. -- `runnerSpec` passes the flag only when exporting and not `cfg.microVM()`, - like `--image`. `serverSpec` and `runnerSpec` then set - `OTEL_EXPORTER_OTLP_ENDPOINT`. +- `ResolveCollectorRoute` errors on a non-loopback `server` or a failed + canary. Host gets a direct endpoint, podman a forward when + `VerifyPastaSupport` passes; otherwise the route is empty and main logs it. +- `envSlice` drops inherited `OTEL_*` keys before the per-agent overlay. +- `runnerSpec` passes the flag only when exporting and not `cfg.microVM()`. + `serverSpec` and `runnerSpec` set `OTEL_EXPORTER_OTLP_ENDPOINT` to the + collector or `--otel-external` URL; `otlp.example.com:4317` fixtures become + URLs. Red, then green: - `TestCreateArgsRendersLoopbackForward`, - `TestExecSpecExportsOTLPEndpointOnlyWhenConfigured` and - `TestBuildSpecMapsLoopbackForward`. -- `TestResolveCollectorRoute`: port 0, remote server (error), host, podman, - failed pasta check and microVM (empty, no error). -- `TestRunnerSpecForwardsCollectorPortOnlyWhenExporting` (omitted on microVM) - and its `serverSpec` twin. -- `TestPastaForwardReachesOnlyCollectorPort`, podman-gated like - `TestKeepIDRemapMapsHostUIDToSpecUID`: an armed agent reaches the forward, - not a host-loopback UDP listener or a TCP port bound after start. It goes red - with `-U,auto` or `-T,auto`. + `TestExecSpecSetsCollectorRouteOnlyWhenConfigured`, + `TestBuildSpecMapsLoopbackForward` and `TestEnvSliceDropsInheritedOTELKeys`. +- `TestResolveCollectorRoute`: port 0, remote server and failed canary + (errors); host, podman, no pasta and microVM. +- `cli.test.ts`: the route beats file endpoints; with no route, a file + endpoint stays. +- `TestServerAndRunnerSpecOTLPEndpoint`: collector, external URL or none. +- Podman-gated, like `TestKeepIDRemapMapsHostUIDToSpecUID`: + `TestPastaForwardReachesOnlyCollectorPort` (no host-loopback UDP or + late-bound TCP; red with `-U,auto` or `-T,auto`) and + `TestVerifyNoPastaForwardsRefusesConfOptions` (`CONTAINERS_CONF_OVERRIDE` + with `pasta_options = ["-T", ""]`). ### CC1 — devenv runs the collector @@ -193,59 +218,47 @@ ADDR [--otel-export-endpoint URL] [--otel-export-headers-file PATH]` prints subcommands. - `processes.otel-collector` renders into the state dir, then execs - `${pkgs.opentelemetry-collector-contrib}/bin/otelcol-contrib` on allocated - loopback ports. The endpoint and headers come from the gitignored - `devenv.local.nix`. -- `compass-server` and `compass-runner` start `after` it. With an endpoint, - both emit to it, and the runner gets `--otel-collector-port`. + `${pkgs.opentelemetry-collector-contrib}/bin/otelcol-contrib`, configured + from the gitignored `devenv.local.nix`. Server and runner start `after` it. Red, then green: today `devenv up` with an endpoint delivers nothing to a -local receiver; afterwards an agent span arrives. `TestRunCollectorConfig` -pins the renderer's bytes. +local receiver; afterwards it receives `compass-server`, `compass-runner` and +`compass-agent` spans. `TestRunCollectorConfig` pins the renderer's bytes. ## Tasks - [ ] CA1 — Collector upstream export, limits, headers and pasta preflight. -- [ ] CB1 — Pinned pasta forward for agents; server and runner emit. +- [ ] CB1 — Pinned forward, route precedence, `--otel-external`. - [ ] CC1 — devenv runs the collector from the shared renderer. +## Deferred (non-load-bearing) + +- Port collision with an operator receiver on 4317/4318 waits for + compass-stack `--collector-*-port` flags. +- `pasta_options` forwards already reach agents on podman's default network; + checking every start is a separate egress change (RIG-5249). + ## Open Questions -1. **Where the collector runs (load-bearing).** Runners can be remote, but the - forward reaches only the runner host's loopback. (a) Refuse the flag unless - `--server` is loopback (compass-stack, devenv). (b) A collector per runner - host forwards to the server's; the flag becomes an endpoint. (c) The runner +1. **Where the collector runs (load-bearing).** The forward reaches only the + runner host's loopback. (a) Refuse the flag unless `--server` is loopback. + (b) A collector per runner host forwards to the server's. (c) The runner relays agent OTLP to a remote collector (OQ2 B). **Recommendation:** (a) now; (b) when a remote runner needs agent telemetry. -2. **Pasta or a runner relay (load-bearing).** A: ship pasta now as a named +2. **Pasta or a runner relay (load-bearing).** A: pasta now as an explicit stepping stone; the microVM follow-up uses a relay over the agent socket, which guestd already forwards (`gatewayPortKey` in - `go/internal/guestd/cmdline.go`). B: build the relay now, overwriting - `compass.session.id` from the gateway binding - (`g.sessions.Session(g.containerName)` in - `go/internal/runner/gateway/publish.go`). B stops forgery on every tier, but - the agent exporter needs a unix-socket transport. Either relay reverses the - frozen ruling in `docs/designs/repo/compass-agent-effect-otel/design.md` - Open Questions 1: OTLP goes to the collector, "never the socket/vsock - gateway". **Recommendation:** A. Pasta is smoke-proven and RIG-4611 needs - podman data now. If the reversal is approved, the relay replaces pasta on - every tier. -3. **Emission gate.** Emitters are wired only with an export endpoint. A - `$HOME/.compass/env` value still wins (`main` in - `packages/compass-agent/src/cli.ts`: "File wins per key it defines"), and - tool subprocesses inherit the endpoint. **Recommendation:** keep the gate; - check the env file when verifying RIG-4611. -4. **Untrusted emitters.** The agent is the untrusted principal (`egress.go`) - and gains a new reach. Every agent appears as `127.0.0.1`, so a forged - `compass.session.id` can poison the D7 join. Billing reads the D5 event log. - **Recommendation:** accept for single-tenant use; multi-tenant waits for + `go/internal/guestd/cmdline.go`). B: build the relay now; it overwrites + `compass.session.id` from `g.sessions.Session(g.containerName)` + (`go/internal/runner/gateway/publish.go`), but the agent exporter needs a + unix-socket transport. Either relay reverses + `docs/designs/repo/compass-agent-effect-otel/design.md` Open Questions 1: + "never the socket/vsock gateway". **Recommendation:** A; an approved relay + replaces pasta on every tier. +3. **Forged session ids (load-bearing).** The untrusted agent stamps + `compass.session.id`, and every agent arrives as `127.0.0.1`. A forged id + files one session's telemetry under another, corrupting every D7 join and + dashboard on that key. Billing reads the D5 event log, unaffected. (a) D7 + treats Plane B session ids as untrusted until a relay stamps them. (b) Hold + the route until OQ2 B. **Recommendation:** (a); multi-tenant waits for OQ2 B. -5. **Unsupported backends.** MicroVM, apple-container and macOS podman get no - agent emitter. **Recommendation:** accept until OQ2 resolves. -6. **Port collision.** An operator receiver on 4317/4318 collides with the - bundled publish. **Recommendation:** follow-up compass-stack - `--collector-*-port` flags. -7. **`--otel-external`.** No child receives that endpoint today. - **Recommendation:** file a follow-up. -8. **Version skew.** devenv runs 0.155.0, the image 0.140.1. - **Recommendation:** accept; both validate the rendered config. From b0c915b09af45748a9a903efe1aa2add884c371c Mon Sep 17 00:00:00 2001 From: mintaka Date: Sun, 11 Oct 2026 21:52:22 -0400 Subject: [PATCH 3/8] docs(design): fold collector-routing review round 2 (RIG-4611) Agents get --otel-external via the runner, inherited OTEL_* are dropped from stack children, --otel-off replaces dummy endpoints in tests, and the pasta checks run support-first through one interface. Spec-impact: none. Refs RIG-4611 Co-authored-by: Matt Wilkinson --- .../compass-agent-collector-routing/design.md | 332 +++++++++--------- 1 file changed, 175 insertions(+), 157 deletions(-) diff --git a/docs/designs/observability/compass-agent-collector-routing/design.md b/docs/designs/observability/compass-agent-collector-routing/design.md index 9a0e635d8..a004659cd 100644 --- a/docs/designs/observability/compass-agent-collector-routing/design.md +++ b/docs/designs/observability/compass-agent-collector-routing/design.md @@ -6,92 +6,86 @@ Amendment record of the observability-architecture design ([`../compass-observability-architecture/design.md`](../compass-observability-architecture/design.md), §D3 and §T4). The parent stays frozen. -Ledger-impact: none. This record adds no `DL-NNN` decision. The PR that lands -it carries this declaration in its body for the design-ledger gate. +Ledger-impact: none. This record adds no `DL-NNN` decision. ## Problem / Intent -Matt ruled on 2026-10-11 (RIG-5144) that agent containers ship OTLP to the -bundled collector, which forwards to an operator upstream. Parent T4 promises -"the Runner-materialized env pointing agent emission at the bundled -collector" but picks no route. This amendment decides the upstream export -(C-a), the agent route (C-b) and the dev wiring (C-c). +Matt ruled on 2026-10-11 (RIG-5144) that agent containers ship OTLP through +the bundled collector to an operator upstream. Parent T4 promises "the +Runner-materialized env pointing agent emission at the bundled collector" but +picks no route. This record decides C-a (upstream export), C-b (agent route) +and C-c (dev wiring). ## Global Constraints -- D3 holds. With no export endpoint, the collector config is today's `nop` - config, byte for byte. With one, nothing touches disk. -- No secret in a podman argv or the 0644 collector config. Endpoints carry no - userinfo, query or fragment; credentials go in the 0600 headers file. -- The runner is co-located with the collector: `--otel-collector-port` is - refused unless the runner's `--server` host is loopback (OQ1). -- Server and runner emit only with an export endpoint or `--otel-external`. - A runner route beats any agent env-file endpoint. -- A pasta forward needs Linux and rootless podman 4.6 or newer (the first - `podman info` reporting pasta). MicroVM, apple-container and macOS podman - start agents without an emitter and log it. -- The collector image stays `DefaultCollectorImage` (0.140.0), so the exporter - key is `otlphttp`. devenv's collector is 0.155.0; both validate the config. -- `baseRuleset` does not change. Describe an operator's receiver generically, - e.g. Grafana Alloy. +- D3 holds: with no export endpoint the collector config keeps today's `nop` + bytes. Exported telemetry stays in memory; only config and headers files + are written. +- No secret in a podman argv or the 0644 config. Endpoints are http(s) URLs + without userinfo, query or fragment; credentials live in the 0600 headers + file. +- Loopback means host `127.0.0.1` or `localhost`; other loopback forms are + refused. +- Server and runner emit only with an export endpoint or `--otel-external`, + and drop inherited `OTEL_*`. An agent route clears the env file's + `OTEL_EXPORTER_OTLP_*`. +- Pasta needs Linux and rootless podman 4.6+. +- `DefaultCollectorImage` stays 0.140.0, so the exporter key is `otlphttp`; + devenv's 0.155.0 validates the same config. +- `baseRuleset` is unchanged. Name operator receivers generically, e.g. + Grafana Alloy. ## Approach -**C-a, upstream export.** `collectorConfigYAML` in -`go/internal/stack/collector_container.go` hard-codes `nop: {}`. A new -`--otel-export-endpoint` swaps it for one `otlphttp` exporter behind a -`memory_limiter` and a 4 MiB request cap, because untrusted agents now reach -the collector. The queue stays in memory. Each start replaces a 0600 copy of -the optional headers file, mounted with `--userns=keep-id:uid=10001,gid=10001`. -A host-loopback upstream is reached through a pasta forward. - -**C-b, agent route: a pinned pasta forward.** The agent container gets -`--network pasta:-T,14318:,-U,none`. passt(1) -defines `-T` as "TCP port forwarding from target namespace to init -namespace", default **auto**. The packet leaves on `lo`, which `baseRuleset` -(`go/internal/runtime/egress.go`) accepts. - -`containers.conf` `pasta_options` are appended to that string: in the smoke, -`-T 24460` opened that host port to an armed agent, and `podman info` does not -show it. So the runner runs a canary, `pasta:-T,none,-U,none`, which must -start and hold no sockets. A configured `-T` or `-U` makes pasta fail ("-T -none conflicts with previous options"). - -The agent merges `$HOME/.compass/env` after its exec env, and "File wins per +**C-a, upstream export.** `collectorConfigYAML` +(`go/internal/stack/collector_container.go`) hard-codes `nop: {}`. A new +`--otel-export-endpoint` swaps in one `otlphttp` exporter behind a +`memory_limiter` and a 4 MiB request cap, since untrusted agents now reach +the collector. Each start replaces a 0600 copy of the optional headers file, +mounted with `--userns=keep-id:uid=10001,gid=10001`. + +**C-b, agent route: a pinned pasta forward.** Agents get +`--network pasta:-T,14318:,-U,none`. passt(1) defines `-T` as +"TCP port forwarding from target namespace to init namespace", default +**auto**. Traffic leaves on `lo`, which `baseRuleset` +(`go/internal/runtime/egress.go`) accepts. A podman 5.8.6 smoke opened only +the forward; `-U,auto` and `-T,auto` controls leaked. + +Podman appends `containers.conf` `pasta_options`, unseen by `podman info`: a +configured `-T 24460` reached an armed agent. So a canary, +`pasta:-T,none,-U,none`, must start and hold no sockets; any configured `-T` +or `-U` fails it. + +The agent merges `$HOME/.compass/env` into `process.env`, and "File wins per key it defines" (`main` in `packages/compass-agent/src/cli.ts`). The file -cannot set `COMPASS_*` keys (`isReservedEnvKey`), so the route travels as -`COMPASS_OTLP_ENDPOINT`. After the merge the agent copies it to -`OTEL_EXPORTER_OTLP_ENDPOINT` and drops per-signal endpoint keys. +cannot set `COMPASS_*` (`isReservedEnvKey`), so the route travels as +`COMPASS_OTLP_ENDPOINT`; after the merge the agent clears the file's +`OTEL_EXPORTER_OTLP_*` keys, so no `grpc` protocol or stale headers survive. -Smoke (podman 5.8.6, armed, empty allowlist): only the forward opened, and a -span reached the upstream. `-U,auto` and `-T,auto` controls leaked. +**Agent targets.** compass-runner's `--otel-agent-endpoint` names the agents' +target. The stack passes the collector URL `http://127.0.0.1:`, +whatever its `ListenAddr`, or the `--otel-external` URL. A remote host joins +an armed allowlist (OQ4); a loopback target needs pasta, else agents get no +emitter. -The host backend dials the host port directly; its children inherit -`os.Environ()` (`envSlice` in `go/internal/runtime/host_backend.go`). +**`--otel-external`.** D3 points every surface at it; today no child receives +it. CB1 passes it to server, runner and agents, so it becomes an http(s) URL: +the Go emitters use `otlptracehttp` (`go/internal/otel/provider.go`). +Refusing a bare `host:port` breaks callers, so CB1's commit carries `!` for +release-please. A new `--otel-off` replaces the dummy `127.0.0.1:4317` tests +pass: no collector, endpoint or route. -**`--otel-external`.** D3 points Compass surfaces at this endpoint, but no -child receives it. CB1 passes it to server and runner, amending D3's input to -an http(s) URL, because the Go emitters use `otlptracehttp` -(`go/internal/otel/provider.go`). Agents stay as shipped. - -**C-c, dev wiring.** CC1 runs nixpkgs `otelcol-contrib` on a config from the -bundled container's renderer. +**C-c.** CC1 runs nixpkgs `otelcol-contrib` on the bundled renderer's config. ## Alternatives considered - **`host.containers.internal`.** Allowlists match hosts, not ports. -- **A shared podman network.** Agents get L3 reach to each other. -- **Podman secrets, or host networking.** Secret state; breaks the `nop` - bytes. -- **Parsing `containers.conf`.** Several sources; the canary reads podman's - merge. -- **The route in `$HOME/.compass/env`.** That is the secrets channel. -- **A runner relay on the agent socket.** See OQ2. +- **A shared podman network.** Agents reach each other. +- **Podman secrets or host networking.** Secret state; breaks `nop` bytes. +- **Parsing `containers.conf`.** Several sources; the canary sees the merge. ## Plan -Order: CA1, then CB1, then CC1. - ### CA1 — Collector upstream export Files: `go/internal/stack/{config,deps,stack,collector_container}.go`, @@ -113,46 +107,46 @@ type CollectorYAMLParams struct { } func CollectorConfigYAML(p CollectorYAMLParams) string // replaces collectorConfigYAML type PastaVerifier interface{ VerifyPastaSupport(ctx context.Context) error } -Deps.PastaVerifier PastaVerifier // buildDeps wires runtime.NewPodmanCLI() +Deps.PastaVerifier PastaVerifier // buildDeps: runtime.NewPodmanCLI() const CollectorHeadersMountPath = "/run/compass/otlp-headers.yaml" const CollectorContainerUpstreamPort = "14319" -// runtime: Linux, podman >= 4.6, Host.Security.Rootless, Host.Pasta.Executable +// runtime: Linux, podman >= 4.6, rootless, Host.Pasta.Executable func (p *PodmanCLI) VerifyPastaSupport(ctx context.Context) error ``` -- Flags `--otel-export-endpoint` and `--otel-export-headers-file` fall back to - `COMPASS_OTEL_EXPORT_*` env. -- `Validate` rejects a non-http(s) URL, userinfo, query, fragment, headers - without an endpoint, an endpoint with `ExternalOTLPEndpoint`, and a loopback - upstream off Linux or not `127.0.0.1`/`localhost`. -- A loopback upstream renders `127.0.0.1:14319` (`TLSServerName` for https) - and adds `--network pasta:-T,14319:,-U,none` after `startCollector` - calls `Deps.PastaVerifier`. -- `Start` renames a 0600 temp copy of the headers over the old one; unlike +- `--otel-export-endpoint` and `--otel-export-headers-file` fall back to + `COMPASS_OTEL_EXPORT_*`. `Validate` refuses a bad URL, headers without an + endpoint, an endpoint with `ExternalOTLPEndpoint`, and a loopback upstream + off Linux or in another form. +- `collectorContainerSpec`, not the renderer, rewrites a loopback upstream to + `127.0.0.1:14319` (`TLSServerName` for https) and adds + `--network pasta:-T,14319:,-U,none`; `startCollector` first calls + `Deps.PastaVerifier`. +- `Start` renames a 0600 temp copy over the old headers; unlike `ensureGatewayToken`, it never keeps an existing file. Red, then green: -- `TestCollectorConfigYAMLExportsUpstream` is red today: `otlphttp` and - `memory_limiter` in all three pipelines, `max_request_body_size`, no `nop`, - no `file_storage`. -- `TestCollectorConfigYAMLRealizesD3Posture` compares - `CollectorConfigYAML(CollectorYAMLParams{})` with today's exact bytes. +- `TestCollectorConfigYAMLExportsUpstream`, red today: `otlphttp` and + `memory_limiter` in all three pipelines, `max_request_body_size`, no `nop` + or `file_storage`. +- `TestCollectorConfigYAMLRealizesD3Posture`: zero params give today's bytes. - `TestCollectorContainerSpecForwardsLoopbackUpstream`, - `TestResolveConfigCollectorExportFlags`, `TestVerifyPastaSupport`, and - `TestStartCollectorRefusesForwardWithoutPasta` (a failing verifier stops - `Up`). -- `TestCollectorStartReplacesHeadersCopy` (0600, rotation, source untouched) - and `TestCollectorRunArgsExportContract` (no header value in argv or config). + `TestResolveConfigCollectorExportFlags`, `TestVerifyPastaSupport`, + `TestStartCollectorRefusesForwardWithoutPasta`. +- `TestCollectorStartReplacesHeadersCopy` (0600, rotation, source untouched); + `TestCollectorRunArgsExportContract` (no header value in argv or config). - Smoke: `compass-stack up` delivers a span and header to a local receiver. -### CB1 — Route emitters to the collector +### CB1 — Route every emitter Files: `go/internal/runtime/{podman,agent,host_backend}.go`, `go/internal/runner/{spec,host,agent_exec}.go`, -`go/cmd/compass-runner/main.go`, `go/internal/stack/{config,spec}.go`, -`packages/compass-agent/src/cli.ts`. +`go/internal/stack/{config,spec,stack}.go`, +`go/internal/stack/adapters/process.go`, +`go/cmd/{compass-runner,compass-stack}/main.go`, +`packages/compass-agent/src/cli.ts`, and the migrated callers. Interfaces: @@ -160,53 +154,78 @@ Interfaces: // runtime type LoopbackForward struct{ Port, HostPort uint16 } // zero = none WorkloadSpec.LoopbackForward, AgentSpec.LoopbackForward LoopbackForward -// canary: --network pasta:-T,none,-U,none; error on failure or any socket +// canary: errors on failure or any socket func (p *PodmanCLI) VerifyNoPastaForwards(ctx context.Context, image string) error // runner +type pastaForwarder interface { // *runtime.PodmanCLI + VerifyPastaSupport(ctx context.Context) error + VerifyNoPastaForwards(ctx context.Context, image string) error +} const AgentCollectorPort = 14318 -type CollectorRoute struct { - Forward runtime.LoopbackForward - Endpoint string // "" = agents get no emitter +type AgentRoute struct { + Forward runtime.LoopbackForward + Endpoint string // "" = no agent emitter + AllowHost string // remote target host } -func ResolveCollectorRoute(ctx context.Context, engine runtime.WorkloadRuntime, hostPort uint16, server *url.URL, image string) (CollectorRoute, error) +func ResolveAgentRoute(ctx context.Context, engine runtime.WorkloadRuntime, target *url.URL, image string) (AgentRoute, error) SpecDefaults.LoopbackForward runtime.LoopbackForward -AgentHostConfig.OTLPEndpoint, AgentEnv.OTLPEndpoint string // execSpec sets COMPASS_OTLP_ENDPOINT +AgentHostConfig.OTLPEndpoint, AgentEnv.OTLPEndpoint string // COMPASS_OTLP_ENDPOINT + +// stack +Config.OTLPOff bool +func (c Config) bundlesCollector() bool // replaces ExternalOTLPEndpoint == "" gates +func childEnv(inherited, overlay []string) []string // adapters; drops inherited OTEL_* ``` ```ts -// cli.ts: called in main right after the env-file merge -export function applyCollectorRoute(env: Record): void +// main passes env.COMPASS_OTLP_ENDPOINT after the env-file merge +export function applyCollectorRoute(route: string | undefined, target = process.env): void ``` -- `createArgs` renders `--network pasta:-T,:,-U,none`; the - zero value leaves the argv unchanged. -- compass-runner gains `--otel-collector-port` (`COMPASS_OTEL_COLLECTOR_PORT`; - 0 is off). -- `ResolveCollectorRoute` errors on a non-loopback `server` or a failed - canary. Host gets a direct endpoint, podman a forward when - `VerifyPastaSupport` passes; otherwise the route is empty and main logs it. -- `envSlice` drops inherited `OTEL_*` keys before the per-agent overlay. -- `runnerSpec` passes the flag only when exporting and not `cfg.microVM()`. - `serverSpec` and `runnerSpec` set `OTEL_EXPORTER_OTLP_ENDPOINT` to the - collector or `--otel-external` URL; `otlp.example.com:4317` fixtures become - URLs. +- `ResolveAgentRoute`: no target, empty; host tier (`runtime.TierOf`), + direct; remote, verbatim plus `AllowHost`. Loopback needs a + `pastaForwarder` passing `VerifyPastaSupport`, else an empty, logged route; + then the canary runs, and its failure errors; then the forward keeps scheme + and host on port 14318. +- compass-runner gains `--otel-agent-endpoint` + (`COMPASS_OTEL_AGENT_ENDPOINT`) and appends `AllowHost` to a configured + policy after `ResolveEgress`. +- `createArgs` renders `--network pasta:-T,:,-U,none`; zero + leaves the argv unchanged. `envSlice` and `childEnv` drop inherited + `OTEL_*`; `ProcessSupervisor.Start` uses `childEnv`, which covers the stack + `embedded.RunStackUp` execs. +- `serverSpec` sets `OTEL_EXPORTER_OTLP_ENDPOINT` and `runnerSpec` adds + `--otel-agent-endpoint`: the collector URL when exporting, else the + `--otel-external` URL, else neither. +- `--otel-off` excludes `--otel-external` and the export flags; `Validate` + refuses a non-URL `ExternalOTLPEndpoint`. +- Migration: `go/e2e/fixture.go` sets `OTLPOff`; the `127.0.0.1:4317` runs in + `go/cmd/compass-stack/{container_postgres,nats}_podman_test.go` pass + `--otel-off`; `TestExternalOTLPUpDown` passes `http://127.0.0.1:4318`; + `otlp.example.com:4317` fixtures become URLs; the `ci.yml` comment names + `--otel-off`. Red, then green: - `TestCreateArgsRendersLoopbackForward`, `TestExecSpecSetsCollectorRouteOnlyWhenConfigured`, - `TestBuildSpecMapsLoopbackForward` and `TestEnvSliceDropsInheritedOTELKeys`. -- `TestResolveCollectorRoute`: port 0, remote server and failed canary - (errors); host, podman, no pasta and microVM. -- `cli.test.ts`: the route beats file endpoints; with no route, a file - endpoint stays. -- `TestServerAndRunnerSpecOTLPEndpoint`: collector, external URL or none. -- Podman-gated, like `TestKeepIDRemapMapsHostUIDToSpecUID`: - `TestPastaForwardReachesOnlyCollectorPort` (no host-loopback UDP or - late-bound TCP; red with `-U,auto` or `-T,auto`) and - `TestVerifyNoPastaForwardsRefusesConfOptions` (`CONTAINERS_CONF_OVERRIDE` - with `pasta_options = ["-T", ""]`). + `TestBuildSpecMapsLoopbackForward`, `TestEnvSliceDropsInheritedOTELKeys`. +- `TestResolveAgentRoute`: no target, host, remote, no capability, http and + https forwards, failed canary; with pasta unsupported a counting fake + proves the canary never ran. +- `TestStartDropsInheritedOTELKeys` (a `trapecho` child): an inherited base + endpoint with the route off; an inherited + `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT` with it on. +- `TestServerAndRunnerSpecOTLPEndpoint`: collector, external, `nop`, off; a + `0.0.0.0` `ListenAddr` keeps the loopback collector URL. +- `TestResolveConfigOTelOff`; `TestConfigValidate` refuses `127.0.0.1:4317`. +- `cli.test.ts`: the route beats a file endpoint and clears a file `grpc` + protocol and headers; with no route a file endpoint stays. +- Podman-gated: `TestPastaForwardReachesOnlyCollectorPort` (no host-loopback + UDP or late-bound TCP; red with `-U,auto` or `-T,auto`) and + `TestVerifyNoPastaForwardsRefusesConfOptions` (a `-T` in + `CONTAINERS_CONF_OVERRIDE`). ### CC1 — devenv runs the collector @@ -214,51 +233,50 @@ Files: `devenv.nix`, `go/cmd/compass-stack/main.go`. Interfaces: `compass-stack collector-config --grpc ADDR --http ADDR --health ADDR [--otel-export-endpoint URL] [--otel-export-headers-file PATH]` prints -`CollectorConfigYAML` with the host headers path. `usageError` names five -subcommands. +`CollectorConfigYAML` with the endpoint verbatim and the host headers path. +`usageError` names five subcommands. -- `processes.otel-collector` renders into the state dir, then execs - `${pkgs.opentelemetry-collector-contrib}/bin/otelcol-contrib`, configured - from the gitignored `devenv.local.nix`. Server and runner start `after` it. +- When the gitignored `devenv.local.nix` sets + `env.COMPASS_OTEL_EXPORT_ENDPOINT`, `processes.otel-collector` renders into + the state dir and execs + `${pkgs.opentelemetry-collector-contrib}/bin/otelcol-contrib`. Server and + runner start `after` it with `OTEL_EXPORTER_OTLP_ENDPOINT`, and the runner + with `--otel-agent-endpoint`, both `http://127.0.0.1:`. -Red, then green: today `devenv up` with an endpoint delivers nothing to a -local receiver; afterwards it receives `compass-server`, `compass-runner` and -`compass-agent` spans. `TestRunCollectorConfig` pins the renderer's bytes. +Red, then green: `devenv up` with an endpoint delivers nothing to a local +receiver today, and afterwards delivers `compass-server`, `compass-runner` and +`compass-agent` spans. `TestRunCollectorConfig` pins a loopback upstream. ## Tasks -- [ ] CA1 — Collector upstream export, limits, headers and pasta preflight. -- [ ] CB1 — Pinned forward, route precedence, `--otel-external`. -- [ ] CC1 — devenv runs the collector from the shared renderer. +- [ ] CA1 — Collector upstream export. +- [ ] CB1 — Route every emitter. +- [ ] CC1 — devenv runs the collector. ## Deferred (non-load-bearing) -- Port collision with an operator receiver on 4317/4318 waits for - compass-stack `--collector-*-port` flags. -- `pasta_options` forwards already reach agents on podman's default network; - checking every start is a separate egress change (RIG-5249). +- Default-network `pasta_options` forwards already reach agents (RIG-5249). ## Open Questions -1. **Where the collector runs (load-bearing).** The forward reaches only the - runner host's loopback. (a) Refuse the flag unless `--server` is loopback. - (b) A collector per runner host forwards to the server's. (c) The runner - relays agent OTLP to a remote collector (OQ2 B). **Recommendation:** (a) - now; (b) when a remote runner needs agent telemetry. -2. **Pasta or a runner relay (load-bearing).** A: pasta now as an explicit - stepping stone; the microVM follow-up uses a relay over the agent socket, - which guestd already forwards (`gatewayPortKey` in - `go/internal/guestd/cmdline.go`). B: build the relay now; it overwrites +1. **Remote runners.** A forward reaches only the runner host's loopback, so + a remote runner names a remote `--otel-agent-endpoint` (a), or its host + runs a forwarding collector (b). **Recommendation:** (a); (b) on demand. +2. **Pasta or a runner relay (load-bearing).** A: pasta now; microVM later + relays over the agent socket guestd forwards (`gatewayPortKey`, + `go/internal/guestd/cmdline.go`). B: a relay now, stamping `compass.session.id` from `g.sessions.Session(g.containerName)` - (`go/internal/runner/gateway/publish.go`), but the agent exporter needs a - unix-socket transport. Either relay reverses - `docs/designs/repo/compass-agent-effect-otel/design.md` Open Questions 1: - "never the socket/vsock gateway". **Recommendation:** A; an approved relay - replaces pasta on every tier. -3. **Forged session ids (load-bearing).** The untrusted agent stamps - `compass.session.id`, and every agent arrives as `127.0.0.1`. A forged id - files one session's telemetry under another, corrupting every D7 join and - dashboard on that key. Billing reads the D5 event log, unaffected. (a) D7 - treats Plane B session ids as untrusted until a relay stamps them. (b) Hold - the route until OQ2 B. **Recommendation:** (a); multi-tenant waits for - OQ2 B. + (`go/internal/runner/gateway/publish.go`); agents need a unix-socket + exporter. Any relay reverses Open Questions 1 of + `docs/designs/repo/compass-agent-effect-otel/design.md`, "never the + socket/vsock gateway". **Recommendation:** A; an approved relay replaces + pasta everywhere. +3. **Forged session ids (load-bearing).** Untrusted agents stamp + `compass.session.id`, so a forged id files telemetry under another session + and corrupts D7 joins; billing reads the D5 event log. (a) D7 treats + Plane B session ids as untrusted until a relay stamps them. (b) Hold the + route until OQ2 B. **Recommendation:** (a); multi-tenant waits for OQ2 B. +4. **Remote agent targets.** Allowlists match hosts, not ports, so agents + reach every port on that host. (a) Accept: the operator named it. (b) + Refuse, silencing agents under `--otel-external`. **Recommendation:** (a); + (b) breaks D3. From a439bdeef6eacee8f01132bd1d58ac2306a16c93 Mon Sep 17 00:00:00 2001 From: mintaka Date: Sun, 11 Oct 2026 23:07:40 -0400 Subject: [PATCH 4/8] docs(design): fold collector-routing review round 3 (RIG-4611) External auth via the headers file, shared endpoint validation for the runner flag, and an explicit off state reaching agents. Spec-impact: none. Refs RIG-4611 Co-authored-by: Matt Wilkinson --- .../compass-agent-collector-routing/design.md | 262 +++++++++--------- 1 file changed, 134 insertions(+), 128 deletions(-) diff --git a/docs/designs/observability/compass-agent-collector-routing/design.md b/docs/designs/observability/compass-agent-collector-routing/design.md index a004659cd..b324b5eba 100644 --- a/docs/designs/observability/compass-agent-collector-routing/design.md +++ b/docs/designs/observability/compass-agent-collector-routing/design.md @@ -6,74 +6,67 @@ Amendment record of the observability-architecture design ([`../compass-observability-architecture/design.md`](../compass-observability-architecture/design.md), §D3 and §T4). The parent stays frozen. -Ledger-impact: none. This record adds no `DL-NNN` decision. +Ledger-impact: none. ## Problem / Intent Matt ruled on 2026-10-11 (RIG-5144) that agent containers ship OTLP through -the bundled collector to an operator upstream. Parent T4 promises "the -Runner-materialized env pointing agent emission at the bundled collector" but -picks no route. This record decides C-a (upstream export), C-b (agent route) -and C-c (dev wiring). +the bundled collector to an operator upstream. Parent T4 points agent +emission "at the bundled collector" but picks no route. This record decides +C-a (upstream export), C-b (agent route) and C-c (dev wiring). ## Global Constraints -- D3 holds: with no export endpoint the collector config keeps today's `nop` - bytes. Exported telemetry stays in memory; only config and headers files - are written. -- No secret in a podman argv or the 0644 config. Endpoints are http(s) URLs - without userinfo, query or fragment; credentials live in the 0600 headers - file. -- Loopback means host `127.0.0.1` or `localhost`; other loopback forms are - refused. +- D3 holds: no export endpoint keeps today's `nop` config bytes, and only + config and headers files touch disk. +- No secret in argv or the 0644 config; credentials stay in the 0600 headers + file, then process env or 0600 files, never podman `-e`. +- `otlpenv.ParseEndpoint` admits http(s) URLs with a host, no userinfo, query + or fragment, and loopback only as `127.0.0.1` or `localhost`. - Server and runner emit only with an export endpoint or `--otel-external`, - and drop inherited `OTEL_*`. An agent route clears the env file's - `OTEL_EXPORTER_OTLP_*`. + and drop inherited `OTEL_*`. An agent route, or `off`, clears the env + file's `OTEL_EXPORTER_OTLP_*`. - Pasta needs Linux and rootless podman 4.6+. -- `DefaultCollectorImage` stays 0.140.0, so the exporter key is `otlphttp`; - devenv's 0.155.0 validates the same config. -- `baseRuleset` is unchanged. Name operator receivers generically, e.g. - Grafana Alloy. +- `DefaultCollectorImage` stays 0.140.0 (`otlphttp`); devenv's 0.155.0 + validates the same config. +- `baseRuleset` is unchanged. Name receivers generically, e.g. Grafana Alloy. ## Approach **C-a, upstream export.** `collectorConfigYAML` -(`go/internal/stack/collector_container.go`) hard-codes `nop: {}`. A new -`--otel-export-endpoint` swaps in one `otlphttp` exporter behind a -`memory_limiter` and a 4 MiB request cap, since untrusted agents now reach -the collector. Each start replaces a 0600 copy of the optional headers file, -mounted with `--userns=keep-id:uid=10001,gid=10001`. +(`go/internal/stack/collector_container.go`) hard-codes `nop: {}`. +`--otel-export-endpoint` swaps in one `otlphttp` exporter, behind a +`memory_limiter` and a 4 MiB request cap because untrusted agents now reach +the collector. **C-b, agent route: a pinned pasta forward.** Agents get -`--network pasta:-T,14318:,-U,none`. passt(1) defines `-T` as -"TCP port forwarding from target namespace to init namespace", default -**auto**. Traffic leaves on `lo`, which `baseRuleset` -(`go/internal/runtime/egress.go`) accepts. A podman 5.8.6 smoke opened only -the forward; `-U,auto` and `-T,auto` controls leaked. +`--network pasta:-T,14318:,-U,none`. passt(1) `-T` forwards +"from target namespace to init namespace", default **auto**. Traffic leaves +on `lo`, which `baseRuleset` (`go/internal/runtime/egress.go`) accepts. A +podman 5.8.6 smoke opened only the forward; `auto` controls leaked. Podman appends `containers.conf` `pasta_options`, unseen by `podman info`: a configured `-T 24460` reached an armed agent. So a canary, -`pasta:-T,none,-U,none`, must start and hold no sockets; any configured `-T` -or `-U` fails it. - -The agent merges `$HOME/.compass/env` into `process.env`, and "File wins per -key it defines" (`main` in `packages/compass-agent/src/cli.ts`). The file -cannot set `COMPASS_*` (`isReservedEnvKey`), so the route travels as -`COMPASS_OTLP_ENDPOINT`; after the merge the agent clears the file's -`OTEL_EXPORTER_OTLP_*` keys, so no `grpc` protocol or stale headers survive. - -**Agent targets.** compass-runner's `--otel-agent-endpoint` names the agents' -target. The stack passes the collector URL `http://127.0.0.1:`, -whatever its `ListenAddr`, or the `--otel-external` URL. A remote host joins -an armed allowlist (OQ4); a loopback target needs pasta, else agents get no -emitter. - -**`--otel-external`.** D3 points every surface at it; today no child receives -it. CB1 passes it to server, runner and agents, so it becomes an http(s) URL: -the Go emitters use `otlptracehttp` (`go/internal/otel/provider.go`). -Refusing a bare `host:port` breaks callers, so CB1's commit carries `!` for -release-please. A new `--otel-off` replaces the dummy `127.0.0.1:4317` tests -pass: no collector, endpoint or route. +`pasta:-T,none,-U,none`, must start and hold no sockets. + +The agent merges `$HOME/.compass/env` into `process.env`; "File wins per key +it defines" (`main`, `packages/compass-agent/src/cli.ts`). The file cannot +set `COMPASS_*` (`isReservedEnvKey`), so the route travels as +`COMPASS_OTLP_ENDPOINT`. + +**Agent targets.** compass-runner's `--otel-agent-endpoint` takes the +collector URL, the `--otel-external` URL, or `off`. Remote hosts join an +armed allowlist; loopback needs pasta. + +**`--otel-external`.** D3 points every surface at it, yet no child receives +it. CB1 passes it on as an http(s) URL for `otlptracehttp`; the commit +carries `!`. `--otel-off` replaces the dummy `127.0.0.1:4317` tests pass. +The headers file now authenticates it as `OTEL_EXPORTER_OTLP_HEADERS`. Exec +env is podman `-e` argv (`execSpec`, `go/internal/runner/agent_exec.go`), so +agents read a 0600 `$HOME/.compass/otlp-headers` written over stdin (OQ4). + +New leaf `go/internal/otel/otlpenv` shares the checks with compass-runner +and keeps the OTel SDK out of the stack core. **C-c.** CC1 runs nixpkgs `otelcol-contrib` on the bundled renderer's config. @@ -90,13 +83,14 @@ pass: no collector, endpoint or route. Files: `go/internal/stack/{config,deps,stack,collector_container}.go`, `go/internal/stack/adapters/collector_container.go`, -`go/internal/runtime/podman.go`, `go/cmd/compass-stack/main.go`. +`go/internal/runtime/podman.go`, `go/internal/otel/otlpenv/otlpenv.go`, +`go/cmd/compass-stack/main.go`. Interfaces: ```go // stack -Config.CollectorExportEndpoint, Config.CollectorExportHeadersFile string +Config.CollectorExportEndpoint, Config.OTLPHeadersFile string CollectorContainerSpec.ExportHeadersFile string // "" = no mount, no keep-id CollectorContainerSpec.UpstreamForwardPort uint16 // 0 = none type CollectorYAMLParams struct { @@ -113,30 +107,36 @@ const CollectorContainerUpstreamPort = "14319" // runtime: Linux, podman >= 4.6, rootless, Host.Pasta.Executable func (p *PodmanCLI) VerifyPastaSupport(ctx context.Context) error + +// otlpenv +func ParseEndpoint(raw string) (*url.URL, error) ``` - `--otel-export-endpoint` and `--otel-export-headers-file` fall back to - `COMPASS_OTEL_EXPORT_*`. `Validate` refuses a bad URL, headers without an - endpoint, an endpoint with `ExternalOTLPEndpoint`, and a loopback upstream - off Linux or in another form. + `COMPASS_OTEL_EXPORT_*`. `Validate` runs `ParseEndpoint` and refuses + headers with neither endpoint, both endpoints, and a loopback upstream off + Linux. - `collectorContainerSpec`, not the renderer, rewrites a loopback upstream to `127.0.0.1:14319` (`TLSServerName` for https) and adds `--network pasta:-T,14319:,-U,none`; `startCollector` first calls `Deps.PastaVerifier`. -- `Start` renames a 0600 temp copy over the old headers; unlike - `ensureGatewayToken`, it never keeps an existing file. +- `Start` renames a 0600 temp copy over the old headers, mounted with + `--userns=keep-id:uid=10001,gid=10001`; unlike `ensureGatewayToken`, it + never keeps an existing file. Red, then green: - `TestCollectorConfigYAMLExportsUpstream`, red today: `otlphttp` and - `memory_limiter` in all three pipelines, `max_request_body_size`, no `nop` - or `file_storage`. + `memory_limiter` in all three pipelines, `max_request_body_size`, no `nop`. - `TestCollectorConfigYAMLRealizesD3Posture`: zero params give today's bytes. +- `TestParseEndpoint` accepts `127.0.0.1`, `localhost` and remote URLs; + refuses `127.0.0.1:4317`, `otlp.example.com:4317`, `127.0.0.2`, `[::1]`, + `0.0.0.0`, no host, userinfo, a query and a bare `#`. - `TestCollectorContainerSpecForwardsLoopbackUpstream`, `TestResolveConfigCollectorExportFlags`, `TestVerifyPastaSupport`, - `TestStartCollectorRefusesForwardWithoutPasta`. -- `TestCollectorStartReplacesHeadersCopy` (0600, rotation, source untouched); - `TestCollectorRunArgsExportContract` (no header value in argv or config). + `TestStartCollectorRefusesForwardWithoutPasta`, + `TestCollectorStartReplacesHeadersCopy`; `TestCollectorRunArgsExportContract` + (no header value in argv or config). - Smoke: `compass-stack up` delivers a span and header to a local receiver. ### CB1 — Route every emitter @@ -144,7 +144,7 @@ Red, then green: Files: `go/internal/runtime/{podman,agent,host_backend}.go`, `go/internal/runner/{spec,host,agent_exec}.go`, `go/internal/stack/{config,spec,stack}.go`, -`go/internal/stack/adapters/process.go`, +`go/internal/stack/adapters/process.go`, `go/internal/otel/otlpenv`, `go/cmd/{compass-runner,compass-stack}/main.go`, `packages/compass-agent/src/cli.ts`, and the migrated callers. @@ -154,8 +154,7 @@ Interfaces: // runtime type LoopbackForward struct{ Port, HostPort uint16 } // zero = none WorkloadSpec.LoopbackForward, AgentSpec.LoopbackForward LoopbackForward -// canary: errors on failure or any socket -func (p *PodmanCLI) VerifyNoPastaForwards(ctx context.Context, image string) error +func (p *PodmanCLI) VerifyNoPastaForwards(ctx context.Context, image string) error // canary // runner type pastaForwarder interface { // *runtime.PodmanCLI @@ -163,6 +162,7 @@ type pastaForwarder interface { // *runtime.PodmanCLI VerifyNoPastaForwards(ctx context.Context, image string) error } const AgentCollectorPort = 14318 +const AgentOTLPOff = "off" type AgentRoute struct { Forward runtime.LoopbackForward Endpoint string // "" = no agent emitter @@ -171,61 +171,70 @@ type AgentRoute struct { func ResolveAgentRoute(ctx context.Context, engine runtime.WorkloadRuntime, target *url.URL, image string) (AgentRoute, error) SpecDefaults.LoopbackForward runtime.LoopbackForward AgentHostConfig.OTLPEndpoint, AgentEnv.OTLPEndpoint string // COMPASS_OTLP_ENDPOINT +AgentHostConfig.OTLPHeaders string // runner env COMPASS_OTEL_AGENT_HEADERS +func parseAgentEndpoint(raw string) (target *url.URL, off bool, err error) // compass-runner -// stack +// otlpenv: YAML map → name=url.PathEscape(value),… +func HeadersValue(path string) (string, error) + +// stack; serverSpec and runnerSpec take the encoded headers Config.OTLPOff bool func (c Config) bundlesCollector() bool // replaces ExternalOTLPEndpoint == "" gates func childEnv(inherited, overlay []string) []string // adapters; drops inherited OTEL_* ``` ```ts -// main passes env.COMPASS_OTLP_ENDPOINT after the env-file merge -export function applyCollectorRoute(route: string | undefined, target = process.env): void +// called after the env-file merge +export function applyCollectorRoute(route: string | undefined, headers: string, target = process.env): void ``` - `ResolveAgentRoute`: no target, empty; host tier (`runtime.TierOf`), direct; remote, verbatim plus `AllowHost`. Loopback needs a `pastaForwarder` passing `VerifyPastaSupport`, else an empty, logged route; - then the canary runs, and its failure errors; then the forward keeps scheme - and host on port 14318. -- compass-runner gains `--otel-agent-endpoint` - (`COMPASS_OTEL_AGENT_ENDPOINT`) and appends `AllowHost` to a configured - policy after `ResolveEgress`. + then a failed canary errors; then the forward keeps scheme and host on + port 14318. +- `--otel-agent-endpoint` (`COMPASS_OTEL_AGENT_ENDPOINT`) fails startup on a + `parseAgentEndpoint` error; `off` yields `AgentRoute{Endpoint: + AgentOTLPOff}`. `main` appends `AllowHost` to a configured policy after + `ResolveEgress`. +- Before a URL-routed agent exec, `WriteAgentFile` writes `OTLPHeaders`, + possibly empty, to `.compass/otlp-headers`. - `createArgs` renders `--network pasta:-T,:,-U,none`; zero - leaves the argv unchanged. `envSlice` and `childEnv` drop inherited - `OTEL_*`; `ProcessSupervisor.Start` uses `childEnv`, which covers the stack - `embedded.RunStackUp` execs. -- `serverSpec` sets `OTEL_EXPORTER_OTLP_ENDPOINT` and `runnerSpec` adds - `--otel-agent-endpoint`: the collector URL when exporting, else the - `--otel-external` URL, else neither. -- `--otel-off` excludes `--otel-external` and the export flags; `Validate` - refuses a non-URL `ExternalOTLPEndpoint`. -- Migration: `go/e2e/fixture.go` sets `OTLPOff`; the `127.0.0.1:4317` runs in + leaves argv unchanged. `envSlice` and `childEnv` (used by + `ProcessSupervisor.Start`) drop inherited `OTEL_*`. +- Server `OTEL_EXPORTER_OTLP_ENDPOINT` and runner `--otel-agent-endpoint` + carry the collector URL when exporting, else the external URL with one + `Up`-time `HeadersValue` as `OTEL_EXPORTER_OTLP_HEADERS` and the runner's + `COMPASS_OTEL_AGENT_HEADERS`. `--otel-off` gives the runner `off`. +- `Validate` runs `ParseEndpoint` on `ExternalOTLPEndpoint` and admits the + headers file with it; `--otel-off` excludes all three. +- Migration: `go/e2e/fixture.go` sets `OTLPOff`; `go/cmd/compass-stack/{container_postgres,nats}_podman_test.go` pass - `--otel-off`; `TestExternalOTLPUpDown` passes `http://127.0.0.1:4318`; - `otlp.example.com:4317` fixtures become URLs; the `ci.yml` comment names - `--otel-off`. + `--otel-off`; `TestExternalOTLPUpDown` and `otlp.example.com:4317` + fixtures use URLs; the `ci.yml` comment names `--otel-off`. Red, then green: -- `TestCreateArgsRendersLoopbackForward`, - `TestExecSpecSetsCollectorRouteOnlyWhenConfigured`, - `TestBuildSpecMapsLoopbackForward`, `TestEnvSliceDropsInheritedOTELKeys`. -- `TestResolveAgentRoute`: no target, host, remote, no capability, http and - https forwards, failed canary; with pasta unsupported a counting fake - proves the canary never ran. +- `TestCreateArgsRendersLoopbackForward`, `TestBuildSpecMapsLoopbackForward`, + `TestEnvSliceDropsInheritedOTELKeys`; + `TestExecSpecSetsCollectorRouteOnlyWhenConfigured`, including `off`. +- `TestResolveAgentRoute` covers each branch; a counting fake proves an + unsupported host skips the canary. +- `TestParseAgentEndpoint` (table): empty, `off`, each `TestParseEndpoint` + refusal. - `TestStartDropsInheritedOTELKeys` (a `trapecho` child): an inherited base - endpoint with the route off; an inherited - `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT` with it on. -- `TestServerAndRunnerSpecOTLPEndpoint`: collector, external, `nop`, off; a - `0.0.0.0` `ListenAddr` keeps the loopback collector URL. + endpoint with the route off; an inherited traces endpoint with it on. +- `TestServerAndRunnerSpecOTLPEndpoint`: collector, external (headers in + `Env`, never `Args`), `nop`, off; a `0.0.0.0` `ListenAddr` keeps loopback. +- `TestHeadersValue` escapes `,`, `;` and `%` and refuses a newline; + `TestStartWritesAgentOTLPHeaders`: stdin, never exec argv. - `TestResolveConfigOTelOff`; `TestConfigValidate` refuses `127.0.0.1:4317`. -- `cli.test.ts`: the route beats a file endpoint and clears a file `grpc` - protocol and headers; with no route a file endpoint stays. -- Podman-gated: `TestPastaForwardReachesOnlyCollectorPort` (no host-loopback - UDP or late-bound TCP; red with `-U,auto` or `-T,auto`) and - `TestVerifyNoPastaForwardsRefusesConfOptions` (a `-T` in - `CONTAINERS_CONF_OVERRIDE`). +- `cli.test.ts`: the route beats a file endpoint, clears file protocol and + headers, and exports `otlp-headers`; `off` with a file endpoint leaves no + endpoint and skips telemetry init; no route keeps a file endpoint. +- Podman-gated: `TestPastaForwardReachesOnlyCollectorPort` (no host UDP or + late TCP; red with `auto`) and `TestVerifyNoPastaForwardsRefusesConfOptions` + (a `-T` in `CONTAINERS_CONF_OVERRIDE`). ### CC1 — devenv runs the collector @@ -233,19 +242,18 @@ Files: `devenv.nix`, `go/cmd/compass-stack/main.go`. Interfaces: `compass-stack collector-config --grpc ADDR --http ADDR --health ADDR [--otel-export-endpoint URL] [--otel-export-headers-file PATH]` prints -`CollectorConfigYAML` with the endpoint verbatim and the host headers path. -`usageError` names five subcommands. +`CollectorConfigYAML`, endpoint verbatim, host headers path. `usageError` +names five subcommands. - When the gitignored `devenv.local.nix` sets - `env.COMPASS_OTEL_EXPORT_ENDPOINT`, `processes.otel-collector` renders into - the state dir and execs - `${pkgs.opentelemetry-collector-contrib}/bin/otelcol-contrib`. Server and - runner start `after` it with `OTEL_EXPORTER_OTLP_ENDPOINT`, and the runner - with `--otel-agent-endpoint`, both `http://127.0.0.1:`. + `env.COMPASS_OTEL_EXPORT_ENDPOINT`, `processes.otel-collector` execs + `${pkgs.opentelemetry-collector-contrib}/bin/otelcol-contrib` on it. Server + and runner start `after` it, with `OTEL_EXPORTER_OTLP_ENDPOINT` and + `--otel-agent-endpoint` at `http://127.0.0.1:`. -Red, then green: `devenv up` with an endpoint delivers nothing to a local -receiver today, and afterwards delivers `compass-server`, `compass-runner` and -`compass-agent` spans. `TestRunCollectorConfig` pins a loopback upstream. +Red, then green: `devenv up` with an endpoint delivers no spans today, then +server, runner and agent spans. `TestRunCollectorConfig` pins a loopback +upstream. ## Tasks @@ -259,24 +267,22 @@ receiver today, and afterwards delivers `compass-server`, `compass-runner` and ## Open Questions -1. **Remote runners.** A forward reaches only the runner host's loopback, so - a remote runner names a remote `--otel-agent-endpoint` (a), or its host - runs a forwarding collector (b). **Recommendation:** (a); (b) on demand. +1. **Remote runners.** Forwards reach only the runner host's loopback. (a) + Name a remote `--otel-agent-endpoint`. (b) Run a forwarding collector + there. **Recommendation:** (a); (b) on demand. 2. **Pasta or a runner relay (load-bearing).** A: pasta now; microVM later - relays over the agent socket guestd forwards (`gatewayPortKey`, + relays over the agent socket (`gatewayPortKey`, `go/internal/guestd/cmdline.go`). B: a relay now, stamping `compass.session.id` from `g.sessions.Session(g.containerName)` - (`go/internal/runner/gateway/publish.go`); agents need a unix-socket - exporter. Any relay reverses Open Questions 1 of - `docs/designs/repo/compass-agent-effect-otel/design.md`, "never the - socket/vsock gateway". **Recommendation:** A; an approved relay replaces - pasta everywhere. -3. **Forged session ids (load-bearing).** Untrusted agents stamp - `compass.session.id`, so a forged id files telemetry under another session - and corrupts D7 joins; billing reads the D5 event log. (a) D7 treats - Plane B session ids as untrusted until a relay stamps them. (b) Hold the + (`go/internal/runner/gateway/publish.go`). Any relay reverses OQ1 of + `docs/designs/repo/compass-agent-effect-otel/design.md`. + **Recommendation:** A; an approved relay replaces pasta everywhere. +3. **Forged session ids (load-bearing).** Agents stamp `compass.session.id`, + so a forged id corrupts D7 joins; billing reads the D5 event log. (a) D7 + treats Plane B ids as untrusted until a relay stamps them. (b) Hold the route until OQ2 B. **Recommendation:** (a); multi-tenant waits for OQ2 B. -4. **Remote agent targets.** Allowlists match hosts, not ports, so agents - reach every port on that host. (a) Accept: the operator named it. (b) - Refuse, silencing agents under `--otel-external`. **Recommendation:** (a); - (b) breaks D3. +4. **External targets.** Allowlists match hosts, not ports, and agents can + read the external headers. (a) Accept: the operator named both, and + `--otel-export-endpoint` keeps credentials in the collector. (b) Refuse, + silencing agents under `--otel-external`. **Recommendation:** (a); (b) + breaks D3. From b6e462ad5447c6aa00ef08bbaeb3a6ccb5ba0e82 Mon Sep 17 00:00:00 2001 From: mintaka Date: Mon, 12 Oct 2026 19:46:03 -0400 Subject: [PATCH 5/8] docs(design)!: agent OTLP rides the agent socket to a host collector (RIG-4611) Rewrite around Matt's RIG-5237 rulings: a collector per runner host, OTLP/HTTP over the agent socket stamped by the Runner gateway. Lifts the effect-otel no-socket ruling for agent telemetry; records DL-514. Spec-impact: none. Refs RIG-4611 Co-authored-by: Matt Wilkinson --- .../designs/decisions/observability/DL-514.md | 6 + .../compass-agent-collector-routing/design.md | 505 +++++++++--------- 2 files changed, 263 insertions(+), 248 deletions(-) create mode 100644 docs/designs/decisions/observability/DL-514.md diff --git a/docs/designs/decisions/observability/DL-514.md b/docs/designs/decisions/observability/DL-514.md new file mode 100644 index 000000000..237935969 --- /dev/null +++ b/docs/designs/decisions/observability/DL-514.md @@ -0,0 +1,6 @@ +--- +id: DL-514 +decision: "Agent OTLP rides the agent socket (Matt, RIG-5237): agents export OTLP/HTTP to the Runner gateway's `/v1/traces`, `/v1/metrics` and `/v1/logs`, which stamp the bound `compass.session.id` and `service.name` and relay to a collector on each runner host; the socket stays the agent's only channel, with no OTLP egress. Lifts the effect-otel ruling that agent OTLP never uses the socket/vsock gateway and the agent deployer contracts' env-file endpoint, and moves loop-otel's SDK session id to `compass.agent.sdk_session.id`" +status: "Active (Matt, 2026-10-12)" +record: ../../observability/compass-agent-collector-routing/design.md#rulings +--- diff --git a/docs/designs/observability/compass-agent-collector-routing/design.md b/docs/designs/observability/compass-agent-collector-routing/design.md index b324b5eba..eada8075e 100644 --- a/docs/designs/observability/compass-agent-collector-routing/design.md +++ b/docs/designs/observability/compass-agent-collector-routing/design.md @@ -1,4 +1,4 @@ -# Agent containers export through the bundled collector +# Agent OTLP rides the agent socket to a host collector Tracker: RIG-4611 @@ -6,283 +6,292 @@ Amendment record of the observability-architecture design ([`../compass-observability-architecture/design.md`](../compass-observability-architecture/design.md), §D3 and §T4). The parent stays frozen. -Ledger-impact: none. +Ledger-impact: DL-514. ## Problem / Intent -Matt ruled on 2026-10-11 (RIG-5144) that agent containers ship OTLP through -the bundled collector to an operator upstream. Parent T4 points agent -emission "at the bundled collector" but picks no route. This record decides -C-a (upstream export), C-b (agent route) and C-c (dev wiring). +Matt ruled (RIG-5144) that agents ship OTLP through a collector to an +operator upstream. Parent T4 picks no route. This record decides the +collector's upstream export (C-a), the agent route (C-b) and dev wiring (C-c). -## Global Constraints - -- D3 holds: no export endpoint keeps today's `nop` config bytes, and only - config and headers files touch disk. -- No secret in argv or the 0644 config; credentials stay in the 0600 headers - file, then process env or 0600 files, never podman `-e`. -- `otlpenv.ParseEndpoint` admits http(s) URLs with a host, no userinfo, query - or fragment, and loopback only as `127.0.0.1` or `localhost`. -- Server and runner emit only with an export endpoint or `--otel-external`, - and drop inherited `OTEL_*`. An agent route, or `off`, clears the env - file's `OTEL_EXPORTER_OTLP_*`. -- Pasta needs Linux and rootless podman 4.6+. -- `DefaultCollectorImage` stays 0.140.0 (`otlphttp`); devenv's 0.155.0 - validates the same config. -- `baseRuleset` is unchanged. Name receivers generically, e.g. Grafana Alloy. - -## Approach +## Rulings -**C-a, upstream export.** `collectorConfigYAML` -(`go/internal/stack/collector_container.go`) hard-codes `nop: {}`. -`--otel-export-endpoint` swaps in one `otlphttp` exporter, behind a -`memory_limiter` and a 4 MiB request cap because untrusted agents now reach -the collector. +Matt, RIG-5237 (2026-10-11 and 2026-10-12): -**C-b, agent route: a pinned pasta forward.** Agents get -`--network pasta:-T,14318:,-U,none`. passt(1) `-T` forwards -"from target namespace to init namespace", default **auto**. Traffic leaves -on `lo`, which `baseRuleset` (`go/internal/runtime/egress.go`) accepts. A -podman 5.8.6 smoke opened only the forward; `auto` controls leaked. +1. **OQ1 B: a collector on each runner host** forwards upstream. +2. **OQ2 B, transport A: agents export OTLP/HTTP over the agent socket.** The + Runner gateway serves `/v1/traces` and `/v1/metrics` and relays to the host + collector. No pasta, and no egress allowlist entry for OTLP. +3. **The socket stays the agent's only channel to its Runner.** -Podman appends `containers.conf` `pasta_options`, unseen by `podman info`: a -configured `-T 24460` reached an armed agent. So a canary, -`pasta:-T,none,-U,none`, must start and hold no sockets. +Amended frozen texts: -The agent merges `$HOME/.compass/env` into `process.env`; "File wins per key -it defines" (`main`, `packages/compass-agent/src/cli.ts`). The file cannot -set `COMPASS_*` (`isReservedEnvKey`), so the route travels as -`COMPASS_OTLP_ENDPOINT`. +- `docs/designs/repo/compass-agent-effect-otel/design.md`, Open Questions + item 1 and Decision 3 ("never the socket/vsock gateway"): lifted for agent + telemetry, for one transport on every tier, Runner-stamped attribution and + no agent egress. +- `docs/designs/observability/compass-agent-loop-otel/design.md`, Decision 3a: + the join key is the gateway-stamped `compass.session.id`; the SDK id moves + to `compass.agent.sdk_session.id`. +- Both agent deployer contracts + (`compass-agent-effect-otel/otel-endpoint-deployer-contract.md`, + `compass-agent-loop-otel/deployer-contract.md`): env-file endpoints are + ignored; the gateway sets `service.name`. +- Parent D3 "agent-side emission stays exactly as shipped": superseded for the + route. Off by default holds. -**Agent targets.** compass-runner's `--otel-agent-endpoint` takes the -collector URL, the `--otel-external` URL, or `off`. Remote hosts join an -armed allowlist; loopback needs pasta. +## Global Constraints -**`--otel-external`.** D3 points every surface at it, yet no child receives -it. CB1 passes it on as an http(s) URL for `otlptracehttp`; the commit -carries `!`. `--otel-off` replaces the dummy `127.0.0.1:4317` tests pass. -The headers file now authenticates it as `OTEL_EXPORTER_OTLP_HEADERS`. Exec -env is podman `-e` argv (`execSpec`, `go/internal/runner/agent_exec.go`), so -agents read a 0600 `$HOME/.compass/otlp-headers` written over stdin (OQ4). +- D3 holds: no export endpoint keeps today's `nop` config bytes; only config + and headers files touch disk. +- No secret in argv or the 0644 config. Upstream credentials live in the 0600 + headers file and server or runner env, never in agents. The host tier shares + the Runner's uid (`go/internal/runtime/host_backend.go`): no such boundary. +- Only the gateway sets `compass.session.id`, `compass.telemetry.origin` and + the agent's `service.name` on relayed telemetry. +- Every endpoint passes `otlpenv.ParseEndpoint` (CA1). +- Server and runner emit or relay only with an endpoint, and drop inherited + `OTEL_*`. +- `DefaultCollectorImage` stays 0.140.0; devenv's 0.155.0 validates the same + config. +- Name receivers generically, e.g. Grafana Alloy. +- Breaking (`!`): `--otel-external` takes a URL; env-file agent endpoints are + ignored; compass-runner refuses per-signal and TLS `OTEL_EXPORTER_OTLP_*` + keys. -New leaf `go/internal/otel/otlpenv` shares the checks with compass-runner -and keeps the OTel SDK out of the stack core. +## Approach -**C-c.** CC1 runs nixpkgs `otelcol-contrib` on the bundled renderer's config. +**C-a: the collector exports.** `collectorConfigYAML` +(`go/internal/stack/collector_container.go`) hard-codes `nop: {}`. With +`--otel-export-endpoint` (fallback `COMPASS_OTEL_EXPORT_ENDPOINT`) it renders +`otlphttp` behind `memory_limiter` (256 MiB, 64 MiB spike) and an 8 MiB +receiver cap. `--otel-export-headers-file` is copied 0600 to +`/run/compass/otlp-headers.yaml` under `--userns=keep-id:uid=10001,gid=10001` +and replaced on every `Start`. + +**C-b: the host collector.** The Runner relays agent OTLP to the endpoint its +own SDK reads, `OTEL_EXPORTER_OTLP_ENDPOINT` plus `_HEADERS` (`setupOtel`, +`go/cmd/compass-runner/main.go`). `compass-stack` sets it on server and runner: + +- `--otel-export-endpoint`: the bundled collector, + `http://127.0.0.1:`. Without it nothing is set (D3). +- `--otel-external `: the operator's own host collector, which owns + batching, retry and upstream (ruling 1). `--otel-export-headers-file` rides + it as `_HEADERS` in `Env`; adding `--otel-export-endpoint` is a usage error. +- `--otel-off`: no collector, nothing set. + +A remote runner host's operator runs the collector (e.g. Grafana Alloy, or +otelcol-contrib with `compass-stack collector-config`). compass-runner refuses +a bad endpoint and per-signal or TLS keys, so relay and exporters agree. + +**C-b: the gateway handlers.** `gateway.Serve` +(`go/internal/runner/gateway/gateway.go`) mounts `POST /v1/traces`, +`/v1/metrics` and `/v1/logs` (Resolved 2) only when `Deps.OTLPRelay` is set. +Each handler: + +1. Resolves `g.sessions.Session(g.containerName)`; unbound or empty is 403, as + `(*Gateway).Publish` fails closed. +2. Answers 415 unless `application/x-protobuf` with identity encoding. +3. Takes one of 4 per-Gateway slots and 16 MiB of a Runner-wide 128 MiB + budget, else answers 503 with `Retry-After: 1` before reading. +4. Arms a 5 s read deadline (as `withBodyReadDeadline`, + `go/server/network_door.go`), reads at most 4 MiB (413), and decodes with + `go.opentelemetry.io/proto/otlp`, promoted to direct (400 if malformed). +5. Deletes `compass.session.id` from every scope, span, event, link, data + point, exemplar and log record. On each resource it sets + `compass.session.id` to the bound id that server spans carry + (`go/internal/delivery/dispatch.go`), `compass.telemetry.origin=agent` and + `service.name=compass-agent`. +6. Re-encodes (over 8 MiB is 413) and POSTs under `r.Context()` with an 8 s + deadline, inside OTel JS's 10 s timeout. Upstream 2xx is 200; 429, 502, 503 + or 504 is 503 with its `Retry-After`; 413 stays; another 4xx is 400; else + 502. Only transient failures retry (`isExportHTTPErrorRetryable`). + +**C-b: the agent.** When it relays, the Runner sets `COMPASS_AGENT_OTLP=1`, a +reserved `COMPASS_*` key the env file cannot set. After the env-file merge the +agent deletes every `OTEL_EXPORTER_OTLP_*` key and logs one `[compass-agent]` +line naming them. With the flag it writes the SDK session id as +`compass.agent.sdk_session.id`. + +- `makeOtelLayer` (`packages/compass-agent/src/transport/otel-layer.ts`) builds + exporters with `url: "http://compass-agent/v1/"` and + `httpAgentOptions: { socketPath }` (Bun probe-proven). +- The SDK loop's exporters (`registerProviders`, `@oh-my-pi/pi-coding-agent`) + read env once, at construction. A `Bun.serve` proxy on `127.0.0.1:0` checks + a random per-process token header, caps bodies at 4 MiB and forwards with + `fetch(…, { unix })`, as `runInGuestProbe` + (`go/internal/runner/e2e_vsock_gateway_microvm_test.go`) does. Its endpoint + and token are set only around `await telemetryHooks.init()`, so tool + subprocesses never inherit them. Egress accepts `oif lo` (`NftScript`). + +| Tier | Socket reaches the agent by | OTLP | +| --- | --- | --- | +| podman | bind mount at `/run/compass/agent.sock` | covered | +| microVM | guestd `gatewayProxy`, a byte splice to vsock | covered | +| host | `COMPASS_AGENT_SOCKET_PATH` in the state dir | covered; no boundary | +| apple-container | `--volume` over virtiofs | gap until the vsock transport lands (`docs/designs/platform/apple-container-macos-runner/design.md`) | + +OTLP adds no gap of its own. ## Alternatives considered -- **`host.containers.internal`.** Allowlists match hosts, not ports. -- **A shared podman network.** Agents reach each other. -- **Podman secrets or host networking.** Secret state; breaks `nop` bytes. -- **Parsing `containers.conf`.** Several sources; the canary sees the merge. +- **Pasta forward** (earlier draft): podman only, agent-asserted attribution; + its unshipped helpers are dropped and RIG-5249 becomes moot. +- **Direct export through the egress allowlist** (effect-otel ruling (a)): + credentials in every agent, forgeable attribution. ## Plan -### CA1 — Collector upstream export +### CA1: collector upstream export -Files: `go/internal/stack/{config,deps,stack,collector_container}.go`, +Files: `go/internal/stack/{collector_container,config}.go`, `go/internal/stack/adapters/collector_container.go`, -`go/internal/runtime/podman.go`, `go/internal/otel/otlpenv/otlpenv.go`, -`go/cmd/compass-stack/main.go`. +`go/cmd/compass-stack/main.go`, new `go/internal/otel/otlpenv`. Interfaces: -```go -// stack -Config.CollectorExportEndpoint, Config.OTLPHeadersFile string -CollectorContainerSpec.ExportHeadersFile string // "" = no mount, no keep-id -CollectorContainerSpec.UpstreamForwardPort uint16 // 0 = none -type CollectorYAMLParams struct { - GRPC, HTTP, Health string // "" = 0.0.0.0: - Export string // "" = nop (D3) - TLSServerName string - HeadersPath string // "" = no headers key -} -func CollectorConfigYAML(p CollectorYAMLParams) string // replaces collectorConfigYAML -type PastaVerifier interface{ VerifyPastaSupport(ctx context.Context) error } -Deps.PastaVerifier PastaVerifier // buildDeps: runtime.NewPodmanCLI() -const CollectorHeadersMountPath = "/run/compass/otlp-headers.yaml" -const CollectorContainerUpstreamPort = "14319" - -// runtime: Linux, podman >= 4.6, rootless, Host.Pasta.Executable -func (p *PodmanCLI) VerifyPastaSupport(ctx context.Context) error - -// otlpenv -func ParseEndpoint(raw string) (*url.URL, error) -``` - -- `--otel-export-endpoint` and `--otel-export-headers-file` fall back to - `COMPASS_OTEL_EXPORT_*`. `Validate` runs `ParseEndpoint` and refuses - headers with neither endpoint, both endpoints, and a loopback upstream off - Linux. -- `collectorContainerSpec`, not the renderer, rewrites a loopback upstream to - `127.0.0.1:14319` (`TLSServerName` for https) and adds - `--network pasta:-T,14319:,-U,none`; `startCollector` first calls - `Deps.PastaVerifier`. -- `Start` renames a 0600 temp copy over the old headers, mounted with - `--userns=keep-id:uid=10001,gid=10001`; unlike `ensureGatewayToken`, it - never keeps an existing file. - -Red, then green: - -- `TestCollectorConfigYAMLExportsUpstream`, red today: `otlphttp` and - `memory_limiter` in all three pipelines, `max_request_body_size`, no `nop`. +- `Config.CollectorExportEndpoint`, `Config.CollectorExportHeadersFile string`. +- `CollectorConfigYAML(p CollectorYAMLParams) string`, with + `CollectorYAMLParams{GRPC, HTTP, Health, Export, HeadersPath string}`. +- `CollectorContainerSpec.ExportHeadersFile string`; + `const CollectorHeadersMountPath = "/run/compass/otlp-headers.yaml"`. +- `otlpenv.ParseEndpoint(raw string, withHeaders bool) (*url.URL, error)`. + +Red→green: + +- `TestCollectorConfigYAMLExportsUpstream` (red today): `otlphttp` and + `memory_limiter` in all three pipelines, an 8388608 body cap, no `nop`. - `TestCollectorConfigYAMLRealizesD3Posture`: zero params give today's bytes. -- `TestParseEndpoint` accepts `127.0.0.1`, `localhost` and remote URLs; - refuses `127.0.0.1:4317`, `otlp.example.com:4317`, `127.0.0.2`, `[::1]`, - `0.0.0.0`, no host, userinfo, a query and a bare `#`. -- `TestCollectorContainerSpecForwardsLoopbackUpstream`, - `TestResolveConfigCollectorExportFlags`, `TestVerifyPastaSupport`, - `TestStartCollectorRefusesForwardWithoutPasta`, - `TestCollectorStartReplacesHeadersCopy`; `TestCollectorRunArgsExportContract` +- `TestParseEndpoint`: accepts http(s) to `127.0.0.1`, `localhost` and remote + hosts; refuses bare `host:port`, `127.0.0.2`, `[::1]`, `0.0.0.0`, userinfo, + query, fragment, and remote `http` with headers. +- `TestCollectorStartReplacesHeadersCopy`; `TestCollectorRunArgsExportContract` (no header value in argv or config). -- Smoke: `compass-stack up` delivers a span and header to a local receiver. +- Smoke: `compass-stack up` delivers a span and the header to a local receiver. + +### CB1: gateway OTLP relay -### CB1 — Route every emitter +Files: `go/internal/runner/gateway/{gateway,otlp}.go`, +`go/internal/otel/otlprelay`, `go/go.mod`. -Files: `go/internal/runtime/{podman,agent,host_backend}.go`, -`go/internal/runner/{spec,host,agent_exec}.go`, -`go/internal/stack/{config,spec,stack}.go`, -`go/internal/stack/adapters/process.go`, `go/internal/otel/otlpenv`, -`go/cmd/{compass-runner,compass-stack}/main.go`, -`packages/compass-agent/src/cli.ts`, and the migrated callers. +Interfaces: + +- `type OTLPRelay interface { Export(ctx context.Context, path string, body []byte) error }`; + `Deps.OTLPRelay` (nil leaves the paths unmounted); + `Deps.OTLPBudget *semaphore.Weighted` (`golang.org/x/sync`). +- `otlprelay.New(endpoint *url.URL, headers map[string]string) *Relay`; a + non-2xx upstream returns `*otlprelay.StatusError{Code int, RetryAfter string}`. +- `const maxOTLPBodyBytes = 4 << 20`, `maxOTLPInFlight = 4`. +- Counter `compass.runner.otlp_relay.requests{signal,outcome}`, nil-guarded + like `compass.delivery.dispatched`; no session label. + +Red→green (hermetic, fake relay): + +- `TestOTLPStampOverridesAgentSession`: forged ids on a resource, span, data + point and log record, and `service.name=compass-server`, arrive as the bound + id and `compass-agent`. +- `TestOTLPUnboundContainerForbidden` (403, relay uncalled); `TestOTLPBodyCap` + (413); `TestOTLPRejectsCompressedBody` (gzip is 415). +- `TestOTLPSaturationShedsWith503`: a fifth request or empty budget, body unread. +- `TestOTLPRelayFailure`: 400 stays 400, 503 keeps `Retry-After`, transport + error is 502. +- `TestOTLPUnmountedWhenRelayNil` (404); `TestRelayPostsWithHeaders`. +- microVM e2e `TestVsockGateway_InGuestOTLPStamped`: a guest POST reaches a + fake `AgentHostConfig.OTLPRelay` with the bound id. The podman leg joins + `TestSocketOwnedAndConnectableInContainer` once its probe image lands. + +### CB2: runner and stack wiring + +Files: `go/cmd/compass-runner/main.go`, `go/internal/runner/{host,agent_exec}.go`, +`go/internal/runtime/host_backend.go`, `go/internal/stack/{config,spec}.go`, +`go/internal/stack/adapters/process.go`, `go/cmd/compass-stack/main.go`. Interfaces: -```go -// runtime -type LoopbackForward struct{ Port, HostPort uint16 } // zero = none -WorkloadSpec.LoopbackForward, AgentSpec.LoopbackForward LoopbackForward -func (p *PodmanCLI) VerifyNoPastaForwards(ctx context.Context, image string) error // canary - -// runner -type pastaForwarder interface { // *runtime.PodmanCLI - VerifyPastaSupport(ctx context.Context) error - VerifyNoPastaForwards(ctx context.Context, image string) error -} -const AgentCollectorPort = 14318 -const AgentOTLPOff = "off" -type AgentRoute struct { - Forward runtime.LoopbackForward - Endpoint string // "" = no agent emitter - AllowHost string // remote target host -} -func ResolveAgentRoute(ctx context.Context, engine runtime.WorkloadRuntime, target *url.URL, image string) (AgentRoute, error) -SpecDefaults.LoopbackForward runtime.LoopbackForward -AgentHostConfig.OTLPEndpoint, AgentEnv.OTLPEndpoint string // COMPASS_OTLP_ENDPOINT -AgentHostConfig.OTLPHeaders string // runner env COMPASS_OTEL_AGENT_HEADERS -func parseAgentEndpoint(raw string) (target *url.URL, off bool, err error) // compass-runner - -// otlpenv: YAML map → name=url.PathEscape(value),… -func HeadersValue(path string) (string, error) - -// stack; serverSpec and runnerSpec take the encoded headers -Config.OTLPOff bool -func (c Config) bundlesCollector() bool // replaces ExternalOTLPEndpoint == "" gates -func childEnv(inherited, overlay []string) []string // adapters; drops inherited OTEL_* -``` - -```ts -// called after the env-file merge -export function applyCollectorRoute(route: string | undefined, headers: string, target = process.env): void -``` - -- `ResolveAgentRoute`: no target, empty; host tier (`runtime.TierOf`), - direct; remote, verbatim plus `AllowHost`. Loopback needs a - `pastaForwarder` passing `VerifyPastaSupport`, else an empty, logged route; - then a failed canary errors; then the forward keeps scheme and host on - port 14318. -- `--otel-agent-endpoint` (`COMPASS_OTEL_AGENT_ENDPOINT`) fails startup on a - `parseAgentEndpoint` error; `off` yields `AgentRoute{Endpoint: - AgentOTLPOff}`. `main` appends `AllowHost` to a configured policy after - `ResolveEgress`. -- Before a URL-routed agent exec, `WriteAgentFile` writes `OTLPHeaders`, - possibly empty, to `.compass/otlp-headers`. -- `createArgs` renders `--network pasta:-T,:,-U,none`; zero - leaves argv unchanged. `envSlice` and `childEnv` (used by - `ProcessSupervisor.Start`) drop inherited `OTEL_*`. -- Server `OTEL_EXPORTER_OTLP_ENDPOINT` and runner `--otel-agent-endpoint` - carry the collector URL when exporting, else the external URL with one - `Up`-time `HeadersValue` as `OTEL_EXPORTER_OTLP_HEADERS` and the runner's - `COMPASS_OTEL_AGENT_HEADERS`. `--otel-off` gives the runner `off`. -- `Validate` runs `ParseEndpoint` on `ExternalOTLPEndpoint` and admits the - headers file with it; `--otel-off` excludes all three. -- Migration: `go/e2e/fixture.go` sets `OTLPOff`; - `go/cmd/compass-stack/{container_postgres,nats}_podman_test.go` pass - `--otel-off`; `TestExternalOTLPUpDown` and `otlp.example.com:4317` - fixtures use URLs; the `ci.yml` comment names `--otel-off`. - -Red, then green: - -- `TestCreateArgsRendersLoopbackForward`, `TestBuildSpecMapsLoopbackForward`, - `TestEnvSliceDropsInheritedOTELKeys`; - `TestExecSpecSetsCollectorRouteOnlyWhenConfigured`, including `off`. -- `TestResolveAgentRoute` covers each branch; a counting fake proves an - unsupported host skips the canary. -- `TestParseAgentEndpoint` (table): empty, `off`, each `TestParseEndpoint` - refusal. -- `TestStartDropsInheritedOTELKeys` (a `trapecho` child): an inherited base - endpoint with the route off; an inherited traces endpoint with it on. -- `TestServerAndRunnerSpecOTLPEndpoint`: collector, external (headers in - `Env`, never `Args`), `nop`, off; a `0.0.0.0` `ListenAddr` keeps loopback. -- `TestHeadersValue` escapes `,`, `;` and `%` and refuses a newline; - `TestStartWritesAgentOTLPHeaders`: stdin, never exec argv. -- `TestResolveConfigOTelOff`; `TestConfigValidate` refuses `127.0.0.1:4317`. -- `cli.test.ts`: the route beats a file endpoint, clears file protocol and - headers, and exports `otlp-headers`; `off` with a file endpoint leaves no - endpoint and skips telemetry init; no route keeps a file endpoint. -- Podman-gated: `TestPastaForwardReachesOnlyCollectorPort` (no host UDP or - late TCP; red with `auto`) and `TestVerifyNoPastaForwardsRefusesConfOptions` - (a `-T` in `CONTAINERS_CONF_OVERRIDE`). - -### CC1 — devenv runs the collector - -Files: `devenv.nix`, `go/cmd/compass-stack/main.go`. - -Interfaces: `compass-stack collector-config --grpc ADDR --http ADDR --health -ADDR [--otel-export-endpoint URL] [--otel-export-headers-file PATH]` prints -`CollectorConfigYAML`, endpoint verbatim, host headers path. `usageError` -names five subcommands. - -- When the gitignored `devenv.local.nix` sets - `env.COMPASS_OTEL_EXPORT_ENDPOINT`, `processes.otel-collector` execs - `${pkgs.opentelemetry-collector-contrib}/bin/otelcol-contrib` on it. Server - and runner start `after` it, with `OTEL_EXPORTER_OTLP_ENDPOINT` and - `--otel-agent-endpoint` at `http://127.0.0.1:`. - -Red, then green: `devenv up` with an endpoint delivers no spans today, then -server, runner and agent spans. `TestRunCollectorConfig` pins a loopback -upstream. +- `AgentHostConfig.OTLPRelay gateway.OTLPRelay`; `(*agentHost).gatewayDeps() + gateway.Deps` replaces both `gateway.Deps{…}` literals in `host.go` and + shares one budget. +- `AgentEnv.Telemetry bool` sets `COMPASS_AGENT_OTLP=1`. +- `otlpenv.HeadersValue(map[string]string) (string, error)`; + `otlpenv.RunnerConfig(environ []string) (*url.URL, map[string]string, error)`. +- `Config.OTLPOff bool`; `ExternalOTLPEndpoint` becomes a URL; + `childEnv(inherited, overlay []string) []string` and `envSlice` drop + inherited `OTEL_*`. + +Red→green: + +- `TestServerAndRunnerSpecOTLPEndpoint`: export, external and off; headers in + `Env`, never `Args`. +- `TestStartDropsInheritedOTELKeys`, `TestEnvSliceDropsInheritedOTELKeys`, + `TestExecSpecTelemetryFlag`. +- `TestResolveConfigOTelModes`: external plus export is refused. +- `TestRunnerConfig`: refuses a bad endpoint, a per-signal or certificate key, + and remote `http` with headers. +- Migration to `--otel-off`: `go/e2e/fixture.go`, `TestContainerPostgresUpDown`, + `TestExternalDatabaseUpDown`, `TestNatsUpDown`, `TestExternalNatsUpDown`. To + a URL: `TestExternalOTLPUpDown`, `TestResolveConfigCollectorFlags`, + `TestConfigValidate`, `TestExternalOTLPSkipsCollector`. + +### CB3: agent exporters over the socket + +Files: `packages/compass-agent/src/transport/{index,otel-layer,otlp-proxy}.ts`, +`packages/compass-agent/src/cli.ts`. + +Interfaces: + +- `makeOtelLayer(socketPath: string): Layer.Layer` (`Layer.empty` + without the flag), called by `createUnixSocketTransport`. +- `startOtlpProxy(socketPath: string): Promise<{ endpoint: string; headers: string; close(): Promise }>`. +- `scrubOtlpEnv(env: NodeJS.ProcessEnv): string[]` returns the dropped keys; + `isTelemetryEndpointConfigured` keys on the flag. + +Red→green (bun:test, a Unix-socket server in a temp dir): + +- A transport span reaches `/v1/traces` only with the flag. +- A no-argument `OTLPTraceExporter` reaches the socket through the proxy; a + tokenless request is refused. +- A child spawned after init sees no `OTEL_EXPORTER_OTLP_*`. +- `scrubOtlpEnv` drops and logs an env-file endpoint. +- Migration to the flag: `cli.test.ts`'s env-file endpoint, "all four + loop-OTel env keys…", "endpoint set ⇒ env defaults…" and "deployer-set + OTEL_SERVICE_NAME…" tests; `otel-layer.test.ts`'s three endpoint tests. + +### CC1: devenv runs the collector + +Interfaces: `compass-stack collector-config --grpc --http --health +[--otel-export-endpoint] [--otel-export-headers-file]` prints +`CollectorConfigYAML`; `usageError` names five subcommands. + +devenv `processes.otel-collector` runs `otelcol-contrib` when the gitignored +`devenv.local.nix` sets `env.COMPASS_OTEL_EXPORT_ENDPOINT`; server and runner +then point at it. + +Red→green: `TestRunCollectorConfig`; `devenv up` with an endpoint delivers an +agent span upstream (none today). + +### CD1: operator docs + +Interfaces: none. `docs/self-host.md` gains an observability section (export +flags, `--otel-external`, `--otel-off`, remote-runner `OTEL_EXPORTER_OTLP_*`); +each deployer contract points there. + +Red→green: none; rumdl and the design-ledger gate pass. ## Tasks -- [ ] CA1 — Collector upstream export. -- [ ] CB1 — Route every emitter. -- [ ] CC1 — devenv runs the collector. - -## Deferred (non-load-bearing) - -- Default-network `pasta_options` forwards already reach agents (RIG-5249). - -## Open Questions - -1. **Remote runners.** Forwards reach only the runner host's loopback. (a) - Name a remote `--otel-agent-endpoint`. (b) Run a forwarding collector - there. **Recommendation:** (a); (b) on demand. -2. **Pasta or a runner relay (load-bearing).** A: pasta now; microVM later - relays over the agent socket (`gatewayPortKey`, - `go/internal/guestd/cmdline.go`). B: a relay now, stamping - `compass.session.id` from `g.sessions.Session(g.containerName)` - (`go/internal/runner/gateway/publish.go`). Any relay reverses OQ1 of - `docs/designs/repo/compass-agent-effect-otel/design.md`. - **Recommendation:** A; an approved relay replaces pasta everywhere. -3. **Forged session ids (load-bearing).** Agents stamp `compass.session.id`, - so a forged id corrupts D7 joins; billing reads the D5 event log. (a) D7 - treats Plane B ids as untrusted until a relay stamps them. (b) Hold the - route until OQ2 B. **Recommendation:** (a); multi-tenant waits for OQ2 B. -4. **External targets.** Allowlists match hosts, not ports, and agents can - read the external headers. (a) Accept: the operator named both, and - `--otel-export-endpoint` keeps credentials in the collector. (b) Refuse, - silencing agents under `--otel-external`. **Recommendation:** (a); (b) - breaks D3. +- [ ] CA1: collector export +- [ ] CB1: gateway relay +- [ ] CB2: runner and stack wiring +- [ ] CB3: agent socket exporters +- [ ] CC1: devenv collector +- [ ] CD1: operator docs + +## Resolved decisions + +1. **SDK-loop exporters use the token proxy, not an SDK patch:** no fork, no + new dependency. Agent-owned providers would miss the SDK's metric and log + hooks, which only `initTelemetryExport`'s private module registers. +2. **`/v1/logs` is mounted with the same stamp.** An env-file endpoint enables + SDK logs today (`resolveSignalConfig`), so log content is unchanged. +3. **The remote host collector is operator-run:** a runner-only + `compass-stack` mode stays additive until RIG-2485 needs it. From 0779a9c02fc5ecf1af98dc712ac4e2059e5e3645 Mon Sep 17 00:00:00 2001 From: mintaka Date: Mon, 12 Oct 2026 22:24:25 -0400 Subject: [PATCH 6/8] docs(design): fold collector-routing review round 4 (RIG-4611) Spec-impact: none. Refs RIG-4611 Co-authored-by: Matt Wilkinson --- .../designs/decisions/observability/DL-514.md | 2 +- .../compass-agent-collector-routing/design.md | 212 +++++++++--------- 2 files changed, 109 insertions(+), 105 deletions(-) diff --git a/docs/designs/decisions/observability/DL-514.md b/docs/designs/decisions/observability/DL-514.md index 237935969..8c0504ca4 100644 --- a/docs/designs/decisions/observability/DL-514.md +++ b/docs/designs/decisions/observability/DL-514.md @@ -1,6 +1,6 @@ --- id: DL-514 -decision: "Agent OTLP rides the agent socket (Matt, RIG-5237): agents export OTLP/HTTP to the Runner gateway's `/v1/traces`, `/v1/metrics` and `/v1/logs`, which stamp the bound `compass.session.id` and `service.name` and relay to a collector on each runner host; the socket stays the agent's only channel, with no OTLP egress. Lifts the effect-otel ruling that agent OTLP never uses the socket/vsock gateway and the agent deployer contracts' env-file endpoint, and moves loop-otel's SDK session id to `compass.agent.sdk_session.id`" +decision: "Agent OTLP rides the agent socket (Matt, RIG-5237): agents export OTLP/HTTP to the Runner gateway's `/v1/traces`, `/v1/metrics` and `/v1/logs`, which stamp the bound `compass.session.id`, `compass.telemetry.origin=agent` and `service.name=compass-agent` and relay to a collector on each runner host; the socket stays the agent's only channel, with no OTLP egress. The stamp attributes per session except on the host tier, which is one trust domain. Lifts the effect-otel ruling that agent OTLP never uses the socket/vsock gateway and the agent deployer contracts' env-file endpoint, and moves loop-otel's SDK session id to `compass.agent.sdk_session.id`" status: "Active (Matt, 2026-10-12)" record: ../../observability/compass-agent-collector-routing/design.md#rulings --- diff --git a/docs/designs/observability/compass-agent-collector-routing/design.md b/docs/designs/observability/compass-agent-collector-routing/design.md index eada8075e..2c1ddf184 100644 --- a/docs/designs/observability/compass-agent-collector-routing/design.md +++ b/docs/designs/observability/compass-agent-collector-routing/design.md @@ -20,17 +20,15 @@ Matt, RIG-5237 (2026-10-11 and 2026-10-12): 1. **OQ1 B: a collector on each runner host** forwards upstream. 2. **OQ2 B, transport A: agents export OTLP/HTTP over the agent socket.** The - Runner gateway serves `/v1/traces` and `/v1/metrics` and relays to the host - collector. No pasta, and no egress allowlist entry for OTLP. + Runner gateway serves `/v1/traces` and `/v1/metrics` (`/v1/logs`: Resolved + 2) and relays to the host collector. No pasta, no OTLP egress entry. 3. **The socket stays the agent's only channel to its Runner.** Amended frozen texts: -- `docs/designs/repo/compass-agent-effect-otel/design.md`, Open Questions - item 1 and Decision 3 ("never the socket/vsock gateway"): lifted for agent - telemetry, for one transport on every tier, Runner-stamped attribution and - no agent egress. -- `docs/designs/observability/compass-agent-loop-otel/design.md`, Decision 3a: +- `docs/designs/repo/compass-agent-effect-otel/design.md` Open Questions item + 1 and Decision 3: lifted for agent telemetry. +- `docs/designs/observability/compass-agent-loop-otel/design.md` Decision 3a: the join key is the gateway-stamped `compass.session.id`; the SDK id moves to `compass.agent.sdk_session.id`. - Both agent deployer contracts @@ -45,100 +43,102 @@ Amended frozen texts: - D3 holds: no export endpoint keeps today's `nop` config bytes; only config and headers files touch disk. - No secret in argv or the 0644 config. Upstream credentials live in the 0600 - headers file and server or runner env, never in agents. The host tier shares - the Runner's uid (`go/internal/runtime/host_backend.go`): no such boundary. + headers file and server or runner env, never in agents. +- Host-tier exception: agents share the Runner's uid + (`go/internal/runtime/host_backend.go`), so they can read its env and reach + any session's socket. Host-tier telemetry is one trust domain, not + per-session evidence. - Only the gateway sets `compass.session.id`, `compass.telemetry.origin` and the agent's `service.name` on relayed telemetry. -- Every endpoint passes `otlpenv.ParseEndpoint` (CA1). +- Every endpoint passes `otlpenv.ParseEndpoint` (CA1): plaintext only to + loopback. - Server and runner emit or relay only with an endpoint, and drop inherited - `OTEL_*`. -- `DefaultCollectorImage` stays 0.140.0; devenv's 0.155.0 validates the same - config. -- Name receivers generically, e.g. Grafana Alloy. + `OTEL_*`. `DefaultCollectorImage` stays 0.140.0. - Breaking (`!`): `--otel-external` takes a URL; env-file agent endpoints are ignored; compass-runner refuses per-signal and TLS `OTEL_EXPORTER_OTLP_*` keys. ## Approach -**C-a: the collector exports.** `collectorConfigYAML` +**C-a: the collector exports (CA1).** `collectorConfigYAML` (`go/internal/stack/collector_container.go`) hard-codes `nop: {}`. With `--otel-export-endpoint` (fallback `COMPASS_OTEL_EXPORT_ENDPOINT`) it renders -`otlphttp` behind `memory_limiter` (256 MiB, 64 MiB spike) and an 8 MiB -receiver cap. `--otel-export-headers-file` is copied 0600 to -`/run/compass/otlp-headers.yaml` under `--userns=keep-id:uid=10001,gid=10001` -and replaced on every `Start`. +`otlphttp` behind `memory_limiter` (`check_interval: 1s`, 256 MiB, 64 MiB +spike) and an 8 MiB receiver cap. `--otel-export-headers-file` is copied 0600 +to `/run/compass/otlp-headers.yaml` on every `Start`. -**C-b: the host collector.** The Runner relays agent OTLP to the endpoint its +**C-b, host collector (CB2).** The Runner relays agent OTLP to the endpoint its own SDK reads, `OTEL_EXPORTER_OTLP_ENDPOINT` plus `_HEADERS` (`setupOtel`, -`go/cmd/compass-runner/main.go`). `compass-stack` sets it on server and runner: +`go/cmd/compass-runner/main.go`). `compass-stack` sets it on server and runner; +any two of these flags is a usage error: - `--otel-export-endpoint`: the bundled collector, `http://127.0.0.1:`. Without it nothing is set (D3). -- `--otel-external `: the operator's own host collector, which owns - batching, retry and upstream (ruling 1). `--otel-export-headers-file` rides - it as `_HEADERS` in `Env`; adding `--otel-export-endpoint` is a usage error. -- `--otel-off`: no collector, nothing set. +- `--otel-external `: the operator's host collector, which owns batching, + retry and upstream (ruling 1); `--otel-export-headers-file` rides it in `Env`. +- `--otel-off`: nothing set, overriding `COMPASS_OTEL_EXPORT_ENDPOINT`. -A remote runner host's operator runs the collector (e.g. Grafana Alloy, or -otelcol-contrib with `compass-stack collector-config`). compass-runner refuses -a bad endpoint and per-signal or TLS keys, so relay and exporters agree. +A remote runner host's operator runs the collector (e.g. Grafana Alloy). +compass-runner refuses a bad endpoint and per-signal or TLS keys. The relay +appends `/v1/` by `path.Join`, as the Go SDK's `otlpconfig` does. -**C-b: the gateway handlers.** `gateway.Serve` +**C-b, gateway handlers (CB1).** `gateway.Serve` (`go/internal/runner/gateway/gateway.go`) mounts `POST /v1/traces`, -`/v1/metrics` and `/v1/logs` (Resolved 2) only when `Deps.OTLPRelay` is set. -Each handler: +`/v1/metrics` and `/v1/logs` only when `Deps.OTLPRelay` is set. Each handler: -1. Resolves `g.sessions.Session(g.containerName)`; unbound or empty is 403, as - `(*Gateway).Publish` fails closed. +1. Resolves `g.sessions.Session(g.containerName)`. Unbound is 503 with + `Retry-After: 1`: `agentHost.Start` starts the agent before recording its + session. 2. Answers 415 unless `application/x-protobuf` with identity encoding. -3. Takes one of 4 per-Gateway slots and 16 MiB of a Runner-wide 128 MiB - budget, else answers 503 with `Retry-After: 1` before reading. -4. Arms a 5 s read deadline (as `withBodyReadDeadline`, - `go/server/network_door.go`), reads at most 4 MiB (413), and decodes with - `go.opentelemetry.io/proto/otlp`, promoted to direct (400 if malformed). +3. Takes one of 8 per-Gateway slots (a flush exports five signals) and 4 MiB + of a Runner-wide 128 MiB budget, else 503 with `Retry-After: 1`, unread. + One 8 s deadline (as `withBodyReadDeadline`, `go/server/network_door.go`) + covers read and upstream. +4. Reads at most 4 MiB (413). An allocation-free `protowire` pre-scan counts + messages and packed-varint bytes; over 65,536 messages or depth 32 is 413. + It reserves the decoded bound (416 B per message, 8 B per packed byte, + twice the body), else 503, then decodes with `go.opentelemetry.io/proto/otlp` + (direct; 400 if malformed). Measured: 4 MiB of empty attributes decodes to + 244 MiB; an empty span, the costliest, to 331 B. 5. Deletes `compass.session.id` from every scope, span, event, link, data point, exemplar and log record. On each resource it sets - `compass.session.id` to the bound id that server spans carry - (`go/internal/delivery/dispatch.go`), `compass.telemetry.origin=agent` and + `compass.session.id` to the bound id, `compass.telemetry.origin=agent` and `service.name=compass-agent`. -6. Re-encodes (over 8 MiB is 413) and POSTs under `r.Context()` with an 8 s - deadline, inside OTel JS's 10 s timeout. Upstream 2xx is 200; 429, 502, 503 - or 504 is 503 with its `Retry-After`; 413 stays; another 4xx is 400; else - 502. Only transient failures retry (`isExportHTTPErrorRetryable`). - -**C-b: the agent.** When it relays, the Runner sets `COMPASS_AGENT_OTLP=1`, a -reserved `COMPASS_*` key the env file cannot set. After the env-file merge the -agent deletes every `OTEL_EXPORTER_OTLP_*` key and logs one `[compass-agent]` -line naming them. With the flag it writes the SDK session id as +6. Re-encodes (over 8 MiB is 413) and POSTs under `r.Context()`. Upstream 2xx + is 200; 429, 502, 503 or 504 is 503 with its `Retry-After`; 413 stays; + another 4xx is 400; else 502. + +**C-b, agent (CB3).** When relaying, the Runner sets the reserved +`COMPASS_AGENT_OTLP=1`. After the env-file merge the agent deletes and logs +every `OTEL_EXPORTER_OTLP_*` key, and records the SDK session id as `compass.agent.sdk_session.id`. - `makeOtelLayer` (`packages/compass-agent/src/transport/otel-layer.ts`) builds exporters with `url: "http://compass-agent/v1/"` and - `httpAgentOptions: { socketPath }` (Bun probe-proven). + `httpAgentOptions: { socketPath }`. - The SDK loop's exporters (`registerProviders`, `@oh-my-pi/pi-coding-agent`) - read env once, at construction. A `Bun.serve` proxy on `127.0.0.1:0` checks - a random per-process token header, caps bodies at 4 MiB and forwards with - `fetch(…, { unix })`, as `runInGuestProbe` - (`go/internal/runner/e2e_vsock_gateway_microvm_test.go`) does. Its endpoint - and token are set only around `await telemetryHooks.init()`, so tool - subprocesses never inherit them. Egress accepts `oif lo` (`NftScript`). + read env at construction. A `Bun.serve` proxy on `127.0.0.1:0` checks a + per-process token header, caps bodies at 4 MiB and forwards with + `fetch(…, { unix, signal: req.signal })`. Endpoint and token are set only + across `await telemetryHooks.init()`; a subprocess spawned during its + dynamic import gains only what its socket already gives. Egress accepts + `oif lo` (`NftScript`). | Tier | Socket reaches the agent by | OTLP | | --- | --- | --- | -| podman | bind mount at `/run/compass/agent.sock` | covered | -| microVM | guestd `gatewayProxy`, a byte splice to vsock | covered | -| host | `COMPASS_AGENT_SOCKET_PATH` in the state dir | covered; no boundary | -| apple-container | `--volume` over virtiofs | gap until the vsock transport lands (`docs/designs/platform/apple-container-macos-runner/design.md`) | +| podman | bind mount at `/run/compass/agent.sock` | covered hermetically | +| microVM | guestd `gatewayProxy` splice to vsock | covered | +| host | `COMPASS_AGENT_SOCKET_PATH` | covered; one trust domain | +| apple-container | `--volume` over virtiofs | none: DL-338's slow-roll holds the agent channel | -OTLP adds no gap of its own. +**C-c: dev wiring (CC1).** devenv runs the same collector config. ## Alternatives considered - **Pasta forward** (earlier draft): podman only, agent-asserted attribution; - its unshipped helpers are dropped and RIG-5249 becomes moot. -- **Direct export through the egress allowlist** (effect-otel ruling (a)): - credentials in every agent, forgeable attribution. + RIG-5249 becomes moot. +- **Direct export through the egress allowlist**: credentials in every agent, + forgeable attribution. ## Plan @@ -155,24 +155,29 @@ Interfaces: `CollectorYAMLParams{GRPC, HTTP, Health, Export, HeadersPath string}`. - `CollectorContainerSpec.ExportHeadersFile string`; `const CollectorHeadersMountPath = "/run/compass/otlp-headers.yaml"`. -- `otlpenv.ParseEndpoint(raw string, withHeaders bool) (*url.URL, error)`. +- `otlpenv.ParseEndpoint(raw string) (*url.URL, error)`; + `otlpenv.SignalURL(base *url.URL, signal string) *url.URL`. Red→green: - `TestCollectorConfigYAMLExportsUpstream` (red today): `otlphttp` and - `memory_limiter` in all three pipelines, an 8388608 body cap, no `nop`. -- `TestCollectorConfigYAMLRealizesD3Posture`: zero params give today's bytes. -- `TestParseEndpoint`: accepts http(s) to `127.0.0.1`, `localhost` and remote - hosts; refuses bare `host:port`, `127.0.0.2`, `[::1]`, `0.0.0.0`, userinfo, - query, fragment, and remote `http` with headers. + `memory_limiter` with `check_interval` in all three pipelines, an 8388608 + body cap, no `nop`. +- `TestCollectorConfigYAMLRealizesD3Posture`: container ports, empty `Export` + and `HeadersPath`, today's bytes. +- `TestParseEndpoint`: `http` only to `127.0.0.1` or `localhost`, `https` + anywhere; refuses bare `host:port`, `127.0.0.2`, `[::1]`, `0.0.0.0`, + userinfo, query, fragment. +- `TestSignalURL`: `https://h/otlp` gives `/otlp/v1/traces`. - `TestCollectorStartReplacesHeadersCopy`; `TestCollectorRunArgsExportContract` (no header value in argv or config). -- Smoke: `compass-stack up` delivers a span and the header to a local receiver. +- Smoke: `otelcol validate` on the pinned image; `compass-stack up` delivers a + span and the header to a local receiver. ### CB1: gateway OTLP relay Files: `go/internal/runner/gateway/{gateway,otlp}.go`, -`go/internal/otel/otlprelay`, `go/go.mod`. +`go/internal/otel/{otlprelay,otlpscan}`, `go/go.mod`. Interfaces: @@ -181,24 +186,27 @@ Interfaces: `Deps.OTLPBudget *semaphore.Weighted` (`golang.org/x/sync`). - `otlprelay.New(endpoint *url.URL, headers map[string]string) *Relay`; a non-2xx upstream returns `*otlprelay.StatusError{Code int, RetryAfter string}`. -- `const maxOTLPBodyBytes = 4 << 20`, `maxOTLPInFlight = 4`. -- Counter `compass.runner.otlp_relay.requests{signal,outcome}`, nil-guarded - like `compass.delivery.dispatched`; no session label. +- `otlpscan.Bound(body []byte, signal string) (uint64, error)`. +- `const maxOTLPBodyBytes = 4 << 20`, `maxOTLPInFlight = 8`, + `maxOTLPMessages = 1 << 16`, `otlpRequestDeadline = 8 * time.Second`. +- Counter `compass.runner.otlp_relay.requests{signal,outcome}`, nil-guarded, + no session label. Red→green (hermetic, fake relay): - `TestOTLPStampOverridesAgentSession`: forged ids on a resource, span, data point and log record, and `service.name=compass-server`, arrive as the bound id and `compass-agent`. -- `TestOTLPUnboundContainerForbidden` (403, relay uncalled); `TestOTLPBodyCap` - (413); `TestOTLPRejectsCompressedBody` (gzip is 415). -- `TestOTLPSaturationShedsWith503`: a fifth request or empty budget, body unread. +- `TestOTLPUnboundContainerRetryable` (503); `TestOTLPBodyCap` (413); + `TestOTLPRejectsCompressedBody` (415). +- `TestOTLPSaturationShedsWith503`: a ninth request or empty budget, unread. +- `TestOTLPDecodeBound`: 70,000 empty attributes is 413; `otlpscan.Bound` + covers measured decode allocation. - `TestOTLPRelayFailure`: 400 stays 400, 503 keeps `Retry-After`, transport error is 502. - `TestOTLPUnmountedWhenRelayNil` (404); `TestRelayPostsWithHeaders`. -- microVM e2e `TestVsockGateway_InGuestOTLPStamped`: a guest POST reaches a - fake `AgentHostConfig.OTLPRelay` with the bound id. The podman leg joins - `TestSocketOwnedAndConnectableInContainer` once its probe image lands. +- microVM e2e `TestVsockGateway_InGuestOTLPStamped` against a fake + `AgentHostConfig.OTLPRelay`. ### CB2: runner and stack wiring @@ -209,8 +217,7 @@ Files: `go/cmd/compass-runner/main.go`, `go/internal/runner/{host,agent_exec}.go Interfaces: - `AgentHostConfig.OTLPRelay gateway.OTLPRelay`; `(*agentHost).gatewayDeps() - gateway.Deps` replaces both `gateway.Deps{…}` literals in `host.go` and - shares one budget. + gateway.Deps` replaces both `host.go` literals, sharing one budget. - `AgentEnv.Telemetry bool` sets `COMPASS_AGENT_OTLP=1`. - `otlpenv.HeadersValue(map[string]string) (string, error)`; `otlpenv.RunnerConfig(environ []string) (*url.URL, map[string]string, error)`. @@ -220,13 +227,13 @@ Interfaces: Red→green: -- `TestServerAndRunnerSpecOTLPEndpoint`: export, external and off; headers in - `Env`, never `Args`. +- `TestServerAndRunnerSpecOTLPEndpoint`: each mode; headers in `Env`, never + `Args`. - `TestStartDropsInheritedOTELKeys`, `TestEnvSliceDropsInheritedOTELKeys`, `TestExecSpecTelemetryFlag`. -- `TestResolveConfigOTelModes`: external plus export is refused. -- `TestRunnerConfig`: refuses a bad endpoint, a per-signal or certificate key, - and remote `http` with headers. +- `TestResolveConfigOTelModes`: two OTel flags are refused; `--otel-off` beats + the env fallback. +- `TestRunnerConfig`: refuses a bad endpoint and per-signal or certificate keys. - Migration to `--otel-off`: `go/e2e/fixture.go`, `TestContainerPostgresUpDown`, `TestExternalDatabaseUpDown`, `TestNatsUpDown`, `TestExternalNatsUpDown`. To a URL: `TestExternalOTLPUpDown`, `TestResolveConfigCollectorFlags`, @@ -249,12 +256,12 @@ Red→green (bun:test, a Unix-socket server in a temp dir): - A transport span reaches `/v1/traces` only with the flag. - A no-argument `OTLPTraceExporter` reaches the socket through the proxy; a - tokenless request is refused. + tokenless request is refused; a client abort cancels the forward. - A child spawned after init sees no `OTEL_EXPORTER_OTLP_*`. - `scrubOtlpEnv` drops and logs an env-file endpoint. - Migration to the flag: `cli.test.ts`'s env-file endpoint, "all four loop-OTel env keys…", "endpoint set ⇒ env defaults…" and "deployer-set - OTEL_SERVICE_NAME…" tests; `otel-layer.test.ts`'s three endpoint tests. + OTEL_SERVICE_NAME…"; `otel-layer.test.ts`'s three endpoint tests. ### CC1: devenv runs the collector @@ -263,19 +270,17 @@ Interfaces: `compass-stack collector-config --grpc --http --health `CollectorConfigYAML`; `usageError` names five subcommands. devenv `processes.otel-collector` runs `otelcol-contrib` when the gitignored -`devenv.local.nix` sets `env.COMPASS_OTEL_EXPORT_ENDPOINT`; server and runner -then point at it. +`devenv.local.nix` sets `env.COMPASS_OTEL_EXPORT_ENDPOINT`. Red→green: `TestRunCollectorConfig`; `devenv up` with an endpoint delivers an -agent span upstream (none today). +agent span upstream. ### CD1: operator docs -Interfaces: none. `docs/self-host.md` gains an observability section (export -flags, `--otel-external`, `--otel-off`, remote-runner `OTEL_EXPORTER_OTLP_*`); -each deployer contract points there. +Interfaces: none. `docs/self-host.md` gains an observability section (flags, +remote-runner `OTEL_EXPORTER_OTLP_*`); each deployer contract points there. -Red→green: none; rumdl and the design-ledger gate pass. +Red→green: none; rumdl and the ledger gate pass. ## Tasks @@ -288,10 +293,9 @@ Red→green: none; rumdl and the design-ledger gate pass. ## Resolved decisions -1. **SDK-loop exporters use the token proxy, not an SDK patch:** no fork, no - new dependency. Agent-owned providers would miss the SDK's metric and log - hooks, which only `initTelemetryExport`'s private module registers. -2. **`/v1/logs` is mounted with the same stamp.** An env-file endpoint enables - SDK logs today (`resolveSignalConfig`), so log content is unchanged. -3. **The remote host collector is operator-run:** a runner-only - `compass-stack` mode stays additive until RIG-2485 needs it. +1. **SDK-loop exporters use the token proxy, not an SDK patch:** no fork. + Agent-owned providers would miss the SDK's metric and log hooks, which + only its dynamically imported, unexported `telemetry-export-otlp` registers. +2. **`/v1/logs` is mounted with the same stamp:** env-file endpoints enable SDK + logs (`resolveSignalConfig`). +3. **The remote host collector is operator-run;** a runner-only mode is additive. From b7eb211cfa95023009ec778f3945df6cd1559e39 Mon Sep 17 00:00:00 2001 From: mintaka Date: Mon, 12 Oct 2026 23:01:28 -0400 Subject: [PATCH 7/8] docs(design): count unpacked repeated scalars in the OTLP decode bound (RIG-4611) Spec-impact: none. Refs RIG-4611 Co-authored-by: Matt Wilkinson --- .../compass-agent-collector-routing/design.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/docs/designs/observability/compass-agent-collector-routing/design.md b/docs/designs/observability/compass-agent-collector-routing/design.md index 2c1ddf184..cbf01eb5a 100644 --- a/docs/designs/observability/compass-agent-collector-routing/design.md +++ b/docs/designs/observability/compass-agent-collector-routing/design.md @@ -95,11 +95,11 @@ appends `/v1/` by `path.Join`, as the Go SDK's `otlpconfig` does. One 8 s deadline (as `withBodyReadDeadline`, `go/server/network_door.go`) covers read and upstream. 4. Reads at most 4 MiB (413). An allocation-free `protowire` pre-scan counts - messages and packed-varint bytes; over 65,536 messages or depth 32 is 413. - It reserves the decoded bound (416 B per message, 8 B per packed byte, - twice the body), else 503, then decodes with `go.opentelemetry.io/proto/otlp` - (direct; 400 if malformed). Measured: 4 MiB of empty attributes decodes to - 244 MiB; an empty span, the costliest, to 331 B. + messages and repeated-scalar elements, packed or unpacked; over 65,536 + messages or depth 32 is 413. It reserves 416 B per message, 16 B per + element (append doubling) and twice the body, else 503, then decodes with + `go.opentelemetry.io/proto/otlp` (direct; 400 if malformed). 4 MiB of empty + attributes measured 244 MiB decoded. 5. Deletes `compass.session.id` from every scope, span, event, link, data point, exemplar and log record. On each resource it sets `compass.session.id` to the bound id, `compass.telemetry.origin=agent` and @@ -201,7 +201,7 @@ Red→green (hermetic, fake relay): `TestOTLPRejectsCompressedBody` (415). - `TestOTLPSaturationShedsWith503`: a ninth request or empty budget, unread. - `TestOTLPDecodeBound`: 70,000 empty attributes is 413; `otlpscan.Bound` - covers measured decode allocation. + covers measured allocation, including two million unpacked `BucketCounts`. - `TestOTLPRelayFailure`: 400 stays 400, 503 keeps `Retry-After`, transport error is 502. - `TestOTLPUnmountedWhenRelayNil` (404); `TestRelayPostsWithHeaders`. From c0753ea30346cc0c0ae7a22e320dc773a1a7a05b Mon Sep 17 00:00:00 2001 From: mintaka Date: Mon, 12 Oct 2026 23:27:46 -0400 Subject: [PATCH 8/8] docs(design): claim DL-515 for the collector rulings on this lane (RIG-4611) Spec-impact: none. Refs RIG-4611 Co-authored-by: Matt Wilkinson --- docs/designs/decisions/observability/{DL-514.md => DL-515.md} | 2 +- .../observability/compass-agent-collector-routing/design.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) rename docs/designs/decisions/observability/{DL-514.md => DL-515.md} (98%) diff --git a/docs/designs/decisions/observability/DL-514.md b/docs/designs/decisions/observability/DL-515.md similarity index 98% rename from docs/designs/decisions/observability/DL-514.md rename to docs/designs/decisions/observability/DL-515.md index 8c0504ca4..01a7e8a9b 100644 --- a/docs/designs/decisions/observability/DL-514.md +++ b/docs/designs/decisions/observability/DL-515.md @@ -1,5 +1,5 @@ --- -id: DL-514 +id: DL-515 decision: "Agent OTLP rides the agent socket (Matt, RIG-5237): agents export OTLP/HTTP to the Runner gateway's `/v1/traces`, `/v1/metrics` and `/v1/logs`, which stamp the bound `compass.session.id`, `compass.telemetry.origin=agent` and `service.name=compass-agent` and relay to a collector on each runner host; the socket stays the agent's only channel, with no OTLP egress. The stamp attributes per session except on the host tier, which is one trust domain. Lifts the effect-otel ruling that agent OTLP never uses the socket/vsock gateway and the agent deployer contracts' env-file endpoint, and moves loop-otel's SDK session id to `compass.agent.sdk_session.id`" status: "Active (Matt, 2026-10-12)" record: ../../observability/compass-agent-collector-routing/design.md#rulings diff --git a/docs/designs/observability/compass-agent-collector-routing/design.md b/docs/designs/observability/compass-agent-collector-routing/design.md index cbf01eb5a..bfdf5d7b4 100644 --- a/docs/designs/observability/compass-agent-collector-routing/design.md +++ b/docs/designs/observability/compass-agent-collector-routing/design.md @@ -6,7 +6,7 @@ Amendment record of the observability-architecture design ([`../compass-observability-architecture/design.md`](../compass-observability-architecture/design.md), §D3 and §T4). The parent stays frozen. -Ledger-impact: DL-514. +Ledger-impact: DL-515. ## Problem / Intent