feat(agent): collect and store host diagnostic bundles

[memetics19]

Summary

Adds read-only diagnostic bundles: pulse-agent --diagnose collects evidence about why a host failed — OOM kills, disk usage, busiest processes, container and systemd state, Proxmox guest status — and Pulse stores it. The goal is answering "why did this break" on a remote host without opening an SSH session.

Collection is on demand only in this PR. Attaching bundles to incidents automatically, and acting on them, are follow-up work.

Type of change

• feat — new feature
• fix — bug fix
• refactor — no behaviour change
• docs
• ci / build — pipeline or tooling
• test
• chore
• Breaking change (! / BREAKING CHANGE: footer)

Changes

agent

• New internal/diagnostics package: collectors for dmesg OOM kills, df usage, ps top processes, docker ps -a, systemctl list-units --failed, and qm list.
• Collectors take an injected Runner, so they are tested against recorded command output rather than the host the tests run on.
• Each command is individually timeout-bounded — a wedged host is exactly when diagnostics matter most.
• Sections degrade independently: a host that denies dmesg, has no Docker, or is not a Proxmox node still produces a useful bundle, with unavailable collectors recording their own error.
• Captures the recent journal for units already in the failed state and recent output for containers already stopped. An OOM kill says a service died; the log says why. Targets are derived from what the host reports as broken, never taken from the server. Bounded to 200 lines, 5 targets per section, 32 KiB each, so the bundle stays under the 1 MiB request cap.
• A failing command's own output is folded into its section error. A bare exit status 1 is useless here; cannot connect to the docker daemon and usage: sudo dmesg are the actual diagnoses.
• --diagnose flag for one-shot collection. Prints the bundle with no server configured, which is the only mode available when Pulse itself is unreachable.
• postJSON extracted from Push so both ingest paths share one request path.

api

• Migration 11 adds incident_diagnostics. incident_id is optional — an on-demand bundle describes a host at a moment, not necessarily an incident — and cascades when set.
• POST /api/ingest/diagnostics, authenticated with the same agent bearer token as metric ingest.
• The bundle is stored verbatim as JSON: the agent owns the bundle schema, so collectors can change without a server-side migration.
• collected_at records receipt time, keeping ordering consistent across agents with skewed clocks. The agent's own timestamp survives inside the payload.

docs

• New docs/diagnostics.md, added to the nav; architecture.md links to it.

chore

• Per-module coverage.* artifacts added to .gitignore.

Breaking changes

None.

Test plan

• cd api && go test ./... -count=1
• cd agent && go test ./... -count=1
• cd cli && go test ./... -count=1
• cd ui && npx tsc --noEmit (UI unchanged)
• gofmt -l clean, go vet ./... clean in all three modules
• Manual check: ran pulse-agent --diagnose on macOS. disk and docker collected; kernel, processes, systemd, and proxmox degraded to per-section errors as designed, and the bundle stayed usable.

Running it live also caught a bug the unit tests had not: /dev was flagged as a full disk, because pseudo filesystems permanently report 100% capacity. A false "disk full" would drive a wrong diagnosis, so pseudo filesystems are now listed but never flagged. Covered by a regression test.

Known limitations

• The parsers are tested against representative command output, not against output captured from a real Proxmox host. qm list, systemctl list-units, and the dmesg OOM line should be checked against a live host before relying on them.
• ps -eo ... --sort=-pcpu is procps-specific. Fine on Debian/Proxmox; it degrades on macOS, so the processes section cannot be smoke-tested on a Mac.
• Reading the kernel ring buffer needs root. Fine for the systemd unit; the kernel section degrades when the agent runs unprivileged.

Known gaps, deliberately left for follow-ups

• The agent binary is not shipped in releases. release.yml cross-compiles only api/cmd/pulse, and install.sh never mentions the agent, so pulse-agent --diagnose is only reachable by building from source. Pre-existing, but this PR is the first change that depends on it.
• Diagnostics are never pruned. PruneIncidentDiagnostics exists but nothing calls it; pruner.Run only prunes check results. The table grows without bound.
• An unknown incident_id returns 500 and leaks the SQL error rather than a 400.

[memetics19]

Review decision: Request changes

The fixed-command design, per-section degradation, authenticated agent ownership, and test seams are solid. I found seven actionable correctness/lifecycle issues in the inline comments. The highest-risk path is a degraded host: sequential timeouts can consume the full parent deadline, and the configured upload then starts with an expired context.

I also verified the three-dot PR diff with git diff --check, gofmt, go vet ./..., full tests, and race-enabled tests in agent, api, and cli; all passed. Current GitHub checks are green. One important coverage gap remains: .github/workflows/ci.yml vets/tests api but not the newly changed agent module. Please add the agent module to CI and include blocking-runner, real stderr, malformed-output, CLI flag/exit, bundle-shape, and retention tests with the fixes.