fix(observability): remove impossible Loki healthcheck that hung the stack

[Gerrrt]

What changed

Loki no longer declares a healthcheck, and Grafana and Alloy now wait on it with condition: service_started instead of service_healthy. A new scripts/check_compose_health.py guard makes it hard to reintroduce the bug, and secrets/*.sops.yaml is now ignored by yamllint.

Why

make up hung on the monitoring host. Every container came up except Loki, which sat in starting forever with nothing logged.

grafana/loki is built FROM gcr.io/distroless/static:nonroot — the image contains /usr/bin/loki and nothing else. No shell, no wget, no curl. A Docker healthcheck can only exec something inside the container, so the wget --spider probe could never succeed no matter how healthy Loki was.

That is not cosmetic. Grafana and Alloy both waited on Loki via depends_on: condition: service_healthy, so neither ever started, and the stack appeared to deadlock with no error surfaced anywhere. Loki itself was fine the entire time; only the probe was impossible.

Readiness is now observed from outside the container instead:

• curl -s localhost:3100/ready on demand
• continuously by Prometheus, which already scrapes loki:3100 as the loki job — InstanceDown{job="loki"} and LokiIngestionStalled both cover it

Downgrading the dependencies to service_started is safe for both dependants: Grafana resolves its Loki datasource lazily per query, and Alloy retries a failed push with backoff while buffering in the meantime. The worst case is an empty panel for a few seconds.

Two follow-ons rode along:

• scripts/check_compose_health.py — docker compose config accepts a service_healthy dependency on a service with no healthcheck without complaint, which is exactly why CI stayed green while the stack deadlocked. The guard closes both halves of the trap: every service_healthy target must actually declare a healthcheck, and no healthcheck may exec a binary on an image with no userland to run it. The second half needs an explicit list — an image reference says nothing about its base, and grafana/loki contains no "distroless" substring to match on. Wired into CI, scripts/validate.sh, and make check-compose-health.
• yamllint ignore for secrets/*.sops.yaml — committing the encrypted secrets file turned the Lint job red on main. SOPS output is machine-generated ciphertext; it does not follow the repo's comment and indentation conventions and never will.

Blast radius

The observability stack on prometheus (10.0.99.20, VLAN 99). Startup ordering only — no config change to any service, no data path touched.

• No change to network segmentation or firewall rules
• No new port published to a VLAN that could not already reach the service
• No credential added outside secrets/*.sops.yaml

Verification

Locally, against the pinned images and real binaries:

compose.yaml OK — 4 healthcheck(s), 2 service_healthy dependency/dependencies, all satisfiable
docker compose config -q     → exit 0
yamllint --strict .          → clean (red before the ignore)
markdownlint-cli2            → 0 issues in 21 files
shellcheck scripts/*.sh      → clean
check_dashboards.py          → 5 dashboards OK (79 panels, 79 PromQL expressions)

The guard matters only if it fails when it should, so both regression paths were tested against fixtures.

Re-pointing Grafana back at loki: condition: service_healthy:

grafana waits for loki to become healthy, but loki declares no healthcheck — it
can never report healthy, so grafana will hang forever          (exit 1)

Re-adding a wget healthcheck to Loki — the subtler path, since it satisfies the check above and hangs the stack anyway:

loki has a healthcheck exec'ing 'wget', but grafana/loki:3.7.4@sha256:… has no
shell and no userland — the probe can never run, so loki stays 'starting'
forever                                                          (exit 1)

Image classification is covered by a 13-case table over bare, tagged, digest-pinned and registry-qualified references, asserting grafana/loki and explicit distroless/scratch bases are flagged while the busybox, alpine and ubuntu based images in this stack are not. The PyYAML fallback was exercised both ways in a throwaway venv: installs and passes where pip is available, and reports PyYAML is required and could not be installed rather than a traceback where it is not.

• make validate passes
• Deployed to the lab and confirmed working — pending; git pull && make up on the monitoring host is the remaining step
• Docs updated (inline rationale lives in compose.yaml next to the removed healthcheck, where the next person tempted to add one will read it)

[Copilot]

Pull request overview

This PR fixes a compose startup deadlock in the observability stack by removing Loki’s unsatisfiable healthcheck and changing Grafana/Alloy to no longer wait on Loki becoming “healthy”. It also adds a CI guard to prevent unsatisfiable service_healthy dependency graphs and updates yamllint ignores for encrypted SOPS files.

Changes:

• Remove Loki healthcheck and switch Grafana/Alloy depends_on from service_healthy to service_started.
• Add scripts/check_compose_health.py and wire it into scripts/validate.sh, CI, and make check-compose-health.
• Ignore secrets/*.sops.yaml in .yamllint.yaml to avoid lint failures on machine-generated ciphertext.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
stacks/observability/compose.yaml	Removes Loki healthcheck and adjusts dependency conditions to prevent stack hang.
scripts/validate.sh	Runs the new compose health dependency check as part of local validation.
scripts/check_compose_health.py	Adds a guard script to detect unsatisfiable compose health dependencies/healthchecks.
Makefile	Adds a check-compose-health target.
.yamllint.yaml	Ignores SOPS-encrypted secret YAML files.
.github/workflows/ci.yml	Runs the new compose health dependency guard in CI.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.