From a52e17894f12fd2f980876a553e212ed4a508827 Mon Sep 17 00:00:00 2001 From: mintaka Date: Sun, 11 Oct 2026 21:17:08 -0400 Subject: [PATCH] docs(testing): document the Go testing tiers (RIG-2908) docs/architecture/testing-tiers.md names the default, pgtest and podman e2e tiers, their tags, cost and CI job, and the rule for picking one. It covers the pgtest DSN order, sharedFixture vs plain NewFixture with WithSite and WithEmbeddedPostgres, the shared-stack rules (register from TestMain, route by marker, await every turn, no -count above 1), and the embedded-postgres smoke leg's role. The e2e and pgtest package docs link to it, and the design record ticks T6 with an as-shipped errata note. Co-authored-by: Matt Wilkinson --- docs/architecture/testing-tiers.md | 101 ++++++++++++++++++ .../ci/compass-test-harness-tiers/design.md | 10 +- go/e2e/clients.go | 3 +- go/e2e/embedded_postgres_test.go | 8 +- go/internal/pgtest/pgtest.go | 6 +- 5 files changed, 118 insertions(+), 10 deletions(-) create mode 100644 docs/architecture/testing-tiers.md diff --git a/docs/architecture/testing-tiers.md b/docs/architecture/testing-tiers.md new file mode 100644 index 000000000..9fb2ad1fd --- /dev/null +++ b/docs/architecture/testing-tiers.md @@ -0,0 +1,101 @@ +# Go testing tiers + +The Go module has three test tiers. Pick the cheapest tier that can fail on +the behavior you are testing. + +| Tier | Build tag | Runs against | Cost | CI job | +| --- | --- | --- | --- | --- | +| Default | none | Fakes and in-process code | Seconds | moon battery | +| pgtest | `pgtest` | A real Postgres | Minutes per package | `pgtest` | +| podman e2e | `podman` | The real stack and the real agent image | Tens of minutes | `e2e` | + +The exact commands and flags are in the `pgtest` and `e2e` jobs of +`.github/workflows/ci.yml`. + +## Which tier + +1. Does the assertion need real SQL: a query, a constraint, RLS, a migration? + Use **pgtest**. +2. Does it need the process or container topology: stack children, the runner, + an agent turn, teardown? Use **podman e2e**. +3. Otherwise use the **default** tier. + +A tagged test that cannot reach its dependency skips. CI checks that the +pgtest and podman suites actually ran, so a skip there does not pass silently. + +## pgtest + +The entry point is `pgtest.RequireDSN(t)` (`go/internal/pgtest`). Open your own +store on the DSN it returns. It finds Postgres in this order: + +- `COMPASS_TEST_DATABASE_DSN` points at a running Postgres. Each test gets a + fresh schema, dropped when the test ends. CI uses this. +- `COMPASS_TEST_USE_CONTAINER=1` starts a throwaway container. Use this + locally. +- With neither set, it fails when a container runtime exists, and skips when + none does. CI sets `COMPASS_REQUIRE_LIVE=1`, so there a missing database + fails. + +For tests that must go through a transaction-mode pooler, use +`pgtest.RequirePoolerDSN(t)`. It needs both `COMPASS_TEST_DATABASE_DSN` and +`COMPASS_TEST_POOLER_DSN` in URL form, and gives each test a fresh database. + +## podman e2e + +Fixture-based legs live in `go/e2e`. `cmd/compass-stack` also has +podman-tagged supervisor tests that do not use the fixture. + +In `go/e2e`, `TestMain` builds the stack binaries once and starts one suite +Postgres. Each leg starts with a `podmanUsable` check and skips when rootless +podman cannot run the agent image. + +### Shared stack: the default + +Most legs call `sharedFixture(t)`. Every such leg in the run uses one stack, +which attaches to the suite Postgres through `ExternalDatabase`. + +- Register options the shared stack needs from a `registerFixtureOptions` + function listed in `TestMain`. Never register from a leg body: canned routes + are fixed when the stub is built. A registered option applies to every shared + leg. +- You can register `WithForgeStub()` and the `WithCanned*` options. + `WithObjectStore` needs a `*testing.T` to start, so it is own-stack only. +- Route each model turn by a marker unique to your leg. The positional canned + script is one counter shared by every leg. +- Await every turn your leg starts before it returns, so no model request + outlives the leg. +- `-count` above 1 is not supported; CI runs with `-count=1`. In a full run, + `zz_legsix_test.go` closes the shared stack at the end of each pass, so a + later pass cannot reach it. Without leg six, handles are create-only and the + stack persists, so the second pass fails with `already_exists`. + +### Own stack: the opt-out + +Call `NewFixture(ctx, t, opts...)` when the leg needs a stack nobody else +touches, such as a restart, a teardown check, or object-store state. It costs a +full cold start: its own stack-owned Postgres, server and runner. + +- `WithSite(site)` reuses one root, state dir and port across two `NewFixture` + calls, so the second `Up` reattaches the same Postgres data. +- `WithEmbeddedPostgres()` states that the leg needs the stack-owned Postgres. + Plain `NewFixture` already uses it; the option fails the test if an external + database is also set. +- `WithForgeStub()`, `WithObjectStore(g)` and the `WithCanned*` options apply to + this leg only. + +`zz_legsix_test.go` closes the shared stack before its restart, so it must run +last. Its file name sorts it last, and it skips under `-shuffle`. + +### The embedded-Postgres smoke leg + +Shared legs use the suite Postgres. Other own-stack legs use the stack-owned +Postgres but do not check its lifecycle. `TestEmbeddedPostgresSmoke` is the +only leg that asserts the supervisor-owns-Postgres lifecycle and its pgid-record +teardown: the record names a live Postgres group after `Up`, and after `Down` +the group is gone and the record is removed. Keep it when you change how the +stack starts or stops Postgres. + +## Related + +- Design record: `docs/designs/infra/ci/compass-test-harness-tiers/design.md`. +- Toolchains and the moon battery: [Build & CI](build-and-ci.md). diff --git a/docs/designs/infra/ci/compass-test-harness-tiers/design.md b/docs/designs/infra/ci/compass-test-harness-tiers/design.md index 5c0dd5232..7f9330752 100644 --- a/docs/designs/infra/ci/compass-test-harness-tiers/design.md +++ b/docs/designs/infra/ci/compass-test-harness-tiers/design.md @@ -482,6 +482,12 @@ docs. - Interfaces: none (prose); MUST cite the same seams this record does. - Test cycle: markdownlint. +*Errata (as shipped).* The page is `docs/architecture/testing-tiers.md`, the +eng-docs Architecture section. It documents `sharedFixture` vs plain +`NewFixture` (no `WithOwnStack`, per the T3 errata), `WithSite`, and +`WithEmbeddedPostgres`. The `-count>1` caveat lives in that page, not in the +`fixture.go` package doc; the `go/e2e` and `pgtest` package docs link to it. + ## Tasks - [x] T1 — extract `go/internal/pgshare` (acquisition + schema isolation + @@ -494,8 +500,8 @@ docs. - [x] T4 — `TestEmbeddedPostgresSmoke` + `WithEmbeddedPostgres()`. - [x] T5 — forge-injection convergence (`mountForgeForTest`). Satisfied on main by `newForgeE2EWire`; see T5 errata. -- [ ] T6 — tier-pyramid contributor doc. Owner: implement lane (depends - T3/T4 landing so the doc describes reality). +- [x] T6 — tier-pyramid contributor doc. Shipped as + `docs/architecture/testing-tiers.md`; see T6 errata. ## Open Questions (1–2 RULED by Matt at freeze; 3–4 deferred) diff --git a/go/e2e/clients.go b/go/e2e/clients.go index ae0cf27c0..6cf08657e 100644 --- a/go/e2e/clients.go +++ b/go/e2e/clients.go @@ -5,7 +5,8 @@ // CLI drives) and hands the harness's later legs (RIG-1785 H2-H6) a Fixture with // authenticated Connect clients and the store DSN. It is podman-build-tagged — // it drives real child processes and the real agent image, so it is out of the -// hermetic unit lane and runs only under `-tags podman`. +// hermetic unit lane and runs only under `-tags podman`. Which tier to test at, +// and sharedFixture vs NewFixture: docs/architecture/testing-tiers.md. package e2e import ( diff --git a/go/e2e/embedded_postgres_test.go b/go/e2e/embedded_postgres_test.go index 91b7edc26..1aa9313a4 100644 --- a/go/e2e/embedded_postgres_test.go +++ b/go/e2e/embedded_postgres_test.go @@ -19,10 +19,10 @@ import ( "github.com/RigelBuild/compass/go/internal/stack" ) -// TestEmbeddedPostgresSmoke is the one e2e leg that runs the product-default -// postgres: the stack spawns and owns it, so this is the only dedicated e2e -// coverage of the supervisor-owns-postgres lifecycle and its pgid-record -// teardown. Every other leg attaches to the suite postgres instead. +// TestEmbeddedPostgresSmoke is the only e2e leg that asserts the +// supervisor-owns-postgres lifecycle and its pgid-record teardown. Shared legs +// use the suite postgres; other own-stack legs run the stack-owned postgres but +// never check how it is recorded or reaped. func TestEmbeddedPostgresSmoke(t *testing.T) { if !podmanUsable(t.Context()) { t.Skip("rootless podman cannot run compass-agent:latest here; skipping the real-stack e2e") diff --git a/go/internal/pgtest/pgtest.go b/go/internal/pgtest/pgtest.go index 0a46d02ef..0fd558fd3 100644 --- a/go/internal/pgtest/pgtest.go +++ b/go/internal/pgtest/pgtest.go @@ -1,3 +1,5 @@ +//go:build pgtest && unix + // Package pgtest is the shared real-Postgres test harness for the compass // backend. A store of record is only proven against the database it targets, so // every store/handler/serve integration test runs against an actual Postgres @@ -25,9 +27,7 @@ // It is build-tagged `pgtest` so it is compiled only into that opt-in test lane, // never the default `go test` gate. It is a non-test file (not `_test.go`) so it // is importable across packages; the build tag keeps it out of production -// binaries all the same. -//go:build pgtest && unix - +// binaries all the same. Which tier to test at: docs/architecture/testing-tiers.md. package pgtest import (