diff --git a/docs/architecture/testing-tiers.md b/docs/architecture/testing-tiers.md new file mode 100644 index 000000000..9fb2ad1fd --- /dev/null +++ b/docs/architecture/testing-tiers.md @@ -0,0 +1,101 @@ +# Go testing tiers + +The Go module has three test tiers. Pick the cheapest tier that can fail on +the behavior you are testing. + +| Tier | Build tag | Runs against | Cost | CI job | +| --- | --- | --- | --- | --- | +| Default | none | Fakes and in-process code | Seconds | moon battery | +| pgtest | `pgtest` | A real Postgres | Minutes per package | `pgtest` | +| podman e2e | `podman` | The real stack and the real agent image | Tens of minutes | `e2e` | + +The exact commands and flags are in the `pgtest` and `e2e` jobs of +`.github/workflows/ci.yml`. + +## Which tier + +1. Does the assertion need real SQL: a query, a constraint, RLS, a migration? + Use **pgtest**. +2. Does it need the process or container topology: stack children, the runner, + an agent turn, teardown? Use **podman e2e**. +3. Otherwise use the **default** tier. + +A tagged test that cannot reach its dependency skips. CI checks that the +pgtest and podman suites actually ran, so a skip there does not pass silently. + +## pgtest + +The entry point is `pgtest.RequireDSN(t)` (`go/internal/pgtest`). Open your own +store on the DSN it returns. It finds Postgres in this order: + +- `COMPASS_TEST_DATABASE_DSN` points at a running Postgres. Each test gets a + fresh schema, dropped when the test ends. CI uses this. +- `COMPASS_TEST_USE_CONTAINER=1` starts a throwaway container. Use this + locally. +- With neither set, it fails when a container runtime exists, and skips when + none does. CI sets `COMPASS_REQUIRE_LIVE=1`, so there a missing database + fails. + +For tests that must go through a transaction-mode pooler, use +`pgtest.RequirePoolerDSN(t)`. It needs both `COMPASS_TEST_DATABASE_DSN` and +`COMPASS_TEST_POOLER_DSN` in URL form, and gives each test a fresh database. + +## podman e2e + +Fixture-based legs live in `go/e2e`. `cmd/compass-stack` also has +podman-tagged supervisor tests that do not use the fixture. + +In `go/e2e`, `TestMain` builds the stack binaries once and starts one suite +Postgres. Each leg starts with a `podmanUsable` check and skips when rootless +podman cannot run the agent image. + +### Shared stack: the default + +Most legs call `sharedFixture(t)`. Every such leg in the run uses one stack, +which attaches to the suite Postgres through `ExternalDatabase`. + +- Register options the shared stack needs from a `registerFixtureOptions` + function listed in `TestMain`. Never register from a leg body: canned routes + are fixed when the stub is built. A registered option applies to every shared + leg. +- You can register `WithForgeStub()` and the `WithCanned*` options. + `WithObjectStore` needs a `*testing.T` to start, so it is own-stack only. +- Route each model turn by a marker unique to your leg. The positional canned + script is one counter shared by every leg. +- Await every turn your leg starts before it returns, so no model request + outlives the leg. +- `-count` above 1 is not supported; CI runs with `-count=1`. In a full run, + `zz_legsix_test.go` closes the shared stack at the end of each pass, so a + later pass cannot reach it. Without leg six, handles are create-only and the + stack persists, so the second pass fails with `already_exists`. + +### Own stack: the opt-out + +Call `NewFixture(ctx, t, opts...)` when the leg needs a stack nobody else +touches, such as a restart, a teardown check, or object-store state. It costs a +full cold start: its own stack-owned Postgres, server and runner. + +- `WithSite(site)` reuses one root, state dir and port across two `NewFixture` + calls, so the second `Up` reattaches the same Postgres data. +- `WithEmbeddedPostgres()` states that the leg needs the stack-owned Postgres. + Plain `NewFixture` already uses it; the option fails the test if an external + database is also set. +- `WithForgeStub()`, `WithObjectStore(g)` and the `WithCanned*` options apply to + this leg only. + +`zz_legsix_test.go` closes the shared stack before its restart, so it must run +last. Its file name sorts it last, and it skips under `-shuffle`. + +### The embedded-Postgres smoke leg + +Shared legs use the suite Postgres. Other own-stack legs use the stack-owned +Postgres but do not check its lifecycle. `TestEmbeddedPostgresSmoke` is the +only leg that asserts the supervisor-owns-Postgres lifecycle and its pgid-record +teardown: the record names a live Postgres group after `Up`, and after `Down` +the group is gone and the record is removed. Keep it when you change how the +stack starts or stops Postgres. + +## Related + +- Design record: `docs/designs/infra/ci/compass-test-harness-tiers/design.md`. +- Toolchains and the moon battery: [Build & CI](build-and-ci.md). diff --git a/docs/designs/infra/ci/compass-test-harness-tiers/design.md b/docs/designs/infra/ci/compass-test-harness-tiers/design.md index 5c0dd5232..7f9330752 100644 --- a/docs/designs/infra/ci/compass-test-harness-tiers/design.md +++ b/docs/designs/infra/ci/compass-test-harness-tiers/design.md @@ -482,6 +482,12 @@ docs. - Interfaces: none (prose); MUST cite the same seams this record does. - Test cycle: markdownlint. +*Errata (as shipped).* The page is `docs/architecture/testing-tiers.md`, the +eng-docs Architecture section. It documents `sharedFixture` vs plain +`NewFixture` (no `WithOwnStack`, per the T3 errata), `WithSite`, and +`WithEmbeddedPostgres`. The `-count>1` caveat lives in that page, not in the +`fixture.go` package doc; the `go/e2e` and `pgtest` package docs link to it. + ## Tasks - [x] T1 — extract `go/internal/pgshare` (acquisition + schema isolation + @@ -494,8 +500,8 @@ docs. - [x] T4 — `TestEmbeddedPostgresSmoke` + `WithEmbeddedPostgres()`. - [x] T5 — forge-injection convergence (`mountForgeForTest`). Satisfied on main by `newForgeE2EWire`; see T5 errata. -- [ ] T6 — tier-pyramid contributor doc. Owner: implement lane (depends - T3/T4 landing so the doc describes reality). +- [x] T6 — tier-pyramid contributor doc. Shipped as + `docs/architecture/testing-tiers.md`; see T6 errata. ## Open Questions (1–2 RULED by Matt at freeze; 3–4 deferred) diff --git a/go/e2e/clients.go b/go/e2e/clients.go index ae0cf27c0..6cf08657e 100644 --- a/go/e2e/clients.go +++ b/go/e2e/clients.go @@ -5,7 +5,8 @@ // CLI drives) and hands the harness's later legs (RIG-1785 H2-H6) a Fixture with // authenticated Connect clients and the store DSN. It is podman-build-tagged — // it drives real child processes and the real agent image, so it is out of the -// hermetic unit lane and runs only under `-tags podman`. +// hermetic unit lane and runs only under `-tags podman`. Which tier to test at, +// and sharedFixture vs NewFixture: docs/architecture/testing-tiers.md. package e2e import ( diff --git a/go/e2e/embedded_postgres_test.go b/go/e2e/embedded_postgres_test.go index 91b7edc26..1aa9313a4 100644 --- a/go/e2e/embedded_postgres_test.go +++ b/go/e2e/embedded_postgres_test.go @@ -19,10 +19,10 @@ import ( "github.com/RigelBuild/compass/go/internal/stack" ) -// TestEmbeddedPostgresSmoke is the one e2e leg that runs the product-default -// postgres: the stack spawns and owns it, so this is the only dedicated e2e -// coverage of the supervisor-owns-postgres lifecycle and its pgid-record -// teardown. Every other leg attaches to the suite postgres instead. +// TestEmbeddedPostgresSmoke is the only e2e leg that asserts the +// supervisor-owns-postgres lifecycle and its pgid-record teardown. Shared legs +// use the suite postgres; other own-stack legs run the stack-owned postgres but +// never check how it is recorded or reaped. func TestEmbeddedPostgresSmoke(t *testing.T) { if !podmanUsable(t.Context()) { t.Skip("rootless podman cannot run compass-agent:latest here; skipping the real-stack e2e") diff --git a/go/internal/pgtest/pgtest.go b/go/internal/pgtest/pgtest.go index 0a46d02ef..0fd558fd3 100644 --- a/go/internal/pgtest/pgtest.go +++ b/go/internal/pgtest/pgtest.go @@ -1,3 +1,5 @@ +//go:build pgtest && unix + // Package pgtest is the shared real-Postgres test harness for the compass // backend. A store of record is only proven against the database it targets, so // every store/handler/serve integration test runs against an actual Postgres @@ -25,9 +27,7 @@ // It is build-tagged `pgtest` so it is compiled only into that opt-in test lane, // never the default `go test` gate. It is a non-test file (not `_test.go`) so it // is importable across packages; the build tag keeps it out of production -// binaries all the same. -//go:build pgtest && unix - +// binaries all the same. Which tier to test at: docs/architecture/testing-tiers.md. package pgtest import (