Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
101 changes: 101 additions & 0 deletions docs/architecture/testing-tiers.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,101 @@
# Go testing tiers

The Go module has three test tiers. Pick the cheapest tier that can fail on
the behavior you are testing.

| Tier | Build tag | Runs against | Cost | CI job |
| --- | --- | --- | --- | --- |
| Default | none | Fakes and in-process code | Seconds | moon battery |
| pgtest | `pgtest` | A real Postgres | Minutes per package | `pgtest` |
| podman e2e | `podman` | The real stack and the real agent image | Tens of minutes | `e2e` |

The exact commands and flags are in the `pgtest` and `e2e` jobs of
`.github/workflows/ci.yml`.

## Which tier

1. Does the assertion need real SQL: a query, a constraint, RLS, a migration?
Use **pgtest**.
2. Does it need the process or container topology: stack children, the runner,
an agent turn, teardown? Use **podman e2e**.
3. Otherwise use the **default** tier.

A tagged test that cannot reach its dependency skips. CI checks that the
pgtest and podman suites actually ran, so a skip there does not pass silently.

## pgtest

The entry point is `pgtest.RequireDSN(t)` (`go/internal/pgtest`). Open your own
store on the DSN it returns. It finds Postgres in this order:

- `COMPASS_TEST_DATABASE_DSN` points at a running Postgres. Each test gets a
fresh schema, dropped when the test ends. CI uses this.
- `COMPASS_TEST_USE_CONTAINER=1` starts a throwaway container. Use this
locally.
- With neither set, it fails when a container runtime exists, and skips when
none does. CI sets `COMPASS_REQUIRE_LIVE=1`, so there a missing database
fails.

For tests that must go through a transaction-mode pooler, use
`pgtest.RequirePoolerDSN(t)`. It needs both `COMPASS_TEST_DATABASE_DSN` and
`COMPASS_TEST_POOLER_DSN` in URL form, and gives each test a fresh database.

## podman e2e

Fixture-based legs live in `go/e2e`. `cmd/compass-stack` also has
podman-tagged supervisor tests that do not use the fixture.

In `go/e2e`, `TestMain` builds the stack binaries once and starts one suite
Postgres. Each leg starts with a `podmanUsable` check and skips when rootless
podman cannot run the agent image.

### Shared stack: the default

Most legs call `sharedFixture(t)`. Every such leg in the run uses one stack,
which attaches to the suite Postgres through `ExternalDatabase`.

- Register options the shared stack needs from a `register<Leg>FixtureOptions`
function listed in `TestMain`. Never register from a leg body: canned routes
are fixed when the stub is built. A registered option applies to every shared
leg.
- You can register `WithForgeStub()` and the `WithCanned*` options.
`WithObjectStore` needs a `*testing.T` to start, so it is own-stack only.
- Route each model turn by a marker unique to your leg. The positional canned
script is one counter shared by every leg.
- Await every turn your leg starts before it returns, so no model request
outlives the leg.
- `-count` above 1 is not supported; CI runs with `-count=1`. In a full run,
`zz_legsix_test.go` closes the shared stack at the end of each pass, so a
later pass cannot reach it. Without leg six, handles are create-only and the
stack persists, so the second pass fails with `already_exists`.

### Own stack: the opt-out

Call `NewFixture(ctx, t, opts...)` when the leg needs a stack nobody else
touches, such as a restart, a teardown check, or object-store state. It costs a
full cold start: its own stack-owned Postgres, server and runner.

- `WithSite(site)` reuses one root, state dir and port across two `NewFixture`
calls, so the second `Up` reattaches the same Postgres data.
- `WithEmbeddedPostgres()` states that the leg needs the stack-owned Postgres.
Plain `NewFixture` already uses it; the option fails the test if an external
database is also set.
- `WithForgeStub()`, `WithObjectStore(g)` and the `WithCanned*` options apply to
this leg only.

`zz_legsix_test.go` closes the shared stack before its restart, so it must run
last. Its file name sorts it last, and it skips under `-shuffle`.

### The embedded-Postgres smoke leg

Shared legs use the suite Postgres. Other own-stack legs use the stack-owned
Postgres but do not check its lifecycle. `TestEmbeddedPostgresSmoke` is the
only leg that asserts the supervisor-owns-Postgres lifecycle and its pgid-record
teardown: the record names a live Postgres group after `Up`, and after `Down`
the group is gone and the record is removed. Keep it when you change how the
stack starts or stops Postgres.

## Related

- Design record: `docs/designs/infra/ci/compass-test-harness-tiers/design.md`.
- Toolchains and the moon battery: [Build & CI](build-and-ci.md).
10 changes: 8 additions & 2 deletions docs/designs/infra/ci/compass-test-harness-tiers/design.md
Original file line number Diff line number Diff line change
Expand Up @@ -482,6 +482,12 @@ docs.
- Interfaces: none (prose); MUST cite the same seams this record does.
- Test cycle: markdownlint.

*Errata (as shipped).* The page is `docs/architecture/testing-tiers.md`, the
eng-docs Architecture section. It documents `sharedFixture` vs plain
`NewFixture` (no `WithOwnStack`, per the T3 errata), `WithSite`, and
`WithEmbeddedPostgres`. The `-count>1` caveat lives in that page, not in the
`fixture.go` package doc; the `go/e2e` and `pgtest` package docs link to it.

## Tasks

- [x] T1 — extract `go/internal/pgshare` (acquisition + schema isolation +
Expand All @@ -494,8 +500,8 @@ docs.
- [x] T4 — `TestEmbeddedPostgresSmoke` + `WithEmbeddedPostgres()`.
- [x] T5 — forge-injection convergence (`mountForgeForTest`). Satisfied on
main by `newForgeE2EWire`; see T5 errata.
- [ ] T6 — tier-pyramid contributor doc. Owner: implement lane (depends
T3/T4 landing so the doc describes reality).
- [x] T6 — tier-pyramid contributor doc. Shipped as
`docs/architecture/testing-tiers.md`; see T6 errata.

## Open Questions (1–2 RULED by Matt at freeze; 3–4 deferred)

Expand Down
3 changes: 2 additions & 1 deletion go/e2e/clients.go
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,8 @@
// CLI drives) and hands the harness's later legs (RIG-1785 H2-H6) a Fixture with
// authenticated Connect clients and the store DSN. It is podman-build-tagged —
// it drives real child processes and the real agent image, so it is out of the
// hermetic unit lane and runs only under `-tags podman`.
// hermetic unit lane and runs only under `-tags podman`. Which tier to test at,
// and sharedFixture vs NewFixture: docs/architecture/testing-tiers.md.
package e2e

import (
Expand Down
8 changes: 4 additions & 4 deletions go/e2e/embedded_postgres_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -19,10 +19,10 @@ import (
"github.com/RigelBuild/compass/go/internal/stack"
)

// TestEmbeddedPostgresSmoke is the one e2e leg that runs the product-default
// postgres: the stack spawns and owns it, so this is the only dedicated e2e
// coverage of the supervisor-owns-postgres lifecycle and its pgid-record
// teardown. Every other leg attaches to the suite postgres instead.
// TestEmbeddedPostgresSmoke is the only e2e leg that asserts the
// supervisor-owns-postgres lifecycle and its pgid-record teardown. Shared legs
// use the suite postgres; other own-stack legs run the stack-owned postgres but
// never check how it is recorded or reaped.
func TestEmbeddedPostgresSmoke(t *testing.T) {
if !podmanUsable(t.Context()) {
t.Skip("rootless podman cannot run compass-agent:latest here; skipping the real-stack e2e")
Expand Down
6 changes: 3 additions & 3 deletions go/internal/pgtest/pgtest.go
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
//go:build pgtest && unix

// Package pgtest is the shared real-Postgres test harness for the compass
// backend. A store of record is only proven against the database it targets, so
// every store/handler/serve integration test runs against an actual Postgres
Expand Down Expand Up @@ -25,9 +27,7 @@
// It is build-tagged `pgtest` so it is compiled only into that opt-in test lane,
// never the default `go test` gate. It is a non-test file (not `_test.go`) so it
// is importable across packages; the build tag keeps it out of production
// binaries all the same.
//go:build pgtest && unix

// binaries all the same. Which tier to test at: docs/architecture/testing-tiers.md.
package pgtest

import (
Expand Down
Loading