feat(store): move the transcript archive flush off the commit path (RIG-5158)

[rigel-mintaka]

Summary

Move the transcript archive flush off CommitConversationFrame. The commit returns once the row lands in Postgres. A Store-owned flusher archives later, one pass at a time per session.

Why

AppendTranscriptEntry ran the compaction flush and the safety valve inline. The object-store PUT had no deadline. A slow or frozen store held the RPC until the agent's 5s attempt timeout, and the agent's serial frame chain stalled behind it. This is the measured root cause of the s3-outage e2e flake (RIG-5157).

Change

Implements the frozen record docs/designs/agent/compass-transcript-flush-off-commit.md (DL-491), both tasks.

• Commit path: AppendTranscriptEntry commits the row and calls requestFlush. primaryFlush is gone. A deduped retry, or a store with no object store, requests nothing.
• Flusher (go/internal/store/transcript_flusher.go):
  • one slot per session, with triggers that land mid-pass coalescing into one more pass;
  • at most maxConcurrentFlushes = 4 passes at once;
  • each pass runs under a 60s deadline and the commit's tenant and system scope;
  • a failed pass is logged at ERROR and counted in compass.store.transcript_flush_failures.
• Lifecycle: Open starts the flusher. Close cancels it and waits for every pass before pool.Close().
• Database guards for the new races:
  • HotTailBytes and HotTailSizes find the checkpoint inside the statement, and valveCut stops before any checkpoint row.
  • PruneTranscriptEntries is now :execrows. A pruning flush rolls back with errFlushRaced unless it deleted exactly the rows it archived.
  • SafetyValveSegments hides valve segments at or below the latest checkpoint.
• Tests and callers: FlushTranscript is the synchronous barrier tests use. The sync-flush tests, the server resume S3 test and the e2e comments are updated.

Verification

• New pgtests pass, and pass 5 of 5 under -race:

  • non-blocking commit (checkpoint and valve);
  • coalescing;
  • global cap;
  • checkpoint inside the cut;
  • DB guard with overlapping stores;
  • pass deadline;
  • Close joins the pass;
  • tenant;
  • segment filter;
  • valveCut cases;
  • runnerhub CommitConversationFrame returns while the PUT hangs.

• Mutations, each killed by its test:

  Mutation	Failing test
  Inline flush restored	non-blocking test
  valveCut checkpoint stop removed	TestValveCut
  Prune count check dropped	DB guard test
  Segment filter dropped	filter test
  Close skips the join	shutdown test
  Semaphore capacity set to 5	global cap test

• go test -tags pgtest -race passes for the full internal/store, internal/runnerhub and server packages on a fresh Postgres.

• golangci-lint reports 0 issues with and without the full tag set. sqlc-drift is clean. go build -tags podman ./e2e/... succeeds.

• Review loop: round 1 found 0 high, 1 medium, 1 low; both are fixed. Round 2 found 0/0/0.

Refs RIG-5158, RIG-5157.

Ledger-impact: none (implements DL-491, already recorded).

[trunk-io]

😎 This pull request was merged.

[linear-code]

RIG-5158

RIG-5157

[rigel-mintaka]

Review loop at head c040cfc: round 1 found 0 high, 1 medium, 1 low. The medium was a missing global-cap test, now TestFlushPassesShareGlobalCap; the low was a stale SegmentKindSafetyValve comment. Both are fixed. Round 2 found 0/0/0. After review, the pre-push hook's nilaway flagged valveCut slicing a possibly-nil tail, so an empty-rows guard was added. TestValveCut passes and nilaway is clean.

[github-actions]

Compass engineering docs preview: https://compass-repo-rig-5158-flush-l8mp.compass-eng-docs.pages.dev

Deployed from compass-repo/rig-5158-flush-impl at c040cfc.