diff --git a/docs/design/Agent-Integration.md b/docs/design/Agent-Integration.md new file mode 100644 index 0000000000..3add53da77 --- /dev/null +++ b/docs/design/Agent-Integration.md @@ -0,0 +1,663 @@ +# Agent integration: an MCP lab server + +Status: accepted 2026-11-14, Slices 0-3 built (proposed 2026-11-01; revised 2026-11-14 from +the week's incidents, +`~/BDSPLab/notes/2026-11-13-week-problems.md`, and an implementation review; checked against the +lab scripts and the user's console rules on 2026-11-14; the user's decisions of 2026-11-14 +are recorded in [Decisions and open questions](#decisions-and-open-questions)). It builds on: +- the lab channel: fork #107, `SerialPrograms/Source/CommonFramework/Windows/Lab.md` + (`Lab/inbox`, `Lab/outbox`, `Lab/state`, the owner token); +- run records: fork #116, `docs/design/Lab-RunRecords.md`; +- the lab scripts on `agents` (`lab/control`, `lab/scripts`) and in `~/BDSPLab` + (`labrun.sh`, `labbuild.sh`, `checkin.sh`, `lablock.sh`, `heavy_gate.sh`, ...). + +Before this rollout nothing here was implemented except what those already have, FRLG RNG's one-off +`RngProgramInstance::run_precheck()` (upstream PokemonAutomation/Arduino-Source#1881, which the +precheck list below generalizes) and the `require-profile` Claude Code hook (done, see +Enforcement). In particular, the app did not expose a frame sequence, a generic +game/screen detector, a preflight API, or an action source. Those are deliverables below, not +pre-existing integration points. Everything here is fork tooling for now (user, 2026-11-14). + +Build status (2026-11-14): Slices 0-3 are built as fork drafts: #478 (Slice 0, contracts and +fixtures, base `agents`, tip ff209b333), #481 (Slice 1, `Lab/state.json` and the snapshot +command, base `lab/tooling`, tip 629bfde88), #482 (Slice 2, read server, on #478, tip +cbfccc2ac) and #483 (Slice 3, control plane, on #482, tip c9b45d06e). Slice 4 is +approved to start, stacked on upstream #1881's branch `upstream/frlg-free-party-slots`. Draft +#476 (`LabStateV1`, commit 7c44e72c3) adds a serializer and its tests only; it does not modify +`LabChannel` or tag transitions, and its second `ProgramState` enum is to be dropped (see +Resources). + +## Requirement + +Agents drive console B through about forty shell scripts around the file channel. Each one +relearns the protocol and the rules, and the rules hold only as long as every agent +remembers them. This week that failed in eight ways (table below): stale frames, a start on +the HOME screen, setup errors found after a 2-10 minute reset, the app run without its +profile, and answers that cost four log greps. + +The goal is one local server that owns console B control, answers state questions from typed +resources, and refuses a bad request before it costs a reset. Agents talk to it over MCP; +the user can use the same tools from a thin CLI. + +Not in scope for the console server: remote control, console A, GitHub writes, branch/PR +maintenance, CI diagnosis, video streaming, or arbitrary game selection from HOME. Heavy +build/test orchestration can be added later as a separate local-operations worker; it must +not expand the console daemon's credentials or failure domain. + +## Problems this solves + +One row per incident from 2026-11-07..13. "R" = resource, "T" = tool, "E" = event. + +| Incident | Prevented by | +|---|---| +| Reused shot tag returned a stale Safari frame while the player was at the Mansion | T `shot()` returns a new frame with its `seq`; never a reused file | +| PNG read while the app was still writing it | T `shot()` returns the image only once written | +| Selecting a settings panel through the inbox blocked the channel | T `run_program` takes program panels only; no raw `select` | +| Panel switch reset controller and video ("requires video feedback") | R `console://B/state` `controller`, `video`; `run_program` waits for both | +| No-HDMI colour bars went unnoticed until a frame was read | R `state.video.signal`; E `video_lost` | +| Game switches (HOME, close, pick tile) done by hand | deferred: arbitrary title selection has no reliable app seam yet | +| PLZA 03.08 started with the game closed and walked on HOME | T `ensure_game(expected)` and the program's `game_open` precheck refuse the run | +| UI stop at 22:25 had to be guessed as the user's | E `stopped{source: ui}`; lease moves to `user` | +| "Hands off B while the user plays" held only by memory | lease: tools refuse while `user` holds it | +| Which save B holds lived in notes (Safari logo vs Mansion door, unsaved Parasect) | R `console://B/save` | +| Max Resets 10 on a long-shot target | schema range plus program precondition | +| Safari needs ≥ 4000 advances | precondition on `run_program` | +| FRLG license check not loaded (Game Settings panel never opened) | precondition: game settings loaded | +| No Sweet Scent user in the last slot | a precheck declared with the program, run before any reset | +| Restart on a post-run summary screen | precondition "known screen" refuses the run; recovery remains explicit | +| PLZA LANGUAGE left on ja | precondition: language matches the console's game | +| Dev-only checkpoint hidden without Developer Mode shifted indices; wrong checkpoint started | options by key and checkpoint id, never by index; refuse a hidden row | +| Audio input not allowlisted for lab-B (UserSetupError spam) | server launch checks the lab profile's settings | +| App run without `--profile`, twice | done 2026-11-14: the `require-profile` PreToolUse hook refuses the call (Enforcement); from Slice 7 the server launches the app | +| Rules retyped per brief and drifting | rules enforced by the server, not restated in briefs | +| Agents parked waiting on the console | long console operations return a request/ticket; E `program_finished` | +| Linter rule changed mid-task | not addressed by this design | +| One upstream sync = merges through 36 branches (+~40 descendants) | out of scope: keep this in PR-maintenance tooling | +| CI red from SystemKeyboardDetector images never pushed to CommandLineTests | out of scope: keep this in CI-diagnosis tooling | +| Branches checked out in several worktrees went stale | out of scope: keep this in PR-maintenance tooling | +| "How far off is the RNG?" took four greps | R `runs://latest` carries RNG hits (target, hit, delta) | +| Results spread over notes, outbox, error dirs with spaces | R `runs://`: one record, frame and video at clean paths | +| Sim passes, console fails (FRLG 13.01, 12.03; PLZA walk_to_objective) | partly: `runs://` keeps the walked poses a sim test replays | +| Disk under the 30 GB floor | future local-operations worker refuses a build under the floor | +| RAM swap from parallel builds | future local-operations worker serializes builds and applies the existing limits | + +## Shape + +``` + agent hosts (Claude Code, Codex) user shell (thin lab CLI) + │ MCP Streamable HTTP on 127.0.0.1 │ + └──────────────────┬───────────────┘ + ▼ + pa-lab console daemon + ┌──────────────────┼───────────────────┐ + │ │ │ + console-B adapter SQLite control DB audit export + Lab/ files only leases, events, JSONL (derived) + requests, tickets + │ + │ Lab/inbox → Lab/outbox + │ Lab/state + Lab/state.json, Lab/save.json, Lab/runs/ + ▼ + SerialPrograms (`--profile lab-B`) +``` + +The daemon is one long-lived Python process because leases, idempotency and the event +sequence must be shared by every client. It uses MCP Streamable HTTP bound only to +`127.0.0.1`, checks an unguessable bearer token and validates `Origin`; stdio would create +one independent lease database per client. Its allowlist is a server-owned config with one +entry, B. It must not reuse the operator's broader `~/BDSPLab/consoles.json`, which also +describes A. + +The durable control plane is SQLite. JSONL is an audit/export format, not the source of +truth for leases, idempotency or event cursors. The daemon initially attaches to an app +started by the existing guarded launch path. It becomes the app supervisor only after that +launch behavior has been extracted and tested; it must reuse the profile, direct-exec and +process-identity guards rather than reimplementing them. + +The console daemon has no GitHub token and does not inspect worktrees. A future +local-operations worker may expose queued build/test tickets over a separate process and +configuration, reusing the existing heavy-gate scripts. `ci_status` and `stack_plan` remain +outside this design. + +The daemon lives with the lab tooling on `agents` (`lab/server/`), in a venv like the +code-health one, with its fake-lab tests beside it. App-side slices go on branches from +`lab/tooling` (#107), where the lab channel lives, so this design PR stays documentation only. + +## Resources (read-only, first implementation slice) + +The app keeps the legacy key/value `Lab/state` (`seq`, `state`, `program`, `controller`, +`owner`, `error`) byte-compatible for existing scripts and atomically writes a new versioned +`Lab/state.json`. The C++ side only serializes. The Python daemon owns strict parsing: it +rejects an unsupported `schema`, keeps unknown fields that matter for diagnostics, and +overlays only server-owned data such as the lease. Shared golden JSON fixtures (Slice 0, #478) +are the cross-language contract: the app's writer is tested against them byte for byte and the +daemon's parser against the same files. Example app file: + +```json +{ + "schema": 1, + "written_at": "2026-11-14T06:10:12.124Z", + "instance": {"id": "d8ea6a9d-1eab-4a40-b355-d88072b289a1", "pid": 4411}, + "panel": {"id": "PokemonFRLG:AutoStory", "kind": "program"}, + "program": { + "state": "stopped", + "run_id": null, + "last_transition": { + "action": "stop", + "source": "ui", + "request_id": null, + "at": "2026-11-14T06:10:10.002Z" + } + }, + "controller": {"state": "ready"}, + "video": { + "state": "live", + "frame_seq": 918274, + "captured_at": "2026-11-14T06:10:12.091Z", + "signal": "unknown" + }, + "observation": { + "game": {"value": "unknown", "observed_at": null}, + "screen": {"value": "unknown", "observed_at": null} + } +} +``` + +`written_at` is a heartbeat even when no logical field changes. `instance.id` changes on +each app start, so a reset frame sequence cannot be mistaken for a newer frame from the old +process. `frame_seq` is one process-wide monotonic counter: it increases for every frame the +app receives and never resets on a panel switch or a camera auto-reset, so frames order by +`(instance.id, frame_seq)` with no stream epoch. `captured_at` is that frame's capture time, +not the time the state file was serialized. +The daemon computes frame age from `captured_at`; it does not persist a decaying +`last_frame_age_ms` value. `program.state` reuses the app's one `ProgramState` enum, mapped +once to the legacy upper-case values and once to lower-case JSON values (`stopped`, +`running`, `stopping`, `not_ready`, `no_program`); no second state enum. +`program.last_transition` is an object, or `null` before the first transition. + +Publication: a logical change (panel, program state, transition, controller, video state) +rewrites `Lab/state.json` immediately; heartbeat-only and frame-only changes (`written_at`, +`frame_seq`, `captured_at`) are throttled to one write every 1-2 s. + +Version 1 distinguishes transport freshness (`live`, `stale`, `missing`) from visual signal +classification (`ok`, `black`, `colour_bars`, `unknown`). The first slice can truthfully +publish `unknown`; black/colour-bar detection becomes a later detector, not a guessed +boolean. Game and screen observations are also `unknown` until a named detector has run. +Every non-unknown observation must include `observed_at` and, when useful, detector and +confidence fields. + +`program.last_transition.source` is emitted where the action originates, not inferred from +the resulting state: `ui`, `lab`, `program`, or `error`. A lab action carries the daemon's +request UUID. `ProgramSession`'s start and stop take an `ActionSource` and a request id, and the +session latches the stop's initiator from `STOPPING` through `STOPPED`, so a later event cannot +overwrite who stopped the run. This is how the daemon distinguishes a user's Stop click from +its own Stop request even if both race. + +**`console:///state`** is the parsed app state plus: + +```json +{"lease": {"kind": "agent", "owner": "agent:frlg-lane", "acquired_at": "...", "expires_at": "..."}} +``` + +It also reports daemon/app reachability and the exact refusal reasons that currently block +an action. Unknown facts stay unknown; absence is never interpreted as success. + +**`console:///save`** is a fact with provenance, not a promise that every game save was +observed. Programs that know they saved atomically write `Lab/save.json` with `schema`, +`game`, `slot`, `checkpoint`, structured position, `saved_at`, `run_id`, and +`producer` (program id and build). The daemon exposes a status of `known`, `stale`, or +`unknown` and the reason. It marks the fact stale after a user lease, a CFR batch, or any +program that may save but does not publish the hook. Only a new program-authored save fact +can return it to known. + +**`runs://latest`, `runs://`**: #116's run record, summarised: +`verdict` (`passed|failed|stopped`, reason, stop source), checkpoint lines +(`13.01 failed mansion_holes after 2 attempts`), RNG hits (`target 4123, hit 4127, +delta +4`), and the failure frame and video copied to `~/BDSPLab/runs//` (no spaces). + +## Leases, requests and events + +Reads need no lease. Every tool that can press a button, start/stop a program or mutate app +selection requires a live lease token, not merely the caller-supplied owner name. The +SQLite state machine is: + +| State | Entry | Exit → target | +|---|---|---| +| `free` | initial state or explicit release | successful `acquire_console` → `agent-idle`; observed UI action or operator claim → `user` | +| `agent-idle` | acquired with a 20-minute TTL, renewed by every tool call | renew → `agent-idle`; explicit release or TTL expiry → `free`; run start → `agent-running`; observed UI action → `user` | +| `agent-running` | the leased caller starts a run | natural terminal event or lab-requested stop (`stop_program`) → `free`; UI-originated transition → `user`; it never expires mid-run | +| `user` | app reports a UI-originated Start/Stop or the operator explicitly claims B | explicit operator release only → `free` | + +A UI action revokes any agent lease before a user lease is installed. There is no guessed +30-minute user timeout: absence of UI telemetry is not proof that the user is done. The +operator releases a `user` lease with the thin CLI's `release-user` command, which reads a +separate operator token file; a hook rule refuses that command from an agent shell. The user +and the agents share one login on the lab Mac, so the rule, not cryptography, is the guard. On +daemon restart, leases and request results survive in SQLite; the daemon reconciles them +against `instance.id`, current program state and the latest transition before allowing a +mutation. + +**Until Slice 7 the existing locks stay in force and the daemon works beside them.** +`labrun.sh` launches the app under `lablock.sh`, and the launcher's train holds that lock for +the app's lifetime; `labctl.sh` and `cfrun.sh` check it before writing a request. From Slice 5 +on, the daemon claims the channel's `owner` token (Lab.md) when it attaches and re-claims it +after every app restart, since that token dies with the app; the app then refuses any +mutating request that does not come from the daemon, which is what makes "raw writes to +`Lab/inbox` are not exposed" hold for stray scripts too. Three gates coexist during the +transition: `lablock` on launching the app, the owner token on writing the channel, the lease +on which agent may act. Slice 7 folds the first into the daemon; the owner token stays as the +app/daemon boundary. + +Every mutating call requires a client-generated request UUID. The unique key +`(client_id, tool, request_id)` stores the normalized arguments and terminal result. An +exact retry returns the stored result; reuse with different arguments is refused. This +prevents a transport retry from starting a program or pressing a button twice. + +Events are rows in SQLite with a monotonically increasing 64-bit `seq`, event time, console, +app instance, request id and JSON payload. MCP resource-update notifications are a wake-up +hint; `next_events(after_seq, timeout <= 60 s)` is the durable catch-up API and returns the +next cursor. JSONL can be exported for humans, but truncating it cannot corrupt cursors. + +| Event | Fields | +|---|---| +| `program_finished`, `program_failed` | run id, verdict, checkpoint | +| `stopped` | source: `ui`, `server`, `program`, `error` | +| `video_lost`, `video_back` | last frame seq | +| `autosave`, `save` | the new `console://B/save` | +| `lease_changed` | old and new owner | +| `request_completed`, `request_failed` | request id, tool, normalized result or refusal | + +Agents do not park on a tool call. Long operations return a ticket/request id; a later +client consumes events or reads request status. The daemon may keep recent resource +notifications in memory, but correctness always comes from the durable cursor. + +## Tools + +Each tool validates before it acts and returns JSON, or a refusal naming the failed check. + +| Tool | Validates / does | +|---|---| +| `acquire_console(console, owner, ttl)` | Atomically acquires `free`; returns an opaque lease token and expiry. Owner is descriptive, not authority. | +| `renew_console(lease, ttl)` / `release_console(lease)` | Operate only on the matching token. A user lease is released through an operator-authenticated CLI action, not an agent token. | +| `shot(console, max_age_ms=2000, timeout_ms=5000)` | Lease not needed. Records `(instance.id, frame_seq)` at call start and waits for a strictly newer frame from the app's video-session snapshot seam. Returns the completed PNG plus the new token and `captured_at`; never reuses a caller tag. Refuses on timeout, stale transport, or a classified bad signal. | +| `ensure_game(console, expected, allowed_screens)` | Read-only verification. Runs/reads the named detectors and returns the observation and age. It never navigates HOME or launches a title. | +| `describe_program(console, panel)` | Returns a stable program id and only the stable parts of the option schema: keys, types, defaults, bounds and which enum entries are available, plus the declared prechecks (each check's name and `needs` line), without touching the console. Dynamic visibility is not part of it. Refuses settings panels and hidden/unavailable programs. | +| `preflight_program(console, panel, options)` | Lease required because detectors may use app resources, but performs no controller action. Rejects unknown keys and disabled values, stages the candidate options, evaluates dynamic visibility and `check_validity`, then has the app run the program's prechecks in order; always restores the prior options. Returns the checks run, the first failure's name and user message, and evidence. | +| `run_program(console, panel, options, preflight_token)` | Lease. Requires a still-current successful preflight bound to app instance, panel, the candidate options' hash, observation versions and lease. Starts once and returns request/run ids. | +| `stop_program(console)` | Lease. Requests Stop once; success is the matching `last_transition{source: lab, request_id}` or a terminal run event, not merely a stopped snapshot. | +| `cfr_batch(console, lines)` | Lease, program stopped. Debug-only allowlisted command batch; marks the save fact stale. Raw writes to `Lab/inbox` are not exposed. | + +`switch_game` is deliberately absent from the first design. Current common routines can +resume or start the selected HOME tile, but cannot reliably select an arbitrary title by +identity. Add a separate proposal only after the app exposes a tested title-selection seam. +Until then, `ensure_game` turns the dangerous case into an early refusal and leaves recovery +to the operator. + +The `shot` freshness rule closes both old races: its token is newer than call start, and the +daemon responds only after the atomically published PNG can be opened. + +The snapshot is a direct video-session seam: it reads frames from the panel's `VideoSession` +frame listener, never constructs `LabConsole`, and needs no owner token, program state or +controller. It must not wait behind the sequential `Lab/inbox` worker, so a 5-second `shot` +is never blocked by a 30-second controller wait or a long CFR batch. As built in #481 +(629bfde88), the `snapshot LABEL [TIMEOUT_MS]` channel command already has the first half: it +waits on that frame listener for the first frame newer than the line (100-60000 ms, default +5000), saves `_