From 315f4f825af54c9477ce0d304b90a4b392c26c6b Mon Sep 17 00:00:00 2001 From: dayfine Date: Sun, 1 Nov 2026 12:48:57 -0800 Subject: [PATCH 1/7] Design: an MCP server for agents to run and develop programs over the lab channel Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01AKYzhrMnPK61XHWEch4a6E --- docs/design/Agent-Integration.md | 264 +++++++++++++++++++++++++++++++ 1 file changed, 264 insertions(+) create mode 100644 docs/design/Agent-Integration.md diff --git a/docs/design/Agent-Integration.md b/docs/design/Agent-Integration.md new file mode 100644 index 0000000000..d1b030c96b --- /dev/null +++ b/docs/design/Agent-Integration.md @@ -0,0 +1,264 @@ +# Agent integration: an MCP server for SerialPrograms + +Status: proposal (2026-11-01). It builds on the lab channel (fork #107, +`SerialPrograms/Source/CommonFramework/Windows/Lab.md`) and on run records (fork #116). +Nothing here is implemented except what #107 already has. + +## Requirement + +Coding agents (Claude Code, Codex) already develop and test programs here. They do it +through shell scripts around the lab channel: `labctl.sh`, `cfrun.sh` and `lablock.sh` +(see `devtools/lab/`). That works, but every agent has to rediscover the same things: +- the file protocol; +- which program identifiers and option keys exist; +- how to tell a battle screen from the overworld; +- how to save evidence; +- the safety rules, such as never pressing keys while a program runs, never touching the + user's console, and never starting a new game over the only save. + +Agents end up filtering logs by date with `grep`, copying screenshots into notes by hand, +and hitting the same traps again (a missed `owner` line, a stale build, a batch with a +typo). The goal is one typed, documented and safe interface that any agent host can use, +to run, observe and develop any program on a real console, without reading the app's +source first. + +Users of this interface: +- **An agent developing a program.** The loop is edit → build → test offline → run on the + console → read the evidence → fix. +- **An agent running or monitoring a program for the user.** Start a farm, watch it, and + report when it stops. +- **The user.** They see the same evidence the agent saw: run records, screenshots, route + pictures. + +## Scope + +In scope: +- **An MCP server.** A separate process, speaking the Model Context Protocol over stdio, that + wraps the lab channel of one or more running SerialPrograms apps. +- **Read access** to: + - program metadata: the program list, option schemas, detectors; + - run records (#116) and their screenshots; + - logs, by run rather than by date. +- **Safety rules enforced in the server,** not left to prompt text: the console allowlist, + locks, owner tokens, no presses while running, and confirmation for save-destroying + options. +- **Thin development helpers:** build and test status, deploying a build to a lab slot, + launching an app with a profile. +- **Small app-side additions** that make the interface typed: an option schema dump, a + detector list with descriptions, and structured run events. Each is its own PR. + +Not in scope: +- **Remote or networked control.** The server runs on the same machine as the apps and + talks through files. A socket transport is possible later, but files keep the app + unchanged and auditable. +- **A general "do anything" tool.** There is no raw shell, and no arbitrary file write + outside the lab folders. +- **Upstream publishing.** No PR or GitHub actions; agents keep their own `gh` rules. +- **Changing how programs work.** Programs never depend on the server. With no lab folder, + the app behaves exactly as today. +- **Video streaming.** Screenshots and the existing `history` clips only. + +## High-level design + +``` + agent host (Claude Code, Codex, ...) + │ MCP over stdio + ▼ + pa-lab MCP server (devtools/mcp/, Python, standard library + the mcp package) + ├─ consoles.json which app profiles and consoles it may touch + ├─ lock + owner lablock.sh semantics, owner token per session + ├─ tools / resources (below) + └─ evidence index run records, outbox pictures, logs by run + │ files: Lab/inbox → Lab/outbox, Lab/state, Lab/runs// + ▼ + SerialPrograms app (one per console, --profile NAME) + └─ lab channel (#107) + run recorder (#116) +``` + +**Why a separate process over files:** +- The lab channel already gives one request file per command batch, journals and a state + file, with app-side checks (refusing presses while running, the owner token). +- The server adds typing, discovery and policy without growing the app. +- A crash on either side leaves plain files behind to inspect. + +### Example user journey: develop a fix and verify it + +The user asks: "FRLG AutoStory fails at the Viridian Forest gate; fix it." + +1. **Look at the evidence.** The agent calls `runs.list(program="PokemonFRLG:AutoStory", + status="failed")`, then `runs.read(id)`. It gets the events (checkpoint 00.08, step + `forest_gate`, the error) and the failure screenshot, and the route picture renders + (#125). It sees that the player was a tile off after a battle. +2. **Change code and test offline.** It edits code, then calls `dev.build(target="lab-frlg")` + and `dev.test(filter="PokemonFRLG")`. Both run the existing build and test commands and + return structured results. +3. **Deploy.** It calls `dev.deploy(build="lab-frlg", slot="B")`, which copies the build and + checks that the binary's mtime is newer than the source. Then + `app.launch(console="B", profile="lab-B")`. +4. **Run.** It calls `console.lock("B")`, then `program.select("PokemonFRLG:AutoStory")` and + `program.set_options({...})`. The server checks the option schema. It refuses + `NEW_GAME_FROM_TITLE=true` without `confirm="overwrite save"`, unless the console's + policy allows it. Then `program.start()`. +5. **Watch.** It calls `program.wait(until="stopped", timeout_s=1800, notify_every_s=300)`. + Progress comes back as run events (`checkpoint 00.08 start`, `battle`, `leg`), not as + raw log lines. +6. **Report.** It calls `runs.read(latest)` and gives the user the result with the record's + pictures. Then it calls `console.release("B")`. + +### Example user journey: monitor a farm for the user + +The user asks: "Start the BBQ farmer on console B and tell me if it stops." + +The agent calls: +1. `console.lock("B")`, then `program.select("PokemonSV:BBQSoloFarmer")`; +2. `program.set_options(preset="user-default")`, then `program.start()`; +3. `program.wait(until="stopped")`. + +When the program stops, the agent returns `runs.read(latest).summary` and the last +screenshot. The agent never presses a button while the farmer runs: the server refuses it, +and so does the app. + +### Example commands (MCP tool calls) + +```jsonc +{"tool": "console.state", "args": {"console": "B"}} +→ {"state": "STOPPED", "program": "NintendoSwitch:CommandFileRunner", "owner": "", "app": "frlg-m28", "binary_mtime": "..."} + +{"tool": "console.batch", "args": {"console": "B", "lines": ["dpad UP 2600 0", "wait 2500", "press B", "wait_for frlg_battle_menu 3500", "shot end"]}} +→ {"ok": true, "lines": [{"n": 1, "ok": true, "ms": [0, 2602]}, ..., {"n": 4, "out": "timeout"}], "images": ["outbox/req_end.png"]} + +{"tool": "program.options", "args": {"program": "PokemonFRLG:AutoStory"}} +→ {"START_CHECKPOINT": {"type": "enum", "values": ["00.01 ...", "00.09 Pewter Gym ..."]}, "NEW_GAME_FROM_TITLE": {"type": "bool", "danger": "overwrites the save"}, ...} +``` + +## Tools and resources + +Tools (verbs) are grouped by what they touch. Each returns structured JSON, and images +as MCP image content. + +| Group | Tools | Notes | +|---|---|---| +| console | `list`, `state`, `lock`, `release`, `shot`, `detect`, `wait_for`, `batch` | `batch` takes command lines from the lab channel's list (`help`), validated before sending. Presses are refused while a program runs, as the app does. | +| program | `list`, `select`, `options`, `set_options`, `start`, `stop`, `wait` | `options` returns the schema. `set_options` checks values against it and the console's policy. `wait` blocks until a state is reached or a timeout passes, and returns new run events. | +| runs | `list`, `read`, `events`, `pictures` | Reads `Lab/runs//` (#116). `pictures` calls the #125 renderer on the run's route events. | +| logs | `tail`, `search` | By run id or time window. Never returns the whole log. | +| dev | `build`, `test`, `deploy`, `launch`, `quit` | Wrap the existing commands (`ninja -j4`, the unit test runner, `bdsp_deploy.sh`, `stage_launch.sh`). One build at a time. `quit` only closes an app this session launched. | + +Resources (nouns the host can list and read): +- `pa://programs`: every program identifier, with title and description. +- `pa://programs//options`: the option schema. +- `pa://detectors`: every detector name, its game, and what it means. +- `pa://consoles`: the allowlist and each console's policy. +- `pa://runs/`: one run record. +- `pa://docs/lab`: Lab.md. + +## Safety model + +Rules live in `devtools/mcp/consoles.json`, which the user owns. Agents can read it but +not change it through the server. + +```json +{ + "A": {"profile": null, "access": "none", "why": "the user's console"}, + "B": {"profile": "lab-B", "access": "full", + "confirm": ["NEW_GAME_FROM_TITLE", "NEW_GAME_INTRO", "close_game"]} +} +``` + +Enforcement: +- **Allowlist.** Consoles with `access: none` are invisible. `read` allows screenshots and + state only. +- **Lock and owner.** Every write takes the console lock (`lablock.sh` semantics) and puts + the session's owner token on each request. Another session's lock is reported with its + holder, never broken. +- **Never press while running.** The server checks the state file before each batch, and + the app enforces it as well. +- **Confirmations.** Listed options and commands need an explicit `confirm` argument + naming what is lost, e.g. "overwrite save". A host can surface that to the user. +- **Own processes only.** `dev.quit` refuses an app the session did not launch, and an app + that is running a program the session did not start. +- **Stale builds.** `program.start` refuses when the deployed binary is older than the + last `dev.build` of that slot, unless `allow_stale=true`. +- **Audit.** Every tool call and its result summary is appended to + `Lab/mcp-audit.jsonl` in the profile. + +## Major new components + +1. **`devtools/mcp/pa_lab_server.py`: the MCP server.** Python 3.9+, the standard library + plus the `mcp` package. + - It owns the tool table, the policy and the request writer: write beside `inbox`, then + move into it, as `labctl.sh` does. It also parses journals into JSON. + - About 600 lines, with unit tests against a fake lab folder: a directory plus a tiny + simulated app loop that answers requests from fixtures. +2. **Option schema export (app side, small).** A lab command `schema` that writes the + selected program's options as JSON: key, type, enum values, default, and a "danger" + note the program sets. + - It reuses the existing config option tree, which already serialises to JSON for the + settings file. Only types and enum labels are new. + - About 150 lines plus tests. +3. **Detector registry descriptions (app side, small).** The lab's detector list (#107) + gets a one-line description per detector, and `help detectors` prints them. +4. **Run events (#116).** Structured events (`checkpoint`, `step`, `route_plan`, `leg`, + `battle`, `failure`, `screenshot`), written with `run_recorder().event()` into + `events.jsonl`. + - Programs opt in. AutoStory first, since its story runner already logs these as text. + - The server reads these events; it never parses free-text logs. +5. **Program skills and docs.** Per-game notes are exposed as resources, not code: + - which detectors to use; + - safe states to stop in; + - known traps, such as "the battle menu appears only after the appearance text is + dismissed" (measured 2026-11-01). + - They live next to each program as `.agent.md`, and the server lists them. + +## Developing new programs with it + +The same loop works for a program that doesn't exist yet: + +1. **Explore by hand.** Use `console.batch` with presses, `shot` and `detect` to learn the + screens. Save the pictures as detector test data. +2. **Write detectors and their screenshot tests.** Run them offline with `dev.test`. +3. **Write the program with an injectable game-steps interface**, as FRLG and BDSP + AutoStory do with `StoryStepExecutor`. A simulation test can then run it against a fake + game without a console. +4. **Run on the console** through `program.*`, and read the run record. + +The server adds nothing program-specific. A program gets better agent support by: +- emitting run events; +- declaring dangerous options; +- shipping an `.agent.md`. + +## Milestones and work plan + +| PR | Content | Size | Tests | +|---|---|---|---| +| 1 | This design doc | doc | review | +| 2 | `devtools/mcp`: the server with the console, program and logs tools over the existing lab channel, and the policy file | ~600 + tests | fake-lab unit tests; one manual session on console B | +| 3 | App: `schema` lab command and detector descriptions | ~200 | unit tests on two programs' option trees | +| 4 | Run events in the run recorder (#116 slice) and the `runs` tools | ~300 | recorder unit tests; an AutoStory run produces events | +| 5 | AutoStory emits checkpoint, step, leg and failure events; the `runs.pictures` tool calls #125 | ~250 | the simulation tests assert the events; one console run | +| 6 | The `dev` tools (build, test, deploy, launch, quit) with the stale-build check | ~250 | unit tests with fake commands | +| 7 | `.agent.md` for FRLG AutoStory, BDSP AutoStory and the BBQ farmer | docs | — | + +Order: PR 2 is useful on its own, because today's batch and labctl flows become typed. PRs +3 and 4 make it discoverable and observable. PRs 5–7 make agents faster on the programs we +work on most. + +Testing story: +- **Every tool is unit-tested** against a fake lab folder: a fixture app that consumes + requests and writes journals. +- **The app-side commands** have C++ unit tests in the existing runner. +- **One scripted console session per milestone** checks the end-to-end path on console B, + recorded as a run record. + +## Open questions + +- **Language:** Python (fast to write; agents already have it) or a C++ tool built with the + app (one toolchain, no `mcp` dependency)? The proposal is Python, because the server never + links app code. +- **Where `consoles.json` lives:** in the repo as an example, with the real one in the + user's profile folder? The proposal is the profile folder, so each machine keeps its own + rules. +- **Upstream or fork-only?** The lab channel and the server are developer tooling. Upstream + may want them behind a developer-mode flag. +- **A socket transport,** so an agent can watch several consoles without polling: wait until + the file transport limits something real. From bd2e96ee6780ab28bacd5b705ba6032fc0a002fd Mon Sep 17 00:00:00 2001 From: dayfine Date: Sun, 1 Nov 2026 12:54:20 -0800 Subject: [PATCH 2/7] Design: agent integration as one lab CLI; MCP an optional thin wrapper last Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01AKYzhrMnPK61XHWEch4a6E --- docs/design/Agent-Integration.md | 364 ++++++++++++++++--------------- 1 file changed, 189 insertions(+), 175 deletions(-) diff --git a/docs/design/Agent-Integration.md b/docs/design/Agent-Integration.md index d1b030c96b..0c0dc305aa 100644 --- a/docs/design/Agent-Integration.md +++ b/docs/design/Agent-Integration.md @@ -1,14 +1,17 @@ -# Agent integration: an MCP server for SerialPrograms +# Agent integration: one lab CLI, MCP later -Status: proposal (2026-11-01). It builds on the lab channel (fork #107, -`SerialPrograms/Source/CommonFramework/Windows/Lab.md`) and on run records (fork #116). -Nothing here is implemented except what #107 already has. +Status: proposal (2026-11-01, revised the same day: CLI first, MCP an optional last +step). It builds on two things: +- the lab channel: fork #107, `SerialPrograms/Source/CommonFramework/Windows/Lab.md`; +- run records: fork #116, `docs/design/Lab-RunRecords.md` on `prototype/program-session-agent`. + +Nothing here is implemented except what those two already have. ## Requirement Coding agents (Claude Code, Codex) already develop and test programs here. They do it -through shell scripts around the lab channel: `labctl.sh`, `cfrun.sh` and `lablock.sh` -(see `devtools/lab/`). That works, but every agent has to rediscover the same things: +through shell scripts around the lab channel: `labctl.sh`, `cfrun.sh` and `lablock.sh` in +`devtools/lab/`. That works, but every agent has to rediscover the same things: - the file protocol; - which program identifiers and option keys exist; - how to tell a battle screen from the overworld; @@ -18,247 +21,258 @@ through shell scripts around the lab channel: `labctl.sh`, `cfrun.sh` and `lablo Agents end up filtering logs by date with `grep`, copying screenshots into notes by hand, and hitting the same traps again (a missed `owner` line, a stale build, a batch with a -typo). The goal is one typed, documented and safe interface that any agent host can use, -to run, observe and develop any program on a real console, without reading the app's -source first. +typo). + +The goal is one documented, safe interface for running, observing and developing any +program on a real console. It returns JSON, and every agent host uses it the same way. Users of this interface: - **An agent developing a program.** The loop is edit → build → test offline → run on the console → read the evidence → fix. - **An agent running or monitoring a program for the user.** Start a farm, watch it, and report when it stops. -- **The user.** They see the same evidence the agent saw: run records, screenshots, route - pictures. +- **The user.** They run the same commands, and see the evidence the agent saw: run + records, screenshots, route pictures. ## Scope In scope: -- **An MCP server.** A separate process, speaking the Model Context Protocol over stdio, that - wraps the lab channel of one or more running SerialPrograms apps. -- **Read access** to: - - program metadata: the program list, option schemas, detectors; - - run records (#116) and their screenshots; - - logs, by run rather than by date. -- **Safety rules enforced in the server,** not left to prompt text: the console allowlist, - locks, owner tokens, no presses while running, and confirmation for save-destroying - options. -- **Thin development helpers:** build and test status, deploying a build to a lab slot, - launching an app with a profile. -- **Small app-side additions** that make the interface typed: an option schema dump, a - detector list with descriptions, and structured run events. Each is its own PR. +- **One lab CLI, `devtools/lab/lab.py`.** Python 3.9+, standard library only. JSON on + stdout, a non-zero exit code on any refusal or failure. + - It holds every guard. + - `labctl.sh` and `cfrun.sh` become thin callers of it, kept for existing notes and skills. + - The agent skills (`console-iteration`, `bdsp-autostory-test`, ...) document `lab.py` + commands instead of raw request files. +- **Small app-side additions that make the CLI typed,** each its own PR: an option schema + command, detector descriptions, structured run events (#116's schema), and per-program + agent notes. +- **Optionally, last:** a thin MCP server over the CLI (see "MCP, later"). Not in scope: -- **Remote or networked control.** The server runs on the same machine as the apps and - talks through files. A socket transport is possible later, but files keep the app - unchanged and auditable. -- **A general "do anything" tool.** There is no raw shell, and no arbitrary file write - outside the lab folders. -- **Upstream publishing.** No PR or GitHub actions; agents keep their own `gh` rules. -- **Changing how programs work.** Programs never depend on the server. With no lab folder, +- **Remote or networked control.** The CLI runs on the machine with the apps and talks + through the lab folders. +- **A general "do anything" tool.** There is no raw shell passthrough, and no file writes + outside the lab folders and the audit log. +- **GitHub or upstream actions.** Agents keep their own `gh` rules. +- **Changing how programs work.** Programs never depend on the CLI. With no `Lab/` folder, the app behaves exactly as today. - **Video streaming.** Screenshots and the existing `history` clips only. ## High-level design ``` - agent host (Claude Code, Codex, ...) - │ MCP over stdio - ▼ - pa-lab MCP server (devtools/mcp/, Python, standard library + the mcp package) - ├─ consoles.json which app profiles and consoles it may touch - ├─ lock + owner lablock.sh semantics, owner token per session - ├─ tools / resources (below) - └─ evidence index run records, outbox pictures, logs by run + agent host (Claude Code, Codex, a shell) optional, later: MCP client + │ runs │ + ▼ ▼ + devtools/lab/lab.py ◄──────────────────────── pa-lab MCP server (thin: one tool = one + ├─ console | program | runs | logs | dev lab.py call, no logic of its own) + ├─ guards: consoles.json, lock + owner, confirms, + │ own-process quit, build freshness, audit log + └─ JSON out │ files: Lab/inbox → Lab/outbox, Lab/state, Lab/runs// ▼ SerialPrograms app (one per console, --profile NAME) └─ lab channel (#107) + run recorder (#116) ``` -**Why a separate process over files:** +**Why a CLI over files:** - The lab channel already gives one request file per command batch, journals and a state file, with app-side checks (refusing presses while running, the owner token). -- The server adds typing, discovery and policy without growing the app. -- A crash on either side leaves plain files behind to inspect. +- The CLI adds typing, discovery and policy without growing the app. +- Any agent with a shell can use it, and so can the user, by hand. A crash on either side + leaves plain files behind to inspect. -### Example user journey: develop a fix and verify it - -The user asks: "FRLG AutoStory fails at the Viridian Forest gate; fix it." +### Commands -1. **Look at the evidence.** The agent calls `runs.list(program="PokemonFRLG:AutoStory", - status="failed")`, then `runs.read(id)`. It gets the events (checkpoint 00.08, step - `forest_gate`, the error) and the failure screenshot, and the route picture renders - (#125). It sees that the player was a tile off after a battle. -2. **Change code and test offline.** It edits code, then calls `dev.build(target="lab-frlg")` - and `dev.test(filter="PokemonFRLG")`. Both run the existing build and test commands and - return structured results. -3. **Deploy.** It calls `dev.deploy(build="lab-frlg", slot="B")`, which copies the build and - checks that the binary's mtime is newer than the source. Then - `app.launch(console="B", profile="lab-B")`. -4. **Run.** It calls `console.lock("B")`, then `program.select("PokemonFRLG:AutoStory")` and - `program.set_options({...})`. The server checks the option schema. It refuses - `NEW_GAME_FROM_TITLE=true` without `confirm="overwrite save"`, unless the console's - policy allows it. Then `program.start()`. -5. **Watch.** It calls `program.wait(until="stopped", timeout_s=1800, notify_every_s=300)`. - Progress comes back as run events (`checkpoint 00.08 start`, `battle`, `leg`), not as - raw log lines. -6. **Report.** It calls `runs.read(latest)` and gives the user the result with the record's - pictures. Then it calls `console.release("B")`. +``` +lab.py console list|state|lock|release|shot|detect|wait-for|batch --console B +lab.py program list|select|options|set|start|stop|wait --console B +lab.py runs list|show|events|report|pictures [RUN] +lab.py logs tail|search --run RUN | --since T +lab.py dev build|test|deploy|launch|quit --slot lab-frlg +``` -### Example user journey: monitor a farm for the user +Examples: -The user asks: "Start the BBQ farmer on console B and tell me if it stops." +```sh +$ lab.py console state --console B +{"console": "B", "state": "STOPPED", "program": "NintendoSwitch:CommandFileRunner", + "owner": "", "build": "frlg-m28", "binary_mtime": "2026-11-01T11:39"} -The agent calls: -1. `console.lock("B")`, then `program.select("PokemonSV:BBQSoloFarmer")`; -2. `program.set_options(preset="user-default")`, then `program.start()`; -3. `program.wait(until="stopped")`. +$ lab.py console batch --console B -- "dpad UP 2600 0" "wait 2500" "press B" \ + "wait_for frlg_battle_menu 3500" "shot end" +{"ok": true, "lines": [{"n": 1, "ok": true, "ms": [0, 2602]}, ..., + {"n": 4, "ok": true, "out": "timeout"}], "images": [".../outbox/req_end.png"]} -When the program stops, the agent returns `runs.read(latest).summary` and the last -screenshot. The agent never presses a button while the farmer runs: the server refuses it, -and so does the app. +$ lab.py program set --console B START_CHECKPOINT='"8"' NEW_GAME_FROM_TITLE=true +{"ok": false, "refused": "NEW_GAME_FROM_TITLE overwrites the save on console B; pass --confirm 'overwrite save'"} -### Example commands (MCP tool calls) +$ lab.py runs report latest --program PokemonFRLG:AutoStory > notes/run.md +``` -```jsonc -{"tool": "console.state", "args": {"console": "B"}} -→ {"state": "STOPPED", "program": "NintendoSwitch:CommandFileRunner", "owner": "", "app": "frlg-m28", "binary_mtime": "..."} +### Example user journey: develop a fix and verify it -{"tool": "console.batch", "args": {"console": "B", "lines": ["dpad UP 2600 0", "wait 2500", "press B", "wait_for frlg_battle_menu 3500", "shot end"]}} -→ {"ok": true, "lines": [{"n": 1, "ok": true, "ms": [0, 2602]}, ..., {"n": 4, "out": "timeout"}], "images": ["outbox/req_end.png"]} +The user asks: "FRLG AutoStory fails at the Viridian Forest gate; fix it." -{"tool": "program.options", "args": {"program": "PokemonFRLG:AutoStory"}} -→ {"START_CHECKPOINT": {"type": "enum", "values": ["00.01 ...", "00.09 Pewter Gym ..."]}, "NEW_GAME_FROM_TITLE": {"type": "bool", "danger": "overwrites the save"}, ...} -``` +1. **Look at the evidence.** The agent runs `lab.py runs list --program + PokemonFRLG:AutoStory --status failed`, then `lab.py runs show RUN`. It gets the + checkpoint 00.08, the step `forest_gate`, the error and the failure frame. `lab.py runs + pictures RUN` draws the walk (#125), and shows the player a tile off after a battle. +2. **Change code and test offline.** It edits code, then runs `lab.py dev build --slot + lab-frlg` and `lab.py dev test --filter PokemonFRLG`. Both wrap the existing ninja and + unit-test commands. +3. **Deploy.** `lab.py dev deploy --slot lab-frlg --console B` copies the build and records + its mtime. Then `lab.py dev launch --console B`. +4. **Run.** It runs `lab.py console lock --console B`, `lab.py program select + PokemonFRLG:AutoStory` and `lab.py program set START_CHECKPOINT=...`. The options are + checked against the program's schema. Then `lab.py program start`. A start is refused if + the deployed binary is older than the last build of that slot. +5. **Watch.** `lab.py program wait --until stopped --timeout 1800` prints new run events as + they arrive (`checkpoint 00.08 start`, `battle`, `leg`), not raw log lines. +6. **Report.** `lab.py runs report latest` writes Markdown with the outcome, build, options, + checkpoint table and failure frames. The agent gives it to the user, then runs `lab.py + console release`. -## Tools and resources +### Example user journey: monitor a farm for the user -Tools (verbs) are grouped by what they touch. Each returns structured JSON, and images -as MCP image content. +The user asks: "Start the BBQ farmer on console B and tell me if it stops." -| Group | Tools | Notes | -|---|---|---| -| console | `list`, `state`, `lock`, `release`, `shot`, `detect`, `wait_for`, `batch` | `batch` takes command lines from the lab channel's list (`help`), validated before sending. Presses are refused while a program runs, as the app does. | -| program | `list`, `select`, `options`, `set_options`, `start`, `stop`, `wait` | `options` returns the schema. `set_options` checks values against it and the console's policy. `wait` blocks until a state is reached or a timeout passes, and returns new run events. | -| runs | `list`, `read`, `events`, `pictures` | Reads `Lab/runs//` (#116). `pictures` calls the #125 renderer on the run's route events. | -| logs | `tail`, `search` | By run id or time window. Never returns the whole log. | -| dev | `build`, `test`, `deploy`, `launch`, `quit` | Wrap the existing commands (`ninja -j4`, the unit test runner, `bdsp_deploy.sh`, `stage_launch.sh`). One build at a time. `quit` only closes an app this session launched. | +The agent runs: +1. `lab.py console lock`, then `lab.py program select PokemonSV:BBQSoloFarmer`; +2. `lab.py program start`; +3. `lab.py program wait --until stopped`. -Resources (nouns the host can list and read): -- `pa://programs`: every program identifier, with title and description. -- `pa://programs//options`: the option schema. -- `pa://detectors`: every detector name, its game, and what it means. -- `pa://consoles`: the allowlist and each console's policy. -- `pa://runs/`: one run record. -- `pa://docs/lab`: Lab.md. +When the program stops, it sends the user `lab.py runs show latest` and the last frame. +Presses are refused while the farmer runs, by the CLI and again by the app. -## Safety model +## Guards (all in lab.py) -Rules live in `devtools/mcp/consoles.json`, which the user owns. Agents can read it but -not change it through the server. +Rules live in `consoles.json` in the user's profile folder; an example sits in the repo. +The CLI reads it and has no command that changes it. ```json { "A": {"profile": null, "access": "none", "why": "the user's console"}, "B": {"profile": "lab-B", "access": "full", - "confirm": ["NEW_GAME_FROM_TITLE", "NEW_GAME_INTRO", "close_game"]} + "confirm": {"NEW_GAME_FROM_TITLE": "overwrite save", "NEW_GAME_INTRO": "overwrite save", + "close_game": "close the game"}} } ``` Enforcement: -- **Allowlist.** Consoles with `access: none` are invisible. `read` allows screenshots and - state only. -- **Lock and owner.** Every write takes the console lock (`lablock.sh` semantics) and puts - the session's owner token on each request. Another session's lock is reported with its - holder, never broken. -- **Never press while running.** The server checks the state file before each batch, and - the app enforces it as well. -- **Confirmations.** Listed options and commands need an explicit `confirm` argument - naming what is lost, e.g. "overwrite save". A host can surface that to the user. -- **Own processes only.** `dev.quit` refuses an app the session did not launch, and an app - that is running a program the session did not start. -- **Stale builds.** `program.start` refuses when the deployed binary is older than the - last `dev.build` of that slot, unless `allow_stale=true`. -- **Audit.** Every tool call and its result summary is appended to - `Lab/mcp-audit.jsonl` in the profile. +- **Allowlist.** Consoles with `access: none` are not listed and every command on them is + refused. `read` allows `state`, `shot`, `detect` and `wait-for` only. +- **Lock and owner.** Every write takes the console lock (the `lablock.sh` semantics, moved + into the CLI) and puts the session's owner token on each request. Another session's + lock is reported with its holder, never broken. +- **Never press while running.** Checked from `Lab/state` before every batch; the app + enforces it again. +- **Confirmations.** Listed options and commands need `--confirm ''` naming what is + lost. +- **Own processes only.** `dev quit` refuses an app the CLI didn't launch in this session + (pid file), or an app running a program it didn't start. +- **Build freshness.** `program start` refuses a deployed binary older than the slot's last + `dev build`, unless `--allow-stale`. +- **Audit.** Every command, its arguments and its result summary are appended to + `Lab/audit.jsonl` in the profile. ## Major new components -1. **`devtools/mcp/pa_lab_server.py`: the MCP server.** Python 3.9+, the standard library - plus the `mcp` package. - - It owns the tool table, the policy and the request writer: write beside `inbox`, then - move into it, as `labctl.sh` does. It also parses journals into JSON. - - About 600 lines, with unit tests against a fake lab folder: a directory plus a tiny - simulated app loop that answers requests from fixtures. -2. **Option schema export (app side, small).** A lab command `schema` that writes the - selected program's options as JSON: key, type, enum values, default, and a "danger" - note the program sets. - - It reuses the existing config option tree, which already serialises to JSON for the - settings file. Only types and enum labels are new. - - About 150 lines plus tests. -3. **Detector registry descriptions (app side, small).** The lab's detector list (#107) - gets a one-line description per detector, and `help detectors` prints them. -4. **Run events (#116).** Structured events (`checkpoint`, `step`, `route_plan`, `leg`, - `battle`, `failure`, `screenshot`), written with `run_recorder().event()` into - `events.jsonl`. - - Programs opt in. AutoStory first, since its story runner already logs these as text. - - The server reads these events; it never parses free-text logs. -5. **Program skills and docs.** Per-game notes are exposed as resources, not code: - - which detectors to use; - - safe states to stop in; - - known traps, such as "the battle menu appears only after the appearance text is - dismissed" (measured 2026-11-01). - - They live next to each program as `.agent.md`, and the server lists them. +1. **`devtools/lab/lab.py`: the CLI.** Five subcommand modules, a request writer (write + beside `inbox`, then move into it), a journal parser, the guards and the audit log. + - About 700 lines. + - Unit tests against a fake lab folder: a fixture "app" loop that consumes requests and + writes journals and state. + - `runs` is #116's proposed `runs.py` (list, show, report), folded in as the `runs` + subcommand rather than a second script. +2. **Option schema (app side).** A lab command `schema` writes the selected program's + options as JSON: key, type, enum labels, default, and an optional "danger" note the + program sets on an option. + - It reuses the config option tree that already writes the settings file. About 150 + lines plus tests. +3. **Detector descriptions (app side).** Each detector in the lab's list (#107) gets a + one-line description; `help detectors` prints them. +4. **Run events: one format, defined once, in #116.** `events.jsonl` objects always carry + `time` and `kind`; the rest is optional (`checkpoint`, `step`, `attempt`, `text`, + `pose`, `frame`, `output`). + - Kinds: `error`, `probe`, `checkpoint`, `step`, `attempt`, `route_plan`, `leg`, + `battle`, `screenshot`. `pose` is the string `" (x, y) facing D[, certainty]"`. + - This design adds no kinds. Programs send events through `run_recorder().event()`, and + story runners and walkers through the `SegmentHooks.event` callback that #116 proposes. + - The CLI, #125's route renderer, and any later MCP wrapper read only this format. +5. **Per-program agent notes.** `.agent.md` next to a program, covering: + - which detectors to use and the safe states to stop in; + - known traps, such as "the battle menu shows only after the appearance text is + dismissed", measured on console B on 2026-11-01; + - the program's dangerous options. + + `lab.py program notes ID` prints the file, and the skills point to it. ## Developing new programs with it The same loop works for a program that doesn't exist yet: -1. **Explore by hand.** Use `console.batch` with presses, `shot` and `detect` to learn the - screens. Save the pictures as detector test data. -2. **Write detectors and their screenshot tests.** Run them offline with `dev.test`. -3. **Write the program with an injectable game-steps interface**, as FRLG and BDSP - AutoStory do with `StoryStepExecutor`. A simulation test can then run it against a fake - game without a console. -4. **Run on the console** through `program.*`, and read the run record. +1. **Explore by hand.** Use `lab.py console batch` with presses, `shot` and `detect` to + learn the screens. Saved frames become detector test data. +2. **Write detectors and their screenshot tests.** Run them offline with `lab.py dev test`. +3. **Write the program with an injectable game-steps interface,** as FRLG and BDSP + AutoStory do with `StoryStepExecutor`, so a simulation test can run it against a fake + game. +4. **Run it on the console** with `lab.py program`, and read the run record with `lab.py + runs`. -The server adds nothing program-specific. A program gets better agent support by: +A program gets better agent support by: - emitting run events; -- declaring dangerous options; +- marking dangerous options in its schema; - shipping an `.agent.md`. +## MCP, later + +A shell agent can always call `lab.py`, or the raw request files, directly, so an MCP +server adds no enforcement. The guards only hold because they live in the one CLI every +path goes through, and in the app. + +What MCP would add is **discovery**, meaning tools and resources a client lists without +reading docs, and access for **clients without a shell**. So it comes last, and only when +one of those is needed: a non-shell client we want to support, or upstream asking for it. + +When it comes, it's a thin server: +- each tool runs one `lab.py` command and returns its JSON; +- resources map to `lab.py program list|options`, the detector descriptions and `runs show`; +- no policy, state or protocol code of its own. + ## Milestones and work plan | PR | Content | Size | Tests | |---|---|---|---| | 1 | This design doc | doc | review | -| 2 | `devtools/mcp`: the server with the console, program and logs tools over the existing lab channel, and the policy file | ~600 + tests | fake-lab unit tests; one manual session on console B | -| 3 | App: `schema` lab command and detector descriptions | ~200 | unit tests on two programs' option trees | -| 4 | Run events in the run recorder (#116 slice) and the `runs` tools | ~300 | recorder unit tests; an AutoStory run produces events | -| 5 | AutoStory emits checkpoint, step, leg and failure events; the `runs.pictures` tool calls #125 | ~250 | the simulation tests assert the events; one console run | -| 6 | The `dev` tools (build, test, deploy, launch, quit) with the stale-build check | ~250 | unit tests with fake commands | -| 7 | `.agent.md` for FRLG AutoStory, BDSP AutoStory and the BBQ farmer | docs | — | +| 2 | `lab.py`: console and program subcommands, the guards, `consoles.json`, the audit log. `labctl.sh` and `cfrun.sh` call it. | ~500 + tests | fake-lab unit tests; one scripted session on console B | +| 3 | `lab.py runs` and `logs` on #116's run records (folds in #116's proposed `runs.py`) | ~250 | unit tests on fixture run folders | +| 4 | App: `schema` lab command, danger notes on options, detector descriptions | ~200 | C++ unit tests on two programs' option trees | +| 5 | `lab.py dev`: build, test, deploy, launch and quit, with the freshness check | ~250 | unit tests with fake commands | +| 6 | Skills and `.agent.md` notes for FRLG AutoStory, BDSP AutoStory and the BBQ farmer | docs | a skill-driven console run | +| 7 (optional) | Thin MCP server over `lab.py` | ~150 | each tool's output matches the CLI's | + +The run events themselves (the hooks in story runners and walkers, and #125 reading +`events.jsonl`) are #116's milestones; PRs 3 and 6 depend on them. -Order: PR 2 is useful on its own, because today's batch and labctl flows become typed. PRs -3 and 4 make it discoverable and observable. PRs 5–7 make agents faster on the programs we -work on most. +Order: PR 2 is useful on its own, because today's labctl and batch flows become typed and +guarded. PRs 3–5 make it observable and discoverable. PR 6 makes agents faster on the +programs we work on most. PR 7 waits for a reason. Testing story: -- **Every tool is unit-tested** against a fake lab folder: a fixture app that consumes - requests and writes journals. +- **Every CLI command is unit-tested** against a fake lab folder. - **The app-side commands** have C++ unit tests in the existing runner. -- **One scripted console session per milestone** checks the end-to-end path on console B, - recorded as a run record. +- **One scripted console-B session per milestone** checks the end-to-end path, and leaves a + run record as evidence. ## Open questions -- **Language:** Python (fast to write; agents already have it) or a C++ tool built with the - app (one toolchain, no `mcp` dependency)? The proposal is Python, because the server never - links app code. -- **Where `consoles.json` lives:** in the repo as an example, with the real one in the - user's profile folder? The proposal is the profile folder, so each machine keeps its own - rules. -- **Upstream or fork-only?** The lab channel and the server are developer tooling. Upstream - may want them behind a developer-mode flag. -- **A socket transport,** so an agent can watch several consoles without polling: wait until - the file transport limits something real. +- **`labctl.sh` and `cfrun.sh`:** keep them as thin wrappers (existing notes and skills call + them), or retire them after the skills move to `lab.py`? The proposal is to keep them for + one release, then retire them. +- **Where `consoles.json` lives:** the proposal is the profile folder, so each machine keeps + its own rules, with an example in the repo. +- **Upstream or fork-only?** The lab channel and the CLI are developer tooling. Upstream may + want them behind developer mode, or not at all. +- **Whether and when to add the MCP wrapper:** only on a concrete need, as above. From db53d1935c91848f394f3067b49cbf8287725d83 Mon Sep 17 00:00:00 2001 From: dayfine Date: Fri, 13 Nov 2026 22:58:52 -0800 Subject: [PATCH 3/7] Design: MCP lab server shaped by the week's incidents Map each 2026-11-07..13 incident to a resource, tool or event; add read-only console state/save and run resources, leases and events, validated tools with declared preconditions, enforcement, and a PR plan that retires lab scripts. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_013VUubhHeYSMGSSDXZkKZss --- docs/design/Agent-Integration.md | 450 +++++++++++++------------------ 1 file changed, 191 insertions(+), 259 deletions(-) diff --git a/docs/design/Agent-Integration.md b/docs/design/Agent-Integration.md index 0c0dc305aa..6a0e7aceab 100644 --- a/docs/design/Agent-Integration.md +++ b/docs/design/Agent-Integration.md @@ -1,278 +1,210 @@ -# Agent integration: one lab CLI, MCP later +# Agent integration: an MCP lab server -Status: proposal (2026-11-01, revised the same day: CLI first, MCP an optional last -step). It builds on two things: -- the lab channel: fork #107, `SerialPrograms/Source/CommonFramework/Windows/Lab.md`; -- run records: fork #116, `docs/design/Lab-RunRecords.md` on `prototype/program-session-agent`. +Status: proposal (2026-11-01; rewritten 2026-11-13 from the week's incidents, +`~/BDSPLab/notes/2026-11-13-week-problems.md`). It builds on: +- the lab channel: fork #107, `SerialPrograms/Source/CommonFramework/Windows/Lab.md` + (`Lab/inbox`, `Lab/outbox`, `Lab/state`, the owner token); +- run records: fork #116, `docs/design/Lab-RunRecords.md`; +- the lab scripts on `agents` (`lab/control`, `lab/scripts`) and in `~/BDSPLab` + (`labrun.sh`, `labbuild.sh`, `checkin.sh`, `lablock.sh`, `heavy_gate.sh`, ...). -Nothing here is implemented except what those two already have. +Nothing here is implemented except what those already have. ## Requirement -Coding agents (Claude Code, Codex) already develop and test programs here. They do it -through shell scripts around the lab channel: `labctl.sh`, `cfrun.sh` and `lablock.sh` in -`devtools/lab/`. That works, but every agent has to rediscover the same things: -- the file protocol; -- which program identifiers and option keys exist; -- how to tell a battle screen from the overworld; -- how to save evidence; -- the safety rules, such as never pressing keys while a program runs, never touching the - user's console, and never starting a new game over the only save. - -Agents end up filtering logs by date with `grep`, copying screenshots into notes by hand, -and hitting the same traps again (a missed `owner` line, a stale build, a batch with a -typo). - -The goal is one documented, safe interface for running, observing and developing any -program on a real console. It returns JSON, and every agent host uses it the same way. - -Users of this interface: -- **An agent developing a program.** The loop is edit → build → test offline → run on the - console → read the evidence → fix. -- **An agent running or monitoring a program for the user.** Start a farm, watch it, and - report when it stops. -- **The user.** They run the same commands, and see the evidence the agent saw: run - records, screenshots, route pictures. - -## Scope - -In scope: -- **One lab CLI, `devtools/lab/lab.py`.** Python 3.9+, standard library only. JSON on - stdout, a non-zero exit code on any refusal or failure. - - It holds every guard. - - `labctl.sh` and `cfrun.sh` become thin callers of it, kept for existing notes and skills. - - The agent skills (`console-iteration`, `bdsp-autostory-test`, ...) document `lab.py` - commands instead of raw request files. -- **Small app-side additions that make the CLI typed,** each its own PR: an option schema - command, detector descriptions, structured run events (#116's schema), and per-program - agent notes. -- **Optionally, last:** a thin MCP server over the CLI (see "MCP, later"). - -Not in scope: -- **Remote or networked control.** The CLI runs on the machine with the apps and talks - through the lab folders. -- **A general "do anything" tool.** There is no raw shell passthrough, and no file writes - outside the lab folders and the audit log. -- **GitHub or upstream actions.** Agents keep their own `gh` rules. -- **Changing how programs work.** Programs never depend on the CLI. With no `Lab/` folder, - the app behaves exactly as today. -- **Video streaming.** Screenshots and the existing `history` clips only. - -## High-level design +Agents drive console B through about forty shell scripts around the file channel. Each one +relearns the protocol and the rules, and the rules hold only as long as every agent +remembers them. This week that failed in eight ways (table below): stale frames, a start on +the HOME screen, setup errors found after a 2-10 minute reset, the app run without its +profile, and answers that cost four log greps. + +The goal is one local server that owns console B and the lab's heavy work, answers state +questions from typed resources, and refuses a bad request before it costs a reset. Agents +talk to it over MCP; the user can use the same tools from a thin CLI. + +Not in scope: remote control, console A, GitHub writes (agents keep their `gh` rules), video +streaming, and any change to how programs behave without a `Lab/` folder. + +## Problems this solves + +One row per incident from 2026-11-07..13. "R" = resource, "T" = tool, "E" = event. + +| Incident | Prevented by | +|---|---| +| Reused shot tag returned a stale Safari frame while the player was at the Mansion | T `shot()` returns a new frame with its `seq`; never a reused file | +| PNG read while the app was still writing it | T `shot()` returns the image only once written | +| Selecting a settings panel through the inbox blocked the channel | T `run_program` takes program panels only; no raw `select` | +| Panel switch reset controller and video ("requires video feedback") | R `console://B/state` `controller`, `video`; `run_program` waits for both | +| No-HDMI colour bars went unnoticed until a frame was read | R `state.video.signal`; E `video_lost` | +| Game switches (HOME, close, pick tile) done by hand | T `switch_game(target)` | +| PLZA 03.08 started with the game closed and walked on HOME | precondition "game in front = program's game, known screen" | +| UI stop at 22:25 had to be guessed as the user's | E `stopped{source: ui}`; lease moves to `user` | +| "Hands off B while the user plays" held only by memory | lease: tools refuse while `user` holds it | +| Which save B holds lived in notes (Safari logo vs Mansion door, unsaved Parasect) | R `console://B/save` | +| Max Resets 10 on a long-shot target | schema range plus program precondition | +| Safari needs ≥ 4000 advances | precondition on `run_program` | +| FRLG license check not loaded (Game Settings panel never opened) | precondition: game settings loaded | +| No Sweet Scent user in the last slot | precondition, checked by the program's precheck before any reset | +| Restart on a post-run summary screen | precondition "known screen"; `switch_game` brings it there | +| PLZA LANGUAGE left on ja | precondition: language matches the console's game | +| Dev-only checkpoint hidden without Developer Mode shifted indices; wrong checkpoint started | options by key and checkpoint id, never by index; refuse a hidden row | +| Audio input not allowlisted for lab-B (UserSetupError spam) | server launch checks the lab profile's settings | +| App run without `--profile`, twice | agents never run the app; the server launches it | +| Rules retyped per brief and drifting | rules enforced by the server, not restated in briefs | +| Agents parked waiting on the heavy gate or the console | queued tools return a ticket; E `build_done`, `program_finished` | +| Linter rule changed mid-task | not addressed: `ci_status` names the linter ref it ran with, nothing more | +| One upstream sync = merges through 36 branches (+~40 descendants) | T `stack_plan(pr)` | +| CI red from SystemKeyboardDetector images never pushed to CommandLineTests | T `ci_status` classifies "missing test image" | +| Branches checked out in several worktrees went stale | T `stack_plan` lists each branch's worktrees and how far behind they are | +| "How far off is the RNG?" took four greps | R `runs://latest` carries RNG hits (target, hit, delta) | +| Results spread over notes, outbox, error dirs with spaces | R `runs://`: one record, frame and video at clean paths | +| Sim passes, console fails (FRLG 13.01, 12.03; PLZA walk_to_objective) | partly: `runs://` keeps the walked poses a sim test replays | +| Disk under the 30 GB floor | queued `build_lab` refuses under the floor | +| RAM swap from parallel builds | `build_lab` queue: one build, `-j4`, `nice`, waits while a program runs | + +## Shape ``` - agent host (Claude Code, Codex, a shell) optional, later: MCP client - │ runs │ - ▼ ▼ - devtools/lab/lab.py ◄──────────────────────── pa-lab MCP server (thin: one tool = one - ├─ console | program | runs | logs | dev lab.py call, no logic of its own) - ├─ guards: consoles.json, lock + owner, confirms, - │ own-process quit, build freshness, audit log - └─ JSON out - │ files: Lab/inbox → Lab/outbox, Lab/state, Lab/runs// - ▼ - SerialPrograms app (one per console, --profile NAME) - └─ lab channel (#107) + run recorder (#116) + agent hosts (Claude Code, Codex) user shell (lab CLI) + │ MCP (localhost) │ + ▼ ▼ + pa-lab server (one process, launchd; owns leases, queues, audit log) + ├─ resources: console://B/state, console://B/save, runs:// + ├─ tools: shot, switch_game, run_program, stop_program, build_lab, ci_status, stack_plan + ├─ events: resource updates + next_events() + └─ config: consoles.json (B only; A has no entry) + │ files: Lab/inbox → Lab/outbox, Lab/state, Lab/save.json, Lab/runs// + ▼ + SerialPrograms, launched by the server with --profile lab-B ``` -**Why a CLI over files:** -- The lab channel already gives one request file per command batch, journals and a state - file, with app-side checks (refusing presses while running, the owner token). -- The CLI adds typing, discovery and policy without growing the app. -- Any agent with a shell can use it, and so can the user, by hand. A crash on either side - leaves plain files behind to inspect. +One long-lived process, not one per client: leases, the build queue and the event log must +be shared by every agent. It is Python with the MCP SDK, in a venv like the code-health one, +and lives with the lab tooling on `agents` (`lab/server/`). App-side changes stay on +`lab/tooling`. -### Commands +## Resources (read-only, first PR) -``` -lab.py console list|state|lock|release|shot|detect|wait-for|batch --console B -lab.py program list|select|options|set|start|stop|wait --console B -lab.py runs list|show|events|report|pictures [RUN] -lab.py logs tail|search --run RUN | --since T -lab.py dev build|test|deploy|launch|quit --slot lab-frlg -``` - -Examples: - -```sh -$ lab.py console state --console B -{"console": "B", "state": "STOPPED", "program": "NintendoSwitch:CommandFileRunner", - "owner": "", "build": "frlg-m28", "binary_mtime": "2026-11-01T11:39"} - -$ lab.py console batch --console B -- "dpad UP 2600 0" "wait 2500" "press B" \ - "wait_for frlg_battle_menu 3500" "shot end" -{"ok": true, "lines": [{"n": 1, "ok": true, "ms": [0, 2602]}, ..., - {"n": 4, "ok": true, "out": "timeout"}], "images": [".../outbox/req_end.png"]} - -$ lab.py program set --console B START_CHECKPOINT='"8"' NEW_GAME_FROM_TITLE=true -{"ok": false, "refused": "NEW_GAME_FROM_TITLE overwrites the save on console B; pass --confirm 'overwrite save'"} - -$ lab.py runs report latest --program PokemonFRLG:AutoStory > notes/run.md -``` - -### Example user journey: develop a fix and verify it - -The user asks: "FRLG AutoStory fails at the Viridian Forest gate; fix it." - -1. **Look at the evidence.** The agent runs `lab.py runs list --program - PokemonFRLG:AutoStory --status failed`, then `lab.py runs show RUN`. It gets the - checkpoint 00.08, the step `forest_gate`, the error and the failure frame. `lab.py runs - pictures RUN` draws the walk (#125), and shows the player a tile off after a battle. -2. **Change code and test offline.** It edits code, then runs `lab.py dev build --slot - lab-frlg` and `lab.py dev test --filter PokemonFRLG`. Both wrap the existing ninja and - unit-test commands. -3. **Deploy.** `lab.py dev deploy --slot lab-frlg --console B` copies the build and records - its mtime. Then `lab.py dev launch --console B`. -4. **Run.** It runs `lab.py console lock --console B`, `lab.py program select - PokemonFRLG:AutoStory` and `lab.py program set START_CHECKPOINT=...`. The options are - checked against the program's schema. Then `lab.py program start`. A start is refused if - the deployed binary is older than the last build of that slot. -5. **Watch.** `lab.py program wait --until stopped --timeout 1800` prints new run events as - they arrive (`checkpoint 00.08 start`, `battle`, `leg`), not raw log lines. -6. **Report.** `lab.py runs report latest` writes Markdown with the outcome, build, options, - checkpoint table and failure frames. The agent gives it to the user, then runs `lab.py - console release`. - -### Example user journey: monitor a farm for the user - -The user asks: "Start the BBQ farmer on console B and tell me if it stops." - -The agent runs: -1. `lab.py console lock`, then `lab.py program select PokemonSV:BBQSoloFarmer`; -2. `lab.py program start`; -3. `lab.py program wait --until stopped`. - -When the program stops, it sends the user `lab.py runs show latest` and the last frame. -Presses are refused while the farmer runs, by the CLI and again by the app. - -## Guards (all in lab.py) - -Rules live in `consoles.json` in the user's profile folder; an example sits in the repo. -The CLI reads it and has no command that changes it. +**`console:///state`**, from `Lab/state` plus new app fields: ```json -{ - "A": {"profile": null, "access": "none", "why": "the user's console"}, - "B": {"profile": "lab-B", "access": "full", - "confirm": {"NEW_GAME_FROM_TITLE": "overwrite save", "NEW_GAME_INTRO": "overwrite save", - "close_game": "close the game"}} -} +{"console": "B", "game": "PokemonFRLG", "screen": "overworld", + "panel": "PokemonFRLG:AutoStory", "program_state": "STOPPED", "controller": "ready", + "video": {"signal": true, "last_frame_age_ms": 33, "frame_seq": 918274}, + "lease": {"owner": "agent:frlg-lane", "since": "22:10", "expires": "22:40"}, + "app": {"pid": 4411, "build": "slot-B-frlg/2026-11-13T21:02"}} ``` -Enforcement: -- **Allowlist.** Consoles with `access: none` are not listed and every command on them is - refused. `read` allows `state`, `shot`, `detect` and `wait-for` only. -- **Lock and owner.** Every write takes the console lock (the `lablock.sh` semantics, moved - into the CLI) and puts the session's owner token on each request. Another session's - lock is reported with its holder, never broken. -- **Never press while running.** Checked from `Lab/state` before every batch; the app - enforces it again. -- **Confirmations.** Listed options and commands need `--confirm ''` naming what is - lost. -- **Own processes only.** `dev quit` refuses an app the CLI didn't launch in this session - (pid file), or an app running a program it didn't start. -- **Build freshness.** `program start` refuses a deployed binary older than the slot's last - `dev build`, unless `--allow-stale`. -- **Audit.** Every command, its arguments and its result summary are appended to - `Lab/audit.jsonl` in the profile. - -## Major new components - -1. **`devtools/lab/lab.py`: the CLI.** Five subcommand modules, a request writer (write - beside `inbox`, then move into it), a journal parser, the guards and the audit log. - - About 700 lines. - - Unit tests against a fake lab folder: a fixture "app" loop that consumes requests and - writes journals and state. - - `runs` is #116's proposed `runs.py` (list, show, report), folded in as the `runs` - subcommand rather than a second script. -2. **Option schema (app side).** A lab command `schema` writes the selected program's - options as JSON: key, type, enum labels, default, and an optional "danger" note the - program sets on an option. - - It reuses the config option tree that already writes the settings file. About 150 - lines plus tests. -3. **Detector descriptions (app side).** Each detector in the lab's list (#107) gets a - one-line description; `help detectors` prints them. -4. **Run events: one format, defined once, in #116.** `events.jsonl` objects always carry - `time` and `kind`; the rest is optional (`checkpoint`, `step`, `attempt`, `text`, - `pose`, `frame`, `output`). - - Kinds: `error`, `probe`, `checkpoint`, `step`, `attempt`, `route_plan`, `leg`, - `battle`, `screenshot`. `pose` is the string `" (x, y) facing D[, certainty]"`. - - This design adds no kinds. Programs send events through `run_recorder().event()`, and - story runners and walkers through the `SegmentHooks.event` callback that #116 proposes. - - The CLI, #125's route renderer, and any later MCP wrapper read only this format. -5. **Per-program agent notes.** `.agent.md` next to a program, covering: - - which detectors to use and the safe states to stop in; - - known traps, such as "the battle menu shows only after the appearance text is - dismissed", measured on console B on 2026-11-01; - - the program's dangerous options. - - `lab.py program notes ID` prints the file, and the skills point to it. - -## Developing new programs with it - -The same loop works for a program that doesn't exist yet: - -1. **Explore by hand.** Use `lab.py console batch` with presses, `shot` and `detect` to - learn the screens. Saved frames become detector test data. -2. **Write detectors and their screenshot tests.** Run them offline with `lab.py dev test`. -3. **Write the program with an injectable game-steps interface,** as FRLG and BDSP - AutoStory do with `StoryStepExecutor`, so a simulation test can run it against a fake - game. -4. **Run it on the console** with `lab.py program`, and read the run record with `lab.py - runs`. - -A program gets better agent support by: -- emitting run events; -- marking dangerous options in its schema; -- shipping an `.agent.md`. - -## MCP, later - -A shell agent can always call `lab.py`, or the raw request files, directly, so an MCP -server adds no enforcement. The guards only hold because they live in the one CLI every -path goes through, and in the app. - -What MCP would add is **discovery**, meaning tools and resources a client lists without -reading docs, and access for **clients without a shell**. So it comes last, and only when -one of those is needed: a non-shell client we want to support, or upstream asking for it. - -When it comes, it's a thin server: -- each tool runs one `lab.py` command and returns its JSON; -- resources map to `lab.py program list|options`, the detector descriptions and `runs show`; -- no policy, state or protocol code of its own. - -## Milestones and work plan - -| PR | Content | Size | Tests | +`signal` is false for no frames, black or colour bars. `game` and `screen` come from +detectors (HOME, title, overworld, summary), `unknown` when none match. + +**`console:///save`**: written by programs to `Lab/save.json` on every save: +`game`, `slot`, `checkpoint`, `position` (`" (x, y)"`), `time`, `run`. It turns +`stale: true` when anything that could save ran unrecorded since: a user lease, the CFR, a +program without the hook. + +**`runs://latest`, `runs://`**: #116's run record, summarised: +`verdict` (`passed|failed|stopped`, reason, stop source), checkpoint lines +(`13.01 failed mansion_holes after 2 attempts`), RNG hits (`target 4123, hit 4127, +delta +4`), and the failure frame and video copied to `~/BDSPLab/runs//` (no spaces). + +## Leases and events + +`acquire_console(owner, ttl)` returns a lease; `renew` and `release` take it. Every tool +that presses, starts or stops needs the caller's live lease; reads need none. The lease is +the `lablock.sh` lock and the channel's owner token in one. + +When the app's program state changes without a server request (Start or Stop clicked in the +window), the server gives the lease to `user` and emits `lease_changed{owner: user}`. Agent +tools then refuse. The user lease ends when the user releases it from the CLI, or after +30 minutes with the program stopped and no UI input. + +Events go to an append-only `events.jsonl`, as MCP resource-update notifications, and through +`next_events(since, timeout ≤ 60 s)` for clients without subscriptions: + +| Event | Fields | +|---|---| +| `program_finished`, `program_failed` | run id, verdict, checkpoint | +| `stopped` | source: `ui`, `server`, `program`, `error` | +| `video_lost`, `video_back` | last frame seq | +| `autosave`, `save` | the new `console://B/save` | +| `lease_changed` | old and new owner | +| `build_done`, `tests_done` | ticket, result | +| `ci_done` | PR, SHA, result, missing images | + +Subagents still never wait: they read the ticket and stop. The main session's background +waits read `events.jsonl` instead of polling logs. + +## Tools + +Each tool validates before it acts and returns JSON, or a refusal naming the failed check. + +| Tool | Validates / does | +|---|---| +| `shot(console)` | Lease not needed. Refuses if `last_frame_age_ms` > 2000 or no signal. Returns the image and `frame_seq`, `time`; the file name carries the seq. | +| `switch_game(console, target)` | Lease, program stopped. HOME, close the current game, launch `target` by title (not tile position), wait for its title screen. | +| `run_program(console, panel, options)` | Lease. Panel is a program; options checked against the program's schema (keys, enum labels, ranges); declared preconditions hold; controller and video healthy. Selects, sets, starts; returns the run id. | +| `stop_program(console)` | Lease. Stops; the event records `source: server`. | +| `cfr_batch(console, lines)` | Lease, program stopped. Command File Runner batch for debugging only; marks the save stale. | +| `build_lab(game, ref)`, `run_tests(slot, filter)` | Queued: one at a time, `-j4`, `nice 15`, waits while a program runs, refuses under the disk floor. Returns a ticket. | +| `ci_status(branch \| pr)` | Per workflow: result, failing tests; a `FileException` on a test image that no CommandLineTests ref holds is reported as `missing_image`, not a code failure. | +| `stack_plan(pr)` | Read-only: parents, children, which branches need a merge after a parent moved, in order; each branch's worktrees and how far they lag. Merges stay with pr-maintenance agents. | + +### Preconditions + +A program declares its preconditions as data next to its option schema. The server checks +the ones it can answer from state, save and options; the rest run in the program's own +precheck (as FRLG `RngProgramInstance::run_precheck()` already does) through a `dry_run` +start that reads the screen and stops before any reset, in seconds. + +| Program | Preconditions | +|---|---| +| FRLG RNG (static, gift, egg, Safari) | a free party slot (gift, egg); Sweet Scent user in the last slot (wild); Safari advances ≥ 4000; Game Settings loaded so the license check is on; game on a known screen, not a summary | +| FRLG / BDSP / PLZA AutoStory | game in front is the program's game; start checkpoint chosen by id and visible (dev-only rows need Developer Mode); `console://B/save` matches the start checkpoint and is not stale; LANGUAGE matches the cartridge | +| Every program | lab profile has audio input allowed or off | + +## Enforcement + +- **Agents never run the app.** The server launches it with `--profile lab-B` (the + `labrun.sh` way: direct exec, never a bare binary) and quits only its own process. A + Claude Code PreToolUse hook and the standing rules refuse Bash calls that name the + SerialPrograms binary or write into `Lab/inbox`, so the server is the only path. +- **Console A is never exposed.** `consoles.json` has no entry for it; no resource, no tool. +- **One train.** Leases replace `lablock.sh`; the app still refuses presses while a program + runs. +- **Heavy work is queued.** The disk floor (25 GB to start, 30 GB target) and the heavy + gate are properties of `build_lab` and `run_tests`, not of agent discipline. +- **Audit.** Every call, its arguments and its result go to `audit.jsonl`. + +## Work plan + +| PR | Content | Size | Retires | |---|---|---|---| -| 1 | This design doc | doc | review | -| 2 | `lab.py`: console and program subcommands, the guards, `consoles.json`, the audit log. `labctl.sh` and `cfrun.sh` call it. | ~500 + tests | fake-lab unit tests; one scripted session on console B | -| 3 | `lab.py runs` and `logs` on #116's run records (folds in #116's proposed `runs.py`) | ~250 | unit tests on fixture run folders | -| 4 | App: `schema` lab command, danger notes on options, detector descriptions | ~200 | C++ unit tests on two programs' option trees | -| 5 | `lab.py dev`: build, test, deploy, launch and quit, with the freshness check | ~250 | unit tests with fake commands | -| 6 | Skills and `.agent.md` notes for FRLG AutoStory, BDSP AutoStory and the BBQ farmer | docs | a skill-driven console run | -| 7 (optional) | Thin MCP server over `lab.py` | ~150 | each tool's output matches the CLI's | - -The run events themselves (the hooks in story runners and walkers, and #125 reading -`events.jsonl`) are #116's milestones; PRs 3 and 6 depend on them. - -Order: PR 2 is useful on its own, because today's labctl and batch flows become typed and -guarded. PRs 3–5 make it observable and discoverable. PR 6 makes agents faster on the -programs we work on most. PR 7 waits for a reason. - -Testing story: -- **Every CLI command is unit-tested** against a fake lab folder. -- **The app-side commands** have C++ unit tests in the existing runner. -- **One scripted console-B session per milestone** checks the end-to-end path, and leaves a - run record as evidence. +| 1 | This design | doc | — | +| 2 | App (`lab/tooling`): `Lab/state` gains `video` (signal, frame age, seq), `stop_source`, `screen`; programs write `Lab/save.json` (FRLG and BDSP AutoStory, FRLG RNG first) | ~250 C++ + tests | — | +| 3 | Server: read-only resources `console://B/state`, `console://B/save`, `runs://`, and `shot()` | ~400 Py + fake-lab tests | `checkin.sh`, `lab_status.sh`, `shotB.sh`, `rec_shots.sh`, log greps for verdicts | +| 4 | Leases and events: `acquire_console`, user lease from UI stops, `events.jsonl`, `next_events` | ~300 Py | `lablock.sh`, `switch_guard.sh`, `run_watch.sh`, `watch_run.sh`, `stage_watch.sh`, `full_run_watch.sh`, `plza_watch*.sh` | +| 5 | App: option `schema` lab command, declared preconditions, `dry_run` precheck; FRLG RNG programs and AutoStory first | ~300 C++ + tests | — | +| 6 | Server: `run_program`, `stop_program`, `switch_game`, `cfr_batch`; launches the app | ~450 Py | `labrun.sh`, `lab_panel.sh`, `start_B.sh`, `stopB.sh`, `run_from_save.sh`, `reload.sh`, `preflight_B.sh`, `stage_run.sh`, `stage_launch.sh`, `cfB.sh`, `cfrB.sh`, `labctl.sh` and `cfrun.sh` as agent entry points | +| 7 | Queued heavy tools: `build_lab`, `run_tests`, disk floor | ~250 Py | `labbuild.sh`, `heavy_gate.sh`, `make_slot_b.sh`, `plza_rebuild.sh` | +| 8 | `ci_status`, `stack_plan` | ~300 Py | `wait_for.sh checks`, `ciwait.sh`, hand-made stack lists | +| 9 | Skills and standing rules point at the tools; drop rules the server enforces | docs | rule text in briefs | + +Order: PRs 2-3 answer "what is the console doing and which save does it hold" without +touching it, which alone covers the stale-frame, save and grep incidents. PR 4 makes +ownership explicit. PRs 5-6 move setup errors from minute 10 to second 1. PRs 7-8 are +independent of the console and can go in parallel with 5-6. `disk_gc.sh` and the launchd +jobs stay. + +Testing: every server tool is unit-tested against a fake lab folder (a fixture loop that +consumes requests and writes journals, state and save files); app-side fields have C++ unit +tests; each console milestone ends with one scripted session on console B that leaves a run +record. ## Open questions -- **`labctl.sh` and `cfrun.sh`:** keep them as thin wrappers (existing notes and skills call - them), or retire them after the skills move to `lab.py`? The proposal is to keep them for - one release, then retire them. -- **Where `consoles.json` lives:** the proposal is the profile folder, so each machine keeps - its own rules, with an example in the repo. -- **Upstream or fork-only?** The lab channel and the CLI are developer tooling. Upstream may - want them behind developer mode, or not at all. -- **Whether and when to add the MCP wrapper:** only on a concrete need, as above. +- **Where the user lease ends:** CLI release or 30 idle minutes, as above, or only the CLI? +- **Upstream or fork-only?** The lab channel and the server are developer tooling; upstream + may want the app-side parts behind Developer Mode, or not at all. +- **Precondition home:** data next to each program's options (proposed), or one table in the + server per program id? From 8cf5938241597666f4763b4d18f3161f1a4505a1 Mon Sep 17 00:00:00 2001 From: dayfine Date: Sat, 14 Nov 2026 00:42:00 -0800 Subject: [PATCH 4/7] Expand agent integration design and rollout plan --- docs/design/Agent-Integration.md | 447 +++++++++++++++++++++++-------- 1 file changed, 333 insertions(+), 114 deletions(-) diff --git a/docs/design/Agent-Integration.md b/docs/design/Agent-Integration.md index 6a0e7aceab..a4a97b5c61 100644 --- a/docs/design/Agent-Integration.md +++ b/docs/design/Agent-Integration.md @@ -1,14 +1,16 @@ # Agent integration: an MCP lab server -Status: proposal (2026-11-01; rewritten 2026-11-13 from the week's incidents, -`~/BDSPLab/notes/2026-11-13-week-problems.md`). It builds on: +Status: proposal (2026-11-01; revised 2026-11-14 from the week's incidents, +`~/BDSPLab/notes/2026-11-13-week-problems.md`, and an implementation review). It builds on: - the lab channel: fork #107, `SerialPrograms/Source/CommonFramework/Windows/Lab.md` (`Lab/inbox`, `Lab/outbox`, `Lab/state`, the owner token); - run records: fork #116, `docs/design/Lab-RunRecords.md`; - the lab scripts on `agents` (`lab/control`, `lab/scripts`) and in `~/BDSPLab` (`labrun.sh`, `labbuild.sh`, `checkin.sh`, `lablock.sh`, `heavy_gate.sh`, ...). -Nothing here is implemented except what those already have. +Nothing here is implemented except what those already have. In particular, the app does +not yet expose a frame sequence, a generic game/screen detector, a preflight API, or an +action source. Those are deliverables below, not existing integration points. ## Requirement @@ -18,12 +20,14 @@ remembers them. This week that failed in eight ways (table below): stale frames, the HOME screen, setup errors found after a 2-10 minute reset, the app run without its profile, and answers that cost four log greps. -The goal is one local server that owns console B and the lab's heavy work, answers state -questions from typed resources, and refuses a bad request before it costs a reset. Agents -talk to it over MCP; the user can use the same tools from a thin CLI. +The goal is one local server that owns console B control, answers state questions from typed +resources, and refuses a bad request before it costs a reset. Agents talk to it over MCP; +the user can use the same tools from a thin CLI. -Not in scope: remote control, console A, GitHub writes (agents keep their `gh` rules), video -streaming, and any change to how programs behave without a `Lab/` folder. +Not in scope for the console server: remote control, console A, GitHub writes, branch/PR +maintenance, CI diagnosis, video streaming, or arbitrary game selection from HOME. Heavy +build/test orchestration can be added later as a separate local-operations worker; it must +not expand the console daemon's credentials or failure domain. ## Problems this solves @@ -36,8 +40,8 @@ One row per incident from 2026-11-07..13. "R" = resource, "T" = tool, "E" = even | Selecting a settings panel through the inbox blocked the channel | T `run_program` takes program panels only; no raw `select` | | Panel switch reset controller and video ("requires video feedback") | R `console://B/state` `controller`, `video`; `run_program` waits for both | | No-HDMI colour bars went unnoticed until a frame was read | R `state.video.signal`; E `video_lost` | -| Game switches (HOME, close, pick tile) done by hand | T `switch_game(target)` | -| PLZA 03.08 started with the game closed and walked on HOME | precondition "game in front = program's game, known screen" | +| Game switches (HOME, close, pick tile) done by hand | deferred: arbitrary title selection has no reliable app seam yet | +| PLZA 03.08 started with the game closed and walked on HOME | T `ensure_game(expected)` and the program's dynamic preflight refuse the run | | UI stop at 22:25 had to be guessed as the user's | E `stopped{source: ui}`; lease moves to `user` | | "Hands off B while the user plays" held only by memory | lease: tools refuse while `user` holds it | | Which save B holds lived in notes (Safari logo vs Mansion door, unsaved Parasect) | R `console://B/save` | @@ -45,82 +49,166 @@ One row per incident from 2026-11-07..13. "R" = resource, "T" = tool, "E" = even | Safari needs ≥ 4000 advances | precondition on `run_program` | | FRLG license check not loaded (Game Settings panel never opened) | precondition: game settings loaded | | No Sweet Scent user in the last slot | precondition, checked by the program's precheck before any reset | -| Restart on a post-run summary screen | precondition "known screen"; `switch_game` brings it there | +| Restart on a post-run summary screen | precondition "known screen" refuses the run; recovery remains explicit | | PLZA LANGUAGE left on ja | precondition: language matches the console's game | | Dev-only checkpoint hidden without Developer Mode shifted indices; wrong checkpoint started | options by key and checkpoint id, never by index; refuse a hidden row | | Audio input not allowlisted for lab-B (UserSetupError spam) | server launch checks the lab profile's settings | | App run without `--profile`, twice | agents never run the app; the server launches it | | Rules retyped per brief and drifting | rules enforced by the server, not restated in briefs | -| Agents parked waiting on the heavy gate or the console | queued tools return a ticket; E `build_done`, `program_finished` | -| Linter rule changed mid-task | not addressed: `ci_status` names the linter ref it ran with, nothing more | -| One upstream sync = merges through 36 branches (+~40 descendants) | T `stack_plan(pr)` | -| CI red from SystemKeyboardDetector images never pushed to CommandLineTests | T `ci_status` classifies "missing test image" | -| Branches checked out in several worktrees went stale | T `stack_plan` lists each branch's worktrees and how far behind they are | +| Agents parked waiting on the console | long console operations return a request/ticket; E `program_finished` | +| Linter rule changed mid-task | not addressed by this design | +| One upstream sync = merges through 36 branches (+~40 descendants) | out of scope: keep this in PR-maintenance tooling | +| CI red from SystemKeyboardDetector images never pushed to CommandLineTests | out of scope: keep this in CI-diagnosis tooling | +| Branches checked out in several worktrees went stale | out of scope: keep this in PR-maintenance tooling | | "How far off is the RNG?" took four greps | R `runs://latest` carries RNG hits (target, hit, delta) | | Results spread over notes, outbox, error dirs with spaces | R `runs://`: one record, frame and video at clean paths | | Sim passes, console fails (FRLG 13.01, 12.03; PLZA walk_to_objective) | partly: `runs://` keeps the walked poses a sim test replays | -| Disk under the 30 GB floor | queued `build_lab` refuses under the floor | -| RAM swap from parallel builds | `build_lab` queue: one build, `-j4`, `nice`, waits while a program runs | +| Disk under the 30 GB floor | future local-operations worker refuses a build under the floor | +| RAM swap from parallel builds | future local-operations worker serializes builds and applies the existing limits | ## Shape ``` - agent hosts (Claude Code, Codex) user shell (lab CLI) - │ MCP (localhost) │ - ▼ ▼ - pa-lab server (one process, launchd; owns leases, queues, audit log) - ├─ resources: console://B/state, console://B/save, runs:// - ├─ tools: shot, switch_game, run_program, stop_program, build_lab, ci_status, stack_plan - ├─ events: resource updates + next_events() - └─ config: consoles.json (B only; A has no entry) - │ files: Lab/inbox → Lab/outbox, Lab/state, Lab/save.json, Lab/runs// - ▼ - SerialPrograms, launched by the server with --profile lab-B + agent hosts (Claude Code, Codex) user shell (thin lab CLI) + │ MCP Streamable HTTP on 127.0.0.1 │ + └──────────────────┬───────────────┘ + ▼ + pa-lab console daemon + ┌──────────────────┼───────────────────┐ + │ │ │ + console-B adapter SQLite control DB audit export + Lab/ files only leases, events, JSONL (derived) + requests, tickets + │ + │ Lab/inbox → Lab/outbox + │ Lab/state + Lab/state.json, Lab/save.json, Lab/runs/ + ▼ + SerialPrograms (`--profile lab-B`) ``` -One long-lived process, not one per client: leases, the build queue and the event log must -be shared by every agent. It is Python with the MCP SDK, in a venv like the code-health one, -and lives with the lab tooling on `agents` (`lab/server/`). App-side changes stay on -`lab/tooling`. +The daemon is one long-lived Python process because leases, idempotency and the event +sequence must be shared by every client. It uses MCP Streamable HTTP bound only to +`127.0.0.1`, checks an unguessable bearer token and validates `Origin`; stdio would create +one independent lease database per client. Its allowlist is a server-owned config with one +entry, B. It must not reuse the operator's broader `~/BDSPLab/consoles.json`, which also +describes A. -## Resources (read-only, first PR) +The durable control plane is SQLite. JSONL is an audit/export format, not the source of +truth for leases, idempotency or event cursors. The daemon initially attaches to an app +started by the existing guarded launch path. It becomes the app supervisor only after that +launch behavior has been extracted and tested; it must reuse the profile, direct-exec and +process-identity guards rather than reimplementing them. -**`console:///state`**, from `Lab/state` plus new app fields: +The console daemon has no GitHub token and does not inspect worktrees. A future +local-operations worker may expose queued build/test tickets over a separate process and +configuration, reusing the existing heavy-gate scripts. `ci_status` and `stack_plan` remain +outside this design. + +## Resources (read-only, first implementation slice) + +The app keeps the legacy key/value `Lab/state` for existing scripts and atomically writes a +new versioned `Lab/state.json`. The daemon rejects an unsupported `schema` and overlays only +server-owned data such as the lease. Example app file: + +```json +{ + "schema": 1, + "written_at": "2026-11-14T06:10:12.124Z", + "instance": {"id": "d8ea6a9d-1eab-4a40-b355-d88072b289a1", "pid": 4411}, + "panel": {"id": "PokemonFRLG:AutoStory", "kind": "program"}, + "program": { + "state": "stopped", + "run_id": null, + "last_transition": { + "action": "stop", + "source": "ui", + "request_id": null, + "at": "2026-11-14T06:10:10.002Z" + } + }, + "controller": {"state": "ready"}, + "video": { + "state": "live", + "frame_seq": 918274, + "captured_at": "2026-11-14T06:10:12.091Z", + "signal": "unknown" + }, + "observation": { + "game": {"value": "unknown", "observed_at": null}, + "screen": {"value": "unknown", "observed_at": null} + } +} +``` + +`written_at` is a heartbeat even when no logical field changes. `instance.id` changes on +each app start, so a reset frame sequence cannot be mistaken for a newer frame from the old +process. `frame_seq` increases for every frame accepted by the app's frame cache and +`captured_at` is that frame's capture time, not the time the state file was serialized. +The daemon computes frame age from `captured_at`; it does not persist a decaying +`last_frame_age_ms` value. + +Version 1 distinguishes transport freshness (`live`, `stale`, `missing`) from visual signal +classification (`ok`, `black`, `colour_bars`, `unknown`). The first slice can truthfully +publish `unknown`; black/colour-bar detection becomes a later detector, not a guessed +boolean. Game and screen observations are also `unknown` until a named detector has run. +Every non-unknown observation must include `observed_at` and, when useful, detector and +confidence fields. + +`program.last_transition.source` is emitted where the action originates, not inferred from +the resulting state: `ui`, `lab`, `program`, or `error`. A lab action carries the daemon's +request UUID. This is how the daemon distinguishes a user's Stop click from its own Stop +request even if both race. + +**`console:///state`** is the parsed app state plus: ```json -{"console": "B", "game": "PokemonFRLG", "screen": "overworld", - "panel": "PokemonFRLG:AutoStory", "program_state": "STOPPED", "controller": "ready", - "video": {"signal": true, "last_frame_age_ms": 33, "frame_seq": 918274}, - "lease": {"owner": "agent:frlg-lane", "since": "22:10", "expires": "22:40"}, - "app": {"pid": 4411, "build": "slot-B-frlg/2026-11-13T21:02"}} +{"lease": {"kind": "agent", "owner": "agent:frlg-lane", "acquired_at": "...", "expires_at": "..."}} ``` -`signal` is false for no frames, black or colour bars. `game` and `screen` come from -detectors (HOME, title, overworld, summary), `unknown` when none match. +It also reports daemon/app reachability and the exact refusal reasons that currently block +an action. Unknown facts stay unknown; absence is never interpreted as success. -**`console:///save`**: written by programs to `Lab/save.json` on every save: -`game`, `slot`, `checkpoint`, `position` (`" (x, y)"`), `time`, `run`. It turns -`stale: true` when anything that could save ran unrecorded since: a user lease, the CFR, a -program without the hook. +**`console:///save`** is a fact with provenance, not a promise that every game save was +observed. Programs that know they saved atomically write `Lab/save.json` with `schema`, +`game`, `slot`, `checkpoint`, structured position, `saved_at`, `run_id`, and +`producer` (program id and build). The daemon exposes a status of `known`, `stale`, or +`unknown` and the reason. It marks the fact stale after a user lease, a CFR batch, or any +program that may save but does not publish the hook. Only a new program-authored save fact +can return it to known. **`runs://latest`, `runs://`**: #116's run record, summarised: `verdict` (`passed|failed|stopped`, reason, stop source), checkpoint lines (`13.01 failed mansion_holes after 2 attempts`), RNG hits (`target 4123, hit 4127, delta +4`), and the failure frame and video copied to `~/BDSPLab/runs//` (no spaces). -## Leases and events +## Leases, requests and events + +Reads need no lease. Every tool that can press a button, start/stop a program or mutate app +selection requires a live lease token, not merely the caller-supplied owner name. The +SQLite state machine is: -`acquire_console(owner, ttl)` returns a lease; `renew` and `release` take it. Every tool -that presses, starts or stops needs the caller's live lease; reads need none. The lease is -the `lablock.sh` lock and the channel's owner token in one. +| State | Entry | Exit | +|---|---|---| +| `free` | initial state or explicit release | successful `acquire_console` | +| `agent-idle` | acquired with bounded TTL | renew, explicit release, TTL expiry, run start, or observed UI action | +| `agent-running` | the leased caller starts a run | program terminal event, explicit leased stop, or observed UI action; it never expires mid-run | +| `user` | app reports a UI-originated Start/Stop or the operator explicitly claims B | explicit operator release only | -When the app's program state changes without a server request (Start or Stop clicked in the -window), the server gives the lease to `user` and emits `lease_changed{owner: user}`. Agent -tools then refuse. The user lease ends when the user releases it from the CLI, or after -30 minutes with the program stopped and no UI input. +A UI action revokes any agent lease before a user lease is installed. There is no guessed +30-minute user timeout: absence of UI telemetry is not proof that the user is done. On +daemon restart, leases and request results survive in SQLite; the daemon reconciles them +against `instance.id`, current program state and the latest transition before allowing a +mutation. -Events go to an append-only `events.jsonl`, as MCP resource-update notifications, and through -`next_events(since, timeout ≤ 60 s)` for clients without subscriptions: +Every mutating call requires a client-generated request UUID. The unique key +`(client_id, tool, request_id)` stores the normalized arguments and terminal result. An +exact retry returns the stored result; reuse with different arguments is refused. This +prevents a transport retry from starting a program or pressing a button twice. + +Events are rows in SQLite with a monotonically increasing 64-bit `seq`, event time, console, +app instance, request id and JSON payload. MCP resource-update notifications are a wake-up +hint; `next_events(after_seq, timeout <= 60 s)` is the durable catch-up API and returns the +next cursor. JSONL can be exported for humans, but truncating it cannot corrupt cursors. | Event | Fields | |---|---| @@ -129,11 +217,11 @@ Events go to an append-only `events.jsonl`, as MCP resource-update notifications | `video_lost`, `video_back` | last frame seq | | `autosave`, `save` | the new `console://B/save` | | `lease_changed` | old and new owner | -| `build_done`, `tests_done` | ticket, result | -| `ci_done` | PR, SHA, result, missing images | +| `request_completed`, `request_failed` | request id, tool, normalized result or refusal | -Subagents still never wait: they read the ticket and stop. The main session's background -waits read `events.jsonl` instead of polling logs. +Agents do not park on a tool call. Long operations return a ticket/request id; a later +client consumes events or reads request status. The daemon may keep recent resource +notifications in memory, but correctness always comes from the durable cursor. ## Tools @@ -141,70 +229,201 @@ Each tool validates before it acts and returns JSON, or a refusal naming the fai | Tool | Validates / does | |---|---| -| `shot(console)` | Lease not needed. Refuses if `last_frame_age_ms` > 2000 or no signal. Returns the image and `frame_seq`, `time`; the file name carries the seq. | -| `switch_game(console, target)` | Lease, program stopped. HOME, close the current game, launch `target` by title (not tile position), wait for its title screen. | -| `run_program(console, panel, options)` | Lease. Panel is a program; options checked against the program's schema (keys, enum labels, ranges); declared preconditions hold; controller and video healthy. Selects, sets, starts; returns the run id. | -| `stop_program(console)` | Lease. Stops; the event records `source: server`. | -| `cfr_batch(console, lines)` | Lease, program stopped. Command File Runner batch for debugging only; marks the save stale. | -| `build_lab(game, ref)`, `run_tests(slot, filter)` | Queued: one at a time, `-j4`, `nice 15`, waits while a program runs, refuses under the disk floor. Returns a ticket. | -| `ci_status(branch \| pr)` | Per workflow: result, failing tests; a `FileException` on a test image that no CommandLineTests ref holds is reported as `missing_image`, not a code failure. | -| `stack_plan(pr)` | Read-only: parents, children, which branches need a merge after a parent moved, in order; each branch's worktrees and how far they lag. Merges stay with pr-maintenance agents. | - -### Preconditions - -A program declares its preconditions as data next to its option schema. The server checks -the ones it can answer from state, save and options; the rest run in the program's own -precheck (as FRLG `RngProgramInstance::run_precheck()` already does) through a `dry_run` -start that reads the screen and stops before any reset, in seconds. +| `acquire_console(console, owner, ttl)` | Atomically acquires `free`; returns an opaque lease token and expiry. Owner is descriptive, not authority. | +| `renew_console(lease, ttl)` / `release_console(lease)` | Operate only on the matching token. A user lease is released through an operator-authenticated CLI action, not an agent token. | +| `shot(console, max_age_ms=2000, timeout_ms=5000)` | Lease not needed. Records `(instance.id, frame_seq)` at call start and waits for a strictly newer frame. Returns the completed PNG plus the new token and `captured_at`; never reuses a caller tag. Refuses on timeout, stale transport, or a classified bad signal. | +| `ensure_game(console, expected, allowed_screens)` | Read-only verification. Runs/reads the named detectors and returns the observation and age. It never navigates HOME or launches a title. | +| `describe_program(console, panel)` | Returns a stable program id, typed option schema, defaults and declared preconditions. Refuses settings panels and hidden/unavailable programs. | +| `preflight_program(console, panel, options)` | Lease required because detectors may use app resources, but performs no controller action. Checks selection, typed options, static facts and the program's dynamic checks; returns every check and evidence. | +| `run_program(console, panel, options, preflight_token)` | Lease. Requires a still-current successful preflight bound to app instance, panel, normalized options, observation versions and lease. Starts once and returns request/run ids. | +| `stop_program(console)` | Lease. Requests Stop once; success is the matching `last_transition{source: lab, request_id}` or a terminal run event, not merely a stopped snapshot. | +| `cfr_batch(console, lines)` | Lease, program stopped. Debug-only allowlisted command batch; marks the save fact stale. Raw writes to `Lab/inbox` are not exposed. | + +`switch_game` is deliberately absent from the first design. Current common routines can +resume or start the selected HOME tile, but cannot reliably select an arbitrary title by +identity. Add a separate proposal only after the app exposes a tested title-selection seam. +Until then, `ensure_game` turns the dangerous case into an early refusal and leaves recovery +to the operator. + +The `shot` freshness rule closes both old races: its token is newer than call start, and the +daemon responds only after the atomically published PNG can be opened. An app-side snapshot +command should serve this read-only path without requiring a ready controller. + +### Program contract and preflight + +There is no generic program preflight API today. It is introduced as two layers rather than +simulating a Start/Stop cycle: + +1. A static contract beside the program descriptor gives stable option keys, types, + constraints, defaults, visibility and declarative fact requirements. The daemon validates + and normalizes options without touching the console. UI row indices and display strings + are never API identifiers. +2. A virtual dynamic preflight on the program instance checks facts only the program can + establish: screen readers, party layout, loaded game settings, save compatibility, and + similar state. It returns structured checks and must not press a controller button, + reset, save, or alter options. + +The result is a short-lived preflight token hashing the app instance, program id, normalized +options, relevant state/save observation versions and lease. `run_program` refuses a stale +token. Programs without a dynamic implementation declare that explicitly; absence never +means a successful check. | Program | Preconditions | |---|---| | FRLG RNG (static, gift, egg, Safari) | a free party slot (gift, egg); Sweet Scent user in the last slot (wild); Safari advances ≥ 4000; Game Settings loaded so the license check is on; game on a known screen, not a summary | | FRLG / BDSP / PLZA AutoStory | game in front is the program's game; start checkpoint chosen by id and visible (dev-only rows need Developer Mode); `console://B/save` matches the start checkpoint and is not stale; LANGUAGE matches the cartridge | -| Every program | lab profile has audio input allowed or off | +| Every program | lab profile has audio input allowed or off; controller/video transport live | ## Enforcement -- **Agents never run the app.** The server launches it with `--profile lab-B` (the - `labrun.sh` way: direct exec, never a bare binary) and quits only its own process. A - Claude Code PreToolUse hook and the standing rules refuse Bash calls that name the - SerialPrograms binary or write into `Lab/inbox`, so the server is the only path. -- **Console A is never exposed.** `consoles.json` has no entry for it; no resource, no tool. +- **One authenticated local entry point.** Streamable HTTP binds to loopback, requires a + bearer token with owner-only file permissions, validates `Origin`, caps request sizes and + records the authenticated client id. The token is not placed in logs or MCP results. +- **Agents never run the app once supervision lands.** The server launches it with + `--profile lab-B` using the extracted `labrun.sh` behavior (direct exec, never a bare + binary) and quits only the PID/instance it created. Before that milestone, it is an + attach-only observer and the existing guarded launcher remains authoritative. +- **Console A is never exposed.** The server's own allowlist has no entry for it; no + resource, no tool. A user-supplied console id cannot become a filesystem path. - **One train.** Leases replace `lablock.sh`; the app still refuses presses while a program runs. -- **Heavy work is queued.** The disk floor (25 GB to start, 30 GB target) and the heavy - gate are properties of `build_lab` and `run_tests`, not of agent discipline. -- **Audit.** Every call, its arguments and its result go to `audit.jsonl`. +- **Request boundaries are narrow.** Panel ids and options are validated against the app's + contract. CFR commands are allowlisted. The server never accepts raw inbox paths, shell + commands or app binary paths. +- **Audit is durable and redacted.** Every authenticated call, normalized arguments, state + decision and terminal result is a SQLite record. A JSONL export omits bearer/lease tokens + and other secrets. +- **Heavy work stays separate.** A future worker owns disk floors, serialization, `-j4` and + `nice`; the console daemon neither shells out to arbitrary builds nor gains GitHub access. ## Work plan -| PR | Content | Size | Retires | -|---|---|---|---| -| 1 | This design | doc | — | -| 2 | App (`lab/tooling`): `Lab/state` gains `video` (signal, frame age, seq), `stop_source`, `screen`; programs write `Lab/save.json` (FRLG and BDSP AutoStory, FRLG RNG first) | ~250 C++ + tests | — | -| 3 | Server: read-only resources `console://B/state`, `console://B/save`, `runs://`, and `shot()` | ~400 Py + fake-lab tests | `checkin.sh`, `lab_status.sh`, `shotB.sh`, `rec_shots.sh`, log greps for verdicts | -| 4 | Leases and events: `acquire_console`, user lease from UI stops, `events.jsonl`, `next_events` | ~300 Py | `lablock.sh`, `switch_guard.sh`, `run_watch.sh`, `watch_run.sh`, `stage_watch.sh`, `full_run_watch.sh`, `plza_watch*.sh` | -| 5 | App: option `schema` lab command, declared preconditions, `dry_run` precheck; FRLG RNG programs and AutoStory first | ~300 C++ + tests | — | -| 6 | Server: `run_program`, `stop_program`, `switch_game`, `cfr_batch`; launches the app | ~450 Py | `labrun.sh`, `lab_panel.sh`, `start_B.sh`, `stopB.sh`, `run_from_save.sh`, `reload.sh`, `preflight_B.sh`, `stage_run.sh`, `stage_launch.sh`, `cfB.sh`, `cfrB.sh`, `labctl.sh` and `cfrun.sh` as agent entry points | -| 7 | Queued heavy tools: `build_lab`, `run_tests`, disk floor | ~250 Py | `labbuild.sh`, `heavy_gate.sh`, `make_slot_b.sh`, `plza_rebuild.sh` | -| 8 | `ci_status`, `stack_plan` | ~300 Py | `wait_for.sh checks`, `ciwait.sh`, hand-made stack lists | -| 9 | Skills and standing rules point at the tools; drop rules the server enforces | docs | rule text in briefs | - -Order: PRs 2-3 answer "what is the console doing and which save does it hold" without -touching it, which alone covers the stale-frame, save and grep incidents. PR 4 makes -ownership explicit. PRs 5-6 move setup errors from minute 10 to second 1. PRs 7-8 are -independent of the console and can go in parallel with 5-6. `disk_gc.sh` and the launchd -jobs stay. - -Testing: every server tool is unit-tested against a fake lab folder (a fixture loop that -consumes requests and writes journals, state and save files); app-side fields have C++ unit -tests; each console milestone ends with one scripted session on console B that leaves a run -record. - -## Open questions - -- **Where the user lease ends:** CLI release or 30 idle minutes, as above, or only the CLI? -- **Upstream or fork-only?** The lab channel and the server are developer tooling; upstream - may want the app-side parts behind Developer Mode, or not at all. -- **Precondition home:** data next to each program's options (proposed), or one table in the - server per program id? +Each slice is independently reviewable and keeps the existing scripts working until its +replacement passes a console-B soak. App slices belong on a new branch from +`design/agent-integration`; changes to this proposal are handed to that PR's owner rather +than pushed onto its branch. + +### Slice 0 — freeze contracts and fixtures + +- Commit version-1 JSON examples for app state, save facts, requests, refusals and events. +- Define enums, required/optional fields, timestamp format, unknown-value rules and schema + compatibility. Decide which fields are app-authored and which are daemon overlays. +- Add golden parse/round-trip tests before wiring live state. + +Acceptance: malformed, missing-version and future-version examples fail with a field-level +error; all golden examples round-trip without losing unknown fields needed for diagnostics. + +### Slice 1 — app observation seam (first coding branch) + +- Keep `Lab/state` byte-compatible and add atomically replaced `Lab/state.json`. +- Give each app process an instance UUID and heartbeat; expose panel id/kind, program state, + controller state, last transition, frame sequence and capture timestamp. +- Thread the existing video-cache sequence/timestamp to the lab seam. Do not invent black, + colour-bar, game or screen observations; publish `unknown` until their detectors exist. +- Add a read-only fresh-snapshot command that does not require controller readiness and + returns the published frame token. + +Acceptance: native tests cover serialization and transition-source mapping; a manual test +restarts the app and proves `(instance, seq)` changes; a snapshot requested during a panel +or controller fault is either a newer openable PNG or a typed timeout, never an old image. + +### Slice 2 — attach-only read server + +- Implement the loopback Streamable HTTP endpoint, bearer/Origin checks, B-only allowlist, + strict state/save parsers and read resources. +- Implement `shot` with the newer-than-call-start rule and `ensure_game` as verification + only. Expose `runs://` by adapting #116 once its format is available. +- Treat stale heartbeat, app instance replacement, partial files and unsupported schemas as + explicit unavailable states. + +Acceptance: fake-lab tests exercise partial writes, stale files, sequence resets, app +restarts, timeouts and concurrent readers; a console-B session can replace `shotB.sh` and +status/log greps without granting mutation authority. + +### Slice 3 — durable control plane + +- Add SQLite migrations for leases, requests/idempotency, events and tickets. +- Implement acquire/renew/release, user takeover, restart reconciliation, + `next_events` and request-status reads. +- Add bounded retention/compaction that preserves every active cursor and unresolved ticket. + +Acceptance: crash/restart and concurrent-client tests prove one lease winner, stable retry +results, no running-lease expiry and gap-free event catch-up. + +### Slice 4 — typed program contract + +- Add `describe_program` app support with stable ids and typed options; keep UI concerns out + of the contract. +- Add the side-effect-free dynamic preflight interface and structured check results. +- Implement it first for FRLG RNG and one AutoStory family, including save provenance and + known-screen checks; missing implementations report `unsupported`. + +Acceptance: native tests cover option normalization, ranges/enums, hidden checkpoint ids and +side-effect-free preflight. Deliberately bad Safari, party, language, save and HOME-screen +setups are refused before a reset. + +### Slice 5 — leased actions + +- Implement `preflight_program`, token binding, `run_program`, `stop_program` and the + restricted `cfr_batch` over request UUIDs. +- Emit the action source at the UI/lab/program/error origin and correlate lab transitions + with the request id. Make terminal events authoritative for completion. +- Leave `switch_game` out. Do not make selection-by-visible-index part of the protocol. + +Acceptance: race tests cover duplicate requests, UI takeover during start/stop, lease loss, +app restart between preflight and start, and a terminal event arriving before the response. +A console-B soak demonstrates that every mutation has exactly one audit/request record. + +### Slice 6 — save facts and run-record adoption + +- Add the versioned save hook to the initial AutoStory/RNG programs and mark all unhooked + mutation paths stale with a reason. +- Adapt #116 run records instead of defining a competing run format; carry stop source, + checkpoint and RNG hit summaries through resources/events. +- Copy/link failure media only after publication is complete, with clean run-id paths. + +Acceptance: a known save survives daemon restart, becomes stale after UI/CFR activity, and +returns to known only after a hooked save. A failed and a stopped run each resolve to one +record with the correct source. + +### Slice 7 — supervised app lifecycle + +- Extract the existing `labrun.sh` profile, audio-allowlist, direct-exec, build-selection and + PID identity checks into reusable guarded operations. +- Let the daemon start/stop only the process instance it owns; reconcile crashes and reject + a second instance. Preserve an operator escape hatch. +- Only after the soak, update hooks/rules so agents use the server instead of launching the + app or writing `Lab/inbox` directly. + +Acceptance: tests use fake processes/PIDs; manual tests cover stale pidfiles, wrong profile, +missing audio permission, crash/restart and refusal to kill an unrelated process. + +### Slice 8 — optional local-operations worker + +- In a separate process and config, wrap the existing heavy gate, disk floor, `-j4`, `nice` + and slot-build scripts behind durable tickets. +- Do not add GitHub/PR or worktree mutation to either service. Keep `disk_gc.sh` and existing + launchd maintenance jobs until separately replaced. + +Acceptance: concurrent fake jobs serialize, disk/RAM refusals are typed, cancellation is +safe, and a console run cannot be starved or stopped by the worker. + +### Rollout and retirement + +Run old scripts and the new read resources in parallel for at least three console sessions, +then compare frame tokens, state transitions and run verdicts. Enable mutation for one +program family at a time. Retire a script only when its replacement has native/fake-lab +coverage, a successful console-B session and a documented operator fallback. Do not delete +the file channel: it remains the app/daemon compatibility boundary. + +## Decisions and remaining boundary + +- User leases end only by explicit operator release. +- Static option/precondition data lives with the program contract; dynamic evidence lives in + the program instance. The server orchestrates and caches results but does not duplicate a + per-program rules table. +- The generic app-side observation/contract seams are candidates for upstream review behind + the existing Lab/Developer boundary. The Python daemon, console inventory, credentials and + local policies remain fork/local tooling. +- Arbitrary game switching, CI diagnosis and stack/worktree planning require separate + designs. They are not hidden follow-ups inside the console server. From 7a2da8fac4d5436d218385afecbea0259e968fea Mon Sep 17 00:00:00 2001 From: dayfine Date: Sat, 14 Nov 2026 00:57:18 -0800 Subject: [PATCH 5/7] Design: fit the lab server to the existing locks, hooks and repo rules Review of 8cf593824 against the lab scripts and the user's console rules. Says where the daemon and the app slices live (agents lab/server, branches from lab/tooling), how leases coexist with lablock and the channel owner token until Slice 7, that no hook guards the app binary today, which AGENTS.md hard rules the app slices must keep, the legacy program.state values, which scripts each slice retires, and which decisions are the user's rules versus proposals awaiting the user's confirmation. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_013VUubhHeYSMGSSDXZkKZss --- docs/design/Agent-Integration.md | 78 +++++++++++++++++++++++++++----- 1 file changed, 66 insertions(+), 12 deletions(-) diff --git a/docs/design/Agent-Integration.md b/docs/design/Agent-Integration.md index a4a97b5c61..691dcab892 100644 --- a/docs/design/Agent-Integration.md +++ b/docs/design/Agent-Integration.md @@ -1,7 +1,8 @@ # Agent integration: an MCP lab server Status: proposal (2026-11-01; revised 2026-11-14 from the week's incidents, -`~/BDSPLab/notes/2026-11-13-week-problems.md`, and an implementation review). It builds on: +`~/BDSPLab/notes/2026-11-13-week-problems.md`, and an implementation review; checked against the +lab scripts and the user's console rules on 2026-11-14). It builds on: - the lab channel: fork #107, `SerialPrograms/Source/CommonFramework/Windows/Lab.md` (`Lab/inbox`, `Lab/outbox`, `Lab/state`, the owner token); - run records: fork #116, `docs/design/Lab-RunRecords.md`; @@ -104,6 +105,10 @@ local-operations worker may expose queued build/test tickets over a separate pro configuration, reusing the existing heavy-gate scripts. `ci_status` and `stack_plan` remain outside this design. +The daemon lives with the lab tooling on `agents` (`lab/server/`), in a venv like the +code-health one, with its fake-lab tests beside it. App-side slices go on branches from +`lab/tooling` (#107), where the lab channel lives, so this design PR stays documentation only. + ## Resources (read-only, first implementation slice) The app keeps the legacy key/value `Lab/state` for existing scripts and atomically writes a @@ -145,7 +150,9 @@ each app start, so a reset frame sequence cannot be mistaken for a newer frame f process. `frame_seq` increases for every frame accepted by the app's frame cache and `captured_at` is that frame's capture time, not the time the state file was serialized. The daemon computes frame age from `captured_at`; it does not persist a decaying -`last_frame_age_ms` value. +`last_frame_age_ms` value. `program.state` carries the legacy channel's values in lower case +(`stopped`, `running`, `stopping`, `not_ready`, `no_program`), so one app enum feeds both +files. Version 1 distinguishes transport freshness (`live`, `stale`, `missing`) from visual signal classification (`ok`, `black`, `colour_bars`, `unknown`). The first slice can truthfully @@ -200,6 +207,17 @@ daemon restart, leases and request results survive in SQLite; the daemon reconci against `instance.id`, current program state and the latest transition before allowing a mutation. +**Until Slice 7 the existing locks stay in force and the daemon works beside them.** +`labrun.sh` launches the app under `lablock.sh`, and the launcher's train holds that lock for +the app's lifetime; `labctl.sh` and `cfrun.sh` check it before writing a request. From Slice 5 +on, the daemon claims the channel's `owner` token (Lab.md) when it attaches and re-claims it +after every app restart, since that token dies with the app; the app then refuses any +mutating request that does not come from the daemon, which is what makes "raw writes to +`Lab/inbox` are not exposed" hold for stray scripts too. Three gates coexist during the +transition: `lablock` on launching the app, the owner token on writing the channel, the lease +on which agent may act. Slice 7 folds the first into the daemon; the owner token stays as the +app/daemon boundary. + Every mutating call requires a client-generated request UUID. The unique key `(client_id, tool, request_id)` stores the normalized arguments and terminal result. An exact retry returns the stored result; reuse with different arguments is refused. This @@ -282,7 +300,11 @@ means a successful check. - **Agents never run the app once supervision lands.** The server launches it with `--profile lab-B` using the extracted `labrun.sh` behavior (direct exec, never a bare binary) and quits only the PID/instance it created. Before that milestone, it is an - attach-only observer and the existing guarded launcher remains authoritative. + attach-only observer and the existing guarded launcher remains authoritative. Until then + the only guard is the standing rule: no Claude Code hook refuses a Bash call that names the + binary today (checked 2026-11-14). A PreToolUse hook that refuses the binary without + `--profile` and any write into `Lab/inbox` needs nothing from this design and should land + first, as its own change to the agent harness. - **Console A is never exposed.** The server's own allowlist has no entry for it; no resource, no tool. A user-supplied console id cannot become a filesystem path. - **One train.** Leases replace `lablock.sh`; the app still refuses presses while a program @@ -299,9 +321,9 @@ means a successful check. ## Work plan Each slice is independently reviewable and keeps the existing scripts working until its -replacement passes a console-B soak. App slices belong on a new branch from -`design/agent-integration`; changes to this proposal are handed to that PR's owner rather -than pushed onto its branch. +replacement passes a console-B soak. App slices go on branches from `lab/tooling` (#107), +server slices on `agents`; changes to this proposal are handed to #131's owner rather than +pushed onto its branch. ### Slice 0 — freeze contracts and fixtures @@ -327,6 +349,17 @@ Acceptance: native tests cover serialization and transition-source mapping; a ma restarts the app and proves `(instance, seq)` changes; a snapshot requested during a panel or controller fault is either a newer openable PNG or a typed timeout, never an old image. +App-side slices follow the repo's hard rules (AGENTS.md), which this design must not bend: + +- the heartbeat and the `Lab/state.json` writer run on the lab channel's existing worker, no + new `std::thread`; locking through `PokemonAutomation::Mutex`; +- files through `Common/Cpp/Filesystem` (atomic replace is write-beside-then-rename) and the + project's JSON classes; no `std::filesystem`; +- the transition source is set where the session's start/stop is called: the widget passes + `ui`, the lab channel passes `lab` with its request id; widgets hold no state; +- program contracts and the dynamic preflight are program code: Qt-free, declared next to + the program descriptor and instance. + ### Slice 2 — attach-only read server - Implement the loopback Streamable HTTP endpoint, bearer/Origin checks, B-only allowlist, @@ -356,7 +389,8 @@ results, no running-lease expiry and gap-free event catch-up. of the contract. - Add the side-effect-free dynamic preflight interface and structured check results. - Implement it first for FRLG RNG and one AutoStory family, including save provenance and - known-screen checks; missing implementations report `unsupported`. + known-screen checks; missing implementations report `unsupported`. Two PRs: the contract + and preflight interface with FRLG RNG, then the first AutoStory family. Acceptance: native tests cover option normalization, ranges/enums, hidden checkpoint ids and side-effect-free preflight. Deliberately bad Safari, party, language, save and HOME-screen @@ -416,14 +450,34 @@ program family at a time. Retire a script only when its replacement has native/f coverage, a successful console-B session and a documented operator fallback. Do not delete the file channel: it remains the app/daemon compatibility boundary. -## Decisions and remaining boundary +| Slice | Retires once soaked | +|---|---| +| 2 | `checkin.sh`, `lab_status.sh`, `shotB.sh`, `rec_shots.sh`, log greps for verdicts | +| 3 | `switch_guard.sh`, `run_watch.sh`, `watch_run.sh`, `stage_watch.sh`, `full_run_watch.sh`, `plza_watch*.sh` | +| 5 | `stopB.sh`, `run_from_save.sh`, `reload.sh`, `preflight_B.sh`, `stage_run.sh`, `cfB.sh`, `cfrB.sh`; `labctl.sh` and `cfrun.sh` as agent entry points | +| 7 | `labrun.sh`, `lab_panel.sh`, `start_B.sh`, `stage_launch.sh`, `lablock.sh` | +| 8 | `labbuild.sh`, `heavy_gate.sh`, `make_slot_b.sh`, `plza_rebuild.sh` | + +`disk_gc.sh` and the launchd jobs stay, as before. + +## Decisions and open questions + +Settled by the user's rules: + +- User leases end only by explicit operator release. A Start or Stop clicked in the app + window, not sent through the channel, means the user holds console B until they say it is + free (2026-11-13); the `user` lease is that rule in code, and no idle timer ends it. +- Console A is never exposed, and the app is never run without `--profile`. + +Proposed here, for the user to confirm: -- User leases end only by explicit operator release. - Static option/precondition data lives with the program contract; dynamic evidence lives in the program instance. The server orchestrates and caches results but does not duplicate a per-program rules table. - The generic app-side observation/contract seams are candidates for upstream review behind - the existing Lab/Developer boundary. The Python daemon, console inventory, credentials and + the existing Lab/Developer boundary; whether and when to hand them upstream is the user's + call (no agent opens an upstream PR). The Python daemon, console inventory, credentials and local policies remain fork/local tooling. -- Arbitrary game switching, CI diagnosis and stack/worktree planning require separate - designs. They are not hidden follow-ups inside the console server. + +Separate designs, not hidden follow-ups inside the console server: arbitrary game switching, +CI diagnosis and stack/worktree planning. From 9f5fdc7f91af157d695d76d2d4242907f1b8999e Mon Sep 17 00:00:00 2001 From: dayfine Date: Sat, 14 Nov 2026 01:21:26 -0800 Subject: [PATCH 6/7] Design: record the 2026-11-14 decisions (fork-only, declared prechecks, hook done) Fork-only for now for the app seams and the mGBA work, kept small enough to upstream later. Preconditions are a list of named check objects declared with the program (prechecks()/run_prechecks(), generalizing FRLG RNG's run_precheck from upstream PokemonAutomation/Arduino-Source#1881); they land in Slice 4 and the server first refuses runs on them in Slice 5. The require-profile PreToolUse hook is done, so Enforcement describes it instead of planning it. The open questions are listed with a recommended answer each. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_013VUubhHeYSMGSSDXZkKZss --- docs/design/Agent-Integration.md | 183 +++++++++++++++++++++---------- 1 file changed, 124 insertions(+), 59 deletions(-) diff --git a/docs/design/Agent-Integration.md b/docs/design/Agent-Integration.md index 691dcab892..10f5cd1ae2 100644 --- a/docs/design/Agent-Integration.md +++ b/docs/design/Agent-Integration.md @@ -2,16 +2,20 @@ Status: proposal (2026-11-01; revised 2026-11-14 from the week's incidents, `~/BDSPLab/notes/2026-11-13-week-problems.md`, and an implementation review; checked against the -lab scripts and the user's console rules on 2026-11-14). It builds on: +lab scripts and the user's console rules on 2026-11-14; the user's decisions of 2026-11-14 +are recorded in [Decisions and open questions](#decisions-and-open-questions)). It builds on: - the lab channel: fork #107, `SerialPrograms/Source/CommonFramework/Windows/Lab.md` (`Lab/inbox`, `Lab/outbox`, `Lab/state`, the owner token); - run records: fork #116, `docs/design/Lab-RunRecords.md`; - the lab scripts on `agents` (`lab/control`, `lab/scripts`) and in `~/BDSPLab` (`labrun.sh`, `labbuild.sh`, `checkin.sh`, `lablock.sh`, `heavy_gate.sh`, ...). -Nothing here is implemented except what those already have. In particular, the app does -not yet expose a frame sequence, a generic game/screen detector, a preflight API, or an -action source. Those are deliverables below, not existing integration points. +Nothing here is implemented except what those already have, FRLG RNG's one-off +`RngProgramInstance::run_precheck()` (upstream PokemonAutomation/Arduino-Source#1881, which the +precheck list below generalizes) and the `require-profile` Claude Code hook (done, see +Enforcement). In particular, the app does not yet expose a frame sequence, a generic +game/screen detector, a preflight API, or an action source. Those are deliverables below, not +existing integration points. Everything here is fork tooling for now (user, 2026-11-14). ## Requirement @@ -42,19 +46,19 @@ One row per incident from 2026-11-07..13. "R" = resource, "T" = tool, "E" = even | Panel switch reset controller and video ("requires video feedback") | R `console://B/state` `controller`, `video`; `run_program` waits for both | | No-HDMI colour bars went unnoticed until a frame was read | R `state.video.signal`; E `video_lost` | | Game switches (HOME, close, pick tile) done by hand | deferred: arbitrary title selection has no reliable app seam yet | -| PLZA 03.08 started with the game closed and walked on HOME | T `ensure_game(expected)` and the program's dynamic preflight refuse the run | +| PLZA 03.08 started with the game closed and walked on HOME | T `ensure_game(expected)` and the program's `game_open` precheck refuse the run | | UI stop at 22:25 had to be guessed as the user's | E `stopped{source: ui}`; lease moves to `user` | | "Hands off B while the user plays" held only by memory | lease: tools refuse while `user` holds it | | Which save B holds lived in notes (Safari logo vs Mansion door, unsaved Parasect) | R `console://B/save` | | Max Resets 10 on a long-shot target | schema range plus program precondition | | Safari needs ≥ 4000 advances | precondition on `run_program` | | FRLG license check not loaded (Game Settings panel never opened) | precondition: game settings loaded | -| No Sweet Scent user in the last slot | precondition, checked by the program's precheck before any reset | +| No Sweet Scent user in the last slot | a precheck declared with the program, run before any reset | | Restart on a post-run summary screen | precondition "known screen" refuses the run; recovery remains explicit | | PLZA LANGUAGE left on ja | precondition: language matches the console's game | | Dev-only checkpoint hidden without Developer Mode shifted indices; wrong checkpoint started | options by key and checkpoint id, never by index; refuse a hidden row | | Audio input not allowlisted for lab-B (UserSetupError spam) | server launch checks the lab profile's settings | -| App run without `--profile`, twice | agents never run the app; the server launches it | +| App run without `--profile`, twice | done 2026-11-14: the `require-profile` PreToolUse hook refuses the call (Enforcement); from Slice 7 the server launches the app | | Rules retyped per brief and drifting | rules enforced by the server, not restated in briefs | | Agents parked waiting on the console | long console operations return a request/ticket; E `program_finished` | | Linter rule changed mid-task | not addressed by this design | @@ -251,8 +255,8 @@ Each tool validates before it acts and returns JSON, or a refusal naming the fai | `renew_console(lease, ttl)` / `release_console(lease)` | Operate only on the matching token. A user lease is released through an operator-authenticated CLI action, not an agent token. | | `shot(console, max_age_ms=2000, timeout_ms=5000)` | Lease not needed. Records `(instance.id, frame_seq)` at call start and waits for a strictly newer frame. Returns the completed PNG plus the new token and `captured_at`; never reuses a caller tag. Refuses on timeout, stale transport, or a classified bad signal. | | `ensure_game(console, expected, allowed_screens)` | Read-only verification. Runs/reads the named detectors and returns the observation and age. It never navigates HOME or launches a title. | -| `describe_program(console, panel)` | Returns a stable program id, typed option schema, defaults and declared preconditions. Refuses settings panels and hidden/unavailable programs. | -| `preflight_program(console, panel, options)` | Lease required because detectors may use app resources, but performs no controller action. Checks selection, typed options, static facts and the program's dynamic checks; returns every check and evidence. | +| `describe_program(console, panel)` | Returns a stable program id, typed option schema, defaults and the declared prechecks (each check's name and `needs` line) without touching the console. Refuses settings panels and hidden/unavailable programs. | +| `preflight_program(console, panel, options)` | Lease required because detectors may use app resources, but performs no controller action. Checks selection and typed options, then has the app run the program's prechecks in order; returns the checks run, the first failure's name and user message, and evidence. | | `run_program(console, panel, options, preflight_token)` | Lease. Requires a still-current successful preflight bound to app instance, panel, normalized options, observation versions and lease. Starts once and returns request/run ids. | | `stop_program(console)` | Lease. Requests Stop once; success is the matching `last_transition{source: lab, request_id}` or a terminal run event, not merely a stopped snapshot. | | `cfr_batch(console, lines)` | Lease, program stopped. Debug-only allowlisted command batch; marks the save fact stale. Raw writes to `Lab/inbox` are not exposed. | @@ -269,28 +273,42 @@ command should serve this read-only path without requiring a ready controller. ### Program contract and preflight -There is no generic program preflight API today. It is introduced as two layers rather than -simulating a Start/Stop cycle: +There is no generic program preflight API today; FRLG RNG has a one-off, +`RngProgramInstance::run_precheck()` (upstream PokemonAutomation/Arduino-Source#1881: the black +border check, then free party slots). The general form is two layers rather than a simulated +Start/Stop cycle: 1. A static contract beside the program descriptor gives stable option keys, types, - constraints, defaults, visibility and declarative fact requirements. The daemon validates - and normalizes options without touching the console. UI row indices and display strings - are never API identifiers. -2. A virtual dynamic preflight on the program instance checks facts only the program can - establish: screen readers, party layout, loaded game settings, save compatibility, and - similar state. It returns structured checks and must not press a controller button, - reset, save, or alter options. - -The result is a short-lived preflight token hashing the app instance, program id, normalized -options, relevant state/save observation versions and lease. `run_program` refuses a stale -token. Programs without a dynamic implementation declare that explicitly; absence never + constraints, defaults and visibility. The daemon validates and normalizes options without + touching the console. UI row indices and display strings are never API identifiers. +2. Preconditions declared with the program (user decision, 2026-11-14). The program overrides + `prechecks()` and returns a list of small named check objects; each has a `name`, a + one-line `needs` description, and `run(env, context)` returning pass/fail plus a message + for the user. A shared `run_prechecks()` runs the list in order and throws + `UserSetupError` on the first failure; the program's start path calls it before any reset, + so one list guards both a Start clicked in the app and a `run_program` from the daemon. + Shared checks are written once and reused by name: `game_open`, `black_border`, + `free_party_slots(n)`, `save_at(place)`. Program-specific checks (Sweet Scent user in the + last slot, Safari advances) sit beside the program. `describe_program` lists the names and + `needs` lines without touching the console, so what a program needs can be read without + running it. A check must not press a controller button, reset, save, or alter options. + +There is no server-side rules table, and no template/policy form: the user rejected that in +favour of the plain list because it has to be readable without running (2026-11-14). The +list generalizes upstream #1881's `run_precheck()`: its two checks become `black_border` then +`free_party_slots(1)`. Nothing changes in upstream #1881 now; that move is Slice 4's first PR, and the +server first refuses a run on a failed check in Slice 5. + +The result of a preflight is a short-lived token hashing the app instance, program id, +normalized options, relevant state/save observation versions and lease. `run_program` refuses +a stale token. A program with no `prechecks()` override reports `unsupported`; absence never means a successful check. -| Program | Preconditions | +| Program | Prechecks | |---|---| -| FRLG RNG (static, gift, egg, Safari) | a free party slot (gift, egg); Sweet Scent user in the last slot (wild); Safari advances ≥ 4000; Game Settings loaded so the license check is on; game on a known screen, not a summary | -| FRLG / BDSP / PLZA AutoStory | game in front is the program's game; start checkpoint chosen by id and visible (dev-only rows need Developer Mode); `console://B/save` matches the start checkpoint and is not stale; LANGUAGE matches the cartridge | -| Every program | lab profile has audio input allowed or off; controller/video transport live | +| FRLG RNG (static, gift, egg, Safari) | `black_border`; `free_party_slots(1)` (gift, egg; upstream #1881's two checks, in that order); Sweet Scent user in the last slot (wild); Safari advances ≥ 4000; Game Settings loaded so the license check is on; game on a known screen, not a summary | +| FRLG / BDSP / PLZA AutoStory | `game_open` (the program's game is in front); start checkpoint chosen by id and visible (dev-only rows need Developer Mode); `save_at(start checkpoint)` against `console://B/save`, not stale (from Slice 6); LANGUAGE matches the cartridge | +| Every program (server-side, not a program precheck) | lab profile has audio input allowed or off; controller/video transport live | ## Enforcement @@ -300,11 +318,20 @@ means a successful check. - **Agents never run the app once supervision lands.** The server launches it with `--profile lab-B` using the extracted `labrun.sh` behavior (direct exec, never a bare binary) and quits only the PID/instance it created. Before that milestone, it is an - attach-only observer and the existing guarded launcher remains authoritative. Until then - the only guard is the standing rule: no Claude Code hook refuses a Bash call that names the - binary today (checked 2026-11-14). A PreToolUse hook that refuses the binary without - `--profile` and any write into `Lab/inbox` needs nothing from this design and should land - first, as its own change to the agent harness. + attach-only observer and the existing guarded launcher remains authoritative. +- **The bare binary is already blocked (done 2026-11-14, not part of this rollout).** + `~/.claude/hooks/require-profile.py` is a Claude Code PreToolUse hook registered for `Bash` + in `~/.claude/settings.json`. It splits the command on `&&`, `||`, `;`, `|`, `&` and + newlines, drops leading `VAR=value` assignments and the bare wrappers `exec`, `nohup`, + `env`, `time`, `caffeinate`, and looks at each segment's program: a path ending in + `SerialPrograms`, `SerialPrograms.app` or `SerialPrograms.app/Contents/MacOS/SerialPrograms`, + or `open … SerialPrograms.app`. If that segment has no `--profile` argument the hook exits 2 + with a message naming the target and the call is refused. Everything else passes: builds + (`cmake --build … --target SerialPrograms`), `pgrep`, `grep`, `ls`, and a launch with + `--profile `. Checked against those cases on 2026-11-14. Known gaps, not design + items: it does not guard writes into `Lab/inbox` (the owner token covers that from Slice 5), + it sees only commands run through Claude Code's Bash tool, and a wrapper with flags + (`caffeinate -i …`) is not unwrapped. - **Console A is never exposed.** The server's own allowlist has no entry for it; no resource, no tool. A user-supplied console id cannot become a filesystem path. - **One train.** Leases replace `lablock.sh`; the app still refuses presses while a program @@ -357,8 +384,11 @@ App-side slices follow the repo's hard rules (AGENTS.md), which this design must project's JSON classes; no `std::filesystem`; - the transition source is set where the session's start/stop is called: the widget passes `ui`, the lab channel passes `lab` with its request id; widgets hold no state; -- program contracts and the dynamic preflight are program code: Qt-free, declared next to - the program descriptor and instance. +- program contracts and the precheck list are program code: Qt-free, declared next to the + program descriptor and instance; +- the seams stay small enough to upstream later without a rewrite (user, 2026-11-14): behind + the Lab/Developer boundary, no change to the shared scheduler or clock, one connection, one + controller, one video source per app instance. ### Slice 2 — attach-only read server @@ -387,19 +417,26 @@ results, no running-lease expiry and gap-free event catch-up. - Add `describe_program` app support with stable ids and typed options; keep UI concerns out of the contract. -- Add the side-effect-free dynamic preflight interface and structured check results. -- Implement it first for FRLG RNG and one AutoStory family, including save provenance and - known-screen checks; missing implementations report `unsupported`. Two PRs: the contract - and preflight interface with FRLG RNG, then the first AutoStory family. +- Add the precheck list: `prechecks()`, `run_prechecks()`, `UserSetupError`, the shared + checks `game_open`, `black_border`, `free_party_slots(n)`, and structured check results; + `describe_program` lists them. +- Move FRLG RNG's `run_precheck()` (upstream #1881) onto the list as `black_border` then + `free_party_slots(1)`. This is the first change to that code and it waits for this slice. + Then one AutoStory family with its known-screen checks; `save_at(place)` waits for Slice 6's + save facts. Programs without a list report `unsupported`. Two PRs: the contract and the + precheck list with FRLG RNG, then the first AutoStory family. Acceptance: native tests cover option normalization, ranges/enums, hidden checkpoint ids and -side-effect-free preflight. Deliberately bad Safari, party, language, save and HOME-screen -setups are refused before a reset. +side-effect-free prechecks. Deliberately bad Safari, party, language and HOME-screen setups +are refused before a reset (bad saves: Slice 6). ### Slice 5 — leased actions - Implement `preflight_program`, token binding, `run_program`, `stop_program` and the restricted `cfr_batch` over request UUIDs. +- This is where the server first refuses a run on a failed precheck: `preflight_program` + runs the list through the app and `run_program` needs its token. Until then the only + enforcement is the program's own `run_prechecks()` at Start. - Emit the action source at the UI/lab/program/error origin and correlate lab transitions with the request id. Make terminal events authoritative for completion. - Leave `switch_game` out. Do not make selection-by-visible-index part of the protocol. @@ -411,7 +448,7 @@ A console-B soak demonstrates that every mutation has exactly one audit/request ### Slice 6 — save facts and run-record adoption - Add the versioned save hook to the initial AutoStory/RNG programs and mark all unhooked - mutation paths stale with a reason. + mutation paths stale with a reason; add the shared `save_at(place)` precheck on top of it. - Adapt #116 run records instead of defining a competing run format; carry stop source, checkpoint and RNG hit summaries through resources/events. - Copy/link failure media only after publication is complete, with clean run-id paths. @@ -426,8 +463,9 @@ record with the correct source. PID identity checks into reusable guarded operations. - Let the daemon start/stop only the process instance it owns; reconcile crashes and reject a second instance. Preserve an operator escape hatch. -- Only after the soak, update hooks/rules so agents use the server instead of launching the - app or writing `Lab/inbox` directly. +- Only after the soak, tighten the `require-profile` hook (or add a sibling) so agents cannot + launch the app at all, with or without `--profile`, and update the rules so they use the + server instead of writing `Lab/inbox` directly. Acceptance: tests use fake processes/PIDs; manual tests cover stale pidfiles, wrong profile, missing audio permission, crash/restart and refusal to kill an unrelated process. @@ -462,22 +500,49 @@ the file channel: it remains the app/daemon compatibility boundary. ## Decisions and open questions -Settled by the user's rules: - -- User leases end only by explicit operator release. A Start or Stop clicked in the app - window, not sent through the channel, means the user holds console B until they say it is - free (2026-11-13); the `user` lease is that rule in code, and no idle timer ends it. -- Console A is never exposed, and the app is never run without `--profile`. - -Proposed here, for the user to confirm: - -- Static option/precondition data lives with the program contract; dynamic evidence lives in - the program instance. The server orchestrates and caches results but does not duplicate a - per-program rules table. -- The generic app-side observation/contract seams are candidates for upstream review behind - the existing Lab/Developer boundary; whether and when to hand them upstream is the user's - call (no agent opens an upstream PR). The Python daemon, console inventory, credentials and - local policies remain fork/local tooling. +Settled by the user: + +- **User lease (2026-11-13, confirmed 2026-11-14).** A Start or Stop clicked in the app + window, not sent through the lab inbox, hands console B to the user; agents stop until the + user says it is free. No timeout ends it; the `user` lease state is that rule in code. +- **Console A is never exposed, and the app is never run without `--profile`.** The daemon's + one-entry allowlist covers the first; the `require-profile` hook (done 2026-11-14, see + Enforcement) covers the second today and is not a rollout item. +- **Fork-only for now (2026-11-14).** The app's generic seams (`Lab/state.json`, the instance + id, the snapshot command, `describe_program` and the precheck list) and the mGBA + controller/video work (its own design, not in this doc) stay on the fork. They are kept + small enough to upstream later without a rewrite: behind the Lab/Developer boundary, no + change to the shared scheduler or clock, one connection, one controller, one video source. + Upstream is pitched only with results, by the user; no agent opens an upstream PR. The + Python daemon, console inventory, credentials and local policies are fork/local tooling in + any case. +- **Preconditions are declared with the program (2026-11-14).** The list of named check + objects in Program contract and preflight; no server-side rules table and no + template/policy form (rejected: the list must be readable without running). It generalizes + FRLG's `RngProgramInstance::run_precheck()` (upstream PokemonAutomation/Arduino-Source#1881); + that code changes only in Slice 4, and the server first refuses a run on a failed check in + Slice 5. Nothing changes in upstream #1881 now. + +Still open, with the recommended answer in parentheses: + +1. Operator authentication for releasing a `user` lease and for the thin CLI in general: the + user and the agents are the same login on the lab Mac. (A separate operator token file read + only by the CLI's `release-user` command, plus a hook rule that refuses that command from an + agent shell; the rule, not cryptography, is the guard.) +2. Agent lease TTL and renewal cadence for `agent-idle`. (20 minutes, renewed by every tool + call; never applies to `agent-running`.) +3. How the daemon is started before Slice 7 makes it the supervisor. (A launchd agent beside + `com.dayfine.pa-sync`, attach-only; never started from an agent's Bash call.) +4. `runs://` before #116 lands. (Slice 2 ships without it; it arrives in Slice 6 with #116's + format.) +5. Which AutoStory family goes first in Slice 4. (FRLG: it shares the cartridge and the + one-save-slot risk with the RNG programs already on the list.) +6. Check order and the `unknown` save fact: does `game_open` run before `black_border`, and + does `save_at` refuse or warn while `console://B/save` is `unknown`? (`game_open` first, + since a black-border check means nothing on HOME; `save_at` refuses on `unknown` and is not + declared before Slice 6.) +7. The hook's known gaps (wrappers with flags, no `Lab/inbox` guard, Bash tool only). (Leave + them; the owner token in Slice 5 and app supervision in Slice 7 close them structurally.) Separate designs, not hidden follow-ups inside the console server: arbitrary game switching, -CI diagnosis and stack/worktree planning. +CI diagnosis and stack/worktree planning, and the mGBA controller/video work. From 5c32c745cc7194ebe1fd05cffb648a31c30caf55 Mon Sep 17 00:00:00 2001 From: dayfine Date: Sat, 14 Nov 2026 13:23:46 -0800 Subject: [PATCH 7/7] Design: settle the open questions and the implementation review Moves the seven open questions to settled with their recommended answers, writes the nine answers from the 2026-11-14 implementation review into the resources, lease, tool and slice text, corrects the legacy Lab/state fields and what 7c44e72c3 does, and records Slices 0-3 as built (#478, #481, #482, #483) and Slice 4 as approved on #1881's branch. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_013VUubhHeYSMGSSDXZkKZss --- docs/design/Agent-Integration.md | 253 ++++++++++++++++++++++--------- 1 file changed, 184 insertions(+), 69 deletions(-) diff --git a/docs/design/Agent-Integration.md b/docs/design/Agent-Integration.md index 10f5cd1ae2..3add53da77 100644 --- a/docs/design/Agent-Integration.md +++ b/docs/design/Agent-Integration.md @@ -1,6 +1,7 @@ # Agent integration: an MCP lab server -Status: proposal (2026-11-01; revised 2026-11-14 from the week's incidents, +Status: accepted 2026-11-14, Slices 0-3 built (proposed 2026-11-01; revised 2026-11-14 from +the week's incidents, `~/BDSPLab/notes/2026-11-13-week-problems.md`, and an implementation review; checked against the lab scripts and the user's console rules on 2026-11-14; the user's decisions of 2026-11-14 are recorded in [Decisions and open questions](#decisions-and-open-questions)). It builds on: @@ -10,12 +11,21 @@ are recorded in [Decisions and open questions](#decisions-and-open-questions)). - the lab scripts on `agents` (`lab/control`, `lab/scripts`) and in `~/BDSPLab` (`labrun.sh`, `labbuild.sh`, `checkin.sh`, `lablock.sh`, `heavy_gate.sh`, ...). -Nothing here is implemented except what those already have, FRLG RNG's one-off +Before this rollout nothing here was implemented except what those already have, FRLG RNG's one-off `RngProgramInstance::run_precheck()` (upstream PokemonAutomation/Arduino-Source#1881, which the precheck list below generalizes) and the `require-profile` Claude Code hook (done, see -Enforcement). In particular, the app does not yet expose a frame sequence, a generic +Enforcement). In particular, the app did not expose a frame sequence, a generic game/screen detector, a preflight API, or an action source. Those are deliverables below, not -existing integration points. Everything here is fork tooling for now (user, 2026-11-14). +pre-existing integration points. Everything here is fork tooling for now (user, 2026-11-14). + +Build status (2026-11-14): Slices 0-3 are built as fork drafts: #478 (Slice 0, contracts and +fixtures, base `agents`, tip ff209b333), #481 (Slice 1, `Lab/state.json` and the snapshot +command, base `lab/tooling`, tip 629bfde88), #482 (Slice 2, read server, on #478, tip +cbfccc2ac) and #483 (Slice 3, control plane, on #482, tip c9b45d06e). Slice 4 is +approved to start, stacked on upstream #1881's branch `upstream/frlg-free-party-slots`. Draft +#476 (`LabStateV1`, commit 7c44e72c3) adds a serializer and its tests only; it does not modify +`LabChannel` or tag transitions, and its second `ProgramState` enum is to be dropped (see +Resources). ## Requirement @@ -115,9 +125,13 @@ code-health one, with its fake-lab tests beside it. App-side slices go on branch ## Resources (read-only, first implementation slice) -The app keeps the legacy key/value `Lab/state` for existing scripts and atomically writes a -new versioned `Lab/state.json`. The daemon rejects an unsupported `schema` and overlays only -server-owned data such as the lease. Example app file: +The app keeps the legacy key/value `Lab/state` (`seq`, `state`, `program`, `controller`, +`owner`, `error`) byte-compatible for existing scripts and atomically writes a new versioned +`Lab/state.json`. The C++ side only serializes. The Python daemon owns strict parsing: it +rejects an unsupported `schema`, keeps unknown fields that matter for diagnostics, and +overlays only server-owned data such as the lease. Shared golden JSON fixtures (Slice 0, #478) +are the cross-language contract: the app's writer is tested against them byte for byte and the +daemon's parser against the same files. Example app file: ```json { @@ -151,12 +165,19 @@ server-owned data such as the lease. Example app file: `written_at` is a heartbeat even when no logical field changes. `instance.id` changes on each app start, so a reset frame sequence cannot be mistaken for a newer frame from the old -process. `frame_seq` increases for every frame accepted by the app's frame cache and -`captured_at` is that frame's capture time, not the time the state file was serialized. +process. `frame_seq` is one process-wide monotonic counter: it increases for every frame the +app receives and never resets on a panel switch or a camera auto-reset, so frames order by +`(instance.id, frame_seq)` with no stream epoch. `captured_at` is that frame's capture time, +not the time the state file was serialized. The daemon computes frame age from `captured_at`; it does not persist a decaying -`last_frame_age_ms` value. `program.state` carries the legacy channel's values in lower case -(`stopped`, `running`, `stopping`, `not_ready`, `no_program`), so one app enum feeds both -files. +`last_frame_age_ms` value. `program.state` reuses the app's one `ProgramState` enum, mapped +once to the legacy upper-case values and once to lower-case JSON values (`stopped`, +`running`, `stopping`, `not_ready`, `no_program`); no second state enum. +`program.last_transition` is an object, or `null` before the first transition. + +Publication: a logical change (panel, program state, transition, controller, video state) +rewrites `Lab/state.json` immediately; heartbeat-only and frame-only changes (`written_at`, +`frame_seq`, `captured_at`) are throttled to one write every 1-2 s. Version 1 distinguishes transport freshness (`live`, `stale`, `missing`) from visual signal classification (`ok`, `black`, `colour_bars`, `unknown`). The first slice can truthfully @@ -167,8 +188,10 @@ confidence fields. `program.last_transition.source` is emitted where the action originates, not inferred from the resulting state: `ui`, `lab`, `program`, or `error`. A lab action carries the daemon's -request UUID. This is how the daemon distinguishes a user's Stop click from its own Stop -request even if both race. +request UUID. `ProgramSession`'s start and stop take an `ActionSource` and a request id, and the +session latches the stop's initiator from `STOPPING` through `STOPPED`, so a later event cannot +overwrite who stopped the run. This is how the daemon distinguishes a user's Stop click from +its own Stop request even if both race. **`console:///state`** is the parsed app state plus: @@ -198,15 +221,18 @@ Reads need no lease. Every tool that can press a button, start/stop a program or selection requires a live lease token, not merely the caller-supplied owner name. The SQLite state machine is: -| State | Entry | Exit | +| State | Entry | Exit → target | |---|---|---| -| `free` | initial state or explicit release | successful `acquire_console` | -| `agent-idle` | acquired with bounded TTL | renew, explicit release, TTL expiry, run start, or observed UI action | -| `agent-running` | the leased caller starts a run | program terminal event, explicit leased stop, or observed UI action; it never expires mid-run | -| `user` | app reports a UI-originated Start/Stop or the operator explicitly claims B | explicit operator release only | +| `free` | initial state or explicit release | successful `acquire_console` → `agent-idle`; observed UI action or operator claim → `user` | +| `agent-idle` | acquired with a 20-minute TTL, renewed by every tool call | renew → `agent-idle`; explicit release or TTL expiry → `free`; run start → `agent-running`; observed UI action → `user` | +| `agent-running` | the leased caller starts a run | natural terminal event or lab-requested stop (`stop_program`) → `free`; UI-originated transition → `user`; it never expires mid-run | +| `user` | app reports a UI-originated Start/Stop or the operator explicitly claims B | explicit operator release only → `free` | A UI action revokes any agent lease before a user lease is installed. There is no guessed -30-minute user timeout: absence of UI telemetry is not proof that the user is done. On +30-minute user timeout: absence of UI telemetry is not proof that the user is done. The +operator releases a `user` lease with the thin CLI's `release-user` command, which reads a +separate operator token file; a hook rule refuses that command from an agent shell. The user +and the agents share one login on the lab Mac, so the rule, not cryptography, is the guard. On daemon restart, leases and request results survive in SQLite; the daemon reconciles them against `instance.id`, current program state and the latest transition before allowing a mutation. @@ -253,11 +279,11 @@ Each tool validates before it acts and returns JSON, or a refusal naming the fai |---|---| | `acquire_console(console, owner, ttl)` | Atomically acquires `free`; returns an opaque lease token and expiry. Owner is descriptive, not authority. | | `renew_console(lease, ttl)` / `release_console(lease)` | Operate only on the matching token. A user lease is released through an operator-authenticated CLI action, not an agent token. | -| `shot(console, max_age_ms=2000, timeout_ms=5000)` | Lease not needed. Records `(instance.id, frame_seq)` at call start and waits for a strictly newer frame. Returns the completed PNG plus the new token and `captured_at`; never reuses a caller tag. Refuses on timeout, stale transport, or a classified bad signal. | +| `shot(console, max_age_ms=2000, timeout_ms=5000)` | Lease not needed. Records `(instance.id, frame_seq)` at call start and waits for a strictly newer frame from the app's video-session snapshot seam. Returns the completed PNG plus the new token and `captured_at`; never reuses a caller tag. Refuses on timeout, stale transport, or a classified bad signal. | | `ensure_game(console, expected, allowed_screens)` | Read-only verification. Runs/reads the named detectors and returns the observation and age. It never navigates HOME or launches a title. | -| `describe_program(console, panel)` | Returns a stable program id, typed option schema, defaults and the declared prechecks (each check's name and `needs` line) without touching the console. Refuses settings panels and hidden/unavailable programs. | -| `preflight_program(console, panel, options)` | Lease required because detectors may use app resources, but performs no controller action. Checks selection and typed options, then has the app run the program's prechecks in order; returns the checks run, the first failure's name and user message, and evidence. | -| `run_program(console, panel, options, preflight_token)` | Lease. Requires a still-current successful preflight bound to app instance, panel, normalized options, observation versions and lease. Starts once and returns request/run ids. | +| `describe_program(console, panel)` | Returns a stable program id and only the stable parts of the option schema: keys, types, defaults, bounds and which enum entries are available, plus the declared prechecks (each check's name and `needs` line), without touching the console. Dynamic visibility is not part of it. Refuses settings panels and hidden/unavailable programs. | +| `preflight_program(console, panel, options)` | Lease required because detectors may use app resources, but performs no controller action. Rejects unknown keys and disabled values, stages the candidate options, evaluates dynamic visibility and `check_validity`, then has the app run the program's prechecks in order; always restores the prior options. Returns the checks run, the first failure's name and user message, and evidence. | +| `run_program(console, panel, options, preflight_token)` | Lease. Requires a still-current successful preflight bound to app instance, panel, the candidate options' hash, observation versions and lease. Starts once and returns request/run ids. | | `stop_program(console)` | Lease. Requests Stop once; success is the matching `last_transition{source: lab, request_id}` or a terminal run event, not merely a stopped snapshot. | | `cfr_batch(console, lines)` | Lease, program stopped. Debug-only allowlisted command batch; marks the save fact stale. Raw writes to `Lab/inbox` are not exposed. | @@ -268,8 +294,19 @@ Until then, `ensure_game` turns the dangerous case into an early refusal and lea to the operator. The `shot` freshness rule closes both old races: its token is newer than call start, and the -daemon responds only after the atomically published PNG can be opened. An app-side snapshot -command should serve this read-only path without requiring a ready controller. +daemon responds only after the atomically published PNG can be opened. + +The snapshot is a direct video-session seam: it reads frames from the panel's `VideoSession` +frame listener, never constructs `LabConsole`, and needs no owner token, program state or +controller. It must not wait behind the sequential `Lab/inbox` worker, so a 5-second `shot` +is never blocked by a 30-second controller wait or a long CFR batch. As built in #481 +(629bfde88), the `snapshot LABEL [TIMEOUT_MS]` channel command already has the first half: it +waits on that frame listener for the first frame newer than the line (100-60000 ms, default +5000), saves `_