From 53ef113c4dd0132777be5c9c3dae45ed0b17b276 Mon Sep 17 00:00:00 2001 From: dayfine Date: Sun, 15 Nov 2026 17:00:02 -0800 Subject: [PATCH 1/7] pa-lab: draft user pages for the planned features --- lab/server/GUIDE.md | 17 +++++++------ lab/server/README.md | 7 ++++++ lab/server/guide/actions.md | 37 ++++++++++++++++++++++++++++ lab/server/guide/app-lifecycle.md | 16 +++++++++++++ lab/server/guide/saves.md | 20 ++++++++++++++++ lab/server/guide/start-checks.md | 40 +++++++++++++++++++++++++++++++ lab/server/guide/worker.md | 14 +++++++++++ 7 files changed, 144 insertions(+), 7 deletions(-) create mode 100644 lab/server/guide/actions.md create mode 100644 lab/server/guide/app-lifecycle.md create mode 100644 lab/server/guide/saves.md create mode 100644 lab/server/guide/start-checks.md create mode 100644 lab/server/guide/worker.md diff --git a/lab/server/GUIDE.md b/lab/server/GUIDE.md index 6cc720690b..209aa05f97 100644 --- a/lab/server/GUIDE.md +++ b/lab/server/GUIDE.md @@ -55,13 +55,16 @@ retried request safe, and an event log. None of it presses buttons yet. ## What comes next -| Slice | What you would get | -|---|---| -| 4 (rest) | FRLG AutoStory's start checks listed and run before any press (fork #488) | -| 5 | Leased actions: an agent asks to run or stop a program; the server checks the program's start checks first and refuses on a failed one, and refuses whenever you hold the console. Every action gets exactly one record. | -| 6 | Save facts: the server knows which save is on the console and refuses a run that would start from the wrong one; run records become readable | -| 7 | The server launches and supervises the lab app itself, so a build always starts with the right profile | -| 8 | Optional: a local worker for routine operations | +Each feature has a user page, marked **DRAFT** until it is built and switched on. The drafts +are updated as each piece lands. + +| Slice | What you would get | Page | +|---|---|---| +| 4 (rest) | Every program lists what it needs before it starts; FRLG AutoStory's list is next (fork #488) | [start-checks.md](guide/start-checks.md) | +| 5 | Leased actions: an agent asks to run or stop a program; the server refuses whenever you hold the console, and (5b) when a start check fails. Every action gets exactly one record. 5a is built in #592, not switched on. | [actions.md](guide/actions.md) | +| 6 | Save facts: the server knows which save is on the console and refuses a run that would start from the wrong one; run records become readable | [saves.md](guide/saves.md) | +| 7 | The server launches and supervises the lab app itself, so a build always starts with the right profile | [app-lifecycle.md](guide/app-lifecycle.md) | +| 8 | Optional: a local worker for builds and test runs | [worker.md](guide/worker.md) | ## Limits diff --git a/lab/server/README.md b/lab/server/README.md index 111f461e8d..1d550342cf 100644 --- a/lab/server/README.md +++ b/lab/server/README.md @@ -293,3 +293,10 @@ every event after an active cursor (one used in the last 7 days), every event of and every pending record; finished records go after 7 days. A cursor behind the compacted range is refused (`invalid_arguments`, field `after_seq`, `evidence.compacted_through`), never served with a silent gap. + +## User pages + +`GUIDE.md` is the user-facing overview; `guide/*.md` has one page per feature, marked DRAFT +until it is built and switched on. A PR that builds or switches on part of a feature updates +its page in the same PR: the status line at the top, the tool tables, and the DRAFT banner when +the feature is done. diff --git a/lab/server/guide/actions.md b/lab/server/guide/actions.md new file mode 100644 index 0000000000..a986f0e5f8 --- /dev/null +++ b/lab/server/guide/actions.md @@ -0,0 +1,37 @@ +> **DRAFT.** This page describes a feature that is only partly built. It is updated as each piece lands. +> Status (2026-11-15): Slice 5a (`run_program`, `stop_program` under a lease) is built in [#592](https://github.com/dayfine/Arduino-Source/pull/592), not merged and not switched on. Slice 5b (preflight, options, `cfr_batch`, a console soak) is not built. + +# Actions: agents start and stop programs through the server + +Today agents act on console B through lab scripts. With actions, an agent asks pa-lab instead, +and the server decides whether it may. + +## How it works for you + +- **You always win.** Clicking Start or Stop in the app gives you the console. Any agent + request in flight fails, and new ones are refused until you release it. +- **An agent must hold the console first.** It takes a time-limited lease; without one, or + after it expires, every action is refused. +- **One action, one record.** Each request has an id. Retrying the same id returns the first + answer and never starts or stops twice. Every action is in the event log. +- **Refusals say why**, for example: you hold the console; another program is running; the app + is not running; the selected panel is a different program; the controller is not ready. + +## Tools + +| Tool | What it does | Status | +|---|---|---| +| `acquire_console` / `renew_console` / `release_console` | take, extend and give back the lease | built (control plane) | +| `run_program(panel)` | start the selected program once | built in #592, not switched on | +| `stop_program()` | stop it once; stopping when nothing runs is a no-op, not an error | built in #592, not switched on | +| `preflight_program(panel, options)` | load candidate options, run the program's [start checks](start-checks.md), then put your options back; `run_program` will need its token | planned (5b) | +| options on `run_program` | run with options other than the panel's current ones | planned (5b) | +| `cfr_batch(lines)` | a short allowlisted controller batch, for debugging only; marks the save as unknown | planned (5b) | + +There is deliberately no "switch game" tool: the app cannot yet pick a game on HOME by name. + +## Switching it on + +The session connection (`.mcp.json`) stays read-only until you decide otherwise. The plan: +5b adds preflight; a soak on console B shows every action has exactly one record; then actions +are enabled one program family at a time. diff --git a/lab/server/guide/app-lifecycle.md b/lab/server/guide/app-lifecycle.md new file mode 100644 index 0000000000..3f2c8a6d04 --- /dev/null +++ b/lab/server/guide/app-lifecycle.md @@ -0,0 +1,16 @@ +> **DRAFT.** This page describes a feature that is not built yet. It is updated as it lands. +> Status (2026-11-15): planned (Slice 7). Today `labrun.sh` launches the lab app. + +# The server runs the lab app + +## What you will get + +- **The right build, the right way, every time.** The server launches the lab app itself, with + the lab profile, the audio settings and the build you picked, and refuses anything else + (no more running a build without `--profile`). +- **Crash handling.** It notices a crash or a hung app, records it, and refuses to start a + second copy. It only ever stops the app it started. +- **You keep an escape hatch.** You can still launch or quit the app yourself; the server sees + that and stands back. +- **Agents stop launching apps directly.** After a soak, agents ask the server instead of + running `labrun.sh` or writing to the app's inbox. diff --git a/lab/server/guide/saves.md b/lab/server/guide/saves.md new file mode 100644 index 0000000000..17ae617815 --- /dev/null +++ b/lab/server/guide/saves.md @@ -0,0 +1,20 @@ +> **DRAFT.** This page describes a feature that is not built yet. It is updated as it lands. +> Status (2026-11-15): planned (Slice 6). `console://B/save` exists and usually says `unknown`. + +# Save facts: knowing which save is on the console + +Most lab mistakes that cost real progress are about the save: starting a checkpoint run from +the wrong save, or a hand session moving the player after the last save. + +## What you will get + +- **The server knows the save.** When a program saves (AutoStory and RNG programs first), it + records where and when. `console://B/save` then shows `known`, with the checkpoint or place. +- **It notices when that stops being true.** Anything that can change the game without a + recorded save (you playing, a controller batch, an unhooked program) marks the save + `stale`, with the reason. Only another recorded save makes it `known` again. +- **Runs check it first.** The `save_at(place)` [start check](start-checks.md) refuses a run + whose start does not match the known save. +- **Run records you can read.** Each run ends as one record (passed, failed, or stopped and by + whom), with checkpoint results, RNG hits and links to failure screenshots, shown through the + server's resources and events. diff --git a/lab/server/guide/start-checks.md b/lab/server/guide/start-checks.md new file mode 100644 index 0000000000..7073024119 --- /dev/null +++ b/lab/server/guide/start-checks.md @@ -0,0 +1,40 @@ +> **DRAFT.** This page describes a feature that is only partly built. It is updated as each piece lands. +> Status (2026-11-15): the check list and `describe_program` are built ([#485](https://github.com/dayfine/Arduino-Source/pull/485), [#511](https://github.com/dayfine/Arduino-Source/pull/511), [#512](https://github.com/dayfine/Arduino-Source/pull/512)); FRLG AutoStory's list is a design note ([#488](https://github.com/dayfine/Arduino-Source/pull/488)); the server does not yet run the checks before a start (that is [actions.md](actions.md), Slice 5b). + +# Start checks: what a program needs before it presses anything + +Every program can declare a short, named list of things that must be true before it starts: +the game is in front, the screen has no black border, a party slot is free, the save is the +one the program expects. Each check has a name and a one-line "needs" sentence, and none of +them presses a button, resets, saves or changes an option. + +## What you get + +- **Read what a program needs without running it.** Ask for `describe_program` on a panel + and you get its options (keys, types, defaults, allowed values) and its start checks with + their "needs" lines. +- **One list guards both doors.** The same list runs when you click Start in the app and when + an agent asks the server to start a run, so a bad setup is refused the same way either way, + before any reset. +- **Clear refusals.** The first failing check stops the start with a message saying which + check failed and what it needs. + +## Checks you will see + +| Check | Needs | +|---|---| +| `game_open` | the program's game is in front, not HOME, user select or a black screen | +| `black_border` | the picture fills the screen (no black border) | +| `free_party_slots(n)` | at least n empty party slots (FRLG RNG gift/egg) | +| `save_at(place)` | the save on the console is the one this run starts from (needs [saves.md](saves.md)) | + +Program-specific checks sit beside the program: for example a Sweet Scent user in the last +slot, or enough Safari advances. A program without a list says `unsupported`; that never +counts as passing. + +## Planned lists + +| Program | Checks | +|---|---| +| FRLG RNG | `black_border`, then `free_party_slots(1)` (gift, egg); Sweet Scent user last (wild); Safari advances ≥ 4000 | +| FRLG / BDSP / PLZA AutoStory | `game_open`; the start checkpoint is chosen and visible; `save_at(start checkpoint)`; LANGUAGE matches the game | diff --git a/lab/server/guide/worker.md b/lab/server/guide/worker.md new file mode 100644 index 0000000000..8255d0b26e --- /dev/null +++ b/lab/server/guide/worker.md @@ -0,0 +1,14 @@ +> **DRAFT.** This page describes an optional feature that is not built yet. It is updated if it lands. +> Status (2026-11-15): planned (Slice 8), optional. + +# Local worker for routine operations + +A separate process that runs heavy local jobs (builds, test runs) as queued tickets, with the +existing guards: the heavy-work gate, the disk floor, `-j4` and `nice`. + +## What you will get + +- Builds and tests queue instead of colliding; refusals say why (disk low, memory, a console + run in progress). +- A console run is never slowed down or stopped by a build. +- It does not touch GitHub, PRs or worktrees, and the existing daily cleanup jobs stay. From fd83d7d4ea041753aa4298a2917b1ba400750b83 Mon Sep 17 00:00:00 2001 From: dayfine Date: Sun, 15 Nov 2026 17:00:58 -0800 Subject: [PATCH 2/7] pa-lab user pages: an example request and answer for each tool --- lab/server/GUIDE.md | 29 +++++++++++++++++ lab/server/guide/actions.md | 54 +++++++++++++++++++++++++++++++ lab/server/guide/app-lifecycle.md | 11 +++++++ lab/server/guide/saves.md | 19 +++++++++++ lab/server/guide/start-checks.md | 18 +++++++++++ lab/server/guide/worker.md | 6 ++++ 6 files changed, 137 insertions(+) diff --git a/lab/server/GUIDE.md b/lab/server/GUIDE.md index 209aa05f97..b6e333f0a8 100644 --- a/lab/server/GUIDE.md +++ b/lab/server/GUIDE.md @@ -31,6 +31,35 @@ check-ins). It returns: | `newest_failure_frame`, `newest_screenshot` | Paths to the latest failure frame and screenshot | | `log_tail` | The last log lines (how many is a parameter) | +Example. Ask in a session: *"what's console B doing?"* Claude calls +`console_status(console="B", log_lines=3)` and gets (real output from 2026-11-15, trimmed): + +```json +{ + "console": "B", + "app": {"pid": 87625, "deployment": "slot-B-plza/f0caeb52e879-b440bb4e"}, + "lab_state": {"state": "STOPPING", "program": "PokemonSV:AutoStory", "controller": "ready", "error": ""}, + "console_state": { + "availability": {"state": "available", "reason": null}, + "video": {"state": "live", "frame_seq": 7445}, + "lease": {"state": "user", "owner": "user", "acquired_at": "2026-11-16T00:00:51.929Z"}, + "blocked": [ + {"code": "console_held_by_user", "message": "The user clicked Stop in the app at 00:00; console B is theirs until they release it."}, + {"code": "program_running", "message": "PokemonSV:AutoStory is stopping."} + ] + }, + "program_state": "2026-11-15 16:00:51.929720 - [Program]: Program State: STOPPING", + "last_run": ["... [Checkpoint] 18.02: passed on attempt 1 (one-shot)"], + "last_press": "16:00:21", + "keepawake": "15:57:35 idle 189s: pressed B through Lab inbox (NO_PROGRAM)", + "newest_failure_frame": ".../Screenshots/AutoStory/failures/20261115-151031611644-by_pad_to_bottom_right.png", + "log_tail": ["...", "...", "..."] +} +``` + +With the app closed, `app` is `null` and `blocked` says +`{"code": "app_unavailable", "message": "The app that wrote Lab/state.json (pid 96761) is not running."}`. + **Two resources**, for clients that read resources instead of calling tools: `console://B/state` (the same console state) and `console://B/save` (what is known about the save on the console; today usually `unknown`, see Slice 6 below). diff --git a/lab/server/guide/actions.md b/lab/server/guide/actions.md index a986f0e5f8..8f9f0bba9f 100644 --- a/lab/server/guide/actions.md +++ b/lab/server/guide/actions.md @@ -30,6 +30,60 @@ and the server decides whether it may. There is deliberately no "switch game" tool: the app cannot yet pick a game on HOME by name. +## Examples + +These are the request and answer shapes from the server's contract files +(`contracts/v1/`), with example ids. They are what an agent sends and gets back; you would +ask Claude in plain words ("take console B and run FRLG 13.01 to 13.02"). + +**Take the console.** `acquire_console(console="B", owner="agent:frlg-lane", ttl_s=1200)`: + +```json +{"lease_token": "", "expires_at": "2026-11-14T06:40:00.000Z"} +``` + +If you hold it, the answer is a refusal instead: + +```json +{"request_id": "5f0c2a8e-...", "tool": "acquire_console", + "refusal": {"code": "console_held_by_user", + "message": "The user holds console B (Stop clicked in the app at 06:10); ask them to release it."}} +``` + +**Start the selected program** (5a: the panel's current options): +`run_program(console="B", request_id="9b1d7e44-...", lease_token=..., panel="PokemonFRLG:AutoStory")`. +The stored record once the app has started it: + +```json +{"request_id": "9b1d7e44-...", "tool": "run_program", "status": "completed", + "created_at": "2026-11-14T06:20:00.000Z", "completed_at": "2026-11-14T06:20:02.150Z", + "result": {"run_id": null, "started_at": "2026-11-14T06:20:02.100Z", + "instance_id": "d8ea6a9d-..."}, + "refusal": null} +``` + +Sending the same `request_id` again returns this same record and starts nothing. Refusal codes +you can get: `console_held_by_user`, `console_held_by_agent`, `program_running`, +`app_not_running`, `panel_not_program`, `channel_owned`, `invalid_arguments`. + +**Stop it.** `stop_program(console="B", request_id=..., lease_token=...)` answers with who +stopped it and when, e.g. `{"sent": true, "stopped_at": "...", "stopped_by": "lab"}`. If nothing +was running: `{"sent": false, "state": "stopped", ...}`, which is not an error. + +**Preflight (planned, 5b).** `preflight_program(console="B", panel="PokemonFRLG:RngStatic", options={...})` +returns a token for `run_program`, or the first failed start check: + +```json +{"tool": "preflight_program", + "refusal": {"code": "precheck_failed", "check": "free_party_slots", + "message": "Free party slots: needs 1 empty slot, found 0.", + "evidence": {"needed": 1, "found": 0}}} +``` + +**Controller batch (planned, 5b, debugging only).** +`cfr_batch(console="B", lines=["shot before", "press B 160 640", "shot after"])` runs the lines +with the program stopped and marks the save `stale`. + ## Switching it on The session connection (`.mcp.json`) stays read-only until you decide otherwise. The plan: diff --git a/lab/server/guide/app-lifecycle.md b/lab/server/guide/app-lifecycle.md index 3f2c8a6d04..f89abe47b2 100644 --- a/lab/server/guide/app-lifecycle.md +++ b/lab/server/guide/app-lifecycle.md @@ -14,3 +14,14 @@ that and stands back. - **Agents stop launching apps directly.** After a soak, agents ask the server instead of running `labrun.sh` or writing to the app's inbox. + +## Example (sketch, not built) + +*"Start the lab app on the newest FRLG build."* The server answers with what it launched: + +```json +{"pid": 13122, "game": "frlg", "commit": "72f365065", "profile": "lab-B", + "deployment": "slot-B-frlg/72f3650657d2-8bf901e7"} +``` + +or refuses, e.g. `{"refusal": {"code": "app_running_program", "message": "PokemonFRLG:AutoStory is running; stop it first."}}`. diff --git a/lab/server/guide/saves.md b/lab/server/guide/saves.md index 17ae617815..039439a122 100644 --- a/lab/server/guide/saves.md +++ b/lab/server/guide/saves.md @@ -18,3 +18,22 @@ the wrong save, or a hand session moving the player after the last save. - **Run records you can read.** Each run ends as one record (passed, failed, or stopped and by whom), with checkpoint results, RNG hits and links to failure screenshots, shown through the server's resources and events. + +## Example (sketch, not built) + +Read `console://B/save` after an AutoStory checkpoint saved: + +```json +{"state": "known", "fact": {"program": "PokemonFRLG:AutoStory", "checkpoint": "18.02", + "saved_at": "2026-11-15T23:44:13.115Z"}} +``` + +After you play by hand or a controller batch runs: + +```json +{"state": "stale", "reason": "user input after the last recorded save (2026-11-15T23:50:02Z)", + "fact": {"checkpoint": "18.02", "saved_at": "2026-11-15T23:44:13.115Z"}} +``` + +A run whose start does not match then gets +`{"refusal": {"code": "precheck_failed", "check": "save_at", "message": "The save on console B is stale ..."}}`. diff --git a/lab/server/guide/start-checks.md b/lab/server/guide/start-checks.md index 7073024119..41ea554e01 100644 --- a/lab/server/guide/start-checks.md +++ b/lab/server/guide/start-checks.md @@ -19,6 +19,24 @@ them presses a button, resets, saves or changes an option. - **Clear refusals.** The first failing check stops the start with a message saying which check failed and what it needs. +## Example + +Ask: *"what does FRLG Static RNG need before it starts?"* Claude calls +`describe_program(console="B", panel="PokemonFRLG:RngStatic")`. The answer lists the options +and the checks; the check part looks like this (shape sketch, field names may change): + +```json +{"program": "PokemonFRLG:RngStatic", + "prechecks": [ + {"name": "black_border", "needs": "The picture fills the screen: no black border."}, + {"name": "free_party_slots", "needs": "At least 1 empty party slot."} + ]} +``` + +A program with no list answers `"prechecks": "unsupported"`. When a check fails at Start, the +program stops before any reset with a message like +`Free party slots: needs 1 empty slot, found 0.` + ## Checks you will see | Check | Needs | diff --git a/lab/server/guide/worker.md b/lab/server/guide/worker.md index 8255d0b26e..44caf658e1 100644 --- a/lab/server/guide/worker.md +++ b/lab/server/guide/worker.md @@ -12,3 +12,9 @@ existing guards: the heavy-work gate, the disk floor, `-j4` and `nice`. run in progress). - A console run is never slowed down or stopped by a build. - It does not touch GitHub, PRs or worktrees, and the existing daily cleanup jobs stay. + +## Example (sketch, not built) + +*"Build the PLZA lab tree."* → `{"ticket": 41, "state": "queued", "ahead": 1}`; later +`{"ticket": 41, "state": "done", "exit": 0, "log": "~/BDSPLab/build-plza.log"}`, or +`{"ticket": 41, "state": "refused", "reason": "console run in progress"}`. From deffd78a73880fdaab74a7ead6d4f023537350e7 Mon Sep 17 00:00:00 2001 From: dayfine Date: Sun, 15 Nov 2026 17:15:57 -0800 Subject: [PATCH 3/7] Code health: param_comment and test_only_api advisories; agent rules on vacuous tests param_comments.py reports uncommented literal arguments in calls of 3+ arguments; exposure.py reports public header names that only tests or their own .cpp use. Both advisory, with tags. autostory-code and reviewer get the three review rules from fork #590. --- .agents/agents/autostory-code.md | 7 + .agents/agents/reviewer.md | 6 + devtools/code_health/README.md | 19 +- devtools/code_health/__main__.py | 7 +- devtools/code_health/allow.py | 6 +- devtools/code_health/exposure.py | 118 ++++++++++++ devtools/code_health/param_comments.py | 174 ++++++++++++++++++ devtools/code_health/tests/test_exposure.py | 102 ++++++++++ .../code_health/tests/test_param_comments.py | 107 +++++++++++ 9 files changed, 540 insertions(+), 6 deletions(-) create mode 100644 devtools/code_health/exposure.py create mode 100644 devtools/code_health/param_comments.py create mode 100644 devtools/code_health/tests/test_exposure.py create mode 100644 devtools/code_health/tests/test_param_comments.py diff --git a/.agents/agents/autostory-code.md b/.agents/agents/autostory-code.md index 23443bbf25..b057d80034 100644 --- a/.agents/agents/autostory-code.md +++ b/.agents/agents/autostory-code.md @@ -66,6 +66,13 @@ being restated. expectation, the name not inside the input; `std::string_view` by value for read-only string params; `std::optional`, not sentinels like `""` or `-1`; pixel readers in `Detectors/` read only and press nothing, presses go in `Actions/`. + - No vacuous tests: a test that only restates the code's arithmetic or constants, or passes for + any implementation, is not added. Test behavior a caller relies on. + - Don't widen a header to test internals: test through the public function, or keep the helper + in the .cpp's anonymous namespace (code health `test_only_api`). + - A literal argument in a call of 3+ arguments gets a parameter comment: + `pbf_press_button(context, BUTTON_A, /*hold=*/160ms, /*release=*/400ms)` (code health + `param_comment`). - **Commit early, finish in one turn.** Commit and push as soon as the scoped checks pass; CI runs the full build and tests after the push. Then, in the foreground and without ending the turn: verify → commit → push → open the PR → confirm the PR's file list. Never end with an unpushed commit. diff --git a/.agents/agents/reviewer.md b/.agents/agents/reviewer.md index 26cd832ec0..f07bc55408 100644 --- a/.agents/agents/reviewer.md +++ b/.agents/agents/reviewer.md @@ -52,6 +52,12 @@ being restated. name not inside the input; `std::string_view` by value for read-only string params; `std::optional`, not sentinels like `""` or `-1`; pixel readers in `Detectors/` read only and press nothing, presses go in `Actions/`; new code in its final file and module; line endings kept. + Flag vacuous tests (they only restate the code's arithmetic or constants, or pass for any + implementation): ask for a test of behavior a caller relies on, or none. Flag a header widened + to test internals: test through the public function, or keep the helper in the .cpp's anonymous + namespace. Flag literal arguments without a parameter comment in calls of 3+ arguments + (`/*hold=*/160ms`). Code health reports the last two as `test_only_api` and `param_comment` + advisories; read them, they don't fail CI. - **Time limit.** At the wall-clock limit, post the findings so far with verdict `NEEDS_REWORK`, say the review is partial, and report. - **Review delivery.** Line 1 is `Reviewed SHA: `; `## Verdict` sits on its own line, diff --git a/devtools/code_health/README.md b/devtools/code_health/README.md index ec3667c778..361fe606c4 100644 --- a/devtools/code_health/README.md +++ b/devtools/code_health/README.md @@ -95,14 +95,15 @@ void load_move_table(){ `// code-health: allow [,] -- ` lets the listed metrics of one function or class exceed their limit. Checks are the metric names in the table above: `file_lines`, `function_lines`, `nesting`, `parameters`, `public_methods`, `public_data`, -`single_method_interface`, `header_symbols`. Four style findings take a tag too: `char_pointer`, -`overload_prefix`, `out_param_order` and `string_concat`, for a C API or a fixed interface that needs them. Other style and layering +`single_method_interface`, `header_symbols`. Six style findings take a tag too: `char_pointer`, +`overload_prefix`, `out_param_order`, `string_concat`, `param_comment` and `test_only_api`, for a C API +or a fixed interface that needs them. Other style and layering findings have no bypass. - **Place it** on the function's or class's first line, or in the comment block directly above it (only `//` comment and `template` lines between). A `file_lines` tag goes in the first 20 lines of the file, and so does a `header_symbols` tag. A style tag (`char_pointer`, `overload_prefix`, - `out_param_order`, `string_concat`) goes on the flagged line or in the comment block directly above it, and waives the finding whether it is an error or advisory. A tag that attaches to nothing fails. + `out_param_order`, `string_concat`, `param_comment`, `test_only_api`) goes on the flagged line or in the comment block directly above it, and waives the finding whether it is an error or advisory. A tag that attaches to nothing fails. - **Justify it:** the text after `--`, on the tag line itself, must be at least 40 characters and 6 words, and say why splitting would make the code worse. Unknown checks, a missing or short justification, or a malformed tag fail. @@ -157,9 +158,21 @@ lexer, not a parser: rules that need types stay in review. | Output parameters last: a non-const `T&` parameter before a by-value, `const&` or `&&` one. Not reported: context handles conventionally first (`*Env`/`*Environment`, `*Context`, `*Actions`, `*Backend`, `*Option`, `*Logger`, `ConsoleHandle`, `VideoStream`, `VideoOverlay`, `VideoFeed`, `AudioFeed`, `CancellableScope`, `UnitTestDatabase`, `AsyncCommandSession`, `std::*stream`); pointers, packs and function types (neither input nor output); constructors, operators, `override`/`final`, `= delete`; out-of-class `A::f` definitions in a `.cpp` (the header declaration is checked); a changed line whose signature keeps its parameter kinds | advisory for now, error before upstream (tag: `out_param_order`) | Google "Function Parameter Ordering" | | Building a string with `+`: a string literal, `std::to_string(...)` or `std::string(...)` next to a binary `+` (also `s += "a" + b`). Not reported: `+=` appends, `++`, numeric/pointer `+`, files including Qt headers (QString) | advisory for now, error before upstream (tag: `string_concat`) | fork AGENTS.md "Strings": use `std::format` | | A comment whose first sentence describes the neighbouring function, none of its own | advisory | CONTRIBUTING; Google "Function Comments" | +| A literal argument without a `/*param=*/` comment in a call of 3+ arguments: numbers (`2`, `0ms`, `1.5f`), `true`/`false`, `nullptr`, brace lists of those (`{0, 0}`, `{}`). With the callee declared once per arity in the same file, a comment naming another parameter too. Not reported: string/char literals, macros, `std::` calls, format/log calls (name contains format/print/log, or a string literal first), all-number tuples and `*Box`/`*Color`/`*Point`/`*Rect` constructors, a leading `nullptr` (null logger or Qt parent), a lone trailing duration in a call of at most 4 arguments (`save_button_shows(console, context, 5s)`); calls outside brace bodies (declarations, default arguments) (`param_comments.py`) | advisory (tag: `param_comment`) | Google "Function Arguments"; user review on fork #590 | +| A public namespace-scope name a header adds (on an added line, new since the base) whose only users outside the header and its own .cpp are test files (`*_Tests.cpp`, `*_Test.cpp`, `Tests/`), or that only its own .cpp uses. Users: one `git grep -o -w` over tracked C++. Not reported: names nothing uses yet, names another kept public declaration of the header mentions (parameter, return or member type), `add_tests*` hooks, test headers (`exposure.py`) | advisory (tag: `test_only_api`) | user review on fork #590: test through the public function, or keep the helper in the .cpp's anonymous namespace | Advisories print as `Advisory:` lines and `::warning` annotations and never fail. +`param_comment` and `test_only_api` are advisory by their measured rates (2026-11-15). On 18 fork +PRs (#567-#591) `param_comment` reported 379 calls, 257 of them in test files and 91 controller +timings (`pbf_press_button(context, BUTTON_A, 160ms, 400ms)`); no lexer misreads in a 42-finding +sample, and it reports exactly the review's sites on #590 (`{0, 0}`, `0ms`) and #572 (`2`). As a +failure it would fail nearly every open program PR. `test_only_api` reported 203 names on 10 of +those PRs and exactly #590's three helpers; on the PLZA slice stack 31 of 64 names gained a +non-test caller later in the stack (API added one PR before its caller), and upstream's +`Common/ControllerStates` C-style headers (constants and `*_equals` used only by their own .cpp) +are reported too. + ## Build and native C++ tests in one command The existing `.github/workflows/cpp-ci-serial-programs-base.yml` already runs diff --git a/devtools/code_health/__main__.py b/devtools/code_health/__main__.py index ed19814e94..8c2ae31202 100644 --- a/devtools/code_health/__main__.py +++ b/devtools/code_health/__main__.py @@ -5,7 +5,7 @@ import sys from pathlib import Path -from . import allow, header_api, layering, style +from . import allow, exposure, header_api, layering, param_comments, style from .classes import measure_classes from .findings import annotations from .git_files import changed_files, default_base, read_revision @@ -22,6 +22,7 @@ def run(root, base_ref=None, github=False): base_ref, how = default_base(root) print(f'Base: {base_ref} ({how}).') changes = changed_files(root, base_ref) + users = exposure.git_users(root) failures, notes, bypassed = [], [], [] advisory = 0 for current, original in changes: @@ -40,6 +41,10 @@ def run(root, base_ref=None, github=False): notes.extend(unneeded) style_errors, style_notes = style.check(current, source, previous_source) style_errors += header_api.prefix_overloads(current, source, previous_source) + # Advisory: both report many sites a reviewer may accept (pbf_* timings, API for a + # later PR in a stack); see the README's Style table for the measured rates. + style_notes += param_comments.check(current, source, previous_source) + style_notes += exposure.check(current, source, previous_source, users) # Waive errors and advisories in one pass, so a tag on either counts as used. kept, waived, unattached = allow.waive(current, source, allowances, style_errors + style_notes) failures.extend([f for f in kept if f in style_errors] + unattached) diff --git a/devtools/code_health/allow.py b/devtools/code_health/allow.py index 791ae9e3cc..baf5b93366 100644 --- a/devtools/code_health/allow.py +++ b/devtools/code_health/allow.py @@ -3,7 +3,8 @@ A tag on a function's or class's first line, or in the comment block directly above it, lets that function's or class's listed metrics exceed their limit up to a hard ceiling. A `file_lines` or `header_symbols` tag goes in the first FILE_TAG_LINES lines. A STYLE_CHECKS tag -(`char_pointer`, `overload_prefix`, `out_param_order`, `string_concat`) waives that style finding +(`char_pointer`, `overload_prefix`, `out_param_order`, `string_concat`, `param_comment`, +`test_only_api`) waives that style finding on its own line or the line below its comment block. Every bypass is printed, quoting its justification, so reviewers see it; malformed, unknown or misplaced tags fail.""" import re @@ -13,7 +14,8 @@ from .findings import Finding # Style findings a tag can bypass; they attach like a function's tag, to the finding's line. -STYLE_CHECKS = ('char_pointer', 'overload_prefix', 'out_param_order', 'string_concat') +STYLE_CHECKS = ('char_pointer', 'overload_prefix', 'out_param_order', 'string_concat', 'param_comment', + 'test_only_api') # File-level metrics: the tag goes in the first FILE_TAG_LINES lines. FILE_CHECKS = ('file_lines',) + tuple(HEADER_LIMITS) CHECKS = (('file_lines', 'function_lines', 'nesting', 'parameters') + tuple(CLASS_LIMITS) diff --git a/devtools/code_health/exposure.py b/devtools/code_health/exposure.py new file mode 100644 index 0000000000..5cd94810eb --- /dev/null +++ b/devtools/code_health/exposure.py @@ -0,0 +1,118 @@ +"""Test-only header exposure: a public name a header adds whose only users are tests. + +A helper moved into a header so a `*_Tests.cpp` can call it widens the API for the tests' +sake (user review on fork #590). Test through the public function, or keep the helper in +the .cpp's anonymous namespace. Names are the header's public namespace-scope names +(`header_api.public_names`); only names new since the base revision, on added lines, count. +Users are the tracked C++ files that mention the name as a word (`git grep -w`), other than +the header itself and its own .cpp. Reported: a new name that only test files use, or that +only its own .cpp uses (tests in the same .cpp need no header declaration). A name with no +users at all is new API for a caller still to come, and a name that another public declaration +of the header mentions (a parameter, return or member type) is part of that API: neither is +reported. + +Not reported: test headers (`*_Tests.h`, `Tests/`) and test registration hooks +(`add_tests*`), which exist to be called by the test runner. +""" +import re +import subprocess +from pathlib import PurePosixPath + +from .findings import Finding +from .git_files import EXTENSIONS +from .header_api import declared_name, normalize, public, public_names +from .style import HEADERS, added_lines, rule, scopes, statements, strip_code + +REGISTRATION = re.compile(r'add_tests\w*') +CITE = 'user review on fork #590' + + +def test_file(path): + """`*_Tests.cpp`, `*_Test.cpp` and `*_Tests.h` files and anything under a `Tests/` directory.""" + pure = PurePosixPath(path) + return 'Tests' in pure.parts[:-1] or re.search(r'_Tests?$', pure.stem) is not None + + +def git_users(root): + """A function from names to {name: tracked first-party C++ files that mention it as a word}. + + One `git grep -o` for all names: a name per call would take minutes on a large change.""" + pathspecs = [f'{top}/*{ext}' for top in ('Common', 'SerialPrograms') for ext in sorted(EXTENSIONS)] + + def users(names): + found = {name: set() for name in names} + if not names: + return found + patterns = [arg for name in sorted(names) for arg in ('-e', name)] + result = subprocess.run(['git', '-C', str(root), 'grep', '-o', '-w', '-F', '-I', '--null', + *patterns, '--', *pathspecs], capture_output=True, text=True) + if result.returncode not in (0, 1): + raise ValueError(f'git grep for header names failed: {result.stderr.strip()}') + for line in result.stdout.splitlines(): + path, _, match = line.partition('\0') + if match in found and '3rdParty' not in path: + found[match].add(path) + return found + return users + + +def own_files(path): + """The header and the sources that implement it: same directory and stem.""" + stem = str(PurePosixPath(path).with_suffix('')) + return {stem + ext for ext in EXTENSIONS} + + +def mentions_by_owner(source): + """{public name: identifiers its declaration mentions}, members of a public class included.""" + owners = {} + for statement, enclosing in scopes(statements(strip_code(normalize(source)))): + if statement.end == '}': + continue + if statement.namespace_scope and public(enclosing): + owner = declared_name(statement) + else: + outer = [name for kind, name in enclosing if kind == 'class'] + owner = outer[0] if outer and public([f for f in enclosing if f[0] != 'class']) else None + if owner: + owners.setdefault(owner, set()).update(re.findall(r'[A-Za-z_]\w*', statement.text)) + return owners + + +def exposure(path, mentions): + """Why a name that `mentions` (files) use is exposed only for tests or its own .cpp, or None.""" + outside = sorted(mentions - own_files(path)) + if any(not test_file(user) for user in outside): + return None + if outside: + return f"public only for {', '.join(PurePosixPath(u).name for u in outside)}" + if mentions & (own_files(path) - {path}): + return 'used only in its own .cpp' + return None + + +def check(path, source, previous_source, users): + """Advisories for public names this header adds that only tests or its own .cpp use. + + `users(names)` returns {name: repository files that mention it} (see `git_users`).""" + if not path.endswith(HEADERS) or test_file(path): + return [] + previous = set(public_names(previous_source)) if previous_source is not None else set() + added = added_lines(normalize(source).split('\n'), + None if previous_source is None else normalize(previous_source).split('\n')) + names = public_names(source) + new = [name for name, line in names.items() + if name not in previous and line in added and not REGISTRATION.fullmatch(name)] + mentions = users(new) if new else {} + exposed = {name: why for name in new for why in [exposure(path, mentions.get(name, set()))] if why} + owners = mentions_by_owner(source) + changed = True + while changed: # a name that a kept public declaration mentions is part of its API + needed = {name for name in exposed for owner, words in owners.items() + if owner != name and owner not in exposed and name in words} + changed = bool(needed) + for name in needed: + del exposed[name] + return [Finding(path, names[name], rule( + f"'{name}' is {why}; test through the public function, or keep it in the .cpp's " + 'anonymous namespace (ignore if a later PR in the stack calls it)', CITE), check='test_only_api') + for name, why in sorted(exposed.items(), key=lambda item: names[item[0]])] diff --git a/devtools/code_health/param_comments.py b/devtools/code_health/param_comments.py new file mode 100644 index 0000000000..0f115dbbd9 --- /dev/null +++ b/devtools/code_health/param_comments.py @@ -0,0 +1,174 @@ +"""Parameter comments on literal arguments: `f(x, /*count=*/3, /*cool=*/0ms)`. + +In a call with three or more arguments, a bare literal says nothing about the parameter it +fills, so it needs a `/*param=*/` comment (Google "Function Arguments"; the user's review rule +on fork #590). Literals are numbers (with suffixes, so `0ms` and `1.5f`), `true`, `false`, +`nullptr` and brace lists of those (`{0, 0}`, `{}`). Only literals on added lines count. + +Not reported, because the call itself already names the values: +- string and character literals: a message, key or label reads as its own name; +- macros (ALL_CAPS callees) and `std::` functions: no parameter names / well-known signatures; +- format and log calls (callee name contains format, print or log, or the first argument is a + string literal): the format string's placeholders say what each argument is; +- a `nullptr` first argument: the repository's null logger or Qt parent slot + (`InternalProgramError(nullptr, PA_CURRENT_FUNCTION, message)`, `critical(nullptr, ...)`); +- value-type constructors (`ImageFloatBox`, `ImagePixelBox`, `Color`, a `*Point` or `*Rect`): + the type names the fields; +- a lone trailing duration in a call of at most four arguments + (`save_button_shows(console, context, 5s)`): the call's timeout or length, and the function + name says which. Longer calls (`ssf_press_button(context, buttons, delay, hold, 0ms)`) pass + several durations, so the slot is ambiguous and is still reported; +- calls whose arguments are all plain numbers: a box, color or point tuple whose type names + the fields (`ImageFloatBox(0.1, 0.2, 0.3, 0.4)`). +When the callee is declared in the same file with exactly one arity match, a comment that names +a different parameter is reported too. Lexical, like `style.py`: calls are only looked for inside +brace bodies, never in declarations. +""" +import bisect +import re + +from .findings import Finding +from .style import NOT_FUNCTIONS, added_lines, brace_kind, default_split, matching, rule, signature, \ + statements, strip_code + +MIN_ARGUMENTS = 3 +SHORT_CALL = 4 +CALL = re.compile(r'(?)\s*)*[A-Za-z_]\w*)\s*\(') +NUMBER = r"[-+]?\s*(?:0[xXbB][\da-fA-F']+|\d[\d']*\.?[\d']*(?:[eE][-+]?\d+)?|\.\d+)[A-Za-z_]*" +PLAIN_NUMBER = re.compile(r"[-+]?\s*(?:0[xX][\da-fA-F']+|\d[\d']*\.?[\d']*(?:[eE][-+]?\d+)?|\.\d+)[fFuUlL]*") +DURATION = re.compile(r"\d[\d']*(?:ms|s|min|h|us|ns)") +SCALAR = NUMBER + r'|true|false|nullptr' +LITERAL = re.compile(r'(?:' + SCALAR + r')|\{\s*(?:(?:' + SCALAR + r')(?:\s*,\s*(?:' + SCALAR + r'))*)?\s*\}') +STRING = re.compile(r'(?:(?:u8|u|U|L)?"[^"]*"\s*)+|\'[^\']*\'') +COMMENT = re.compile(r'/\*\s*([A-Za-z_]\w*)\s*=\s*\*/') +MACRO = re.compile(r'[A-Z][A-Z0-9_]*') +VALUE_TYPE = re.compile(r'\w*(?:Box|Color|Point|Rect)') +FORMAT_LIKE = re.compile(r'format|print|log', re.I) +CITE = ('Google "Function Arguments"', 'user review on fork #590') + + +def body_mask(code_text, parsed): + """Per offset of `code_text`, whether it lies inside a function body or a nested brace. + + `statements` emits one statement per `{`, in order, so the k-th `{` pairs with the k-th one.""" + opening = iter([brace_kind(s.text) for s in parsed if s.end == '{']) + mask, stack = bytearray(len(code_text)), [] + for offset, c in enumerate(code_text): + if c == '{': + stack.append(next(opening, 'other')) + elif c == '}' and stack: + stack.pop() + mask[offset] = 'other' in stack + return mask + + +def arguments(code_text, open_index, close_index): + """(start, end) offsets of each top-level argument between the parentheses.""" + spans, depth, start = [], 0, open_index + 1 + for k in range(open_index + 1, close_index): + c = code_text[k] + depth += (c in '([{') - (c in ')]}') + if c == ',' and depth == 0: + spans.append((start, k)) + start = k + 1 + if code_text[start:close_index].strip() or spans: + spans.append((start, close_index)) + return spans + + +def exempt(name, args): + """Whether the call names its values another way (see the module docstring).""" + short = re.split(r'::|\.|->', name.replace(' ', ''))[-1] + if short in NOT_FUNCTIONS or MACRO.fullmatch(short) or name.replace(' ', '').startswith('std::'): + return True + if VALUE_TYPE.fullmatch(short) or FORMAT_LIKE.search(short) or STRING.fullmatch(args[0]): + return True + literals = [arg for arg in args if LITERAL.fullmatch(arg)] + if len(args) <= SHORT_CALL and literals == [args[-1]] and DURATION.fullmatch(args[-1]): + return True + return all(PLAIN_NUMBER.fullmatch(arg) for arg in args) + + +def declared_parameters(parsed): + """{function name: [parameter names]} for names declared once per arity in this file.""" + found = {} + for statement in parsed: + sig = signature(statement) + if sig is None: + continue + names = [] + for param in sig.params: + last = re.search(r'([A-Za-z_]\w*)\s*(?:\[[^\]]*\])?$', default_split(param)[0]) + names.append(last.group(1) if last else '') + key = (sig.name.split('::')[-1], len(names)) + found.setdefault(key, set()).add(tuple(names)) + return {key: list(next(iter(v))) for key, v in found.items() if len(v) == 1} + + +def check(path, source, previous_source): + """Advisories for uncommented literal arguments on lines this change adds.""" + raw = source.replace('\r\n', '\n').replace('\r', '\n') + previous = None if previous_source is None else \ + previous_source.replace('\r\n', '\n').replace('\r', '\n').split('\n') + added = added_lines(raw.split('\n'), previous) + if not added: + return [] + code = strip_code(raw) + code_text = '\n'.join(code) + parsed = statements(code) + mask = body_mask(code_text, parsed) + params = declared_parameters(parsed) + line_of = line_index(code_text) + findings = [] + for match in CALL.finditer(code_text): + if not mask[match.start()]: + continue + open_index = match.end() - 1 + close_index = matching(code_text, open_index) + if close_index < 0: + continue + spans = arguments(code_text, open_index, close_index) + args = [code_text[a:b].strip() for a, b in spans] + if len(args) < MIN_ARGUMENTS or exempt(match.group(1), args): + continue + finding = call_finding(path, match.group(1), spans, (code_text, raw), line_of, added, params) + if finding: + findings.append(finding) + return findings + + +def call_finding(path, name, spans, texts, line_of, added, params): + """One finding per call: its uncommented literals, or comments naming the wrong parameter. + + `texts` is (blanked code, raw source); offsets match because blanking keeps columns.""" + code_text, raw = texts + short = re.split(r'::|\.|->', name.replace(' ', ''))[-1] + declared = params.get((short, len(spans))) + missing, wrong, line = [], [], None + for index, (start, end) in enumerate(spans): + arg = code_text[start:end].strip() + at = line_of(start + len(code_text[start:end]) - len(code_text[start:end].lstrip())) + if at not in added or (index == 0 and arg == 'nullptr'): + continue + comment = COMMENT.search(raw[start:end]) + if comment and declared and declared[index] and comment.group(1) != declared[index]: + wrong.append(f'/*{comment.group(1)}=*/ names parameter \'{declared[index]}\'') + line = line or at + elif not comment and LITERAL.fullmatch(arg): + missing.append(arg) + line = line or at + if not missing and not wrong: + return None + problems = [] + if missing: + plural = len(missing) > 1 + problems.append(f'literal argument{"s" if plural else ""} {", ".join(missing)} ' + f'need{"" if plural else "s"} a /*param=*/ comment') + problems += wrong + return Finding(path, line, rule(f'{short}(): {"; ".join(problems)}', *CITE), check='param_comment') + + +def line_index(text): + """A function from an offset in `text` to its 1-based line.""" + starts = [0] + [k + 1 for k, c in enumerate(text) if c == '\n'] + return lambda offset: bisect.bisect_right(starts, offset) diff --git a/devtools/code_health/tests/test_exposure.py b/devtools/code_health/tests/test_exposure.py new file mode 100644 index 0000000000..0868ac5c2e --- /dev/null +++ b/devtools/code_health/tests/test_exposure.py @@ -0,0 +1,102 @@ +import subprocess +import tempfile +import unittest +from pathlib import Path + +from devtools.code_health.exposure import check, git_users, test_file + + +DIR = 'SerialPrograms/Source/VideoGames/PokemonLZA/Programs/' +HEADER = DIR + 'PokemonLZA_BallThrow.h' +OWN_CPP = DIR + 'PokemonLZA_BallThrow.cpp' +TESTS = DIR + 'PokemonLZA_BallThrow_Tests.cpp' +CALLER = DIR + 'Farming/PokemonLZA_InPlaceCatcher.cpp' + + +def header(body): + return ('namespace PokemonAutomation{\nnamespace PokemonLZA{\n' + body + + '\n} // namespace PokemonLZA\n} // namespace PokemonAutomation\n') + + +# Fork #590 before its review: two helpers exposed next to the public throw_ball(). +BALL_THROW = header( + 'struct BallThrowOptions{\n int aim = 500;\n};\n' + 'struct BallScrolls{\n int count;\n};\n' + 'BallScrolls ball_scrolls(int right_scrolls);\n' + 'int throw_buttons_hold(const BallThrowOptions& options);\n' + 'void throw_ball(Context& context, const BallThrowOptions& options);\n' + 'void add_tests_BallThrow(UnitTestDatabase& database);') + + +def fake_users(table): + return lambda names: {name: set(table.get(name, ())) for name in names} + + +def names(findings): + return [f.message.split("'")[1] for f in findings] + + +class TestOnlyExposureTests(unittest.TestCase): + def test_flags_helpers_only_tests_or_their_own_cpp_use(self): + users = fake_users({ + 'BallThrowOptions': [HEADER, OWN_CPP, CALLER], + 'BallScrolls': [HEADER, OWN_CPP, TESTS], + 'ball_scrolls': [HEADER, OWN_CPP, TESTS], + 'throw_buttons_hold': [HEADER, OWN_CPP], + 'throw_ball': [HEADER, OWN_CPP, CALLER], + 'add_tests_BallThrow': [HEADER, OWN_CPP, DIR + '../PokemonLZA_Tests.cpp'], + }) + found = check(HEADER, BALL_THROW, None, users) + self.assertEqual(['BallScrolls', 'ball_scrolls', 'throw_buttons_hold'], names(found)) + self.assertIn('public only for PokemonLZA_BallThrow_Tests.cpp', found[1].message) + self.assertIn('used only in its own .cpp', found[2].message) + self.assertEqual({'test_only_api'}, {f.check for f in found}) + + def test_names_without_users_or_used_by_other_public_declarations_pass(self): + # A type in a kept function's signature is part of its API; a name nothing uses yet + # is new API for a later caller. + users = fake_users({ + 'BallThrowOptions': [HEADER, OWN_CPP, TESTS], + 'BallScrolls': [HEADER, OWN_CPP, TESTS], + 'ball_scrolls': [HEADER, OWN_CPP, CALLER], + 'throw_ball': [HEADER, OWN_CPP, CALLER], + }) + self.assertEqual([], check(HEADER, BALL_THROW, None, users)) + + def test_only_names_new_since_the_base_count(self): + users = fake_users({'ball_scrolls': [HEADER, OWN_CPP, TESTS], 'BallScrolls': [HEADER, OWN_CPP]}) + self.assertEqual([], check(HEADER, BALL_THROW, BALL_THROW, users)) + + def test_sources_and_test_headers_are_not_checked(self): + users = fake_users({'ball_scrolls': [OWN_CPP, TESTS]}) + self.assertEqual([], check(OWN_CPP, BALL_THROW, None, users)) + self.assertEqual([], check(DIR + 'PokemonLZA_BallThrow_Tests.h', BALL_THROW, None, users)) + + def test_test_files(self): + for path, expected in ((TESTS, True), (DIR + 'Foo_Test.cpp', True), + ('SerialPrograms/Source/Tests/CommandLineTests.cpp', True), + (OWN_CPP, False), (DIR + 'SpeedTest.cpp', False)): + with self.subTest(path=path): + self.assertEqual(expected, test_file(path)) + + +class GitUsersTests(unittest.TestCase): + def test_one_grep_maps_each_name_to_the_files_that_mention_it_as_a_word(self): + with tempfile.TemporaryDirectory() as temp: + root = Path(temp) + for path, text in (('Common/a.h', 'int helper();\nint helper_two();\n'), + ('Common/a_Tests.cpp', 'int x = helper();\n'), + ('Common/b.cpp', 'int y = helper_two();\n'), + ('Common/notes.txt', 'helper\n')): + (root / path).parent.mkdir(parents=True, exist_ok=True) + (root / path).write_text(text) + subprocess.check_call(['git', '-C', temp, 'init', '-q']) + subprocess.check_call(['git', '-C', temp, 'add', '.']) + found = git_users(root)(['helper', 'helper_two', 'absent']) + self.assertEqual({'helper': {'Common/a.h', 'Common/a_Tests.cpp'}, + 'helper_two': {'Common/a.h', 'Common/b.cpp'}, + 'absent': set()}, found) + + +if __name__ == '__main__': + unittest.main() diff --git a/devtools/code_health/tests/test_param_comments.py b/devtools/code_health/tests/test_param_comments.py new file mode 100644 index 0000000000..d78beca90f --- /dev/null +++ b/devtools/code_health/tests/test_param_comments.py @@ -0,0 +1,107 @@ +import unittest + +from devtools.code_health import allow +from devtools.code_health.param_comments import check + + +CPP = 'SerialPrograms/Source/VideoGames/PokemonLZA/Programs/Example.cpp' + + +def in_function(body): + return ('namespace PokemonAutomation{\n' + 'void example(ProControllerContext& context){\n' + body + '\n}\n' + '} // namespace PokemonAutomation\n') + + +def messages(source, previous=None): + return [f.message for f in check(CPP, source, previous)] + + +class LiteralArgumentTests(unittest.TestCase): + def test_flags_uncommented_literals_in_long_calls(self): + # The two calls fork #590's review asked to comment, and #572's middle literal. + for line, literal in ( + ('pbf_controller_state(context, BUTTON_ZL, DPAD_NONE, {0, 0}, {0, 0}, lock_on);', '{0, 0}, {0, 0}'), + ('ssf_press_button(context, BUTTON_ZL | BUTTON_ZR, aim, hold, 0ms);', '0ms'), + ('auto db = make_story_select_database(story_list(), 2, story_list().size());', '2'), + ('pbf_press_button(context, BUTTON_A, 160ms, 400ms);', '160ms, 400ms'), + ('sign_room_walk(Direction::EAST, 2, {3, 3}, true);', '2, {3, 3}, true'), + ('reader.read(image, nullptr, -1.5f);', 'nullptr, -1.5f'), + ('run(context, {}, options);', '{}')): + with self.subTest(line=line): + found = messages(in_function(line)) + self.assertEqual(1, len(found), found) + self.assertIn(f' {literal} need', found[0]) + + def test_commented_literals_pass(self): + source = in_function( + 'pbf_controller_state(\n' + ' context, BUTTON_ZL, DPAD_NONE,\n' + ' /*left_joystick=*/{0, 0}, /*right_joystick=*/{0, 0},\n' + ' /*duration=*/lock_on\n' + ');\n' + 'ssf_press_button(context, BUTTONS, /*delay=*/aim, /*hold=*/hold, /*cool=*/0ms);') + self.assertEqual([], messages(source)) + + def test_exemptions(self): + for line in ( + 'f(a, b);', # fewer than three arguments + 'f(a, b, c);', # no literal + 'save_button_shows(console, context, 5s);', # lone trailing duration + 'env.log(std::format("{} {}", 1, 2), COLOR_RED, true);', # log call + 'std::string s = std::format("{} of {}", index, 3);', # format call + 'report( "story {} failed", 3, true);', # format string first + 'auto x = std::clamp(value, 0, 255);', # std:: signature + 'PA_CHECK(value, 3, true);', # macro + 'throw InternalProgramError(nullptr, PA_CURRENT_FUNCTION, "bad row");', + 'ImageFloatBox box(0.1, 0.2, 0.3, 0.4);', # value tuple + 'return ImageFloatBox(left + 0.1 * i, top, 0.015, 0.2);', # value type + 'box.critical(nullptr, "Error", text);', # null Qt parent + 'select(image, "label", "other", name);', # strings name themselves + 'if (a(1) && b(2) && c(3)){\n go();\n}'): # three one-argument calls + with self.subTest(line=line): + self.assertEqual([], messages(in_function(line))) + + def test_long_call_lone_duration_is_reported(self): + self.assertEqual(1, len(messages(in_function('wait(console, context, BUTTON_A, timeout, 0ms);')))) + + def test_leading_nullptr_is_exempt_but_not_the_rest(self): + found = messages(in_function('execute(nullptr, false, 2000ms);')) + self.assertEqual(1, len(found), found) + self.assertIn(' false, 2000ms need', found[0]) + + def test_declarations_and_namespace_scope_are_not_calls(self): + source = ('namespace PokemonAutomation{\n' + 'void walk(int tiles, bool run = true, Milliseconds wait = 0ms, int retries = 3);\n' + 'const ImageFloatBox BOX(0.1, 0.2, 0.3, 0.4);\n' + '} // namespace PokemonAutomation\n') + self.assertEqual([], messages(source)) + + def test_only_added_lines_count(self): + old = in_function('pbf_press_button(context, BUTTON_A, 160ms, 400ms);') + new = in_function('pbf_press_button(context, BUTTON_A, 160ms, 400ms);\nok();') + self.assertEqual([], messages(new, old)) + self.assertEqual(1, len(messages(new))) + + def test_comment_must_name_the_parameter_declared_in_this_file(self): + source = ('namespace PokemonAutomation{\n' + 'void walk(int tiles, bool run, int retries){\n}\n' + 'void example(){\n' + ' walk(/*tiles=*/2, /*running=*/true, /*retries=*/3);\n' + '}\n' + '} // namespace PokemonAutomation\n') + found = messages(source) + self.assertEqual(1, len(found), found) + self.assertIn("/*running=*/ names parameter 'run'", found[0]) + + def test_finding_carries_its_tag_check(self): + source = in_function(' // code-health: allow param_comment -- the controller state reads as a table of joystick axes\n' + ' pbf_controller_state(context, BUTTON_ZL, DPAD_NONE, {0, 0}, {0, 0}, lock_on);') + findings = check(CPP, source, None) + allowances, errors = allow.parse(CPP, source) + kept, waived, unattached = allow.waive(CPP, source, allowances, findings) + self.assertEqual(([], 1, [], []), (kept, len(waived), unattached, errors)) + + +if __name__ == '__main__': + unittest.main() From 23a2a7e7692f34b3280b1c7cdc9404cf46b011fd Mon Sep 17 00:00:00 2001 From: dayfine Date: Sun, 15 Nov 2026 17:27:50 -0800 Subject: [PATCH 4/7] pa_lab: preflight_program, token-bound run_program options, cfr_batch (Slice 5b) --- lab/server/contracts/v1/invalid_cases.json | 26 +++ lab/server/pa_lab/__main__.py | 2 +- lab/server/pa_lab/actions.py | 239 ++++++++++++++++++--- lab/server/pa_lab/cfr.py | 214 ++++++++++++++++++ lab/server/pa_lab/contracts.py | 3 +- lab/server/pa_lab/control.py | 45 +++- lab/server/pa_lab/db.py | 17 ++ lab/server/pa_lab/lab_reader.py | 8 +- lab/server/pa_lab/mcp_server.py | 58 +++-- lab/server/pa_lab/preflight.py | 223 +++++++++++++++++++ lab/server/tests/fake_lab.py | 101 ++++++++- lab/server/tests/test_actions.py | 88 +++++--- lab/server/tests/test_cfr.py | 104 +++++++++ 13 files changed, 1043 insertions(+), 85 deletions(-) create mode 100644 lab/server/pa_lab/cfr.py create mode 100644 lab/server/pa_lab/preflight.py create mode 100644 lab/server/tests/test_cfr.py diff --git a/lab/server/contracts/v1/invalid_cases.json b/lab/server/contracts/v1/invalid_cases.json index acb79d83c2..66ff4d1ee4 100644 --- a/lab/server/contracts/v1/invalid_cases.json +++ b/lab/server/contracts/v1/invalid_cases.json @@ -1910,6 +1910,32 @@ "contains": "unknown field" } }, + { + "name": "run without preflight token", + "kind": "request", + "input": { + "json": { + "schema": 1, + "request_id": "9b1d7e44-6a2f-4c0e-8f13-2d5c8a90b6e1", + "client_id": "claude:frlg-lane", + "tool": "run_program", + "console": "B", + "lease_token": "example-lease-token-not-real", + "arguments": { + "panel": "PokemonFRLG:AutoStory", + "options": { + "START_CHECKPOINT": "13.01", + "END_CHECKPOINT": "13.02", + "SAVE_MODE": "none" + } + } + } + }, + "expectation": { + "path": "arguments.preflight_token", + "contains": "required field is missing" + } + }, { "name": "run with a preflight token that is not a string", "kind": "request", diff --git a/lab/server/pa_lab/__main__.py b/lab/server/pa_lab/__main__.py index a6761fadcd..8f28b01a7b 100644 --- a/lab/server/pa_lab/__main__.py +++ b/lab/server/pa_lab/__main__.py @@ -36,7 +36,7 @@ def main(argv=None) -> int: serve.add_argument('--read-only', action='store_true', help='serve only console_status and the resources: no inbox tools, no control plane') serve.add_argument('--actions', action='store_true', - help='also serve run_program and stop_program (Slice 5a): they start and stop the program') + help='also serve preflight_program, run_program, stop_program and cfr_batch (Slice 5): they start and stop the program and press buttons') stdio = sub.add_parser('serve-stdio', help='run the read-only MCP server on stdio (for .mcp.json)') stdio.add_argument('--profile-dir', required=True, help="the console's SerialPrograms profile folder") stdio.add_argument('--keepawake-log', default='~/BDSPLab/keepawake.log') diff --git a/lab/server/pa_lab/actions.py b/lab/server/pa_lab/actions.py index f7e6a628f5..db87bccff2 100644 --- a/lab/server/pa_lab/actions.py +++ b/lab/server/pa_lab/actions.py @@ -32,10 +32,23 @@ `console://B/state` read or action call. If the app never took it, it is withdrawn from the inbox and `failed` with `timeout`. -Not in 5a: `preflight_program` (the app has no command that runs a program's prechecks without -starting it, so the program's own prechecks at Start are the only ones run), staging options -(`options` must be `{}`: the panel's current options run), `select`, `cfr_batch`, and claiming the -channel's owner token (a claimed channel is refused). +Slice 5b adds three tools on the same record, inbox and settle path: + +- `preflight_program(panel, options)`: reads `describe_program` (no lease needed), then in one + transaction checks the lease, the panel, `blocked`, the channel, that the program declares + prechecks, and the candidate options against its static contract (`pa_lab.preflight`); then + sends `preflight ` (the app stages the options, runs `check_validity` and the prechecks + in order, and restores the prior options). A pass stores a token bound to the app instance, + panel, options hash, last transition, save status and lease, good for `TOKEN_TTL_S`. +- `run_program` needs a current token for exactly its panel and options and sends `set KEY JSON` + lines for the options, then `start 10`, in one request (the app stops at a failing line). +- `cfr_batch(lines)`: allowlisted console lines (`pa_lab.cfr`) with the program stopped. The save + fact is marked stale (`cfr_batch`) before the request is written, since any press may change + the game. + +Not yet: the app's `preflight` command (none exists, so a preflight sent to today's app fails its +line and is `app_unavailable`), claiming the channel's owner token (a claimed channel is +refused), and `select`. """ from __future__ import annotations @@ -47,20 +60,25 @@ from pathlib import Path from typing import Any, Callable, Dict, List, Optional, Tuple -from pa_lab import control, shot, timestamps +from pa_lab import app_query, cfr, control, preflight, shot, timestamps from pa_lab.control import Refused -from pa_lab.lab_reader import ConsoleReader +from pa_lab.lab_reader import ConsoleReader, Unavailable -TOOLS = ('run_program', 'stop_program') +TOOLS = ('preflight_program', 'run_program', 'stop_program', 'cfr_batch') # `start SECONDS`: how long the app waits for controller=ready before refusing. The daemon checks # the controller first, so this only covers a reconnect between the check and the line. START_CONTROLLER_WAIT_S = 10 # How long a call waits for its answer before returning a pending ticket. The app polls its inbox # every 500 ms; a stop then runs in the background until the program reaches `stopped`. -ANSWER_WAIT_S = {'run_program': START_CONTROLLER_WAIT_S + 3.0, 'stop_program': 10.0} +ANSWER_WAIT_S = {'run_program': START_CONTROLLER_WAIT_S + 3.0, 'stop_program': 10.0, + # the prechecks may open a menu or go Home and back + 'preflight_program': 30.0} +# A cfr_batch waits for its worst case (pa_lab.cfr) plus this. +CFR_SLACK_S = 5.0 POLL_S = 0.1 JOURNAL_ERROR_RE = re.compile(r'^error \d+ \([^)]*\): (.*)$') +JOURNAL_LINE_RE = re.compile(r'^(?:ok|error) (\d+) ') PROGRAM_IS_RE = re.compile(r'^Program is ([A-Z_]+)\.$') ENDED_SOURCES = ('program', 'error') @@ -142,21 +160,74 @@ def _need_free_channel(app: Dict[str, Any]) -> None: 'the daemon does not claim it in Slice 5a.', evidence={'reason': 'channel_owned'}) -def check_run(state: Dict[str, Any], lease: Any, args: Dict[str, Any]) -> None: - """run_program's checks after the lease, in order. Raises Refused; None means send `start`.""" - app = _need_app(state) - if args['options']: - raise Refused('invalid_arguments', "Slice 5a runs the panel's current options; pass options {}. " - 'Staging options comes with preflight_program (Slice 5b).', field='options') - if 'preflight_token' in args: - raise Refused('invalid_arguments', 'There is no preflight_program in Slice 5a; omit preflight_token.', - field='preflight_token') - if app['panel']['kind'] == 'program' and app['panel']['id'] != args['panel']: - raise Refused('invalid_arguments', 'The selected panel is {}, not {}; run_program does not change ' - 'the selection.'.format(app['panel']['id'], args['panel']), field='panel', +def _need_panel(app: Dict[str, Any], panel: str) -> None: + if app['panel']['kind'] == 'program' and app['panel']['id'] != panel: + raise Refused('invalid_arguments', 'The selected panel is {}, not {}; the server does not change ' + 'the selection.'.format(app['panel']['id'], panel), field='panel', evidence={'selected': app['panel']['id']}) + + +def save_now(reader: ConsoleReader) -> Dict[str, Any]: + """`console://B/save` as a token binds it; an unreadable save.json is its own status.""" + try: + return reader.save_resource() + except Unavailable as e: + return {'status': 'unreadable', 'reason': e.reason, 'since': None} + + +def check_preflight(state: Dict[str, Any], program: Dict[str, Any], lease: Any, args: Dict[str, Any]) -> None: + """preflight_program's checks after the lease, against `describe_program`'s reply. Raises Refused.""" + app = _need_app(state) + _need_panel(app, args['panel']) + _first_blocked(state) + _need_free_channel(app) + if program['id'] != args['panel']: + raise Refused('invalid_arguments', 'The app describes {}, not {}.'.format(program['id'], args['panel']), + field='panel', evidence={'selected': program['id']}) + if program['prechecks'] == app_query.PRECHECKS_UNSUPPORTED: + raise Refused('prechecks_unsupported', '{} declares no start checks; the server does not run a program ' + 'whose checks it cannot run.'.format(args['panel'])) + preflight.check_options(program, args['options']) + + +def check_run(state: Dict[str, Any], save: Dict[str, Any], plane: control.ControlPlane, now: float, + lease: Any, args: Dict[str, Any]) -> None: + """run_program's checks after the lease, in order. Raises Refused; None means send `set` lines + `start`.""" + app = _need_app(state) + _need_panel(app, args['panel']) _first_blocked(state) _need_free_channel(app) + issued = plane.preflight(args['preflight_token']) + if issued is None or issued['console'] != state['console']: + raise Refused('preflight_stale', 'That preflight_token is not one this server issued for console {}; run ' + 'preflight_program first.'.format(state['console']), evidence={'reason': 'unknown_token'}) + age = now - timestamps.parse(issued['issued_at']) + if age > preflight.TOKEN_TTL_S: + raise Refused('preflight_stale', 'The preflight is {:.0f} s old (limit {} s); run preflight_program ' + 'again.'.format(age, preflight.TOKEN_TTL_S), + evidence={'reason': 'expired', 'issued_at': issued['issued_at']}) + changed = preflight.changed(issued['binding'], preflight.binding(state, save, args['panel'], args['options'], + lease['token_hash'])) + if changed: + raise Refused('preflight_stale', 'Since the preflight, {} changed; run preflight_program again.'.format( + ', '.join(changed)), evidence={'reason': 'changed', 'changed': changed, + 'preflight_request_id': issued['request_id']}) + + +def check_cfr(state: Dict[str, Any], lease: Any, args: Dict[str, Any]) -> None: + """cfr_batch's checks after the lease: the app is up, a program panel is selected and stopped, + controller and video ready, the channel free, every line allowed. Raises Refused.""" + app = _need_app(state) + if app['panel']['kind'] != 'program': + raise Refused('panel_not_program', 'The selected panel is not a program; console lines run on its console.') + _first_blocked(state) + _need_free_channel(app) + cfr.check(args['lines']) + + +def _replaced(described: str, instance: str, lease: Any, args: Dict[str, Any]) -> None: + raise Refused('app_unavailable', 'The app restarted between describe_program and the preflight.', + evidence={'reason': 'instance_replaced', 'instance_id': described, 'new_instance_id': instance}) def check_stop(state: Dict[str, Any], lease: Any, args: Dict[str, Any]) -> Optional[Dict[str, Any]]: @@ -181,12 +252,39 @@ def __init__(self, plane: control.ControlPlane, consoles: Dict[str, ConsoleReade self.consoles = consoles self.sleep = sleep + def preflight_program(self, client_id: str, request: Dict[str, Any]) -> Dict[str, Any]: + request = dict(request, tool='preflight_program') + reader = self._reader(request.get('console')) + program = None + if not self._known(client_id, request): + # Read-only and outside the transaction: the program's contract to check the options against. + try: + described = app_query.describe_program(reader, {'console': reader.console}, sleep=self.sleep) + except shot.Refused as e: + raise Refused(**e.refusal) from None + program = described['program'] + check = (lambda state: functools.partial(check_preflight, state, program)) + command = (lambda args: 'preflight ' + preflight.dumps(args['options'])) + return self._act(client_id, request, check, command, expect_instance=described['instance_id'] + if program is not None else None) + def run_program(self, client_id: str, request: Dict[str, Any]) -> Dict[str, Any]: - return self._act(client_id, dict(request, tool='run_program'), check_run, - 'start {}'.format(START_CONTROLLER_WAIT_S)) + request = dict(request, tool='run_program') + reader = self._reader(request.get('console')) + save = save_now(reader) + check = (lambda state: functools.partial(check_run, state, save, self.plane, reader.now())) + command = (lambda args: '\n'.join(preflight.set_lines(args['options']) + + ['start {}'.format(START_CONTROLLER_WAIT_S)])) + return self._act(client_id, request, check, command) def stop_program(self, client_id: str, request: Dict[str, Any]) -> Dict[str, Any]: - return self._act(client_id, dict(request, tool='stop_program'), check_stop, 'stop') + return self._act(client_id, dict(request, tool='stop_program'), + lambda state: functools.partial(check_stop, state), lambda args: 'stop') + + def cfr_batch(self, client_id: str, request: Dict[str, Any]) -> Dict[str, Any]: + return self._act(client_id, dict(request, tool='cfr_batch'), + lambda state: functools.partial(check_cfr, state), lambda args: '\n'.join(args['lines']), + mark_save=True) def settle_pending(self, console: str) -> None: """Settles every pending action on `console` from what the app shows now (no waiting).""" @@ -205,21 +303,38 @@ def _reader(self, console: Any) -> ConsoleReader: console, ', '.join(sorted(self.consoles)))) return reader - def _act(self, client_id: str, request: Dict[str, Any], - check: Callable[..., Optional[Dict[str, Any]]], command: str) -> Dict[str, Any]: + def _known(self, client_id: str, request: Dict[str, Any]) -> bool: + """True when this (client, tool, request_id) already has a record: a retry, answered from it.""" + request_id = request.get('request_id') + return isinstance(request_id, str) and self.plane.record(client_id, request['tool'], request_id) is not None + + def _act(self, client_id: str, request: Dict[str, Any], make_check: Callable[[Dict[str, Any]], Callable[..., Any]], + command: Callable[[Dict[str, Any]], str], expect_instance: Optional[str] = None, + mark_save: bool = False) -> Dict[str, Any]: reader = self._reader(request.get('console')) self.settle_pending(reader.console) state = reader.console_state() instance = state['app']['instance']['id'] if state['app'] is not None else '' - record, act = self.plane.begin_action(client_id, request, instance, functools.partial(check, state)) + check = make_check(state) + if expect_instance is not None and state['app'] is not None and expect_instance != instance: + check = functools.partial(_replaced, expect_instance, instance) + record, act = self.plane.begin_action(client_id, request, instance, check) tool = request['tool'] action = self.plane.action(client_id, tool, record['request_id']) if not act: if record['status'] == 'pending' and action is not None: return self._settle(reader, action, wait_s=0) return record - shot.write_request(reader.lab_dir, action['inbox_name'], record['request_id'], command) - return self._settle(reader, action, wait_s=ANSWER_WAIT_S[tool]) + if mark_save: + self.plane.mark_save_stale(reader.console, tool, record['request_id']) + shot.write_request(reader.lab_dir, action['inbox_name'], record['request_id'], command(record['arguments'])) + return self._settle(reader, action, wait_s=self._answer_wait(action)) + + def _answer_wait(self, action: Dict[str, Any]) -> float: + if action['tool'] == 'cfr_batch': + record = self.plane.record(action['client_id'], action['tool'], action['request_id']) + return cfr.check(record['arguments']['lines']) / 1000.0 + CFR_SLACK_S + return ANSWER_WAIT_S[action['tool']] def _settle(self, reader: ConsoleReader, action: Dict[str, Any], wait_s: float) -> Dict[str, Any]: key = (action['client_id'], action['tool'], action['request_id']) @@ -232,12 +347,13 @@ def _settle(self, reader: ConsoleReader, action: Dict[str, Any], wait_s: float) break self.sleep(POLL_S) age = reader.now() - timestamps.parse(action['written_at']) - if age >= ANSWER_WAIT_S[action['tool']]: + wait = self._answer_wait(action) + if age >= wait: if _withdraw(reader.lab_dir / 'inbox' / (action['inbox_name'] + '.txt')): return self.plane.finish_action(*key, refusal=_refusal( 'timeout', 'The app did not take the {} request within {:g} s (busy with another ' 'request?); it was withdrawn and nothing was sent.'.format( - action['tool'], ANSWER_WAIT_S[action['tool']]), evidence={'request': action['inbox_name']})) + action['tool'], wait), evidence={'request': action['inbox_name']})) state = reader.console_state() if state['app'] is None and state['availability']['reason'] == 'app_not_running': return self.plane.finish_action(*key, refusal=_refusal( @@ -248,9 +364,12 @@ def _settle(self, reader: ConsoleReader, action: Dict[str, Any], wait_s: float) def _judge(self, reader: ConsoleReader, action: Dict[str, Any]) -> Optional[Dict[str, Any]]: """The outcome the app shows now, as finish_action keyword arguments, or None to wait.""" lines = _journal(reader.lab_dir, action['inbox_name']) - app = reader.console_state()['app'] + state = reader.console_state() + app = state['app'] if app is None: return None + if action['tool'] in ('preflight_program', 'cfr_batch'): + return self._judge_line(reader, state, action, lines) tool, request_id = action['tool'], action['request_id'] program = app['program'] transition = program['last_transition'] @@ -296,3 +415,61 @@ def _judge(self, reader: ConsoleReader, action: Dict[str, Any]) -> Optional[Dict stopped_by = transition['source'] if transition is not None and transition['action'] == 'stop' else None return {'result': dict(base, sent=True, stopped_at=None, stopped_by=stopped_by, correlated_by='state')} return {'refusal': _app_error(tool, error, lines)} + + def _judge_line(self, reader: ConsoleReader, state: Dict[str, Any], action: Dict[str, Any], + lines: Optional[List[str]]) -> Optional[Dict[str, Any]]: + """A preflight's or cfr_batch's outcome: they change no program state, so only the journal answers. + An app restart or a UI Start/Stop after the request ends it first.""" + app = state['app'] + instance = app['instance']['id'] + tool = action['tool'] + transition = app['program']['last_transition'] + if instance != action['instance_id']: + return {'refusal': _refusal('app_unavailable', 'The app restarted before the {} settled.'.format(tool), + evidence={'reason': 'instance_replaced', 'instance_id': action['instance_id'], + 'new_instance_id': instance})} + if transition is not None and transition['source'] == 'ui' and transition['at'] >= action['written_at']: + return {'refusal': _refusal('console_held_by_user', 'The user clicked {} in the app during the {}; ' + 'console {} is theirs.'.format(transition['action'].capitalize(), tool, + reader.console), + evidence={'transition': transition})} + if lines is None: + return None + base = {'console': reader.console, 'instance_id': instance} + record = self.plane.record(action['client_id'], tool, action['request_id']) + if tool == 'cfr_batch': + return self._cfr_outcome(base, lines) + error = _line_error(lines) + if error is not None: + if error.startswith('owned by'): + return {'refusal': _app_error(tool, error, lines)} + return {'refusal': _refusal('app_unavailable', "The app's preflight line failed: {}".format(error), + evidence={'journal': lines})} + args = record['arguments'] + try: + reply = preflight.parse_reply(lines) + except Refused as e: + return {'refusal': e.refusal} + refusal = preflight.outcome(reply, args['panel'], lines) + if refusal is not None: + return {'refusal': refusal} + token_hash = self.plane.request_token_hash(action['client_id'], tool, action['request_id']) + bound = preflight.binding(state, save_now(reader), args['panel'], args['options'], token_hash) + token = preflight.new_token(bound) + issued = self.plane.issue_preflight(reader.console, action['request_id'], token, bound) + return {'result': dict(base, panel=args['panel'], checks=reply['checks'], options_hash=bound['options_hash'], + preflight_token=token, issued_at=issued, + expires_at=timestamps.format(timestamps.parse(issued) + preflight.TOKEN_TTL_S))} + + @staticmethod + def _cfr_outcome(base: Dict[str, Any], lines: List[str]) -> Dict[str, Any]: + """Every line ran (`done`), or the batch stopped at `failed_line` (1-based in the batch).""" + error = _line_error(lines) + if error is not None and error.startswith('owned by'): + return {'refusal': _app_error('cfr_batch', error, lines)} + output = [line for line in lines if not JOURNAL_LINE_RE.match(line) and line != 'end'] + if error is None: + return {'result': dict(base, outcome='done', output=output)} + failed = next(int(m.group(1)) for m in map(JOURNAL_LINE_RE.match, lines) if m and m.group(0).startswith('error')) + # Line 1 of the request is `request `; the batch starts on line 2. + return {'result': dict(base, outcome='line_failed', failed_line=failed - 1, error=error, output=output)} diff --git a/lab/server/pa_lab/cfr.py b/lab/server/pa_lab/cfr.py new file mode 100644 index 0000000000..ea6a785194 --- /dev/null +++ b/lab/server/pa_lab/cfr.py @@ -0,0 +1,214 @@ +"""The `cfr_batch` allowlist (Slice 5b of fork #131, from #547): which console lines a leased agent may send. + +A batch is 1-200 lines in the app's console-command syntax (Lab.md "Console"), each checked here +before anything is stored or sent. Only controller and screen-reading commands pass; every +number is bounded, and the batch's worst-case duration must fit in `BATCH_MAX_MS`, so the call +that waits for its journal is bounded too. Left out on purpose: + +- channel commands (`select`, `set`, `start`, `stop`, `owner`, `request`, `keepawake`, `popups`, + `snapshot`, `describe_program`, `state`, `help`): the server has its own tools for these; +- `close_game` (loses unsaved progress) and `history` (writes a recording); +- the `CAPTURE` button (writes to the console's album); +- the `image` argument of `detect` / `game_open`, which names a file; +- game-registered commands, comments (`#`) and blank lines; +- any line not in canonical form (single spaces, no leading or trailing space). + +`check(lines)` returns the batch's worst-case duration in ms or raises `control.Refused` +(`invalid_arguments`, `field` = `arguments.lines[i]`, or `arguments.lines` for the budget). +""" + +from __future__ import annotations + +import re +from typing import Callable, Dict, List, Optional + +from pa_lab import control + +BATCH_MAX_MS = 60000 +MAX_HOLD_MS = 10000 +MAX_WAIT_MS = 30000 +MAX_TAPS = 20 +MAX_SHOTS = 10 +LINE_OVERHEAD_MS = 200 # per line: the app's own bookkeeping +SCREEN_READ_MS = 1000 # shot, detect, game_open: one frame read and maybe a PNG write + +BUTTONS = ('A', 'B', 'X', 'Y', 'L', 'R', 'ZL', 'ZR', 'PLUS', 'MINUS', 'LCLICK', 'RCLICK', 'HOME') +DIRECTIONS = ('UP', 'DOWN', 'LEFT', 'RIGHT') +INT_RE = re.compile(r'^(0|[1-9][0-9]{0,5})$') +AXIS_RE = re.compile(r'^-?(0|1)(\.[0-9]{1,3})?$') +STILL_RE = re.compile(r'^(0|[1-9][0-9]{0,2})(\.[0-9]{1,2})?$') +LABEL_RE = re.compile(r'^[A-Za-z0-9_-]{1,64}$') +DETECTOR_RE = re.compile(r'^[a-z][a-z0-9_]{0,39}$') + +__all__ = ['check', 'BATCH_MAX_MS', 'COMMANDS'] + + +class _Bad(ValueError): + pass + + +def _int(word: str, name: str, minimum: int, maximum: int) -> int: + if not INT_RE.match(word) or not minimum <= int(word) <= maximum: + raise _Bad('{} must be an integer {}..{}, not {!r}'.format(name, minimum, maximum, word)) + return int(word) + + +def _axis(word: str, name: str) -> None: + if not AXIS_RE.match(word) or abs(float(word)) > 1: + raise _Bad('{} must be a number -1..1 with at most 3 decimals, not {!r}'.format(name, word)) + + +def _still(word: str) -> None: + if not STILL_RE.match(word): + raise _Bad('still must be a number 0..999 with at most 2 decimals, not {!r}'.format(word)) + + +def _buttons(word: str) -> None: + parts = word.split('+') + if any(part not in BUTTONS for part in parts) or len(set(parts)) != len(parts): + raise _Bad('buttons must be one or more of {} joined with +, not {!r}'.format(' '.join(BUTTONS), word)) + + +def _arity(args: List[str], least: int, most: int, usage: str) -> None: + if not least <= len(args) <= most: + raise _Bad('usage: ' + usage) + + +def _hold_release(args: List[str], hold: int, release: int) -> int: + if len(args) > 0: + hold = _int(args[0], 'hold', 0, MAX_HOLD_MS) + if len(args) > 1: + release = _int(args[1], 'release', 0, MAX_HOLD_MS) + return hold + release + + +def _press(args: List[str]) -> int: + _arity(args, 1, 3, 'press BTN[+BTN] [hold] [release]') + _buttons(args[0]) + return _hold_release(args[1:], 160, 160) + + +def _dpad(args: List[str]) -> int: + _arity(args, 1, 4, 'dpad UP|DOWN|LEFT|RIGHT [hold] [release] [BTN[+BTN]]') + if args[0] not in DIRECTIONS: + raise _Bad('direction must be one of {}'.format(' '.join(DIRECTIONS))) + if len(args) == 4: + _buttons(args[3]) + return _hold_release(args[1:3], 100, 200) + + +def _stick(args: List[str]) -> int: + _arity(args, 3, 6, 'stick L|R x y [hold] [release] [BTN[+BTN]]') + if args[0] not in ('L', 'R'): + raise _Bad('stick must be L or R') + _axis(args[1], 'x') + _axis(args[2], 'y') + if len(args) == 6: + _buttons(args[5]) + return _hold_release(args[3:5], 160, 160) + + +def _wait(args: List[str]) -> int: + _arity(args, 1, 1, 'wait ms') + return _int(args[0], 'ms', 0, MAX_WAIT_MS) + + +def _taps(args: List[str]) -> int: + _arity(args, 2, 6, 'taps UP|DOWN|LEFT|RIGHT N [all] [hold] [release] [still]') + if args[0] not in DIRECTIONS: + raise _Bad('direction must be one of {}'.format(' '.join(DIRECTIONS))) + count = _int(args[1], 'N', 1, MAX_TAPS) + rest = args[3:] if len(args) > 2 and args[2] == 'all' else args[2:] + if len(rest) > 3: + raise _Bad('usage: taps UP|DOWN|LEFT|RIGHT N [all] [hold] [release] [still]') + if len(rest) == 3: + _still(rest[2]) + return count * (_hold_release(rest[:2], 160, 640) + LINE_OVERHEAD_MS) + + +def _walk(args: List[str]) -> int: + _arity(args, 2, 5, 'walk x y [max] [still] [BTN[+BTN]]') + _axis(args[0], 'x') + _axis(args[1], 'y') + longest = _int(args[2], 'max', 200, MAX_HOLD_MS) if len(args) > 2 else 5000 + if len(args) > 3: + _still(args[3]) + if len(args) > 4: + _buttons(args[4]) + return longest + 2 * LINE_OVERHEAD_MS + + +def _shot(args: List[str]) -> int: + _arity(args, 1, 1, 'shot LABEL') + _label(args[0]) + return SCREEN_READ_MS + + +def _shots(args: List[str]) -> int: + _arity(args, 2, 3, 'shots LABEL count [interval]') + _label(args[0]) + count = _int(args[1], 'count', 1, MAX_SHOTS) + interval = _int(args[2], 'interval', 100, 5000) if len(args) > 2 else 500 + return count * (interval + SCREEN_READ_MS) + + +def _label(word: str) -> None: + if not LABEL_RE.match(word): + raise _Bad('label must be 1-64 of A-Z a-z 0-9 _ -') + + +def _detector(word: str) -> None: + if not DETECTOR_RE.match(word): + raise _Bad('detector must be a name of a-z 0-9 _ (no image path)') + + +def _detect(args: List[str]) -> int: + _arity(args, 1, 1, 'detect DETECTOR') + _detector(args[0]) + return SCREEN_READ_MS + + +def _wait_for(args: List[str]) -> int: + _arity(args, 1, 2, 'wait_for|expect DETECTOR [timeout]') + _detector(args[0]) + return (_int(args[1], 'timeout', 0, MAX_WAIT_MS) if len(args) > 1 else 5000) + SCREEN_READ_MS + + +def _game_open(args: List[str]) -> int: + _arity(args, 0, 0, 'game_open') + return SCREEN_READ_MS + + +COMMANDS: Dict[str, Callable[[List[str]], int]] = { + 'press': _press, 'dpad': _dpad, 'stick': _stick, 'wait': _wait, 'taps': _taps, 'walk': _walk, + 'shot': _shot, 'shots': _shots, 'detect': _detect, 'wait_for': _wait_for, 'expect': _wait_for, + 'game_open': _game_open, +} + + +def check_line(line: str) -> int: + """One line's worst-case duration in ms; raises ValueError with the reason it is not allowed.""" + if not isinstance(line, str) or line != ' '.join(line.split()) or not line: + raise _Bad('not in canonical form: one command, words separated by single spaces') + words = line.split(' ') + handler: Optional[Callable[[List[str]], int]] = COMMANDS.get(words[0]) + if handler is None: + raise _Bad('{!r} is not an allowed cfr_batch command; allowed: {}'.format( + words[0], ', '.join(sorted(COMMANDS)))) + return handler(words[1:]) + LINE_OVERHEAD_MS + + +def check(lines: List[str]) -> int: + """The batch's worst-case duration in ms, or control.Refused naming the first bad line.""" + total = 0 + for index, line in enumerate(lines): + try: + total += check_line(line) + except _Bad as e: + raise control.Refused('invalid_arguments', 'Line {} ({!r}): {}.'.format(index + 1, line, e), + field='arguments.lines[{}]'.format(index)) + if total > BATCH_MAX_MS: + raise control.Refused('invalid_arguments', 'The batch may take up to {} ms; the limit is {} ms. ' + 'Split it.'.format(total, BATCH_MAX_MS), field='arguments.lines', + evidence={'worst_case_ms': total, 'limit_ms': BATCH_MAX_MS}) + return total diff --git a/lab/server/pa_lab/contracts.py b/lab/server/pa_lab/contracts.py index a5c2599737..fdc3f2aeb2 100644 --- a/lab/server/pa_lab/contracts.py +++ b/lab/server/pa_lab/contracts.py @@ -600,8 +600,7 @@ def _page_order(obj, path, ctx): 'release_console': Obj([], open=False), 'preflight_program': Obj([PANEL_ARG, F('options', AnyObject())], open=False), 'run_program': Obj([PANEL_ARG, F('options', AnyObject()), - F('preflight_token', Str(), required=False, - doc='from preflight_program (Slice 5b); Slice 5a has no preflight and refuses one')], + F('preflight_token', Str(), doc='from a passed preflight_program with the same panel and options')], open=False), 'stop_program': Obj([], open=False), 'cfr_batch': Obj([F('lines', ListOf(Str(), min_items=1, max_items=200))], open=False), diff --git a/lab/server/pa_lab/control.py b/lab/server/pa_lab/control.py index 70b440adf9..d97c9c6ef7 100644 --- a/lab/server/pa_lab/control.py +++ b/lab/server/pa_lab/control.py @@ -36,6 +36,7 @@ MAX_WAIT_S = 60.0 DEFAULT_PAGE = 100 MAX_PAGE = 500 +PREFLIGHT_KEEP_S = 86400 # a passed preflight's row; its token is good for far less (preflight.TOKEN_TTL_S) POLL_S = 0.2 # another process's writes are seen by polling; this process's wake the waiters @@ -289,7 +290,7 @@ def begin_action(self, client_id: str, request: Dict[str, Any], instance_id: str return self._record(self._request_row(conn, client_id, req['tool'], req['request_id'])), True def action(self, client_id: str, tool: str, request_id: str) -> Optional[Dict[str, Any]]: - """What a run_program / stop_program request sent: console, instance_id, inbox_name, written_at.""" + """What an action request (run, stop, preflight, cfr_batch) sent: console, instance_id, inbox_name, written_at.""" row = self.db.read().execute('SELECT * FROM actions WHERE client_id = ? AND tool = ? AND request_id = ?', (client_id, tool, request_id)).fetchone() return None if row is None else {k: row[k] for k in row.keys()} @@ -299,12 +300,49 @@ def record(self, client_id: str, tool: str, request_id: str) -> Optional[Dict[st return None if row is None else self._record(row) def pending_actions(self, console: str) -> List[Dict[str, Any]]: - """Unresolved run_program / stop_program requests on `console`, oldest first.""" + """Unresolved action requests on `console`, oldest first.""" rows = self.db.read().execute( "SELECT a.* FROM actions a JOIN requests r USING (client_id, tool, request_id) " "WHERE a.console = ? AND r.status = 'pending' ORDER BY a.written_at", (console,)).fetchall() return [{k: row[k] for k in row.keys()} for row in rows] + def request_token_hash(self, client_id: str, tool: str, request_id: str) -> Optional[str]: + """The hash of the lease token a request was made with (never the token).""" + row = self._request_row(self.db.read(), client_id, tool, request_id) + return None if row is None else row['token_hash'] + + # ---- preflights and save marks (Slice 5b) --------------------------------------------------- + + def issue_preflight(self, console: str, request_id: str, token: str, binding: Dict[str, Any]) -> str: + """Stores a passed preflight under its token's hash; returns when it was issued.""" + self._check_console(console) + with self._writing() as conn: + stamp = self._stamp() + conn.execute('INSERT INTO preflights (token_hash, console, request_id, binding, issued_at) ' + 'VALUES (?, ?, ?, ?, ?)', (_hash(token), console, request_id, _json(binding), stamp)) + return stamp + + def preflight(self, token: str) -> Optional[Dict[str, Any]]: + """The stored preflight a token names: console, request_id, binding, issued_at; None if unknown.""" + row = self.db.read().execute('SELECT * FROM preflights WHERE token_hash = ?', (_hash(token),)).fetchone() + if row is None: + return None + return {'console': row['console'], 'request_id': row['request_id'], 'binding': json.loads(row['binding']), + 'issued_at': row['issued_at']} + + def mark_save_stale(self, console: str, reason: str, request_id: str) -> str: + """Records that something (a cfr_batch) may have changed the game since the last save fact.""" + self._check_console(console) + with self._writing() as conn: + stamp = self._stamp() + conn.execute('INSERT OR REPLACE INTO save_marks (console, reason, at, request_id) VALUES (?, ?, ?, ?)', + (console, reason, stamp, request_id)) + return stamp + + def save_mark(self, console: str) -> Optional[Dict[str, Any]]: + row = self.db.read().execute('SELECT * FROM save_marks WHERE console = ?', (console,)).fetchone() + return None if row is None else {'reason': row['reason'], 'at': row['at'], 'request_id': row['request_id']} + def finish_action(self, client_id: str, tool: str, request_id: str, result: Optional[Dict[str, Any]] = None, refusal: Optional[Dict[str, Any]] = None, run_started: Optional[Tuple[str, Dict[str, Any]]] = None) -> Dict[str, Any]: @@ -440,6 +478,9 @@ def compact(self) -> Dict[str, int]: if now - timestamps.parse(row['completed_at']) > r.request_ttl_s] conn.executemany('DELETE FROM requests WHERE client_id = ? AND tool = ? AND request_id = ?', done) conn.executemany('DELETE FROM actions WHERE client_id = ? AND tool = ? AND request_id = ?', done) + expired = [row['token_hash'] for row in conn.execute('SELECT token_hash, issued_at FROM preflights') + if now - timestamps.parse(row['issued_at']) > PREFLIGHT_KEEP_S] + conn.executemany('DELETE FROM preflights WHERE token_hash = ?', [(h,) for h in expired]) self._appended = 0 return {'events': events, 'requests': len(done), 'cursors': len(stale)} diff --git a/lab/server/pa_lab/db.py b/lab/server/pa_lab/db.py index c16691330d..a7fa975b24 100644 --- a/lab/server/pa_lab/db.py +++ b/lab/server/pa_lab/db.py @@ -101,6 +101,23 @@ PRIMARY KEY (client_id, tool, request_id) )""", ], + # 3 (Slice 5b): passed preflights, keyed by their token's hash, with what they were checked + # against; and the newest cfr_batch per console, which makes the save fact stale. + [ + """CREATE TABLE preflights ( + token_hash TEXT PRIMARY KEY, + console TEXT NOT NULL, + request_id TEXT NOT NULL, + binding TEXT NOT NULL, + issued_at TEXT NOT NULL + )""", + """CREATE TABLE save_marks ( + console TEXT PRIMARY KEY, + reason TEXT NOT NULL, + at TEXT NOT NULL, + request_id TEXT NOT NULL + )""", + ], ] SCHEMA_VERSION = len(MIGRATIONS) diff --git a/lab/server/pa_lab/lab_reader.py b/lab/server/pa_lab/lab_reader.py index 7c3cd14a3b..7d4eec3a7c 100644 --- a/lab/server/pa_lab/lab_reader.py +++ b/lab/server/pa_lab/lab_reader.py @@ -168,7 +168,9 @@ def console_state(self) -> Dict[str, Any]: def save_resource(self) -> Dict[str, Any]: """`console://B/save`: the last program-authored save fact and whether it still holds. - A save.json that exists but cannot be read raises Unavailable: it is not "no fact". + A save.json that exists but cannot be read raises Unavailable: it is not "no fact". The fact + is `stale` after a UI Start/Stop (`user_lease`) or a `cfr_batch` (Slice 5b; the control + plane's save mark) dated after `saved_at`; the newer of the two names the reason. """ if not self.save_path.exists(): doc = {'schema': 1, 'console': self.console, 'status': 'unknown', 'reason': 'no_fact', @@ -185,6 +187,10 @@ def save_resource(self) -> Dict[str, Any]: if transition is not None and transition['source'] == 'ui' and \ timestamps.parse(transition['at']) > timestamps.parse(fact['saved_at']): doc.update(status='stale', reason='user_lease', since=transition['at']) + mark = self.control.save_mark(self.console) if self.control is not None else None + if mark is not None and timestamps.parse(mark['at']) > timestamps.parse(fact['saved_at']) and \ + (doc['status'] == 'known' or timestamps.parse(mark['at']) > timestamps.parse(doc['since'])): + doc.update(status='stale', reason=mark['reason'], since=mark['at']) return contracts.validate('save_resource', doc).data diff --git a/lab/server/pa_lab/mcp_server.py b/lab/server/pa_lab/mcp_server.py index 57ed5785d8..d3dc454d6a 100644 --- a/lab/server/pa_lab/mcp_server.py +++ b/lab/server/pa_lab/mcp_server.py @@ -8,8 +8,8 @@ Resources: `console://B/state` (console_state), `console://B/save` (save_resource). Tools: `shot`, `ensure_game`, `describe_program`, `game_open`; with a control plane also `acquire_console`, `renew_console`, `release_console`, `next_events` and `request_status`; with program actions -(`serve --actions`, Slice 5a) also `run_program` and `stop_program`, the only tools that start or -stop a program. Nothing here presses a button. +(`serve --actions`, Slices 5a/5b) also `preflight_program`, `run_program`, `stop_program` and +`cfr_batch`, the only tools that start or stop a program or press a button. Read-only mode (`read_only=True`, `serve-stdio`): only `console_status` and the resources. Every tool that writes into `Lab/inbox` (`shot`, `describe_program`, `game_open`) and every control tool is @@ -57,10 +57,11 @@ 'game_open says whether a game is in front. Unknown observations are never success.') ACTION_INSTRUCTIONS = ( - 'Lab console B. Reads need no lease. run_program and stop_program start and stop the selected program: ' - 'acquire_console first, give every call a new request_id (retry with the same one), and follow a pending ' - 'record with request_status or next_events. A refusal names the check that failed; when the user holds ' - 'the console, stop and wait for them.') + 'Lab console B. Reads need no lease. To start the selected program: acquire_console, preflight_program with ' + 'the panel and options, then run_program with the same panel, options and the preflight_token; stop_program ' + 'stops it. cfr_batch is for debugging only and marks the save stale. Give every call a new request_id ' + '(retry with the same one), and follow a pending record with request_status or next_events. A refusal names ' + 'the check that failed; when the user holds the console, stop and wait for them.') TOOL_LIST = [ { @@ -200,24 +201,49 @@ def _schema(properties: Dict[str, Any], required) -> Dict[str, Any]: _ACTING = {'readOnlyHint': False, 'destructiveHint': True, 'idempotentHint': True} ACTION_TOOL_LIST = [ - {'name': 'run_program', 'title': 'Start the selected program', - 'description': "Starts the program selected in the app with its current options, once per request_id. Needs " - 'the lease token. panel must be the selected program id (this does not change the selection) ' - 'and options must be {} (staging options comes with preflight_program, Slice 5b). Refused ' - 'while the user holds the console, a program runs, the controller or video is not ready, or ' - 'the app is not running. Returns the request record: completed (the app shows this ' - "request's start; the lease is agent_running until the run ends), failed, or pending " - '(follow it with request_status or next_events).', + {'name': 'preflight_program', 'title': "Run the selected program's start checks", + 'description': 'Checks candidate options and runs the selected program\'s start checks without starting it, ' + 'then puts the prior options back; presses no button except what a check needs to look (and ' + 'returns to where it was). Needs the lease token. options are checked against describe_program ' + '(unknown keys, types, bounds, disabled entries) before anything is sent. A program that ' + 'declares no checks is refused (prechecks_unsupported). Completed: the checks and a ' + 'preflight_token for run_program, good for 120 s and only while the app instance, panel, ' + 'options, last start/stop, save status and lease stay the same. Failed: precheck_failed names ' + 'the check and what it needs.', 'inputSchema': _schema({'request_id': _REQUEST_ID, 'lease_token': _LEASE_TOKEN, 'panel': {'type': 'string', 'description': "':', the selected program"}, - 'options': {'type': 'object', 'maxProperties': 0}}, + 'options': {'type': 'object', 'description': 'option key -> value as describe_program ' + 'lists them; a group is an object; {} checks the current options'}}, ['request_id', 'lease_token', 'panel', 'options']), 'annotations': _ACTING}, + {'name': 'run_program', 'title': 'Start the selected program', + 'description': 'Starts the program selected in the app once per request_id, after loading options (the same ' + 'panel and options as the preflight; the panel keeps them after the run). Needs the lease token ' + 'and a current preflight_token (preflight_stale says what changed). panel must be the selected ' + 'program id (this does not change the selection). Refused while the user holds the console, a ' + 'program runs, the controller or video is not ready, or the app is not running. Returns the ' + "request record: completed (the app shows this request's start; the lease is agent_running " + 'until the run ends), failed, or pending (follow it with request_status or next_events).', + 'inputSchema': _schema({'request_id': _REQUEST_ID, 'lease_token': _LEASE_TOKEN, + 'panel': {'type': 'string', 'description': "':', the selected program"}, + 'options': {'type': 'object'}, + 'preflight_token': {'type': 'string', 'description': 'from preflight_program'}}, + ['request_id', 'lease_token', 'panel', 'options', 'preflight_token']), + 'annotations': _ACTING}, {'name': 'stop_program', 'title': 'Stop the running program', 'description': 'Stops the running program once per request_id. Needs the lease token. Completed when the app ' "shows this request's stop and the program is stopped, or the run ended by itself first; " 'sent false when nothing was running. Refused while the user holds the console.', 'inputSchema': _schema({'request_id': _REQUEST_ID, 'lease_token': _LEASE_TOKEN}, ['request_id', 'lease_token']), 'annotations': _ACTING}, + {'name': 'cfr_batch', 'title': 'Send a short batch of console lines (debugging)', + 'description': 'Debugging only. Sends 1-200 console lines (press, dpad, stick, wait, taps, walk, shot, shots, ' + 'detect, wait_for, expect, game_open; numbers bounded, at most 60 s in all) once per ' + 'request_id, with the program stopped. Needs the lease token. Marks console://B/save stale ' + '(reason cfr_batch) and makes any preflight_token stale. Completed with outcome done, or ' + 'line_failed with failed_line and the error; output holds what the lines printed.', + 'inputSchema': _schema({'request_id': _REQUEST_ID, 'lease_token': _LEASE_TOKEN, + 'lines': {'type': 'array', 'items': {'type': 'string'}, 'minItems': 1, 'maxItems': 200}}, + ['request_id', 'lease_token', 'lines']), 'annotations': _ACTING}, ] @@ -438,7 +464,7 @@ def _call_action(self, client: str, name: str, arguments: Any) -> Dict[str, Any] 'console': args.pop('console', None), 'arguments': args} if 'lease_token' in args: request['lease_token'] = args.pop('lease_token') - call = self.actions.run_program if name == 'run_program' else self.actions.stop_program + call = getattr(self.actions, name) try: record = call(client, request) except control.Refused as e: diff --git a/lab/server/pa_lab/preflight.py b/lab/server/pa_lab/preflight.py new file mode 100644 index 0000000000..082ff3c552 --- /dev/null +++ b/lab/server/pa_lab/preflight.py @@ -0,0 +1,223 @@ +"""`preflight_program`'s pure parts (Slice 5b of fork #131): candidate options against the +program's static contract, the app's `preflight` reply, and the token's binding. + +**Options.** `check_options(contract, options)` checks a candidate against `describe_program`'s +option entries (Lab.md "Describe program") before anything is sent: every key must be one the +program saves, a value must have the entry's type and lie within its bounds, an enum/select value +must be an enabled slug, a group's value is an object checked against the group's own entries +(plus `Enabled` when the group is toggleable). A `type: other` option cannot be staged. Dynamic +visibility and `check_validity` depend on the other options' values; only the app can evaluate +them, on the staged candidate. + +**The app command** (proposed for the app's channel; not built yet, see README "Slice 5b"): +`preflight ` stages the candidate options of the selected program on the UI thread, rejects +unknown keys and disabled values, evaluates `check_validity`, runs the program's prechecks in +order without starting it, and restores the prior options through a scope guard. It prints one +line, `preflight `: + + {"schema": 1, "id": ":", "verdict": "passed" | "failed" | "invalid_options" | "unsupported", + "checks": [{"name", "status": "passed" | "failed" | "skipped", "message"}], + "failure": {"check", "message", "evidence"?} | null, # verdict failed + "invalid": {"key", "message"} | null, # verdict invalid_options + "restored": true | false} + +**The token** is the hash of what the preflight was checked against (`binding`): the app instance, +the program id, the candidate options' hash, the observation versions a run depends on (the last +start/stop transition and `console://B/save`'s status) and the lease (its token's hash), plus a +random salt so it cannot be guessed. The daemon stores the binding under the token's hash; +`run_program` recomputes it from what it reads now and refuses a token whose binding changed +(`preflight_stale`, `evidence.changed`) or that is older than `TOKEN_TTL_S`. +""" + +from __future__ import annotations + +import hashlib +import json +import re +import secrets +from typing import Any, Dict, List, Optional + +from pa_lab.control import Refused + +TOKEN_TTL_S = 120 +PREFLIGHT_RE = re.compile(r'^preflight (\{.*\})$') +VERDICTS = ('passed', 'failed', 'invalid_options', 'unsupported') +CHECK_STATUSES = ('passed', 'failed', 'skipped') +BINDING_FIELDS = ('console', 'instance_id', 'panel', 'options_hash', 'transition', 'save', 'lease') + +__all__ = ['check_options', 'options_hash', 'set_lines', 'parse_reply', 'binding', 'new_token', 'changed', + 'TOKEN_TTL_S'] + + +def _json(value: Any) -> str: + return json.dumps(value, sort_keys=True, separators=(',', ':'), ensure_ascii=False) + + +def dumps(value: Any) -> str: + """Canonical one-line JSON: what the daemon writes into a request and hashes.""" + return _json(value) + + +def options_hash(options: Dict[str, Any]) -> str: + return hashlib.sha256(_json(options).encode('utf-8')).hexdigest() + + +# ---- candidate options against the static contract + +def _bad(path: str, message: str) -> Refused: + return Refused('invalid_arguments', '{}: {}.'.format(path, message), field=path) + + +def _number(value: Any) -> bool: + return isinstance(value, (int, float)) and not isinstance(value, bool) + + +def _check_value(entry: Dict[str, Any], value: Any, path: str) -> None: + kind = entry.get('type') + if kind == 'boolean': + if not isinstance(value, bool): + raise _bad(path, 'expected true or false') + elif kind == 'string': + if not isinstance(value, str): + raise _bad(path, 'expected a string') + elif kind in ('integer', 'number'): + if kind == 'integer' and not (isinstance(value, int) and not isinstance(value, bool)): + raise _bad(path, 'expected an integer') + if not _number(value): + raise _bad(path, 'expected a number') + low, high = entry.get('min'), entry.get('max') + if (low is not None and value < low) or (high is not None and value > high): + raise _bad(path, '{} is outside {}..{}'.format(value, '' if low is None else low, + '' if high is None else high)) + elif kind in ('enum', 'select'): + slugs = {e.get('slug'): e.get('enabled') for e in entry.get('entries') or []} + if not isinstance(value, str) or value not in slugs: + raise _bad(path, 'expected one of {}'.format(', '.join(str(s) for s in slugs))) + if slugs[value] is not True: + raise _bad(path, "'{}' is disabled in this program".format(value)) + elif kind == 'group': + if not isinstance(value, dict): + raise _bad(path, 'expected an object (a group of options)') + entries = list(entry.get('options') or []) + if entry.get('toggleable') and not any(e.get('key') == 'Enabled' for e in entries): + entries.append({'key': 'Enabled', 'type': 'boolean'}) + _check_object(entries, value, path) + else: + raise _bad(path, 'the program contract does not describe this option (type {}); it cannot be staged'.format( + kind)) + + +def _check_object(entries: List[Dict[str, Any]], options: Dict[str, Any], path: str) -> None: + by_key = {e.get('key'): e for e in entries} + for key, value in options.items(): + sub = '{}.{}'.format(path, key) + if key not in by_key: + raise _bad(sub, 'unknown option; the program saves {}'.format(', '.join(str(k) for k in by_key))) + _check_value(by_key[key], value, sub) + + +def check_options(program: Dict[str, Any], options: Dict[str, Any]) -> None: + """Raises Refused(invalid_arguments, field=options.) at the first bad option.""" + _check_object(program['options'], options, 'options') + + +def set_lines(options: Dict[str, Any], prefix: str = '') -> List[str]: + """The app's `set KEY JSON` lines that load `options`, in the caller's order; an object value is + a group (`set GROUP.KEY`). Only for options `check_options` passed (no `type: other`).""" + out: List[str] = [] + for key, value in options.items(): + if isinstance(value, dict): + out.extend(set_lines(value, prefix + key + '.')) + else: + out.append('set {}{} {}'.format(prefix, key, _json(value))) + return out + + +# ---- the app's reply + +def _reply_error(message: str, lines: List[str]) -> Refused: + return Refused('app_unavailable', 'The app\'s preflight reply is unusable: {}.'.format(message), + evidence={'journal': lines}) + + +def parse_reply(lines: List[str]) -> Dict[str, Any]: + """The one `preflight ` line of a journal, checked; Refused(app_unavailable) otherwise.""" + matches = [PREFLIGHT_RE.match(line) for line in lines] + matches = [m for m in matches if m] + if len(matches) != 1: + raise _reply_error('expected one preflight line, found {}'.format(len(matches)), lines) + try: + doc = json.loads(matches[0].group(1)) + except ValueError: + raise _reply_error('invalid JSON', lines) from None + if not isinstance(doc, dict) or doc.get('schema') != 1: + raise _reply_error('schema must be 1', lines) + if not isinstance(doc.get('id'), str) or doc.get('verdict') not in VERDICTS or \ + not isinstance(doc.get('restored'), bool) or not isinstance(doc.get('checks'), list): + raise _reply_error('id, verdict, checks and restored are required', lines) + for check in doc['checks']: + if not isinstance(check, dict) or not isinstance(check.get('name'), str) or \ + check.get('status') not in CHECK_STATUSES or not isinstance(check.get('message', ''), str): + raise _reply_error('a check needs a name, a status and a message', lines) + failure = doc.get('failure') + if doc['verdict'] == 'failed' and not (isinstance(failure, dict) and isinstance(failure.get('check'), str) + and isinstance(failure.get('message'), str)): + raise _reply_error('a failed verdict needs failure.check and failure.message', lines) + invalid = doc.get('invalid') + if doc['verdict'] == 'invalid_options' and not (isinstance(invalid, dict) and isinstance(invalid.get('key'), str) + and isinstance(invalid.get('message'), str)): + raise _reply_error('an invalid_options verdict needs invalid.key and invalid.message', lines) + return doc + + +def outcome(doc: Dict[str, Any], panel: str, lines: List[str]) -> Optional[Dict[str, Any]]: + """The refusal a reply means, or None when every check passed and the options were restored.""" + if not doc['restored']: + return {'code': 'app_unavailable', 'message': 'The app did not restore the prior options after the preflight; ' + 'check the panel before any run.', 'evidence': {'reason': 'options_not_restored', 'reply': doc}} + if doc['id'] != panel: + return {'code': 'app_unavailable', 'message': 'The app preflighted {}, not {}.'.format(doc['id'], panel), + 'evidence': {'reason': 'panel_changed', 'reply': doc}} + if doc['verdict'] == 'unsupported': + return {'code': 'prechecks_unsupported', 'message': '{} declares no start checks; the server does not run a ' + 'program whose checks it cannot run.'.format(panel)} + if doc['verdict'] == 'invalid_options': + key = doc['invalid']['key'] + return {'code': 'invalid_arguments', 'field': 'options.' + key, + 'message': 'The app refused option {}: {}'.format(key, doc['invalid']['message'])} + if doc['verdict'] == 'failed': + failure = doc['failure'] + refusal = {'code': 'precheck_failed', 'check': failure['check'], 'message': failure['message']} + if isinstance(failure.get('evidence'), dict): + refusal['evidence'] = failure['evidence'] + return refusal + if any(check['status'] == 'failed' for check in doc['checks']): + return {'code': 'app_unavailable', 'message': 'The app said passed but lists a failed check.', + 'evidence': {'journal': lines}} + return None + + +# ---- the token + +def transition_key(app: Dict[str, Any]) -> Optional[str]: + t = app['program']['last_transition'] + return None if t is None else '{}|{}|{}|{}'.format(t['action'], t['source'], t['request_id'], t['at']) + + +def binding(state: Dict[str, Any], save: Dict[str, Any], panel: str, options: Dict[str, Any], + lease_token_hash: Optional[str]) -> Dict[str, Any]: + """What a preflight was checked against and a run must still find (field names: BINDING_FIELDS).""" + app = state['app'] + return {'console': state['console'], 'instance_id': app['instance']['id'], 'panel': panel, + 'options_hash': options_hash(options), 'transition': transition_key(app), + 'save': [save.get('status'), save.get('reason'), save.get('since')], 'lease': lease_token_hash} + + +def new_token(bound: Dict[str, Any]) -> str: + salt = secrets.token_hex(16) + return 'pf1.' + hashlib.sha256((salt + _json(bound)).encode('utf-8')).hexdigest() + + +def changed(issued: Dict[str, Any], now: Dict[str, Any]) -> List[str]: + """The binding fields that differ; 'lease' rather than the hashes.""" + return [field for field in BINDING_FIELDS if issued.get(field) != now.get(field)] diff --git a/lab/server/tests/fake_lab.py b/lab/server/tests/fake_lab.py index 8142b14988..0197f8e0a7 100644 --- a/lab/server/tests/fake_lab.py +++ b/lab/server/tests/fake_lab.py @@ -9,7 +9,7 @@ from pathlib import Path import tempfile -from pa_lab import timestamps +from pa_lab import cfr, timestamps FIXTURES = Path(__file__).resolve().parent.parent / 'contracts' / 'v1' @@ -94,6 +94,18 @@ def _atomic(path, text): 'schema': 1, } +def describe_reply(**changes): + """The app's `describe_program` line for PROGRAM_CONTRACT with `changes` (e.g. prechecks=...).""" + return 'describe_program ' + json.dumps(dict(PROGRAM_CONTRACT, **changes), sort_keys=True, + separators=(',', ':')) + '\n' + + +PRECHECKS = [ + {'name': 'game_open', 'needs': "The program's game is in front.", 'skippable': False}, + {'name': 'free_party_slots', 'needs': '1 free party slot.', 'skippable': True}, +] +CURRENT_OPTIONS = {'LANGUAGE': 'en', 'START_SEGMENT': 0, 'NOTIFICATIONS': {'Enabled': True}} + PNG = b'\x89PNG\r\n\x1a\n' + b'\x00\x00\x00\rIHDR' + b'\x00' * 17 + b'\x00\x00\x00\x00IEND\xaeB`\x82' @@ -133,8 +145,7 @@ def __init__(self, lab, clock): self.reply_dir = None self.png = PNG self.replies = { - 'describe_program': 'describe_program ' + json.dumps(PROGRAM_CONTRACT, sort_keys=True, - separators=(',', ':')) + '\n', + 'describe_program': describe_reply(), 'game_open': 'game_open yes\n', } self.line_errors = {} @@ -144,6 +155,15 @@ def __init__(self, lab, clock): self.after = {} self._stopping_left = None self.hold = set() + # Slice 5b: the panel's current options, the proposed `preflight` command and console lines. + self.options = copy.deepcopy(CURRENT_OPTIONS) + self.check_outcomes = {} # precheck name -> (status, message[, evidence]); default passed + self.invalid = None # (key, message): the app's check_validity refuses the candidate + self.restore = True # False: the app forgets to restore the prior options + self.preflight_id = None # the program id the reply names (default: the selected panel) + self.staged_seen = [] # the options the prechecks ran against, per preflight + self.line_fail = None # (batch line index, error): that console line fails + self.lines_run = [] self.requests = [] self.pending = [] clock.on_sleep.append(self.tick) @@ -181,6 +201,12 @@ def pick_up(self, request): if words[0] in ('start', 'stop'): self.start_or_stop(request.stem, text, line, words[0]) return + if words[0] == 'preflight': + self.preflight(request.stem, line) + return + if words[0] in cfr.COMMANDS: + self.console_lines(request.stem, text.splitlines()[1:]) + return pending = {'name': request.stem, 'line': line, 'label': words[1], 'timeout_ms': int(words[2]), 'started': self.clock.now(), 'after': self.lab.state['video']['frame_seq'] or 0} if self.journal_error is not None: @@ -211,12 +237,79 @@ def start_or_stop(self, name, text, line, command): if state != ('stopped' if command == 'start' else 'running'): self.finish(pending, '', 'Program is {}.'.format(state.upper())) return + printed = '' + for set_line in (l for l in lines if l.startswith('set ')): + key, value = set_line.split(' ', 2)[1:] + target = self.options + parts = key.split('.') + for part in parts[:-1]: + target = target.setdefault(part, {}) + target[parts[-1]] = json.loads(value) + printed += 'set {} = {}\n'.format(key, value) self.transition(command, 'lab', request_id, 'running' if command == 'start' else 'stopping') if command == 'stop': self._stopping_left = self.stop_ticks if command in self.after: self.after[command](self) - self.finish(pending, '') + self.finish(pending, printed) + + def preflight(self, name, line): + """The proposed `preflight ` (pa_lab/preflight.py): stage, check, run the prechecks in + order up to the first failure, restore (unless `restore` is off), print one reply line.""" + pending = {'name': name, 'line': line} + if 'preflight' in self.line_errors: # today's app: no such command + self.finish(pending, '', self.line_errors['preflight']) + return + prior = copy.deepcopy(self.options) + candidate = json.loads(line.split(' ', 1)[1]) + staged = copy.deepcopy(prior) + for key, value in candidate.items(): + if isinstance(value, dict): + staged.setdefault(key, {}).update(value) + else: + staged[key] = value + self.options = staged + reply = {'schema': 1, 'id': self.preflight_id or self.lab.state['panel']['id'], 'checks': [], + 'failure': None, 'invalid': None, 'restored': True, 'verdict': 'passed'} + contract = json.loads(self.replies['describe_program'].split(' ', 1)[1]) + if self.invalid is not None: + reply.update(verdict='invalid_options', invalid={'key': self.invalid[0], 'message': self.invalid[1]}) + elif contract['prechecks'] == 'unsupported': + reply['verdict'] = 'unsupported' + else: + self.staged_seen.append(copy.deepcopy(staged)) + for check in contract['prechecks']: + outcome = self.check_outcomes.get(check['name'], ('passed', '')) + reply['checks'].append({'name': check['name'], 'status': outcome[0], 'message': outcome[1]}) + if outcome[0] == 'failed': + reply.update(verdict='failed', failure=dict({'check': check['name'], 'message': outcome[1]}, + **({'evidence': outcome[2]} if len(outcome) > 2 else {}))) + break + if self.restore: + self.options = prior + else: + reply['restored'] = False + self.finish(pending, 'preflight {}\n'.format(json.dumps(reply, sort_keys=True, separators=(',', ':')))) + + def console_lines(self, name, lines): + """Console lines as the channel runs them: in order, one journal line each, stopping at the first + failure (`line_fail`). Presses are refused unless the program is stopped.""" + journal = '' + for index, line in enumerate(lines): + number = index + 2 # line 1 is `request ` + error = None + if self.line_fail is not None and self.line_fail[0] == index: + error = self.line_fail[1] + elif self.lab.state['program']['state'] != 'stopped' and line.split()[0] not in ('shot', 'detect'): + error = 'Program is {}.'.format(self.lab.state['program']['state'].upper()) + if error is not None: + journal += 'error {} ({}): {}\n'.format(number, line, error) + break + self.lines_run.append(line) + if line.startswith('shot '): + journal += str(self.lab.dir / 'outbox' / '{}_{}.png'.format(name, line.split()[1])) + '\n' + journal += 'ok {} {} @0-41ms\n'.format(number, line) + (self.lab.dir / 'outbox' / (name + '.out')).write_text(journal + 'end\n', encoding='utf-8') def transition(self, action, source, request_id=None, state=None): """A start or stop as the app records it; `state` defaults to where the action leads.""" diff --git a/lab/server/tests/test_actions.py b/lab/server/tests/test_actions.py index 64a05dc1c8..14cd6fdaee 100644 --- a/lab/server/tests/test_actions.py +++ b/lab/server/tests/test_actions.py @@ -11,7 +11,7 @@ from pa_lab import actions, contracts, control, lab_reader, mcp_server from tests.control_fixtures import START, acquire, new_id, open_plane, temp_db_path -from tests.fake_lab import FakeApp, FakeClock, FakeLab, golden +from tests.fake_lab import PRECHECKS, FakeApp, FakeClock, FakeLab, describe_reply, golden PANEL = 'PokemonFRLG:AutoStory' CLIENT = 'claude:frlg-lane' @@ -26,11 +26,19 @@ def stopped_app(): return doc +NO_TOKEN = 'pf1.never-issued' + + def run_request(token, request_id=None, panel=PANEL, options=None, **arguments): return {'schema': 1, 'request_id': request_id or new_id(), 'console': 'B', 'lease_token': token, 'arguments': dict({'panel': panel, 'options': {} if options is None else options}, **arguments)} +def preflight_request(token, request_id=None, panel=PANEL, options=None): + return {'schema': 1, 'request_id': request_id or new_id(), 'console': 'B', 'lease_token': token, + 'arguments': {'panel': panel, 'options': {} if options is None else options}} + + def stop_request(token, request_id=None): return {'schema': 1, 'request_id': request_id or new_id(), 'console': 'B', 'lease_token': token, 'arguments': {}} @@ -42,6 +50,7 @@ def setUp(self): self.lab = FakeLab(self) self.lab.write_state(stopped_app()) self.app = FakeApp(self.lab, self.clock) + self.app.replies['describe_program'] = describe_reply(prechecks=PRECHECKS) self.alive = True self.path = temp_db_path(self) self.open_daemon() @@ -61,11 +70,26 @@ def acquired(self): def run_program(self, request): return self.actions.run_program(CLIENT, request) + def preflight(self, request): + return self.actions.preflight_program(CLIENT, request) + + def run_req(self, token, request_id=None, panel=PANEL, options=None, **arguments): + """A run request with a token from a preflight of the same panel and options made now; when + that preflight does not pass, a token never issued (for refusals checked before the token).""" + if 'preflight_token' not in arguments: + try: + record = self.preflight(preflight_request(token, panel=panel, options=options)) + except control.Refused: + record = {'status': 'refused'} + passed = record['status'] == 'completed' + arguments['preflight_token'] = record['result']['preflight_token'] if passed else NO_TOKEN + return run_request(token, request_id, panel, options, **arguments) + def stop_program(self, request): return self.actions.stop_program(CLIENT, request) def started(self, token): - record = self.run_program(run_request(token)) + record = self.run_program(self.run_req(token)) self.assertEqual(record['status'], 'completed', record) return record @@ -99,10 +123,11 @@ def once(): class RunTests(ActionTest): def test_a_run_starts_once_and_the_lease_runs(self): token = self.acquired() - request = run_request(token) + request = self.run_req(token) record = self.run_program(request) self.assertEqual(record['status'], 'completed') - self.assertEqual(record['arguments'], {'panel': PANEL, 'options': {}}) + self.assertEqual(record['arguments'], {'panel': PANEL, 'options': {}, + 'preflight_token': request['arguments']['preflight_token']}) self.assertEqual(record['result']['correlated_by'], 'request_id') self.assertIsNone(record['result']['ended']) self.assertEqual(self.sent(), [(request['request_id'], 'start 10')]) @@ -114,7 +139,7 @@ def test_a_run_starts_once_and_the_lease_runs(self): def test_an_exact_retry_returns_the_stored_record_and_sends_nothing(self): token = self.acquired() - request = run_request(token) + request = self.run_req(token) first = self.run_program(request) self.clock.sleep(5) again = self.run_program(request) @@ -123,7 +148,7 @@ def test_an_exact_retry_returns_the_stored_record_and_sends_nothing(self): def test_a_retry_of_a_refusal_returns_the_same_refusal(self): token = self.acquired() - request = run_request(token, panel='PokemonBDSP:AutoStory') + request = self.run_req(token, panel='PokemonBDSP:AutoStory') first = self.run_program(request) self.lab.update_state(**{'panel.id': 'PokemonBDSP:AutoStory'}) again = self.run_program(request) @@ -133,11 +158,11 @@ def test_a_retry_of_a_refusal_returns_the_same_refusal(self): def test_a_duplicate_while_the_first_is_in_flight_gets_the_pending_record(self): token = self.acquired() - request = run_request(token) + request = self.run_req(token) seen = {} self.on_first_sleep(lambda: seen.update( duplicate=self.run_program(request), - other=self.run_program(run_request(token)))) + other=self.run_program(run_request(token, preflight_token=NO_TOKEN)))) record = self.run_program(request) self.assertEqual(seen['duplicate']['status'], 'pending') self.assertEqual((seen['other']['status'], seen['other']['refusal']['code']), @@ -148,9 +173,9 @@ def test_a_duplicate_while_the_first_is_in_flight_gets_the_pending_record(self): def test_the_same_request_id_with_other_arguments_is_refused_unstored(self): token = self.acquired() request_id = new_id() - self.run_program(run_request(token, request_id)) + self.run_program(self.run_req(token, request_id)) with self.assertRaises(control.Refused) as raised: - self.run_program(run_request(token, request_id, panel='PokemonBDSP:AutoStory')) + self.run_program(self.run_req(token, request_id, panel='PokemonBDSP:AutoStory')) self.assertEqual(raised.exception.refusal['code'], 'request_id_reused') def test_refusals_send_nothing_and_are_stored(self): @@ -186,9 +211,6 @@ def owner_claimed(test): dict(name='the app is not running', input=(app_gone, {}), expectation='app_unavailable'), dict(name='another panel is selected', input=(other_panel, {}), expectation='invalid_arguments'), dict(name='the settings page is selected', input=(settings_page, {}), expectation='panel_not_program'), - dict(name='options to stage', input=(None, {'options': {'START_SEGMENT': 3}}), - expectation='invalid_arguments'), - dict(name='a preflight token', input=(None, {'preflight_token': 'x'}), expectation='invalid_arguments'), dict(name='the controller is not ready', input=(controller, {}), expectation='controller_not_ready'), dict(name='a lab script owns the channel', input=(owner_claimed, {}), expectation='console_held_by_agent'), ] @@ -199,7 +221,7 @@ def owner_claimed(test): setup, changes = case['input'] if setup is not None: setup(self) - request = run_request(changes.pop('lease_token', token), **changes) + request = run_request(changes.pop('lease_token', token), preflight_token=NO_TOKEN, **changes) record = self.run_program(request) self.assertEqual((record['status'], record['refusal']['code']), ('refused', case['expectation'])) self.assertEqual(self.run_program(request), record) @@ -258,11 +280,11 @@ class RaceTests(ActionTest): def test_ui_takeover_during_start(self): token = self.acquired() self.app.after['start'] = lambda app: app.transition('stop', 'ui') - record = self.run_program(run_request(token)) + record = self.run_program(self.run_req(token)) self.assertEqual((record['status'], record['refusal']['code']), ('failed', 'console_held_by_user')) self.assertTrue(record['refusal']['evidence']['journal_ok']) self.assertEqual(self.lease(), 'user') - retry = self.run_program(run_request(token)) + retry = self.run_program(self.run_req(token)) self.assertEqual(retry['refusal']['code'], 'console_held_by_user') self.assertEqual(len(self.sent()), 1) @@ -288,13 +310,13 @@ def test_lease_lost_by_expiry_before_the_start(self): token = self.acquired() self.clock.sleep(1199) self.clock.sleep(1) - record = self.run_program(run_request(token)) + record = self.run_program(self.run_req(token)) self.assertEqual(record['refusal']['code'], 'lease_expired') self.assertEqual(self.sent(), []) def test_app_restart_between_requests(self): token = self.acquired() - request = run_request(token) + request = self.run_req(token) self.run_program(request) self.assertEqual(self.lease(), 'agent_running') restarted = stopped_app() @@ -320,9 +342,10 @@ def test_app_restart_during_the_start(self): def restart(app): app.lab.update_state(**{'instance.id': NEW_INSTANCE, 'program.state': 'stopped', 'program.last_transition': None}) + request = self.run_req(token) self.app.hold.add('start') self.on_first_sleep(lambda: restart(self.app)) - record = self.run_program(run_request(token)) + record = self.run_program(request) self.assertEqual((record['status'], record['refusal']['code']), ('failed', 'app_unavailable')) self.assertEqual(record['refusal']['evidence']['reason'], 'instance_replaced') self.assertEqual(self.lease(), 'agent_idle') @@ -333,7 +356,7 @@ def test_the_run_ends_before_the_start_is_answered(self): self.setUp() token = self.acquired() self.app.after['start'] = lambda app: app.transition('stop', source) - record = self.run_program(run_request(token)) + record = self.run_program(self.run_req(token)) self.assertEqual(record['status'], 'completed') self.assertEqual(record['result']['ended']['source'], source) self.assertEqual(record['result']['correlated_by'], 'time') @@ -354,7 +377,7 @@ def test_the_run_ends_before_the_stop_line(self): def test_the_run_ends_after_the_status_read_but_before_the_lease_runs(self): """observe applies the run's end before finish_action: the lease must not enter agent_running.""" token = self.acquired() - request = run_request(token) + request = self.run_req(token) self.app.hold.add('start') record = self.run_program(request) self.assertEqual(record['status'], 'pending') @@ -372,8 +395,9 @@ def test_the_run_ends_after_the_status_read_but_before_the_lease_runs(self): class TicketTests(ActionTest): def test_a_request_the_app_never_took_is_withdrawn(self): token = self.acquired() + request = self.run_req(token) self.app.serve = False - record = self.run_program(run_request(token)) + record = self.run_program(request) self.assertEqual((record['status'], record['refusal']['code']), ('failed', 'timeout')) self.assertEqual(self.inbox(), []) self.app.serve = True @@ -383,7 +407,7 @@ def test_a_request_the_app_never_took_is_withdrawn(self): def test_a_taken_request_stays_pending_and_settles_after_a_daemon_restart(self): token = self.acquired() - request = run_request(token) + request = self.run_req(token) self.app.hold.add('start') record = self.run_program(request) self.assertEqual(record['status'], 'pending') @@ -398,7 +422,7 @@ def test_a_taken_request_stays_pending_and_settles_after_a_daemon_restart(self): def test_a_pending_request_fails_when_the_app_exits(self): token = self.acquired() self.app.hold.add('start') - request = run_request(token) + request = self.run_req(token) self.assertEqual(self.run_program(request)['status'], 'pending') self.alive = False self.actions.settle_pending('B') @@ -417,7 +441,10 @@ def test_the_tools_are_listed_only_with_actions(self): server = mcp_server.LabMcp({'B': self.reader}, control_plane=self.plane, actions=self.actions if case['input']['actions'] else None) names = [t['name'] for t in server.call('tools/list', {})['tools']] - self.assertEqual('run_program' in names and 'stop_program' in names, case['expectation']) + self.assertEqual({'preflight_program', 'run_program', 'stop_program', 'cfr_batch'} <= set(names), + case['expectation']) + self.assertEqual(bool({'preflight_program', 'run_program', 'stop_program', 'cfr_batch'} & set(names)), + case['expectation']) def test_read_only_mode_refuses_actions(self): with self.assertRaises(ValueError): @@ -426,14 +453,19 @@ def test_read_only_mode_refuses_actions(self): def test_run_and_stop_through_tools_call(self): server = mcp_server.LabMcp({'B': self.reader}, control_plane=self.plane, actions=self.actions) token = self.acquired() - run = server.call('tools/call', {'name': 'run_program', 'arguments': { + checked = server.call('tools/call', {'name': 'preflight_program', 'arguments': { 'console': 'B', 'request_id': new_id(), 'lease_token': token, 'panel': PANEL, 'options': {}}}, client=CLIENT) + self.assertFalse(checked['isError'], checked) + preflight_token = checked['structuredContent']['result']['preflight_token'] + run = server.call('tools/call', {'name': 'run_program', 'arguments': { + 'console': 'B', 'request_id': new_id(), 'lease_token': token, 'panel': PANEL, 'options': {}, + 'preflight_token': preflight_token}}, client=CLIENT) self.assertFalse(run['isError'], run) self.assertEqual(run['structuredContent']['status'], 'completed') refused = server.call('tools/call', {'name': 'run_program', 'arguments': { - 'console': 'B', 'request_id': new_id(), 'lease_token': token, 'panel': PANEL, 'options': {}}}, - client=CLIENT) + 'console': 'B', 'request_id': new_id(), 'lease_token': token, 'panel': PANEL, 'options': {}, + 'preflight_token': preflight_token}}, client=CLIENT) self.assertTrue(refused['isError']) self.assertEqual(refused['structuredContent']['refusal']['code'], 'program_running') contracts.validate('refusal', refused['structuredContent']) diff --git a/lab/server/tests/test_cfr.py b/lab/server/tests/test_cfr.py new file mode 100644 index 0000000000..8f43eae7ae --- /dev/null +++ b/lab/server/tests/test_cfr.py @@ -0,0 +1,104 @@ +"""The cfr_batch allowlist (pa_lab/cfr.py): which lines pass, what each may cost, and the budget. + +Run from the repo root: python3 -m unittest discover -s lab/server/tests -t lab/server -v +""" + +import unittest + +from pa_lab import cfr, control + + +class AllowedLineTests(unittest.TestCase): + def test_allowed_lines_and_their_worst_case(self): + cases = [ + dict(name='press with defaults', input='press A', expectation=160 + 160 + 200), + dict(name='press two buttons, hold and release', input='press L+R 100 1500', expectation=1800), + dict(name='HOME is a button', input='press HOME 100 1000', expectation=1300), + dict(name='dpad with defaults', input='dpad DOWN', expectation=100 + 200 + 200), + dict(name='dpad holding B', input='dpad UP 160 400 B', expectation=760), + dict(name='stick with a button', input='stick L 0 1 1000 160 B', expectation=1360), + dict(name='stick with decimals', input='stick R -0.5 0.25', expectation=520), + dict(name='wait', input='wait 1500', expectation=1700), + dict(name='taps counts every tap', input='taps LEFT 3', expectation=3 * (160 + 640 + 200) + 200), + dict(name='taps all with timings', input='taps RIGHT 2 all 80 400 12.5', expectation=2 * 680 + 200), + dict(name='walk default max', input='walk 0 1', expectation=5000 + 400 + 200), + dict(name='walk with max, still and B', input='walk -1 0 2000 10 B', expectation=2600), + dict(name='shot', input='shot after_a', expectation=1200), + dict(name='shots', input='shots door 3 250', expectation=3 * 1250 + 200), + dict(name='detect', input='detect home', expectation=1200), + dict(name='wait_for default timeout', input='wait_for black', expectation=6200), + dict(name='expect with timeout', input='expect home 3000', expectation=4200), + dict(name='game_open', input='game_open', expectation=1200), + ] + for case in cases: + with self.subTest(case['name']): + self.assertEqual(cfr.check_line(case['input']), case['expectation']) + + +class RefusedLineTests(unittest.TestCase): + def test_lines_outside_the_allowlist(self): + cases = [ + dict(name='channel: start', input='start', expectation='not an allowed'), + dict(name='channel: select', input='select PokemonBDSP:AutoStory', expectation='not an allowed'), + dict(name='channel: set', input='set START_SEGMENT "0"', expectation='not an allowed'), + dict(name='channel: stop', input='stop', expectation='not an allowed'), + dict(name='channel: owner release', input='owner release', expectation='not an allowed'), + dict(name='channel: a second request id', input='request 3e8a6c21-7f90-4b2d-a1c4-6d0e9b5f8a73', + expectation='not an allowed'), + dict(name='channel: keepawake', input='keepawake off', expectation='not an allowed'), + dict(name='channel: popups', input='popups close', expectation='not an allowed'), + dict(name='close_game', input='close_game', expectation='not an allowed'), + dict(name='history', input='history run', expectation='not an allowed'), + dict(name='snapshot', input='snapshot frame 2000', expectation='not an allowed'), + dict(name='state', input='state', expectation='not an allowed'), + dict(name='a comment', input='# press A', expectation='not an allowed'), + dict(name='blank', input='', expectation='canonical'), + dict(name='two spaces', input='press A', expectation='canonical'), + dict(name='trailing space', input='press A ', expectation='canonical'), + dict(name='a tab', input='press\tA', expectation='canonical'), + dict(name='a newline smuggles a second command', input='press A\nstart', expectation='canonical'), + dict(name='CAPTURE button', input='press CAPTURE', expectation='buttons'), + dict(name='repeated button', input='press A+A', expectation='buttons'), + dict(name='lower-case button', input='press a', expectation='buttons'), + dict(name='hold too long', input='press A 10001', expectation='hold'), + dict(name='negative release', input='press A 100 -1', expectation='release'), + dict(name='leading zero', input='wait 0100', expectation='ms'), + dict(name='too many arguments', input='press A 1 2 3', expectation='usage'), + dict(name='bad direction', input='dpad NORTH', expectation='direction'), + dict(name='stick out of range', input='stick L 1.5 0', expectation='x must'), + dict(name='stick in exponent form', input='stick L 1e0 0', expectation='x must'), + dict(name='wait too long', input='wait 30001', expectation='ms'), + dict(name='too many taps', input='taps UP 21', expectation='N must'), + dict(name='walk max too long', input='walk 0 1 10001', expectation='max'), + dict(name='shot label with a slash', input='shot ../x', expectation='label'), + dict(name='too many shots', input='shots s 11', expectation='count'), + dict(name='detect on an image path', input='detect home /tmp/x.png', expectation='usage'), + dict(name='detector with a path', input='detect ../home', expectation='detector'), + dict(name='game_open on an image', input='game_open image', expectation='usage'), + ] + for case in cases: + with self.subTest(case['name']): + with self.assertRaises(ValueError) as caught: + cfr.check_line(case['input']) + self.assertIn(case['expectation'], str(caught.exception)) + + +class BatchTests(unittest.TestCase): + def test_the_first_bad_line_is_named(self): + with self.assertRaises(control.Refused) as caught: + cfr.check(['press A', 'start', 'stop']) + self.assertEqual(caught.exception.refusal['code'], 'invalid_arguments') + self.assertEqual(caught.exception.refusal['field'], 'arguments.lines[1]') + + def test_a_batch_over_the_budget_is_refused(self): + with self.assertRaises(control.Refused) as caught: + cfr.check(['wait 30000', 'wait 30000']) + self.assertEqual(caught.exception.refusal['field'], 'arguments.lines') + self.assertEqual(caught.exception.refusal['evidence']['limit_ms'], cfr.BATCH_MAX_MS) + + def test_a_batch_within_the_budget_returns_its_worst_case(self): + self.assertEqual(cfr.check(['shot before', 'press B 160 640', 'shot after']), 1200 + 1000 + 1200) + + +if __name__ == '__main__': + unittest.main() From f6b8189a71495e09b8c14a103b3ceedab2206014 Mon Sep 17 00:00:00 2001 From: dayfine Date: Sun, 15 Nov 2026 17:29:38 -0800 Subject: [PATCH 5/7] pa_lab: preflight and cfr_batch fake-lab tests --- lab/server/pa_lab/preflight.py | 5 +- lab/server/tests/test_cfr_batch.py | 108 ++++++++++++++ lab/server/tests/test_preflight.py | 227 +++++++++++++++++++++++++++++ 3 files changed, 338 insertions(+), 2 deletions(-) create mode 100644 lab/server/tests/test_cfr_batch.py create mode 100644 lab/server/tests/test_preflight.py diff --git a/lab/server/pa_lab/preflight.py b/lab/server/pa_lab/preflight.py index 082ff3c552..e0f2032dd2 100644 --- a/lab/server/pa_lab/preflight.py +++ b/lab/server/pa_lab/preflight.py @@ -122,8 +122,9 @@ def check_options(program: Dict[str, Any], options: Dict[str, Any]) -> None: def set_lines(options: Dict[str, Any], prefix: str = '') -> List[str]: - """The app's `set KEY JSON` lines that load `options`, in the caller's order; an object value is - a group (`set GROUP.KEY`). Only for options `check_options` passed (no `type: other`).""" + """The app's `set KEY JSON` lines that load `options`, in key order (a record keeps them as + canonical JSON); an object value is a group (`set GROUP.KEY`). Only for options `check_options` + passed (no `type: other`).""" out: List[str] = [] for key, value in options.items(): if isinstance(value, dict): diff --git a/lab/server/tests/test_cfr_batch.py b/lab/server/tests/test_cfr_batch.py new file mode 100644 index 0000000000..c1863a2aa0 --- /dev/null +++ b/lab/server/tests/test_cfr_batch.py @@ -0,0 +1,108 @@ +"""cfr_batch against the fake app (Slice 5b, fork #131): a leased batch with the program stopped, +the save fact marked stale before anything is pressed, the journal's outcome, and the refusals +that send nothing. + +Run from the repo root: python3 -m unittest discover -s lab/server/tests -t lab/server -v +""" + +import unittest + +from pa_lab import contracts +from tests.control_fixtures import new_id +from tests.test_actions import CLIENT, ActionTest +from tests.test_preflight import save_known + + +def batch_request(token, lines, request_id=None): + return {'schema': 1, 'request_id': request_id or new_id(), 'console': 'B', 'lease_token': token, + 'arguments': {'lines': lines}} + + +class CfrBatchTest(ActionTest): + def cfr(self, request): + return self.actions.cfr_batch(CLIENT, request) + + def batches_sent(self): + return [text.splitlines()[1:] for text in self.app.requests + if text.splitlines()[1].split()[0] in ('press', 'dpad', 'shot', 'wait', 'stick')] + + def test_a_batch_runs_once_and_marks_the_save_stale(self): + self.lab.write_save(save_known(self)) + self.assertEqual(self.reader.save_resource()['status'], 'known') + token = self.acquired() + request = batch_request(token, ['shot before', 'press B 160 640', 'shot after']) + record = self.cfr(request) + self.assertEqual(record['status'], 'completed', record) + self.assertEqual(record['result']['outcome'], 'done') + self.assertEqual(len(record['result']['output']), 2) # the two shot paths + self.assertEqual(self.app.lines_run, ['shot before', 'press B 160 640', 'shot after']) + save = self.reader.save_resource() + self.assertEqual((save['status'], save['reason']), ('stale', 'cfr_batch')) + self.assertEqual(self.cfr(request), record) # an exact retry presses nothing again + self.assertEqual(len(self.batches_sent()), 1) + contracts.validate('request_record', record) + + def test_a_failing_line_stops_the_batch_and_is_named(self): + token = self.acquired() + self.app.line_fail = (1, 'timeout: home not seen within 3000 ms') + record = self.cfr(batch_request(token, ['press HOME', 'expect home 3000', 'press A'])) + self.assertEqual(record['status'], 'completed') + self.assertEqual((record['result']['outcome'], record['result']['failed_line']), ('line_failed', 2)) + self.assertEqual(self.app.lines_run, ['press HOME']) + + def test_refusals_send_nothing_and_leave_the_save_alone(self): + def other_program(test): + test.app.transition('start', 'lab', new_id()) + + def ui_click(test): + test.app.transition('stop', 'ui') + + def settings_page(test): + test.lab.update_state(**{'panel.id': None, 'panel.kind': 'other', 'program.state': 'no_program'}) + + def owner_claimed(test): + test.lab.update_state(**{'channel.owner_claimed': True}) + + cases = [ + dict(name='a program runs', input=(other_program, ['press A']), expectation='program_running'), + dict(name='the user holds the console', input=(ui_click, ['press A']), expectation='console_held_by_user'), + dict(name='the settings page', input=(settings_page, ['press A']), expectation='panel_not_program'), + dict(name='a lab script owns the channel', input=(owner_claimed, ['press A']), + expectation='console_held_by_agent'), + dict(name='a channel command', input=(None, ['press A', 'start']), expectation='invalid_arguments'), + dict(name='close_game', input=(None, ['close_game']), expectation='invalid_arguments'), + dict(name='over the time budget', input=(None, ['wait 30000', 'wait 30000']), + expectation='invalid_arguments'), + ] + for case in cases: + with self.subTest(case['name']): + self.setUp() + self.lab.write_save(save_known(self)) + token = self.acquired() + setup, lines = case['input'] + if setup is not None: + setup(self) + record = self.cfr(batch_request(token, lines)) + self.assertEqual((record['status'], record['refusal']['code']), ('refused', case['expectation'])) + self.assertEqual(self.app.lines_run, []) + self.assertEqual(self.inbox(), []) + self.assertIsNone(self.plane.save_mark('B')) + + def test_without_a_lease_nothing_is_pressed(self): + record = self.cfr(batch_request('not-a-lease', ['press A'])) + self.assertEqual(record['refusal']['code'], 'lease_invalid') + self.assertEqual(self.app.lines_run, []) + + def test_a_batch_the_app_never_took_is_withdrawn_and_the_save_stays_stale(self): + """The mark is written before the request: a press may have happened if the daemon died after.""" + self.lab.write_save(save_known(self)) + token = self.acquired() + self.app.serve = False + record = self.cfr(batch_request(token, ['press A'])) + self.assertEqual((record['status'], record['refusal']['code']), ('failed', 'timeout')) + self.assertEqual(self.inbox(), []) + self.assertEqual(self.reader.save_resource()['reason'], 'cfr_batch') + + +if __name__ == '__main__': + unittest.main() diff --git a/lab/server/tests/test_preflight.py b/lab/server/tests/test_preflight.py new file mode 100644 index 0000000000..b99346a2ce --- /dev/null +++ b/lab/server/tests/test_preflight.py @@ -0,0 +1,227 @@ +"""preflight_program and the token run_program needs (Slice 5b, fork #131), against the fake app's +proposed `preflight` command: options checked before anything is sent, the reply's verdicts, the +prior options restored (and a reply that says they were not), and every way a token goes stale. + +Run from the repo root: python3 -m unittest discover -s lab/server/tests -t lab/server -v +""" + +import json +import unittest + +from pa_lab import contracts, preflight +from tests.fake_lab import CURRENT_OPTIONS, describe_reply, golden +from tests.test_actions import CLIENT, NEW_INSTANCE, NO_TOKEN, ActionTest, preflight_request, run_request + + +class PreflightTest(ActionTest): + def preflights_sent(self): + """The `preflight ` lines the app took, as parsed candidates.""" + return [json.loads(text.splitlines()[-1].split(' ', 1)[1]) for text in self.app.requests + if text.splitlines()[-1].startswith('preflight ')] + + def passed(self, token, options=None): + record = self.preflight(preflight_request(token, options=options)) + self.assertEqual(record['status'], 'completed', record) + return record + + +class OptionTests(PreflightTest): + def test_bad_options_are_refused_before_anything_is_sent(self): + cases = [ + dict(name='unknown key', input={'START_CHECKPOINT': '13.01'}, expectation='options.START_CHECKPOINT'), + dict(name='disabled enum entry', input={'LANGUAGE': 'ja'}, expectation='options.LANGUAGE'), + dict(name='enum slug not listed', input={'LANGUAGE': 'fr'}, expectation='options.LANGUAGE'), + dict(name='integer above max', input={'START_SEGMENT': 41}, expectation='options.START_SEGMENT'), + dict(name='integer given as a string', input={'START_SEGMENT': '3'}, expectation='options.START_SEGMENT'), + dict(name='boolean given as an integer', input={'START_SEGMENT': True}, expectation='options.START_SEGMENT'), + dict(name='group given a scalar', input={'NOTIFICATIONS': False}, expectation='options.NOTIFICATIONS'), + dict(name='unknown key in a group', input={'NOTIFICATIONS': {'Email': True}}, + expectation='options.NOTIFICATIONS.Email'), + ] + for case in cases: + with self.subTest(case['name']): + self.setUp() + token = self.acquired() + record = self.preflight(preflight_request(token, options=case['input'])) + self.assertEqual((record['status'], record['refusal']['code'], record['refusal']['field']), + ('refused', 'invalid_arguments', case['expectation'])) + self.assertEqual(self.preflights_sent(), []) + self.assertEqual(self.app.options, CURRENT_OPTIONS) + + def test_good_options_reach_the_app_and_the_checks_see_them(self): + token = self.acquired() + options = {'START_SEGMENT': 3, 'NOTIFICATIONS': {'Enabled': False}} + record = self.passed(token, options) + self.assertEqual(self.preflights_sent(), [options]) + self.assertEqual(self.app.staged_seen, [{'LANGUAGE': 'en', 'START_SEGMENT': 3, + 'NOTIFICATIONS': {'Enabled': False}}]) + self.assertEqual(self.app.options, CURRENT_OPTIONS) # put back after the checks + self.assertEqual([c['name'] for c in record['result']['checks']], ['game_open', 'free_party_slots']) + self.assertEqual(record['result']['options_hash'], preflight.options_hash(options)) + contracts.validate('request_record', record) + + +class VerdictTests(PreflightTest): + def test_a_failed_check_is_named_and_the_options_are_restored(self): + token = self.acquired() + self.app.check_outcomes['game_open'] = ('failed', 'The game is not open: the console shows HOME.', + {'screen': 'home'}) + record = self.preflight(preflight_request(token, options={'START_SEGMENT': 7})) + self.assertEqual(record['status'], 'failed') + self.assertEqual(record['refusal'], {'code': 'precheck_failed', 'check': 'game_open', + 'message': 'The game is not open: the console shows HOME.', + 'evidence': {'screen': 'home'}}) + self.assertEqual(self.app.options, CURRENT_OPTIONS) + self.assertEqual(self.plane.preflight(NO_TOKEN), None) + self.assertIn(('request_failed', 'preflight_program'), self.event_types()) + + def test_a_reply_that_did_not_restore_the_options_never_gives_a_token(self): + """Even when every check passed: the panel now holds the candidate, not what the user set.""" + token = self.acquired() + self.app.restore = False + record = self.preflight(preflight_request(token, options={'START_SEGMENT': 7})) + self.assertEqual((record['status'], record['refusal']['code']), ('failed', 'app_unavailable')) + self.assertEqual(record['refusal']['evidence']['reason'], 'options_not_restored') + self.assertIsNone(record['result']) + + def test_other_verdicts(self): + def unsupported(test): + test.app.replies['describe_program'] = describe_reply(prechecks='unsupported') + + def app_refuses_combination(test): + test.app.invalid = ('START_SEGMENT', 'segment 3 needs NEW_GAME off') + + def no_preflight_command(test): # today's app + test.app.line_errors['preflight'] = 'unknown command: preflight' + + def other_program_answers(test): + test.app.preflight_id = 'PokemonBDSP:AutoStory' + + cases = [ + dict(name='the program declares no checks (nothing sent)', input=unsupported, + expectation=('refused', 'prechecks_unsupported', 0)), + dict(name='check_validity refuses the combination', input=app_refuses_combination, + expectation=('failed', 'invalid_arguments', 1)), + dict(name="today's app has no preflight command", input=no_preflight_command, + expectation=('failed', 'app_unavailable', 1)), + dict(name='the reply names another program', input=other_program_answers, + expectation=('failed', 'app_unavailable', 1)), + ] + for case in cases: + with self.subTest(case['name']): + self.setUp() + token = self.acquired() + case['input'](self) + record = self.preflight(preflight_request(token, options={'START_SEGMENT': 3})) + self.assertEqual((record['status'], record['refusal']['code'], len(self.preflights_sent())), + case['expectation']) + + def test_a_preflight_is_leased_and_retried_from_its_record(self): + record = self.preflight(preflight_request('not-a-lease')) + self.assertEqual(record['refusal']['code'], 'lease_invalid') + token = self.acquired() + request = preflight_request(token) + first = self.preflight(request) + again = self.preflight(request) + self.assertEqual(first, again) + self.assertEqual(len(self.preflights_sent()), 1) + + +class TokenTests(PreflightTest): + def run_with(self, token, preflight_token, options=None): + return self.run_program(run_request(token, options=options, preflight_token=preflight_token)) + + def test_a_current_token_starts_the_run_with_its_options(self): + token = self.acquired() + options = {'START_SEGMENT': 3, 'NOTIFICATIONS': {'Enabled': False}} + checked = self.passed(token, options) + record = self.run_with(token, checked['result']['preflight_token'], options) + self.assertEqual(record['status'], 'completed', record) + start = [t for t in self.app.requests if t.splitlines()[-1].startswith('start')][0].splitlines() + # In key order: the record stores the options as canonical JSON. + self.assertEqual(start[1:], ['set NOTIFICATIONS.Enabled false', 'set START_SEGMENT 3', 'start 10']) + self.assertEqual(self.app.options['START_SEGMENT'], 3) + + def test_stale_tokens_are_refused_and_nothing_is_sent(self): + def run_and_end(test): + test.app.transition('start', 'lab', 'aaaaaaaa-1111-4222-8333-444444444444') + test.app.transition('stop', 'program') + + def app_restart(test): + test.lab.update_state(**{'instance.id': NEW_INSTANCE}) + + def save_written(test): + test.clock.sleep(1) + fact = save_known(test) + fact['saved_at'] = test.lab.state['written_at'] + test.lab.write_save(fact) + + def cfr_batch(test): + record = test.actions.cfr_batch(CLIENT, {'schema': 1, 'request_id': '0f0f0f0f-1111-4222-8333-444444444444', + 'console': 'B', 'lease_token': test.token, + 'arguments': {'lines': ['press B']}}) + test.assertEqual(record['status'], 'completed', record) + + def too_old(test): + test.clock.sleep(preflight.TOKEN_TTL_S + 1) + + cases = [ + dict(name='the program ran and ended', input=(run_and_end, None), expectation=['transition']), + dict(name='the app restarted', input=(app_restart, None), expectation=['instance_id']), + dict(name='a save was written', input=(save_written, None), expectation=['save']), + dict(name='a cfr_batch ran', input=(cfr_batch, None), expectation=['save']), + dict(name='other options', input=(None, {'START_SEGMENT': 4}), expectation=['options_hash']), + dict(name='too old', input=(too_old, None), expectation='expired'), + dict(name='never issued', input=(None, None, NO_TOKEN), expectation='unknown_token'), + ] + for case in cases: + with self.subTest(case['name']): + self.setUp() + self.token = self.acquired() + self.lab.write_save(save_known(self)) + checked = self.passed(self.token, {'START_SEGMENT': 3}) + setup, options = case['input'][:2] + if setup is not None: + setup(self) + run_token = case['input'][2] if len(case['input']) > 2 else checked['result']['preflight_token'] + record = self.run_with(self.token, run_token, options or {'START_SEGMENT': 3}) + self.assertEqual((record['status'], record['refusal']['code']), ('refused', 'preflight_stale'), + record) + evidence = record['refusal']['evidence'] + if isinstance(case['expectation'], list): + self.assertEqual(evidence['changed'], case['expectation']) + else: + self.assertEqual(evidence['reason'], case['expectation']) + self.assertEqual(self.sent('start'), []) + + def test_a_token_is_bound_to_its_lease(self): + token = self.acquired() + checked = self.passed(token) + self.plane.execute(CLIENT, {'schema': 1, 'request_id': '1f1f1f1f-1111-4222-8333-444444444444', + 'tool': 'release_console', 'console': 'B', 'lease_token': token, + 'arguments': {}}) + other = self.acquired() + record = self.run_with(other, checked['result']['preflight_token']) + self.assertEqual(record['refusal']['evidence']['changed'], ['lease']) + + def test_a_token_runs_once(self): + """The run's own start changes the last transition, so the same token cannot start a second run.""" + token = self.acquired() + preflight_token = self.passed(token)['result']['preflight_token'] + self.assertEqual(self.run_with(token, preflight_token)['status'], 'completed') + self.app.transition('stop', 'program') + again = self.run_with(token, preflight_token) + self.assertEqual((again['refusal']['code'], again['refusal']['evidence']['changed']), + ('preflight_stale', ['transition'])) + self.assertEqual(len(self.sent('start')), 1) + + +def save_known(test): + """A save fact from before the test, so console://B/save is `known`.""" + fact = golden('save_fact', 'frlg_checkpoint') + fact['saved_at'] = '2026-11-14T05:00:00.000Z' + return fact + + +if __name__ == '__main__': + unittest.main() From 77611f24f22f8909abef87507a86de80eb234c59 Mon Sep 17 00:00:00 2001 From: dayfine Date: Sun, 15 Nov 2026 17:30:23 -0800 Subject: [PATCH 6/7] pa_lab: a cfr_batch makes a preflight token stale even without a save fact --- lab/server/pa_lab/actions.py | 11 ++++++++--- lab/server/pa_lab/preflight.py | 5 +++-- lab/server/tests/test_preflight.py | 8 ++++++-- 3 files changed, 17 insertions(+), 7 deletions(-) diff --git a/lab/server/pa_lab/actions.py b/lab/server/pa_lab/actions.py index db87bccff2..174b7b501d 100644 --- a/lab/server/pa_lab/actions.py +++ b/lab/server/pa_lab/actions.py @@ -168,11 +168,16 @@ def _need_panel(app: Dict[str, Any], panel: str) -> None: def save_now(reader: ConsoleReader) -> Dict[str, Any]: - """`console://B/save` as a token binds it; an unreadable save.json is its own status.""" + """`console://B/save` as a token binds it, with the newest cfr_batch's time (`cfr_at`): a batch + makes a token stale even when there is no save fact to mark. An unreadable save.json is its own + status.""" try: - return reader.save_resource() + doc = dict(reader.save_resource()) except Unavailable as e: - return {'status': 'unreadable', 'reason': e.reason, 'since': None} + doc = {'status': 'unreadable', 'reason': e.reason, 'since': None} + mark = reader.control.save_mark(reader.console) if reader.control is not None else None + doc['cfr_at'] = None if mark is None else mark['at'] + return doc def check_preflight(state: Dict[str, Any], program: Dict[str, Any], lease: Any, args: Dict[str, Any]) -> None: diff --git a/lab/server/pa_lab/preflight.py b/lab/server/pa_lab/preflight.py index e0f2032dd2..6ab881ebaf 100644 --- a/lab/server/pa_lab/preflight.py +++ b/lab/server/pa_lab/preflight.py @@ -23,7 +23,7 @@ **The token** is the hash of what the preflight was checked against (`binding`): the app instance, the program id, the candidate options' hash, the observation versions a run depends on (the last -start/stop transition and `console://B/save`'s status) and the lease (its token's hash), plus a +start/stop transition, `console://B/save`'s status and the newest `cfr_batch`) and the lease (its token's hash), plus a random salt so it cannot be guessed. The daemon stores the binding under the token's hash; `run_program` recomputes it from what it reads now and refuses a token whose binding changed (`preflight_stale`, `evidence.changed`) or that is older than `TOKEN_TTL_S`. @@ -211,7 +211,8 @@ def binding(state: Dict[str, Any], save: Dict[str, Any], panel: str, options: Di app = state['app'] return {'console': state['console'], 'instance_id': app['instance']['id'], 'panel': panel, 'options_hash': options_hash(options), 'transition': transition_key(app), - 'save': [save.get('status'), save.get('reason'), save.get('since')], 'lease': lease_token_hash} + 'save': [save.get('status'), save.get('reason'), save.get('since'), save.get('cfr_at')], + 'lease': lease_token_hash} def new_token(bound: Dict[str, Any]) -> str: diff --git a/lab/server/tests/test_preflight.py b/lab/server/tests/test_preflight.py index b99346a2ce..93bff377c0 100644 --- a/lab/server/tests/test_preflight.py +++ b/lab/server/tests/test_preflight.py @@ -170,6 +170,8 @@ def too_old(test): dict(name='the app restarted', input=(app_restart, None), expectation=['instance_id']), dict(name='a save was written', input=(save_written, None), expectation=['save']), dict(name='a cfr_batch ran', input=(cfr_batch, None), expectation=['save']), + dict(name='a cfr_batch ran with no save fact', input=(cfr_batch, None, None, 'no fact'), + expectation=['save']), dict(name='other options', input=(None, {'START_SEGMENT': 4}), expectation=['options_hash']), dict(name='too old', input=(too_old, None), expectation='expired'), dict(name='never issued', input=(None, None, NO_TOKEN), expectation='unknown_token'), @@ -178,12 +180,14 @@ def too_old(test): with self.subTest(case['name']): self.setUp() self.token = self.acquired() - self.lab.write_save(save_known(self)) + if 'no fact' not in case['input']: + self.lab.write_save(save_known(self)) checked = self.passed(self.token, {'START_SEGMENT': 3}) setup, options = case['input'][:2] if setup is not None: setup(self) - run_token = case['input'][2] if len(case['input']) > 2 else checked['result']['preflight_token'] + run_token = case['input'][2] if len(case['input']) > 2 and case['input'][2] else \ + checked['result']['preflight_token'] record = self.run_with(self.token, run_token, options or {'START_SEGMENT': 3}) self.assertEqual((record['status'], record['refusal']['code']), ('refused', 'preflight_stale'), record) From 23187e5a84d7450212d64c9d7d22eea5fccbe7e8 Mon Sep 17 00:00:00 2001 From: dayfine Date: Sun, 15 Nov 2026 17:31:56 -0800 Subject: [PATCH 7/7] pa_lab: README Slice 5b section and user pages for preflight, options and cfr_batch --- lab/server/GUIDE.md | 11 +++- lab/server/README.md | 100 +++++++++++++++++++++++++++---- lab/server/guide/actions.md | 89 +++++++++++++++++++-------- lab/server/guide/start-checks.md | 6 +- 4 files changed, 168 insertions(+), 38 deletions(-) diff --git a/lab/server/GUIDE.md b/lab/server/GUIDE.md index 7653e22b78..e67c6d9874 100644 --- a/lab/server/GUIDE.md +++ b/lab/server/GUIDE.md @@ -85,12 +85,19 @@ retried request safe, and an event log. None of it presses buttons yet. **Starting and stopping a program (Slice 5a).** `run_program` and `stop_program` exist, but only when the server is started with `serve --actions`, which nothing uses yet; your session connection stays read-only. An agent holding the console's lease can start the program selected in the app -(with its current options) and stop it. It is refused whenever you hold the console (you clicked +(5b: only after its start checks pass with the options it will use) and stop it. It is refused whenever you hold the console (you clicked Start or Stop), when another program is running, when the app is not running, or when the selected program is not the one asked for. If you click Start or Stop while an agent's start or stop is under way, yours wins and the agent's request is marked failed. Every start and stop is one record, and a retried request never starts the program twice. +**Start checks before a start, options, debug batches (Slice 5b, server side).** Also only with +`serve --actions`: an agent first runs the program's start checks with the options it wants +(`preflight_program`), and can start only with the token that returns, within two minutes and +only if nothing changed. `cfr_batch` sends a short allowlisted button batch for debugging and +marks the save stale. The app cannot yet run the start checks on request, so on the real console +nothing can be started this way yet. See [guide/actions.md](guide/actions.md). + ## What comes next Each feature has a user page, marked **DRAFT** until it is built and switched on. The drafts @@ -99,7 +106,7 @@ are updated as each piece lands. | Slice | What you would get | Page | |---|---|---| | 4 (rest) | Every program lists what it needs before it starts; FRLG AutoStory's list is next (fork #488) | [start-checks.md](guide/start-checks.md) | -| 5 | Leased actions: an agent asks to run or stop a program; the server refuses whenever you hold the console, and (5b) when a start check fails. Every action gets exactly one record. 5a is built in #592 and 5b's server side in #SLICE5B; neither is switched on, and the app does not yet run start checks on request. | [actions.md](guide/actions.md) | +| 5 | Leased actions: an agent asks to run or stop a program; the server refuses whenever you hold the console, and (5b) when a start check fails. Every action gets exactly one record. 5a is built in #592 and 5b's server side on branch `lab/server-slice-5b`; neither is switched on, and the app does not yet run start checks on request. | [actions.md](guide/actions.md) | | 6 | Save facts: the server knows which save is on the console and refuses a run that would start from the wrong one; run records become readable | [saves.md](guide/saves.md) | | 7 | The server launches and supervises the lab app itself, so a build always starts with the right profile | [app-lifecycle.md](guide/app-lifecycle.md) | | 8 | Optional: a local worker for builds and test runs | [worker.md](guide/worker.md) | diff --git a/lab/server/README.md b/lab/server/README.md index 232662176f..c750e8794b 100644 --- a/lab/server/README.md +++ b/lab/server/README.md @@ -5,8 +5,9 @@ The local server designed in `docs/design/Agent-Integration.md` (fork #131, bran Slice 2: the attach-only read server ([below](#slice-2-the-attach-only-read-server)). Slice 3: the durable control plane ([below](#slice-3-the-durable-control-plane)): leases, request records, tickets and events in SQLite. Slice 5a: leased `run_program` and `stop_program` -([below](#slice-5a-run_program-and-stop_program)), served only by `serve --actions`, which nothing is -wired to yet. Nothing presses a button: that is Slice 5b's `cfr_batch`. +([below](#slice-5a-run_program-and-stop_program)); Slice 5b: `preflight_program`, options bound to +its token, and `cfr_batch` ([below](#slice-5b-preflight_program-options-on-run_program-cfr_batch)). +All are served only by `serve --actions`, which nothing is wired to yet. New here? Start with [GUIDE.md](GUIDE.md): what pa-lab is and what it gives you today. @@ -24,7 +25,7 @@ New here? Start with [GUIDE.md](GUIDE.md): what pa-lab is and what it gives you | `pa_lab/tools.py`, `pa_lab/shot.py` | `ensure_game` (verification only) and `shot` (newer than the call) | | `pa_lab/mcp_server.py` | MCP over Streamable HTTP on `127.0.0.1`: bearer, Host/Origin, sessions, size caps | | `pa_lab/app_query.py` | `describe_program` and `game_open`: one-line reads through the app's channel (#504) | -| `tests/fake_lab.py` | A fake `Lab/` folder and a fake app answering `snapshot`, `describe_program`, `game_open`, `start` and `stop` requests, in temp dirs | +| `tests/fake_lab.py` | A fake `Lab/` folder and a fake app answering `snapshot`, `describe_program`, `game_open`, `set` + `start`, `stop`, the proposed `preflight` and console lines, in temp dirs | | `tests/test_{lab_reader,ensure_game,shot,config,mcp_server}.py` | Slice 2's fake-lab tests | | `tests/test_app_query.py` | `describe_program` and `game_open` against the fake app | | `pa_lab/db.py` | The SQLite file: migrations (`PRAGMA user_version`), owner-only file, `BEGIN IMMEDIATE` writes | @@ -32,6 +33,8 @@ New here? Start with [GUIDE.md](GUIDE.md): what pa-lab is and what it gives you | `tests/test_{db,control,control_concurrency,control_mcp}.py` | Slice 3's tests: restarts, threads and processes racing, kills mid-write | | `pa_lab/actions.py` | Slice 5a: `run_program` / `stop_program` through the control plane and the app's `start` / `stop` channel commands | | `tests/test_actions.py` | Slice 5a's refusals, retries and races against the fake app | +| `pa_lab/preflight.py`, `pa_lab/cfr.py` | Slice 5b: options against the static contract, the `preflight` reply, the token's binding; the `cfr_batch` allowlist | +| `tests/test_{preflight,cfr_batch,cfr}.py` | Slice 5b: option refusals, verdicts, restore, stale tokens; batches, the save mark; the allowlist | | `pa_lab/status.py`, `pa_lab/stdio_server.py`, `smoke_readonly.py` | Read-only stdio mode for `.mcp.json`: `console_status` ([below](#read-only-stdio-mode-mcpjson)) | | `tests/test_status_readonly.py` | `console_status` and the read-only gate against a fake profile | @@ -350,13 +353,90 @@ a stop) and resolves the record: instance, after the start's `ok` line), not by id, and its `started_at` is unknown. A `run_id` on the stop transition, or the app keeping the start beside the stop, would make that match exact. -**Left for Slice 5b.** `preflight_program`: the app has no channel command that runs a program's -prechecks (or stages candidate options) without starting it, so 5a sends no preflight, runs the -panel's current options, and the program's own `run_prechecks()` at Start are the only checks; -`run_program`'s `preflight_token` became optional in the v1 request contract for this and 5b makes -it required again. Also 5b: staging `options`, claiming the channel's owner token (a claimed -channel is refused today), `cfr_batch`, and the console-B soak (every mutation has exactly one -record). +**Changed by Slice 5b.** `run_program` needs a `preflight_token` again (required in the v1 +request contract) and takes `options`; the 5a refusals for a non-empty `options` and for a token +are gone. See below. + +## Slice 5b: `preflight_program`, options on `run_program`, `cfr_batch` + +Served with the 5a tools by `serve --actions`; still not wired to anything, and `.mcp.json` stays +read-only. Each call is one request record on the same `begin_action` / inbox / settle path as 5a +(an exact retry gets the stored record; a pending earlier action on the console refuses the next). + +```text +preflight_program(console, request_id, lease_token, panel, options) +run_program(console, request_id, lease_token, panel, options, preflight_token) +cfr_batch(console, request_id, lease_token, lines) +``` + +**`preflight_program`** (`pa_lab/preflight.py`, `actions.check_preflight`). First, without a lease +and outside the transaction, `describe_program` (an exact retry skips it). Then, in order: + +| Check | Refusal (stored `refused`, nothing sent) | +|---|---| +| lease, no pending action, app live, `panel` selected, `blocked`, channel free | as for `run_program` (5a) | +| the app restarted between `describe_program` and here | `app_unavailable`, `instance_replaced` | +| the program declares a precheck list | `prechecks_unsupported` (absence never passes) | +| every option key is one the program saves; value of the entry's type, within `min`/`max`; an enum/select slug that is listed and enabled; a group is an object of its own entries (plus `Enabled` when toggleable); no `type: other` | `invalid_arguments`, `field` = `options.` | + +Then the daemon writes `request ` / `preflight ` and waits up to 30 s (the +checks may open a menu or go Home and back) for the one reply line, `preflight ` (format in +`pa_lab/preflight.py`). The record resolves: + +| The reply | Record | +|---|---| +| `restored: false` (whatever the verdict) | `failed`, `app_unavailable`, `evidence.reason: options_not_restored`; no token | +| `id` is not `panel` | `failed`, `app_unavailable`, `panel_changed` | +| `verdict: unsupported` | `failed`, `prechecks_unsupported` | +| `verdict: invalid_options` (the app's `check_validity` or its own key check) | `failed`, `invalid_arguments`, `field: options.` | +| `verdict: failed` | `failed`, `precheck_failed`, `check` = the check's name, its message and evidence | +| `verdict: passed` | `completed`: `checks`, `options_hash`, `preflight_token`, `issued_at`, `expires_at` | +| the line failed (today's app: no `preflight` command) | `failed`, `app_unavailable` with the journal; `owned by` → `console_held_by_agent` | +| a UI Start/Stop or another app instance before the reply | `failed`, `console_held_by_user` / `app_unavailable` `instance_replaced` | + +**The token.** A pass stores (table `preflights`, migration 3) the token's hash with its binding: +console, app instance, panel, the candidate options' hash, the last start/stop transition, +`console://B/save` (status, reason, since) with the newest `cfr_batch`'s time, and the hash of the +request's lease token. The token is a salted hash of that binding. `run_program` reads the same +things and, after the 5a checks, refuses with `preflight_stale` when the token is unknown +(`evidence.reason: unknown_token`), older than 120 s (`expired`), or any bound field differs +(`changed`, `evidence.changed` lists them, e.g. `["transition"]`). A run's own start changes the +transition, so a token starts at most one run. + +**Options on `run_program`.** The request is `request `, one `set KEY JSON` line per option +(groups as `GROUP.KEY`, in key order), then `start 10`; the app stops at the first failing line, so +a refused `set` starts nothing (`app_unavailable` with the journal). The panel keeps the options +after the run, as when a user sets them and clicks Start. + +**`cfr_batch`** (`pa_lab/cfr.py`, the allowlist from #547). 1-200 lines; only `press`, `dpad`, +`stick`, `wait`, `taps`, `walk`, `shot`, `shots`, `detect`, `wait_for`, `expect`, `game_open`, in +canonical form, with bounded numbers and a worst case of at most 60 s for the batch; no channel +command, `close_game`, `history`, `CAPTURE`, image path, comment or blank line +(`invalid_arguments`, `field: arguments.lines[i]`). The other checks: lease, a program panel +(`panel_not_program`), `blocked` (program stopped, controller ready, video live), channel free. +Before the request is written, the console's save mark (table `save_marks`) is set, so +`console://B/save` is `stale` / `cfr_batch` from then on (a press may have happened even if the +daemon dies next) and every earlier preflight token is stale. The daemon waits for the worst case ++ 5 s; the record is `completed` with `outcome: done`, or `line_failed` with `failed_line` +(1-based in the batch) and `error`, and `output` (what the lines printed, e.g. shot paths). An app +that never took the request: withdrawn, `failed`, `timeout`; the mark stays. + +**The app's side (not built).** The app has no command that stages options and runs a program's +prechecks without starting it, and `lab/tooling` has only the precheck *summaries* (#511): the +runnable list (`Precheck::run(env, context)`, #485) is on the upstream-bound FRLG branch, and a +check needs a `SingleSwitchProgramEnvironment` and controller context that exist only inside a +running program. A `preflight` command therefore needs a precheck-only run of the program session +(stage on the UI thread, a scope guard restoring the options, the checks on the program thread), +not just a channel command, so 5b stops at the server and the fake. Until the app has it, every +preflight on the real app is `failed` / `app_unavailable` (unknown command), so `run_program` +cannot start anything there. + +**Known gaps.** The panel's options are not versioned in `state.json`, so a token cannot see an +option the user changes in the UI between the preflight and the run (only a Start/Stop); the app +could print an options hash in the reply and in `state.json`. The `set` echoes in the start's +journal are not compared with what was sent (the preflight already refused clamped or disabled +values). The channel's owner token is still not claimed (a claimed channel is refused), and the +console-B soak (every mutation has exactly one record) is still to run. ## User pages diff --git a/lab/server/guide/actions.md b/lab/server/guide/actions.md index 8f9f0bba9f..496b471cb8 100644 --- a/lab/server/guide/actions.md +++ b/lab/server/guide/actions.md @@ -1,5 +1,5 @@ > **DRAFT.** This page describes a feature that is only partly built. It is updated as each piece lands. -> Status (2026-11-15): Slice 5a (`run_program`, `stop_program` under a lease) is built in [#592](https://github.com/dayfine/Arduino-Source/pull/592), not merged and not switched on. Slice 5b (preflight, options, `cfr_batch`, a console soak) is not built. +> Status (2026-11-15): Slice 5a (`run_program`, `stop_program` under a lease) is built in [#592](https://github.com/dayfine/Arduino-Source/pull/592); Slice 5b's server side (`preflight_program`, options on `run_program`, `cfr_batch`) is built on branch `lab/server-slice-5b`, tested against a fake app only. Neither is merged or switched on. The app cannot yet run a program's start checks on request, so on the real console every preflight fails and no run can start; a console soak comes after that. # Actions: agents start and stop programs through the server @@ -15,18 +15,22 @@ and the server decides whether it may. - **One action, one record.** Each request has an id. Retrying the same id returns the first answer and never starts or stops twice. Every action is in the event log. - **Refusals say why**, for example: you hold the console; another program is running; the app - is not running; the selected panel is a different program; the controller is not ready. + is not running; the selected panel is a different program; the controller is not ready; a + start check failed. +- **Start checks first.** An agent must run the program's start checks with the exact options + it will use, then start within two minutes. If anything changes in between (you click Start or + Stop, the app restarts, a save is written, a debug batch runs, other options), it must check + again. A program that lists no start checks cannot be started this way at all. ## Tools | Tool | What it does | Status | |---|---|---| | `acquire_console` / `renew_console` / `release_console` | take, extend and give back the lease | built (control plane) | -| `run_program(panel)` | start the selected program once | built in #592, not switched on | +| `preflight_program(panel, options)` | check the options, run the program's [start checks](start-checks.md) without starting it, put your options back; returns a token for `run_program` | server built (5b), needs an app command that does not exist yet | +| `run_program(panel, options, preflight_token)` | load the options and start the selected program once; only with a current token for the same options | built (5a + 5b), not switched on | | `stop_program()` | stop it once; stopping when nothing runs is a no-op, not an error | built in #592, not switched on | -| `preflight_program(panel, options)` | load candidate options, run the program's [start checks](start-checks.md), then put your options back; `run_program` will need its token | planned (5b) | -| options on `run_program` | run with options other than the panel's current ones | planned (5b) | -| `cfr_batch(lines)` | a short allowlisted controller batch, for debugging only; marks the save as unknown | planned (5b) | +| `cfr_batch(lines)` | a short allowlisted controller batch, for debugging only; marks the save as stale | built (5b), not switched on | There is deliberately no "switch game" tool: the app cannot yet pick a game on HOME by name. @@ -50,42 +54,77 @@ If you hold it, the answer is a refusal instead: "message": "The user holds console B (Stop clicked in the app at 06:10); ask them to release it."}} ``` -**Start the selected program** (5a: the panel's current options): -`run_program(console="B", request_id="9b1d7e44-...", lease_token=..., panel="PokemonFRLG:AutoStory")`. -The stored record once the app has started it: +**Check before starting.** `preflight_program(console="B", request_id=..., lease_token=..., +panel="PokemonFRLG:AutoStory", options={"START_SEGMENT": 3})`. The result (from the server's +tests against a fake app; real console output comes later): + +```json +{"panel": "PokemonFRLG:AutoStory", "instance_id": "d8ea6a9d-...", + "checks": [{"name": "game_open", "status": "passed", "message": ""}, + {"name": "free_party_slots", "status": "passed", "message": ""}], + "options_hash": "22386fd0...", "preflight_token": "pf1.be3fc628...", + "issued_at": "2026-11-14T06:00:00.150Z", "expires_at": "2026-11-14T06:02:00.150Z"} +``` + +When a check fails, the record is failed and names it; your options are back as they were: + +```json +{"status": "failed", + "refusal": {"code": "precheck_failed", "check": "free_party_slots", + "message": "Your party is full. Leave a party slot free so the program can check the Pokemon it receives.", + "evidence": {"needed": 1, "party_size": 6}}} +``` + +**Start it** with the same panel and options and the token: +`run_program(console="B", request_id="9b1d7e44-...", lease_token=..., panel="PokemonFRLG:AutoStory", +options={"START_SEGMENT": 3}, preflight_token="pf1.be3fc628...")`. The server sets the options in +the app, then clicks Start; the panel keeps those options afterwards. The stored record once the +app has started it: ```json {"request_id": "9b1d7e44-...", "tool": "run_program", "status": "completed", "created_at": "2026-11-14T06:20:00.000Z", "completed_at": "2026-11-14T06:20:02.150Z", "result": {"run_id": null, "started_at": "2026-11-14T06:20:02.100Z", - "instance_id": "d8ea6a9d-..."}, + "instance_id": "d8ea6a9d-...", "correlated_by": "request_id"}, "refusal": null} ``` -Sending the same `request_id` again returns this same record and starts nothing. Refusal codes -you can get: `console_held_by_user`, `console_held_by_agent`, `program_running`, -`app_not_running`, `panel_not_program`, `channel_owned`, `invalid_arguments`. +Sending the same `request_id` again returns this same record and starts nothing. If something +changed since the check, the answer says what: + +```json +{"code": "preflight_stale", "message": "Since the preflight, save changed; run preflight_program again.", + "evidence": {"reason": "changed", "changed": ["save"]}} +``` + +Refusal codes you can get: `console_held_by_user`, `console_held_by_agent`, `program_running`, +`app_unavailable`, `panel_not_program`, `invalid_arguments`, `preflight_stale`; from the check +also `precheck_failed` and `prechecks_unsupported`. **Stop it.** `stop_program(console="B", request_id=..., lease_token=...)` answers with who stopped it and when, e.g. `{"sent": true, "stopped_at": "...", "stopped_by": "lab"}`. If nothing was running: `{"sent": false, "state": "stopped", ...}`, which is not an error. -**Preflight (planned, 5b).** `preflight_program(console="B", panel="PokemonFRLG:RngStatic", options={...})` -returns a token for `run_program`, or the first failed start check: +**Controller batch (debugging only).** +`cfr_batch(console="B", request_id=..., lease_token=..., lines=["shot before", "press B 160 640", "shot after"])` +runs only with the program stopped. Channel commands (`start`, `select`, `set`, ...), `close_game`, +the Capture button and anything over 60 s are refused before anything is pressed. The result +(fake-app test output): ```json -{"tool": "preflight_program", - "refusal": {"code": "precheck_failed", "check": "free_party_slots", - "message": "Free party slots: needs 1 empty slot, found 0.", - "evidence": {"needed": 1, "found": 0}}} +{"outcome": "done", "instance_id": "d8ea6a9d-...", + "output": [".../Lab/outbox/palab_cfr_batch_21d67be8..._before.png", + ".../Lab/outbox/palab_cfr_batch_21d67be8..._after.png"]} ``` -**Controller batch (planned, 5b, debugging only).** -`cfr_batch(console="B", lines=["shot before", "press B 160 640", "shot after"])` runs the lines -with the program stopped and marks the save `stale`. +A line that fails stops the batch: `{"outcome": "line_failed", "failed_line": 2, "error": "..."}`. +After any batch, `console://B/save` says `stale` with reason `cfr_batch`, since the presses may +have changed the game. ## Switching it on -The session connection (`.mcp.json`) stays read-only until you decide otherwise. The plan: -5b adds preflight; a soak on console B shows every action has exactly one record; then actions -are enabled one program family at a time. +The session connection (`.mcp.json`) stays read-only until you decide otherwise. What is left +before actions can be switched on: the app command that runs a program's start checks without +starting it (with your options put back afterwards); a soak on console B that shows every action +has exactly one record; then actions are enabled one program family at a time, starting with +programs that list their start checks. diff --git a/lab/server/guide/start-checks.md b/lab/server/guide/start-checks.md index 41ea554e01..af1157ba5c 100644 --- a/lab/server/guide/start-checks.md +++ b/lab/server/guide/start-checks.md @@ -1,5 +1,5 @@ > **DRAFT.** This page describes a feature that is only partly built. It is updated as each piece lands. -> Status (2026-11-15): the check list and `describe_program` are built ([#485](https://github.com/dayfine/Arduino-Source/pull/485), [#511](https://github.com/dayfine/Arduino-Source/pull/511), [#512](https://github.com/dayfine/Arduino-Source/pull/512)); FRLG AutoStory's list is a design note ([#488](https://github.com/dayfine/Arduino-Source/pull/488)); the server does not yet run the checks before a start (that is [actions.md](actions.md), Slice 5b). +> Status (2026-11-15): the check list and `describe_program` are built ([#485](https://github.com/dayfine/Arduino-Source/pull/485), [#511](https://github.com/dayfine/Arduino-Source/pull/511), [#512](https://github.com/dayfine/Arduino-Source/pull/512)); FRLG AutoStory's list is a design note ([#488](https://github.com/dayfine/Arduino-Source/pull/488)); the server side of running them before a start is built on branch `lab/server-slice-5b` ([actions.md](actions.md), Slice 5b), but the app cannot yet run them without starting the program, so today they still run only when the program starts. # Start checks: what a program needs before it presses anything @@ -18,6 +18,10 @@ them presses a button, resets, saves or changes an option. before any reset. - **Clear refusals.** The first failing check stops the start with a message saying which check failed and what it needs. +- **Checked before an agent starts anything.** An agent asks the server to run the checks with + the options it wants (`preflight_program`); your own option values are put back afterwards, + pass or fail. It can start the program only right after the checks pass, with exactly those + options. A program without a list cannot be started by an agent at all. ## Example