Skip to content

Lab: run records, one folder per program run - #116

Draft
dayfine wants to merge 19 commits into
lab/run-records-basefrom
prototype/program-session-agent
Draft

dayfine wants to merge 19 commits into
lab/run-records-basefrom
prototype/program-session-agent

Conversation

@dayfine

@dayfine dayfine commented Nov 1, 2026 •

Copy link
Copy Markdown
Owner

Depends on #107 and #132

Lab runs had no record of their own: failure frames went missing when a program retried internally, the log had to be filtered by date, and build and options were unknown afterwards. With the run records of #132, this PR writes each run to Lab/runs/<run_id>/ (meta.json with the program, version, git commit and settings; result.json; events.jsonl; log.txt appended as the run goes, up to 8 MiB; frames and probes), records runs only when the profile has a Lab/ folder, and adds labctl.sh wait RUN_ID to read them. The base is lab/run-records-base (#107 with main and #132 merged in), so the diff is the Lab half only; the design, scope and PR plan are in docs/design/Lab-RunRecords.md. Tested with the RunRecord and LabChannel unit test suites in CI, labctl_wait_test.sh, and a local macOS build; no console run yet.

🤖 Generated with Claude Code

https://claude.ai/code/session_01AKYzhrMnPK61XHWEch4a6E

dayfine and others added 13 commits October 31, 2026 23:17
Adds a scope review to DESIGN.md: what the console loop needs from a run
record, how the proposal would have helped on the recorded failures, what to
keep, add, drop and simplify in the first slice (typed program progress in,
sessions.* and request-ID deduplication out, runs as files under Lab/), how
CFR fits, and how the loop would use it.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AKYzhrMnPK61XHWEch4a6E
RunRecord.h holds what one run of a program did: run ID, program, start
and end time, status (running, stopping, terminal), exactly one result
(completed, cancelled or failed, with the primary error and the owned
failure frame), the events the program reported and a bounded log tail.
RunRecordStore keeps the last 32 records of this app instance;
SessionRunRecorder is one session's end of it, a RunRecorder for the
active run and a Logger that copies lines into the run's tail. Nothing
depends on the UI, JSON or a game.

Native tests cover the IDs and capacity, the one-result rule, the event,
frame and log bounds, frame release for old runs and the session recorder.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AKYzhrMnPK61XHWEch4a6E
start_program() opens a run record once check_validity() passes, before
the program thread exists; the thread closes it with one result whichever
way it ends: a normal return or ProgramFinishedException is completed, the
stop path is cancelled, and a setup, download, connection or execution
error is failed with the exception's message and, for exceptions that
carry one, its frame. A stop requested after a failure was observed does
not overwrite it, and a normal return racing a stop stays completed. A
dispatch failure closes the record too and leaves the session stopped
instead of throwing out of start_program(). Errors reported through
report_error() become "error" events, diagnostics rather than the outcome.

The session's logger now chains through its SessionRunRecorder, and the
console sessions take the logger they write through, so a run's log tail
holds the program's and its consoles' lines and nothing else.
ProgramEnvironment::run_recorder() is the program's progress hook.

The lifecycle tests drive a ProgramSession whose program is a scripted
fake with no console: rejected start, immediate completion, setup failure,
failure with a frame, cancellation, a late stop, dispatch failure, and a
second run preserving the first.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AKYzhrMnPK61XHWEch4a6E
…ns.* prototype

LabRunWriter copies the run records into files on the channel's poll:
result.json (status, outcome, error, times, frame, last event; final once
"status" is "terminal"), events.jsonl (one line per event, appended as the
program reports them), log.txt (the run's log tail) and frame-*.png, with
the newest 32 run folders kept. "start" prints run=<id> and the state file
gains a run= line, so a script can wait on that run's result.json. A shot
or detect taken during a run, or within two minutes after it ended, is
appended to the run's events.jsonl as a probe and its screenshot copied in.

The session_agent command, the SessionAgentPrototype sources and the
terminal demo go; nothing of sessions.* remains.

LabChannel tests cover result.json's shape at the indents labctl.sh greps,
events.jsonl lines and frame names, the writer with probes and pruning,
and run= in the state text.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AKYzhrMnPK61XHWEch4a6E
…nd the README

"labctl.sh wait RUN_ID" polls Lab/runs/RUN_ID/result.json until it is
terminal (LABCTL_RUN_WAIT, default two hours) and prints the outcome, the
last event's checkpoint, step and attempt, the error, and the frame, events
and log paths; exit 0 for completed, 1 for failed or cancelled, 2 on a
timeout. "start" now echoes the run= line of its journal.
labctl_wait_test.sh checks it against hand-written result files.

Lab.md documents the run folder, result.json and events.jsonl, the run=
state line and start's reply; the README gets a "Run records" section with
the console-iteration loop that replaces run_watch.sh.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AKYzhrMnPK61XHWEch4a6E
The proposal moves from Integrations/SessionAgentPrototype/DESIGN.md to
CommonFramework/RunRecords.md, next to RunRecord.h. Its body before the
scope review becomes a short "What is implemented" section; the scope
review of 2026-11-01 stays as the spec; the sections after it (session IDs,
request IDs, evidence IDs, quotas) go with the prototype they described.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AKYzhrMnPK61XHWEch4a6E
The previous commit gave MultiConsoleSystemSession a base-logger parameter
but missed this caller, which broke the build. A panel has no run, so its
consoles write through the global logger as before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AKYzhrMnPK61XHWEch4a6E
The UpdatePopupDetector and ShinySoundDetector tests construct a
ConsoleSystemSession directly; give them the global logger, as before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AKYzhrMnPK61XHWEch4a6E
What a run record is for, what it contains and when it is written, the
components, and how #116 splits into a framework PR on main and the Lab
half on #107.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AKYzhrMnPK61XHWEch4a6E
The framework half of the run records moved to common/run-records (on
main); lab/run-records-base joins it with lab/tooling (#107). The
framework files are taken from there, so this branch's diff against it
is the Lab half only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AKYzhrMnPK61XHWEch4a6E
…h Lab/

- The lab channel enables the app's run store when it finds the Lab/
  folder; without it nothing is recorded.
- meta.json, written when a run starts, names the program, the app's
  version, the git commit it was built from and its settings.
- log.txt is appended on each poll from the store's numbered log, up to
  8 MiB; lines the writer missed, and lines past the cap, are counted in
  the file. result.json has log_lines and dropped_log_lines in place of
  log_truncated.
- The design doc takes in RunRecords.md (the prototype's scope review),
  records the decisions and the split into #132 and #116.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AKYzhrMnPK61XHWEch4a6E
@dayfine dayfine changed the title Prototype: agent requests over existing program sessions Lab: run records, one folder per program run Nov 1, 2026
@dayfine
dayfine changed the base branch from lab/tooling to lab/run-records-base November 1, 2026 21:28
@dayfine dayfine added game:shared Shared or game-neutral code lab-tooling Lab channel, CFR, scripts, run records, build labels Nov 1, 2026
dayfine and others added 2 commits November 1, 2026 15:17
The code-health fixes of #132.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AKYzhrMnPK61XHWEch4a6E
- The lab channel switches on the app's store with
  enable_app_run_records() and reads the newest run ID from it (a lab app
  runs one program at a time) instead of ProgramSession::last_run_id().
- The run-writer test is four tests over a shared temporary folder:
  a running run, an ended run with its probe, missed log lines, pruning.
- Docs name record_run_event(env, event).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AKYzhrMnPK61XHWEch4a6E
dayfine added a commit that referenced this pull request Nov 1, 2026
For lab builds: lab/tooling (#107) with the run records. #116's branch
already joins main and #132 (common/run-records), so this is lab/tooling
plus exactly that tree.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AKYzhrMnPK61XHWEch4a6E
dayfine added a commit that referenced this pull request Nov 1, 2026
The code-health fixes of #132 and #116.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AKYzhrMnPK61XHWEch4a6E
#132 names the executable; the git commit moves here.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AKYzhrMnPK61XHWEch4a6E
dayfine added a commit that referenced this pull request Nov 1, 2026
Per the decision on the design: main takes no CMake change. A record
names the app's version and the path of its executable (a lab build's
folder names the build); the git commit (cmake/BuildCommit.cmake) moves
to the lab half, #116.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AKYzhrMnPK61XHWEch4a6E
Per the decision on the design, the git commit lives in the lab half:
cmake/BuildCommit.cmake (moved from #132) writes PROGRAM_COMMIT on every
build, rewriting BuildCommit.cpp only when the commit changes. meta.json
names it beside the record's version and executable path. The design doc
records the five decisions.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AKYzhrMnPK61XHWEch4a6E
dayfine added a commit that referenced this pull request Nov 1, 2026
#132 names the executable; the git commit moves to #116.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AKYzhrMnPK61XHWEch4a6E
dayfine added a commit that referenced this pull request Nov 1, 2026
The executable path from #132; the build commit in #116.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AKYzhrMnPK61XHWEch4a6E
@dayfine

dayfine commented Nov 2, 2026

Copy link
Copy Markdown
Owner Author

Notes from a full day of the BDSP v4 lab loop (00.04 → 01.xx on console B, runs r43–r71, 2 agents testing in parallel). What would have saved the most time, most valuable first:

  1. Flag passes that needed fallbacks. The worst problems today were the ones reported as PASS: the Trainers' School door took three tries (lamp line-up, then nudges) and a checkpoint save went through only on the blind fallback. Both said "one-shot", and the user caught them by watching. Each step should count its retries and fallbacks into result.json, and a run with any of them should be marked "passed with fallbacks".
  2. Keep a frame for every failed step, whichever path reports it. The save's menu failure was caught by its own retry loop, and a short_leg outcome never throws, so neither left a frame and I reproduced both by hand. A frame when a waiting step starts (a talk, an NPC wait) would also have shown the wandering clown's position at once.
  3. Record each route: planned tiles, shortest tiles, replans, and where each leg ended. Every leg landed where planned, but one plan walked 38 tiles for 12. We now catch that in a unit test, but the record should show it for any route.
  4. A lab state agents can read (idle / running + run id). Agents asked me for "console idle" windows before running unit tests, because test mode can rewrite the lab app's settings. labctl wait plus a headless test binary (in progress on tests/headless-runner) would remove the handoffs.
  5. Record manual play too. When the loop got stuck, I played stretches by hand with lab-channel batches and screenshots. Those commands and frames should land in the same run record, so the end-of-run report puts the automated and manual parts together without my notes file.
  6. Turn a failure frame into a test image in one step. Copying the frame into CommandLineTests, pushing it (the first push failed with HTTP 400) and registering the test took several manual steps. A helper that writes the image path and a test stub would make "every failure frame becomes a test" cheap.

Items 1–3 fit #116's events and records as already designed (structured step events plus frames). Items 4 and 5 are new: lab state and a manual-play record type.

🤖 Generated with Claude Code

https://claude.ai/code/session_01AKYzhrMnPK61XHWEch4a6E

@dayfine

dayfine commented Nov 2, 2026

Copy link
Copy Markdown
Owner Author

One more from the same night: after about 14 hours of lab runs, the lab app stopped taking commands. CPU was at ~95% and frames were being dropped. By then the profile held nearly 2,000 outbox files (2.1 GB, mostly failure videos), 1,200 done journal entries, 1,750 route screenshots (3.9 GB) and a 45 MB shared log. The cause isn't proven, but anything a long session writes should be bounded:

  • Keep the newest N per folder, with a byte cap, as Lab/runs/ already does with 32. Apply it to outbox, done, screenshots and the log, and move older files to an archive folder instead of deleting them.
  • Videos only when a run fails, kept for the last few failures.
  • Never scan a growing folder on every poll. Read the inbox by name, and append the journal without listing it.
  • Keep things on disk rather than in memory only. Records must survive an app restart, and they are what agents read afterwards. The aim is bounded persistence.

🤖 Generated with Claude Code

https://claude.ai/code/session_01AKYzhrMnPK61XHWEch4a6E

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AKYzhrMnPK61XHWEch4a6E
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

game:shared Shared or game-neutral code lab-tooling Lab channel, CFR, scripts, run records, build

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant