diff --git a/docs/development/runlog-behavior-v2-plan.md b/docs/development/runlog-behavior-v2-plan.md new file mode 100644 index 0000000..04dfb2f --- /dev/null +++ b/docs/development/runlog-behavior-v2-plan.md @@ -0,0 +1,185 @@ +# Run-log format `behavior_v2` + large-file commit path — PLAN (not started) + +Owner: Michael. Drafted 2026-09-06 from the analysis of rig03-sr run `rydc2tql` +(40 s trials, 51.4 MB, failed to auto-commit). Status: **approved direction, no code +yet**. Implementation lands as three PRs (bridge, Studio, dashboard) in that order. + +## Problem + +The Studio commits run logs through GitHub's Contents API, which rejects files over +~35 MiB (measured: 35 OK, 40 → HTTP 422). A 20 s-trial P3 full run is ~26 MB; a 40 s +run is ~51 MB; an hour-long run would be ~85 MB. Composition of the 51 MB file: + +| Line type | Lines | Bytes | Share | +|---|---|---|---| +| `arena_command` echo (one per 0x70 closed-loop frame command, ~100 Hz) | 217,204 | 41.1 MB | 76% | +| behavior frame `[ms,fc,idx,ft,x,y,hd]` (behavior_v1) | 218,708 | 12.7 MB | 24% | +| runner events + session/metadata lines | 860 | 0.1 MB | <1% | + +Each `arena_command` line is a 189-byte object whose payload is four numbers; the rest is +constant strings and derivable fields. gzip compresses the whole file 8.0× (6.7 MB). + +## Goals / non-goals + +- **Lossless.** Every value recoverable from a v1 file must be recoverable from v2: + bit-identical numbers (no rounding), every event, every field. Verified by a + round-trip test (v1 → v2 → v1 canonical JSON equality). +- Hour-long runs commit from the browser with headroom; the dashboard and the replay + viewer read v1 and v2 transparently. +- Non-goals: changing what is measured; changing the behavior frame array (already + 58 B/line, would need lossy rounding to shrink); Git LFS (Contents API returns the + pointer, breaking every reader; bandwidth quotas). + +## Part 1 — `behavior_v2` line format (bridge `fictrac-bridge/bridge.py`) + +The level name in `frame_schema` / `log_control` becomes `behavior_v2`. Frame arrays +are **unchanged** (`[ms,fc,idx,ft,x,y,hd]`, same columns, same precision). What +changes is the *other* line types: + +1. **Schema line gains the arena-echo layout**: + `{"type":"frame_schema","level":"behavior_v2","cols":[...],"arena_cols":["t_off","dt","hex","status","rx_off"],"t0":}` + `t0` = the session's `logging_started.ms`; all `*_off` values are ms offsets from + it (small ints), which is what makes the arrays short. +2. **`arena_command` → `"a"` array line**: `["a", t_off, dt, "03702e00", status, rx_off]` + - `t_off = t − t0`, `rx_off = rx_ms − t0` (both kept — rx_ms is the bridge receive + time, not derivable from t+dt exactly). + - `hex` = the `head` bytes without spaces (v1 `head` is reconstructed by inserting + spaces; `len` = hex length / 2; `echo` = second byte; `ok` = status === 0). + - `status` = integer, or `null` when the command timed out (v1 `status: null`). + - `error` is `null` in every v1 line observed; when non-null it is appended as a + 7th element so nothing is dropped: `["a", …, rx_off, "error text"]`. + - Constant v1 fields `type:"log"`, `event:"arena_command"`, `dir:"browser→bridge"` + are implied by the `"a"` tag and restored on export. + - Size: 217k lines × ~24 B ≈ 5.3 MB (measured on the real file) vs 41.1 MB. +3. **Runner / session / metadata / config lines stay verbatim JSON objects** (0.1 MB; + readability matters more than bytes there). `run_metadata` gains + `"log_format":"behavior_v2"` for readers that skip the schema line. +4. **Lossless proof**: `tests/test-bridge-behavior.py` gets `v1→v2→v1` round-trip + cases built from real line samples (ok, status-1 reject, timeout/null status, + non-null error, unknown extra key → must raise, not drop). Add a CLI + `bridge.py --convert in.jsonl out.jsonl` (both directions) so existing course + files can be re-encoded for the migration and so the dashboard can be tested on + real data before a rig produces v2. + **Corpus test (gate for every PR in this series):** `scripts/runlog-v2-corpus.py` + runs against the local course-repo clone (`/Users/reiserm/Documents/GitHub/ + cshl-2026-course`, 165 run logs / 1.2 GB, all v1) and, for EVERY file: converts + v1→v2→v1 and asserts canonical-JSON identity line by line; records v1 / v2 / v2.gz + sizes; fails on any line whose keys are not in the known v1 set (older July logs, + legacy `_a`/`_b` protocols and `full`-level logs are exactly the variants the five + hand-picked samples would miss). Output = one table committed to the PR + description. Not part of `pixi run test` (needs the clone), like the dashboard's + `test-analysis.js`. +5. **Runtime log level — four places must change, bridge first.** Today the level is a + Studio setting (File ▾ → Run logging `#fmLogLevel`, localStorage `studio_log_level`, + advanced-only) that the runner ASSERTS at run start via `log_control`. The chain: + - `fictrac-bridge/bridge.py` `set_level()` + the `log_control` handler (line ~515) + accept only `behavior_v1 | full` and **silently ignore anything else** — so a new + Studio asking an old bridge for `behavior_v2` would get a v1 file with no error. + Bridge change: accept `behavior_v2` (new default for a fresh bridge), write the + compact lines, and **acknowledge the level actually in force** — reply to + `log_control` with `{"type":"log_control_ack","level":…}` and advertise + `"levels":[…]` in the hello reply so the browser can tell. + - `js/fictrac-bridge-client.js` `setLogLevel()` whitelist (line ~274) gains + `behavior_v2`; it records the acked level and emits a warning event on mismatch. + - `arena_studio.html`: `#fmLogLevel` gains `behavior_v2 (compact — default)`; the + stored-value default becomes `behavior_v2` when nothing is stored; an explicitly + stored `behavior_v1` is honored for one release (then dropped); the run-start banner + names the level the bridge acknowledged, and warns "bridge too old for behavior_v2 — + logging behavior_v1" when the ack disagrees. `full` (25-col FicTrac) unchanged. + - Console `#cFtLogLevel` read-only mirror shows the acked level. + Ship order: bridge PR first (backwards compatible — old Studios never send v2), then + the Studio PR. Benches must `git pull` + restart `pixi run bridge` (the existing + stale-bridge lesson) — the ack makes a stale bridge visible instead of silent. + +Expected: 40 s full run 51 MB → ~18 MB; 1 h run → ~30 MB. Still tight vs 35 MiB for +hour-long runs — hence Part 2. + +## Part 2 — Studio commit path (`arena_studio.html` classic shell + `js/studio-github.js`) + +1. **gzip on commit.** After `bridge.exportLog()`, compress in the browser with the + native `CompressionStream('gzip')` (Chrome ≥ 80) and commit `.jsonl.gz` via + the existing `contentBytes` path (already used for `.pat`). Lossless by + definition; 8× (v1) to ~6× (v2, already denser). The uncompressed JSONL is never + committed for recorded runs any more; `.jsonl.gz` is the archival format. + Filename stays colon-free. + Expected sizes: 40 s run ≈ 2.5 MB; 1 h ≈ 4 MB. +2. **Size-based fallback to the Git Database API** for any payload > 30 MiB after + compression (defensive; only multi-hour runs or `full`-level logs get there): + `POST /git/blobs` (base64) → `GET /git/trees/` → `POST /git/trees` (base_tree + + one entry) → `POST /git/commits` → `PATCH /git/refs/heads/`. Same token, + same `WRITABLE_PREFIXES` allowlist, one new orchestration `directCommitLarge` in + `studio-github.js` with pure request builders + tests (mocked fetch, like + `directCommit`). GitHub hard limit is 100 MiB per file. +3. **Manual "⇪ Push log" and the run-summary line** report the compressed size and + which path was used. The existing 401 → `Studio.dropInvalidToken` handling applies. +4. **Docs**: `cshl-pipeline-test-plan.md` size-limits table (35 MiB row → "raw; gz + ≈ 6–8× smaller; >30 MiB gz uses the Git Database path"), `data-repo-setup.md`, + release notes, CLAUDE.md gotcha ("run logs are `.jsonl.gz`; readers must inflate"). + +## Part 3 — Readers (dashboard `dashboard/data-browser/`, replay viewer, kinematics) + +1. **Inflate on read**: `github-repo.js` / `app.js` detect `.gz` (name or magic bytes + `1f 8b`) and pipe through `DecompressionStream('gzip')` before `parseJsonl`. + Local-file and same-origin-URL loaders too. Contents API raw reads of ≤100 MB are + supported, so a 4 MB `.gz` is trivial. +2. **Parse v2**: `analysis-core.js parseJsonl` reads `frame_schema` (or + `run_metadata.log_format`); `"a"` arrays are expanded to the v1 `arena_command` + object shape *once at parse time*, so `p3LedEpochs`, stall forensics and every + existing consumer keep working unchanged. Frames are identical in v1/v2. +3. **Catalog**: filename parser accepts `.jsonl.gz`; size column shows compressed + size with the raw size in hover. +4. Tests: `tests/test-analysis.js` gets a v2 + gz fixture (generated by the Part 1 + converter from the existing short-P3 fixture) and asserts identical page output + for v1 and v2 of the same run (same PI values, same bundle counts). The corpus + script additionally builds every P3 page for the v1 and v2 version of EVERY P3 + run in the clone and diffs the CSV rows (parity across the whole corpus). +5. **Other readers (inventory 2026-09-06, `grep arena_command|frame_schema`):** + `js/runlog-replay.js` (replay viewer) and the Studio's run-log adapter + (`js/studio-runlog-adapter.js` / Alt replay) read log FILES and need the same + inflate + v2→v1 expansion helper — put it in ONE shared classic dual-export module + (`js/runlog-format.js`: `inflateIfGzip(bytes)`, `expandV2Line(arr, schema)`, + `detectFormat(firstLines)`) used by the dashboard, the replay viewer and the + adapter, with its own Node test. `js/kinematics.js` and the live scope consume + frame arrays / websocket events only — unchanged. `js/arena-session.js` PRODUCES + the arena_command objects over the websocket — unchanged (compaction is at the + bridge's write step). + +## Migration of existing files + +- Nothing is re-written in place. Old `.jsonl` (v1) stay readable forever. +- Optional one-shot: convert + gzip the existing 27 rig03-sr / bench files with the + Part 1 CLI and commit as `.jsonl.gz` alongside (or instead — Michael's call), which + also shrinks the course repo by ~250 MB. + +## Test matrix (what "done" means) + +| Layer | Test | Where it runs | +|---|---|---| +| Format | v1→v2→v1 round trip on curated samples incl. null status, non-null error, unknown key → raise | `tests/test-bridge-behavior.py` (pixi + CI) | +| Format | Round trip + size table + parity over all 165 existing logs | `scripts/runlog-v2-corpus.py` (local clone) | +| Bridge | `log_control` ack + hello `levels`; v2 default; v1 still selectable | `tests/test-bridge-behavior.py` | +| Bridge client | whitelist accepts v2; mismatch warning event | `tests/test-fictrac-bridge-client.js` | +| Studio | gzip bytes committed via `contentBytes`; `.jsonl.gz` name; size threshold picks `directCommitLarge`; 401 drops token | `tests/test-studio-github.js` (mocked fetch) + browser with mocked api.github.com | +| Shared reader | gzip magic detection, `expandV2Line` exact v1 shape, `detectFormat` | new `tests/test-runlog-format.js` | +| Dashboard | v1 vs v2.gz identical pages/PI/bundles; catalog accepts `.jsonl.gz` | `dashboard/data-browser/tests/test-analysis.js` | +| Replay / adapter | opens v2.gz, same event stream as v1 | existing replay tests + fixture | +| Bench | real runs on a rig (below) | manual | + +## Verification (end-to-end, on a rig) + +1. Bridge on `behavior_v2`: run a short P3; log shows `"a"` lines; `--convert` round + trip of that file is byte-identical in canonical JSON. +2. Studio: recorded 20 s run → `.jsonl.gz` committed (size shown); 40 s run → same, + ~2.5 MB; force the Git Database path with a size threshold override and confirm the + blob/tree/commit/ref sequence lands one file. +3. Dashboard: open the `.gz` from the repo and from a local file; P3 pages identical to + the v1 run of the same protocol; Heisenberg bundles identical. +4. Old v1 `.jsonl` still open everywhere. + +## Open decisions for Michael + +- Keep committing an uncompressed copy for very small runs (readability on GitHub) or + always `.gz`? (Recommend always `.gz`; GitHub can't render 20 MB JSONL anyway.) +- Convert the existing course-repo logs to v2+gz, or leave history as is? +- Default log level after the release: `behavior_v2` everywhere, including course + benches (recommend yes; readers handle both).