From 0f390d5ed9fc248fbab00c083853d4923e7bb299 Mon Sep 17 00:00:00 2001 From: Michael B Reiser Date: Sun, 6 Sep 2026 18:37:12 -0400 Subject: [PATCH 1/2] =?UTF-8?q?feat(readers):=20open=20.jsonl.gz=20+=20beh?= =?UTF-8?q?avior=5Fv2=20run=20logs=20everywhere=20=E2=80=94=20shared=20js/?= =?UTF-8?q?runlog-format.js=20(PR=203=20of=20runlog-behavior-v2-plan)=20?= =?UTF-8?q?=E2=80=94=20Studio=20v0.73?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Reader side of docs/development/runlog-behavior-v2-plan.md (Part 3). Stacks on the Studio PR (v0.72). With this, everything that opens a run-log FILE inflates gzip on the magic bytes and expands behavior_v2's compact arena echoes back to the v1 `arena_command` object at parse time, so every existing consumer (p3LedEpochs, stall forensics, replay frame decoding) is unchanged. - js/runlog-format.js (new; classic dual-export, vendored byte-identical at dashboard/data-browser/vendor/): isGzip, isRunlogName, stripGz, inflateIfGzip, readRunlogText (string|bytes|ArrayBuffer|Blob → text), readRunlogPrefixText (truncation-tolerant gunzip for the catalog's 64 KB metadata reads), isArenaArray, expandV2Line (exact v1 key order; timeout ⇒ status/echo/ok all null), compactV1Line, detectFormat, createNormalizer() (per-file state machine), convertV1ToV2Text / convertV2ToV1Text (JS mirror of the bridge converter, for tests + parity). - Dashboard: analysis-core.js parseJsonl normalizes every line, run.logFormat + run.rawBytes, parseFilename strips .gz; github-repo.js fetchRaw reads bytes and inflates (prefix mode inflates a truncated gz); app.js loaders (repo / URL / local server / dropped file) go through readRunlogText, catalog filters accept .jsonl.gz, new size column (compressed size; inflated size + format in the hover), ?v= cache busters bumped, file input accepts .gz. - Replay: js/runlog-replay.js parseRunLog normalizes records (loads js/runlog-format.js first in arena_studio.html; degrades to "v2 echoes skipped" + console warning under a stale cache); js/arena-studio-alt.js replay picker lists .jsonl.gz and inflates the picked file. arena_replay_viewer.html (protocol-fed) and js/studio-runlog-adapter.js (no file reads) unchanged. - Tests: tests/test-runlog-format.js (new, 86 checks, in pixi run test — also enforces the vendored copy); tests/test-runlog-replay.js v2 case identical to the equivalent v1 (76 checks); dashboard test-analysis.js re-reads the P3 fixture as v2 + gz asserting identical frames, arena_command objects, preference indices, LED epochs and page CSV rows, plus metadata from a gz prefix; dashboard/data-browser/tests/corpus-v2-parity.js runs the same over a whole clone (164/164 on cshl-2026-course origin/main). - Browser-verified: the dashboard opens a converted .jsonl.gz via ?data= (52,369 frames / 29 steps, size column "1.6 MB gz", no console errors). - Docs: dashboard README, release notes v0.73, plan doc status + PR 3 notes, CLAUDE.md reader rule. Co-Authored-By: Claude Fable 5.1 --- CLAUDE.md | 7 +- arena_studio.html | 5 +- dashboard/data-browser/README.md | 23 +- dashboard/data-browser/analysis-core.js | 20 +- dashboard/data-browser/app.js | 81 ++- dashboard/data-browser/github-repo.js | 47 +- dashboard/data-browser/index.html | 9 +- dashboard/data-browser/styles.css | 6 + .../data-browser/tests/corpus-v2-parity.js | 178 +++++++ dashboard/data-browser/tests/test-analysis.js | 85 ++++ .../data-browser/vendor/runlog-format.js | 481 ++++++++++++++++++ .../development/arena-studio-release-notes.md | 9 + docs/development/runlog-behavior-v2-plan.md | 43 +- js/arena-studio-alt.js | 14 +- js/runlog-format.js | 481 ++++++++++++++++++ js/runlog-replay.js | 27 +- pixi.toml | 2 +- tests/test-runlog-format.js | 466 +++++++++++++++++ tests/test-runlog-replay.js | 98 ++++ 19 files changed, 2025 insertions(+), 57 deletions(-) create mode 100644 dashboard/data-browser/tests/corpus-v2-parity.js create mode 100644 dashboard/data-browser/vendor/runlog-format.js create mode 100644 js/runlog-format.js create mode 100644 tests/test-runlog-format.js diff --git a/CLAUDE.md b/CLAUDE.md index 2614fb4..db626f2 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -296,8 +296,11 @@ fix flows to every page automatically; two hand-written HTML pages never will. `commitRunLog` gzips the bridge export (`GH.gzipBytes`) and commits `runlogs//.jsonl.gz` via `GH.commitFile`, which routes >30 MiB payloads through the Git Database API (`GH.directCommitLarge`) because the Contents API - rejects ~35 MiB+ files. Every reader (dashboard, replay viewer, adapter) must inflate - on the gzip magic and accept both `behavior_v1` and `behavior_v2` line formats. The + rejects ~35 MiB+ files. Every reader must go through **`js/runlog-format.js`** + (`readRunlogText` to inflate on the gzip magic, `createNormalizer().normalize(rec)` per + parsed line so `behavior_v2` `["a",…]` echoes become the v1 `arena_command` object) — + the dashboard uses an exact vendored copy at `dashboard/data-browser/vendor/`, and + `tests/test-runlog-format.js` fails when the copies diverge (re-copy after editing). The log level is a runtime setting (File ▾ → Run logging, localStorage `studio_log_level`, default `behavior_v2`); the runner asserts it via `log_control` and the bridge ACKS the level it will actually write (`bridge.waitForLogLevelAck`) — a pre-3.0 bridge diff --git a/arena_studio.html b/arena_studio.html index a5928ab..0f23f4c 100644 --- a/arena_studio.html +++ b/arena_studio.html @@ -2918,7 +2918,7 @@

Import error