[linear3735]

Why

The public231 benchmark has no command for replaying changing screenshot sequences against official JEV and native Omni with the same ordered requests. This adds that collector on top of upstream #52, also tracked by fork #1. This fork PR targets codex/decision-benchmarks-codex.

How

Start with benchmarks/legs/changed_image_replay.py. It reuses DecisionClient, validates the entire manifest before sending requests, preserves repeated images and option order, and records every attempt with raw responses, hashes and completion counts.

What

Add python -m benchmarks.legs.changed_image_replay for /v1/decide and /v1/systemone, with usage in benchmarks/README.md and offline tests. Callers supply the screenshot manifest, service provenance and a fresh output directory. --cache-state records a declaration; it does not reset caches. Response checks validate the response, including probability normalization within 1e-6 for rounding; they do not establish model agreement, cache correctness or task success.

Usage: changed-image replay commands. The tested native service is the combined Omni revision in fork #7, with #5 and #6 as dependencies.

Verification

Current follow-up: benchmark suite 129 passed, zero skipped, including the existing XML skip guard. Full local suite: 1100 passed, 198 subtests passed, 43 optional/environment-dependent skips. Ruff format/check, ty and smoke passed. The existing required benchmark job now covers the whole tests/benchmarks directory. Kimi and a fresh non-author Codex reviewed the same frozen source and found no blocking issues. CI for this head is tracked separately. Commands:

python -m pytest tests/benchmarks -q -p no:cacheprovider
ruff check benchmarks/legs/changed_image_replay.py tests/benchmarks/test_changed_image_replay.py
ruff format --check benchmarks/legs/changed_image_replay.py tests/benchmarks/test_changed_image_replay.py

• uv run ruff format --check . && uv run ruff check . && uv run ty check
• uv run pytest -q and scripts/smoke.sh
• CHANGELOG.md and the docs say what the code does now

Current PR head f09748b967792d52014abb3b0c3b1acbd0ba069c is based on ThinkFlowLab#52 5c59dc4229c47d2deea31bd3fdb70ce844eaa3d5; the original replay commit was rebased onto that dependency without conflicts. This follow-up has CPU checks only; the historical GPU provenance below is retained separately.

Demo / evidence

• Video link, or exemption/blocker with follow-up: no video attached. Add an application, Agents decision-agent and Omni inference recording from one run before marking ready.
• Application/task → agent actions → independently checked result: not yet demonstrated; CPU tests use HTTP fixtures. The separate H800 run verified the real frontend→worker replay, but did not execute agent actions or complete an application task.
• Historical System1-Agents / System1-Omni commits, checkpoint and worker/frontend provenance: Agents base 40bf742e2b84ad6157bf2990dd5e4bab73571702, head 6fcb65e1498e3419699ebc861adb6b6c7ef38547 (tree 4584a1e8a7349524ef100e0a7e68792c87edec63). A bounded H800 replay used Omni c5f17cf79fc5aa4a722a876688d0da2d809808e1 (tree 52f103c30fe6e235cad26fcb482c12c4dca21e4f), JEV-27B-VL f34b598d4ef4bcefd337bee8d8e7ddd3b7733ccc and the shared ABI6 library. Independently checked 22 sequences / 88 requests (choice 40, noul 24, score 24), with zero HTTP/selection failures and maximum absolute probability difference 0.00239793 against the fixed official references (limit 0.025). This is screenshot-manifest replay, not an application-agent success test. Raw records are retained locally; public evidence link remains pending publication approval.

For an agent-assisted self-review, use the self-review skill.
For recording and upload instructions, use the agent video guide.