feat(benchmark): leak-proof frozen-repo snapshot builder (#9259)

[JSONbored]

What

An agent must see exactly the repository state a maintainer saw at commit T and nothing after it. A future-information leak inflates every score built on top, and it does so invisibly — a leaked snapshot still produces perfectly well-formed numbers. That makes leak-proofing the deliverable rather than a property of the code, which is why all filtering lives in a pure, exhaustively tested core (scripts/frozen-repo-snapshot-core.ts) instead of inline in the CLI where it would be untestable.

Carried, each filtered to its state at T: open PRs and issues, with labels filtered to those applied at or before T (label application times come from the issue-events timeline — a label object alone carries no timestamp, so without that every label would be unfilterable), plus gate decisions recorded at or before T.

Never carried: anything created after T — never admitted, so a downstream bug can't reintroduce it — and the outcome of any included unit. An open PR's eventual merge/close is precisely what the agent is being asked to predict, so a snapshot carrying it would be an answer key rather than a benchmark. The FrozenWorkUnit type has no outcome field at all.

The T boundary is inclusive and defined once, in atOrBefore, so no field can quietly disagree with another. An unreadable timestamp is treated as not-visible: a date nobody can parse might be from after T, and the cost of excluding it is lost context rather than an inflated score.

Two real defects found while testing

Both would have broken reproducibility silently, and both are fixed in the source rather than worked around in the tests:

• checksumSnapshot spread its argument, so handing it a whole FrozenRepoSnapshot — the natural re-verification call — folded the existing snapshotChecksum into its own preimage and returned a different, wrong digest with no error. The committed fields are now listed explicitly, which makes the function total over both call shapes and makes an added field a compile error right at the commitment, which is where you want to notice it.
• The decision sort tied on (workUnitId, decidedAt, action), so two decisions differing only in reasonCode fell back to input order — and the snapshot checksum moved when the CLI happened to read them the other way round. That's the exact reproducibility property this module exists to guarantee. The chain now ends in a canonical-serialization comparison, total by construction, so adding a field to FrozenDecision can't reintroduce the hole.

Also simplified canonicalize's comparator from three arms to two: object keys are unique by definition, so the "equal keys" arm was dead code no test could reach.

The CLI refuses to write a leak

auditSnapshotForLeaks re-derives the no-future-information property independently of the builder, and the CLI exits non-zero rather than writing a snapshot that fails it. An inflated benchmark is worse than no benchmark, because its numbers look fine.

Tests (18, 100% of the core's branches)

The invariants the issue asks for, asserted directly: a fixture full of post-T records producing a snapshot that provably excludes them (checked by scanning the serialized snapshot for every future marker, so a leak by any route fails); identical checksums from differently-ordered inputs; a snapshot built at T that doesn't move when a year of post-T history is appended; the outcome-never-included regression; boundary behavior at exactly T in both directions; fail-safe handling of unreadable timestamps in both the visible and open-at-T decisions; both checksumSnapshot call shapes; every leak shape the audit catches; and the full comparator chain including a byte-for-byte duplicate.

scripts/** isn't Codecov-measured, but the core carries the correctness here, so it's covered to the same bar the rest of the harness is.

Closes #9259

[loopover-orb]

Warning

⏸️ LoopOver review result - manual review recommended

_{Review updated: 2026-07-29 04:25:43 UTC}

3 files · 1 AI reviewer · 1 blocker · CI green · clean

⏸️ Suggested Action - Manual Review

Review summary
This adds a pure, exhaustively tested core (`frozen-repo-snapshot-core.ts`) that filters PRs/issues/decisions to their state at a frozen instant T, deliberately excluding outcome fields and post-T data, plus a thin CLI wrapper and an audit function as a belt-and-braces leak check. The checksum-spread regression described in the PR body is real and now fixed by listing fields explicitly instead of spreading the argument, and the test suite exercises boundary inclusivity, label-timestamp filtering, decision tie-breaking, and checksum stability thoroughly. The main correctness gap is in the untested CLI wrapper: it fetches issues/PRs and label-history events with `per_page=100` but never paginates past page one, so any repo with more than 100 open+closed issues/PRs (or any single issue with >100 label events) will silently produce an incomplete — not leaked, but incomplete — snapshot.

Blockers

• scripts/frozen-repo-snapshot.ts:78 — the `issues?state=all&per_page=100` call has no pagination loop, so repos with more than 100 issues/PRs will silently drop older records from the snapshot with no error or warning.
• scripts/frozen-repo-snapshot.ts:108 — `fetchLabelHistory`'s `/events?per_page=100` call is likewise unpaginated, so an issue with a long label-application history can silently lose earlier `labeled` events, causing `freezeWorkUnit` to omit labels that were genuinely applied before T.

Nits — 5 non-blocking

• scripts/frozen-repo-snapshot.ts:117-120 — the D1 SQL is built by string interpolation with manual quote-escaping (`repo.replace(/'/g, "''")`); prefer wrangler's bound-parameter support if available, since hand-rolled escaping is easy to get wrong later even though `repo` is a trusted CLI arg here.
• scripts/frozen-repo-snapshot.ts:160 — the nested nullish/ternary nesting in nested closures (`main`) is getting deep; consider extracting the issue→work-unit mapping loop body into a small named function for readability.
• scripts/frozen-repo-snapshot-core.ts — `compareCanonical`'s fallback comment claims the two-armed key comparator is safe because object keys are unique, but worth double-checking `canonicalize`'s `(a < b ? -1 : 1)` never receives `a === b` some other way (e.g. via prototype-polluted keys) — likely fine, just flagging for a second pair of eyes given how load-bearing determinism is here.
• scripts/frozen-repo-snapshot.ts:150,155 — `console.log`/`console.error` in the CLI are appropriate for a script, but consider whether stdout output should be gated behind a `--quiet` flag for use in automation pipelines.
• No pagination test exists for the CLI (understandably, since it's the IO layer) — worth adding at least a follow-up TODO or issue reference once pagination is fixed, so it doesn't get silently forgotten.

Concerns raised — review before merging

• scripts/frozen-repo-snapshot.ts:78 — the `issues?state=all&per_page=100` call has no pagination loop, so repos with more than 100 issues/PRs will silently drop older records from the snapshot with no error or warning.

📋 Copy for AI agents — paste into your coding agent

Fix the following blocker(s) from this PR review:

1. scripts/frozen-repo-snapshot.ts:78 — the \`issues?state=all&per\_page=100\` call has no pagination loop, so repos with more than 100 issues/PRs will silently drop older records from the snapshot with no error or warning.

Decision drivers

• ❌ Code review — 1 blocker (1 reviewer)
• ❌ Gate result — Blocking (Repo-configured hard blocker found.)

Context & advisory signals — never blocks the verdict

Signal	Result	Evidence
Linked issue	✅ Linked	#9259
Related work	✅ No active overlap found	No same-issue or scoped active PR overlap found.
Change scope	✅ 20/20	Low review scope from cached public metadata (1 linked issue).
Validation posture	✅ 25/25	PR body includes validation/test evidence.
Contributor workload	✅ 10/10	Author activity: 14 registered-repo PR(s), 13 merged, 284 issue(s).
Contributor context	✅ Confirmed Gittensor contributor	JSONbored; Gittensor profile; 14 PR(s), 284 issue(s).
Improvement	✅ Minor	risk: clean · value: minor

Linked issue satisfaction

Addressed
The PR delivers the pure core (buildFrozenRepoSnapshot, atOrBefore boundary, label/outcome filtering) plus a thin CLI wrapper, implements sha256 checksumming over canonicalized sorted-key content mirroring checksumCases, and includes the exact invariant tests requested (post-T fixture exclusion, checksum reproducibility across input order, build-time independence).

Review context

• Author: JSONbored
• Role context: owner (maintainer lane)
• Public audience mode: oss maintainer
• Lane context: Repository is registered but has no active allocation in the current snapshot.
• Public profile languages: Python, TypeScript, Ruby, Go, MDX, Shell, Solidity, JavaScript
• Official Gittensor activity: 14 PR(s), 284 issue(s).
• PR-specific overlap: none found.

Contributor next steps

• Start here: Treat this as maintainer-lane context rather than normal contributor-lane activity.
• Then work through the remaining 1 step in the Signals table above.

Signal definitions

• Related work = same linked issue, overlapping active PRs, or title/path similarity.
• Change scope = cached public metadata such as size labels, draft state, and review-burden hints.
• Validation posture = whether the PR provides enough public validation/test evidence for maintainer review.
• Contributor workload = public contributor activity and cleanup pressure, not a repo-wide quality failure.
• Contributor context = public GitHub/Gittensor identity context; non-Gittensor status is not a blocker.

🧪 Chat with LoopOver

Ask LoopOver a question about this PR directly in a comment — grounded only in the same cached, public-safe facts shown above, never a new claim.

• @loopover ask <question> answers contribution-quality Q&A with source citations and freshness.
• @loopover chat <question> answers in natural prose from cached decision-pack facts via local inference (maintainer/collaborator; read-only).
• A plain-language @loopover mention with a real question is routed to the closest matching read-only command automatically — no exact syntax required.

Full command reference: https://loopover.ai/docs/loopover-commands

🧪 Experimental — new and may change.

Decision record

• action: hold · clause: ai_consensus_defect
• config: 2e5d4bc1b1ee52da1fe1b50635060fd382a5ce3739172ecb942293b10b178690 · pack: oss-anti-slop · ci: passed
• model: claude-code · prompt: 7ac46a3f8ed0bb19bf9274163ef5821c41a4b11125e4c2ffc182d26d1c35f412 · confidence: 0.72
• record: 9fe08f9f88496e61b8dc2b2f5ec09056752c2328cde5303a4c4f4cccbbba26e8 (schema v5, head 3383519)

_{🟩 Safe / merged · 🟦 Advisory · 🟨 Held for review · 🟥 Blocked / closed}

---

💰 Earn for open-source contributions like this. Gittensor lets GitHub contributors earn for the work they already do — register to start earning →.

Checked by LoopOver, a quiet PR intelligence layer for OSS maintainers.

• Re-run LoopOver review

[superagent-security]

Superagent didn't find any vulnerabilities or security issues in this PR.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 89.96%. Comparing base (3200abc) to head (4690960).
✅ All tests successful. No failed tests found.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #9605      +/-   ##
==========================================
- Coverage   90.24%   89.96%   -0.28%     
==========================================
  Files         900      900              
  Lines      112914   112245     -669     
  Branches    26776    26685      -91     
==========================================
- Hits       101896   100981     -915     
- Misses       9687     9932     +245     
- Partials     1331     1332       +1     

Flag	Coverage Δ
backend	95.52% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.
see 16 files with indirect coverage changes

[cloudflare-workers-and-pages]

Deploying with    Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status	Name	Latest Commit	Updated (UTC)
🔵 In progress View logs	loopover-ui	4cdb151	Jul 29 2026, 04:02 AM