feat(benchmark): anti-overfit controls — repo-level seeded split, caps, rotation (#9263)

[JSONbored]

Stacked: #9595 (#9260) → #9598 (#9261) → #9599 (#9262) → this. Retarget as the stack lands.

What

A public leaderboard creates a direct incentive to overfit, and an overfit agent is worse than useless — it looks best exactly when it generalizes worst. Four decisions, each recorded with its reasoning in the module header, because each is a knob an adversary optimizes against.

1. Split granularity is the repo, not the work unit

The split reuses splitBacktestCorpus verbatim (requirement 1 — no second split mechanism), and that reuse is asserted, not assumed: a test recomputes the assignment through the primitive directly and requires the identical held-out repo set. But it keys on the repo, so a repo's work units never straddle the boundary.

Work-unit-level splitting leaks badly: two PRs in the same repo share a maintainer, a review culture, a CI setup and often the same files, so an agent that saw repo A's visible PRs has effectively seen the answer pattern for repo A's held-out PRs. Per-repo splitting is what makes "held out" mean "generalizes to a repo it has never seen" — the property #9216 requirement 3 actually names.

2. Held-out scores do not publish per submission

Each published held-out score is an oracle query; differencing successive ones recovers per-unit membership within a few dozen submissions. Publication happens at evaluation close, or on a fixed public cadence counted from opensAt — never on demand, so a submitter cannot trigger a publication instant by submitting.

gateHeldOutPublication drops the held-out value rather than hiding it behind a flag, so a caller cannot forget to check the flag and serialize a field that is present in memory. That's the difference between a policy and a habit — and there's a regression test asserting no held-out work-unit id or repo name appears anywhere in a serialized open-window payload.

3. Submission cap per (agent, window)

Unbounded resubmission against a fixed corpus is gradient descent on the test set by brute force. Scoped per window rather than per day (which just spreads the same brute force over more days) or globally (which would punish an agent that keeps competing across rotations). Refusals are named, so a submitter can tell a cap they can wait out from a window that will never reopen.

4. Rotation retires a window, it does not extend it

A benchmark public for a year is training data with extra steps. A retired window stays published for historical comparison but stops being a scoring basis — and refuses submissions even with quota left.

Tests (11 node:test cases, 100% of branches)

Repo-disjointness of the two slices; equality with the primitive's own assignment; determinism, seed-dependence, and stability as the corpus grows (an already-assigned repo never moves); both degenerate fractions, the empty corpus, and the out-of-range throw; window states across all four boundaries; cap enforcement including an over-count that never reports negative room and explicit cap overrides; retired/pending refusals; publication at close and on cadence but not between; the drop-not-flag regression; and the membership-non-inference regression.

Closes #9263

[superagent-security]

Superagent didn't find any vulnerabilities or security issues in this PR.

[loopover-orb]

Warning

⏸️ LoopOver review result - manual review recommended

_{Review updated: 2026-07-29 01:36:57 UTC}

3 files · 1 AI reviewer · 1 blocker · CI green · clean

⏸️ Suggested Action - Manual Review

Review summary
This adds a self-contained anti-overfit module (repo-level split reuse, submission cap, window retirement, held-out publication gating) with thorough tests for each documented decision, and wires it into the package barrel. The repo-granularity split correctly reuses `splitBacktestCorpus` and the tests assert byte-identical assignment against the primitive directly, which is a real (not fabricated) verification of decision 1. The weak spot is `heldOutPublicationDecision`'s cadence check: it requires bit-exact millisecond alignment (`elapsedMs % periodMs === 0`), which no real periodic caller supplying actual wall-clock time will ever satisfy, so 'scheduled_cadence' publication as designed will practically never fire outside the crafted test fixtures.

Blockers

• packages/loopover-engine/src/calibration/benchmark-anti-overfit.ts: `heldOutPublicationDecision`'s cadence branch (`elapsedMs % periodMs === 0`) demands exact millisecond equality between `now` and a period boundary since `opensAt`; a real caller invoking this from a periodic checker (cron/poll) with actual `Date.now()` values will essentially never hit that exact modulus, so 'scheduled_cadence' publication as implemented will not fire in production even though the accompanying test only exercises hand-crafted boundary-aligned timestamps (test/benchmark-anti-overfit.test.ts 'a configured cadence publishes on the fixed schedule only').

Nits — 5 non-blocking

• packages/loopover-engine/src/calibration/benchmark-anti-overfit.ts: consider `Math.floor(elapsedMs / periodMs) > previousPeriod` or an explicit 'has this period boundary already published' state so the cadence decision is actually reachable by a real poller instead of requiring an exact instant.
• The description is truncated mid-sentence ('Unbounded resubmission against a fixed') so I can't confirm the full rationale for the cap value matches what's implemented, though the code and tests are internally consistent with what is visible.
• packages/loopover-engine/src/index.ts: the new barrel line is placed correctly alongside the other `./calibration/*` exports, consistent with existing convention — no issue, just confirming placement is right.
• Change the cadence check in benchmark-anti-overfit.ts to a floor/threshold comparison (e.g. `Math.floor(elapsedMs / periodMs) >= 1` combined with externally-tracked last-published period) rather than exact modulo equality, so a real periodic caller can actually trigger 'scheduled_cadence'.
• Add a test that calls `heldOutPublicationDecision` with a realistic non-boundary-aligned `now` shortly after a cadence boundary has passed, to confirm the intended behavior of 'publish once the boundary has been crossed' rather than only at the exact instant.

Concerns raised — review before merging

• packages/loopover-engine/src/calibration/benchmark-anti-overfit.ts: `heldOutPublicationDecision`'s cadence branch (`elapsedMs % periodMs === 0`) demands exact millisecond equality between `now` and a period boundary since `opensAt`; a real caller invoking this from a periodic checker (cron/poll) with actual `Date.now()` values will essentially never hit that exact modulus, so 'scheduled_cadence' publication as implemented will not fire in production even though the accompanying test only exercises hand-crafted boundary-aligned timestamps (test/benchmark-anti-overfit.test.ts 'a configured cadence publishes on the fixed schedule only').

📋 Copy for AI agents — paste into your coding agent

Fix the following blocker(s) from this PR review:

1. packages/loopover-engine/src/calibration/benchmark-anti-overfit.ts: \`heldOutPublicationDecision\`'s cadence branch \(\`elapsedMs % periodMs === 0\`\) demands exact millisecond equality between \`now\` and a period boundary since \`opensAt\`; a real caller invoking this from a periodic checker \(cron/poll\) with actual \`Date.now\(\)\` values will essentially never hit that exact modulus, so 'scheduled\_cadence' publication as implemented will not fire in production even though the accompanying test only exercises hand-crafted boundary-aligned timestamps \(test/benchmark-anti-overfit.test.ts 'a configured cadence publishes on the fixed schedule only'\).

Decision drivers

• ❌ Code review — 1 blocker (1 reviewer)
• ❌ Gate result — Blocking (Repo-configured hard blocker found.)

Context & advisory signals — never blocks the verdict

Signal	Result	Evidence
Linked issue	✅ Linked	#9263
Related work	✅ No active overlap found	No same-issue or scoped active PR overlap found.
Change scope	✅ 20/20	Low review scope from cached public metadata (1 linked issue).
Validation posture	✅ 25/25	PR body includes validation/test evidence.
Contributor workload	✅ 10/10	Author activity: 14 registered-repo PR(s), 13 merged, 306 issue(s).
Contributor context	✅ Confirmed Gittensor contributor	JSONbored; Gittensor profile; 14 PR(s), 306 issue(s).
Improvement	✅ Minor	risk: clean · value: minor

Linked issue satisfaction

Addressed
The PR reuses splitBacktestCorpus verbatim at repo granularity, gates held-out publication to close/fixed cadence (with a test proving membership isn't inferable from serialized output), enforces a per-(agent,window) submission cap with named refusal reasons, and models window retirement/rotation, with each decision's rationale recorded in the module header as requested. Tests cover split determin

Review context

• Author: JSONbored
• Role context: owner (maintainer lane)
• Public audience mode: oss maintainer
• Lane context: Repository is registered but has no active allocation in the current snapshot.
• Public profile languages: Python, TypeScript, Ruby, Go, MDX, Shell, Solidity, JavaScript
• Official Gittensor activity: 14 PR(s), 306 issue(s).
• PR-specific overlap: none found.

Contributor next steps

• Start here: Treat this as maintainer-lane context rather than normal contributor-lane activity.
• Then work through the remaining 2 steps in the Signals table above.

Signal definitions

• Related work = same linked issue, overlapping active PRs, or title/path similarity.
• Change scope = cached public metadata such as size labels, draft state, and review-burden hints.
• Validation posture = whether the PR provides enough public validation/test evidence for maintainer review.
• Contributor workload = public contributor activity and cleanup pressure, not a repo-wide quality failure.
• Contributor context = public GitHub/Gittensor identity context; non-Gittensor status is not a blocker.

🧪 Chat with LoopOver

Ask LoopOver a question about this PR directly in a comment — grounded only in the same cached, public-safe facts shown above, never a new claim.

• @loopover ask <question> answers contribution-quality Q&A with source citations and freshness.
• @loopover chat <question> answers in natural prose from cached decision-pack facts via local inference (maintainer/collaborator; read-only).
• A plain-language @loopover mention with a real question is routed to the closest matching read-only command automatically — no exact syntax required.

Full command reference: https://loopover.ai/docs/loopover-commands

🧪 Experimental — new and may change.

Decision record

• action: hold · clause: ai_consensus_defect
• config: 9c7526ec175c73835dc091b2ee466015239ee6d983a078af72ca7409e061c99b · pack: oss-anti-slop · ci: passed
• model: claude-code · prompt: 7ac46a3f8ed0bb19bf9274163ef5821c41a4b11125e4c2ffc182d26d1c35f412 · confidence: 0.72
• record: daab498f6d7db0041850eade719c7e841e52e4a774e9f9405bd0da33261af06b (schema v5, head 12b2b8d)

_{🟩 Safe / merged · 🟦 Advisory · 🟨 Held for review · 🟥 Blocked / closed}

---

💰 Earn for open-source contributions like this. Gittensor lets GitHub contributors earn for the work they already do — register to start earning →.

Checked by LoopOver, a quiet PR intelligence layer for OSS maintainers.

• Re-run LoopOver review

[cloudflare-workers-and-pages]

Deploying with    Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status	Name	Latest Commit	Preview URL	Updated (UTC)
✅ Deployment successful! View logs	loopover-ui	12b2b8d	Commit Preview URL Branch Preview URL	Jul 29 2026, 12:04 AM

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 89.93%. Comparing base (534dfd5) to head (12b2b8d).
⚠️ Report is 1 commits behind head on main.
✅ All tests successful. No failed tests found.

Additional details and impacted files

@@             Coverage Diff             @@
##             main    #9600       +/-   ##
===========================================
+ Coverage   75.83%   89.93%   +14.09%     
===========================================
  Files         279      885      +606     
  Lines       58864   112107    +53243     
  Branches     6340    26627    +20287     
===========================================
+ Hits        44641   100819    +56178     
+ Misses      13953     9992     -3961     
- Partials      270     1296     +1026     

Flag	Coverage Δ
backend	95.32% <6.06%> (?)
engine	66.73% <100.00%> (+0.18%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
...r-engine/src/calibration/benchmark-anti-overfit.ts	100.00% <100.00%> (ø)
packages/loopover-engine/src/index.ts	100.00% <100.00%> (ø)

... and 742 files with indirect coverage changes