feat(benchmark): untrusted candidate sandbox policy and fail-closed run verdict (#9264) - #9606
Merged
Merged
Conversation
…un verdict (#9264) Candidate agents are untrusted by construction: arbitrary third-party code submitted to be scored. Two facts drive this module. ATTESTATION IS NOT A SANDBOX. Attestation proves WHAT ran; it constrains nothing about what that code can reach while running. So every property here holds with or without SNP hardware and none may be skipped because attestation is present -- they are enforced by the container runtime, which exists today, rather than waiting on the TEE fleet (#8535/#8536). THE NETWORK IS THE WHOLE BALLGAME. A candidate that can reach the network can fetch the realized future -- the outcomes it is being asked to predict -- and score perfectly having learned nothing. #9259 leak-proofs the DATA; this leak-proofs the RUNTIME. Both are required: a perfect snapshot handed to a networked process is not a benchmark. The policy is pure and produces an argv ARRAY, never a shell string: a snapshot path is data we do not fully control, and as a string it would be a command-injection seam. Asserted with a path full of shell metacharacters. Exactly two mounts exist -- the task read-only, one output file writable -- so no host filesystem of ours is reachable. decideCandidateRunVerdict is FAIL-CLOSED, matching the attested-run path's posture: a run that could not establish its claimed isolation does not score at all. Not scored-and-flagged, not scored at a lower tier -- a networked agent's score is not a weaker measurement of the same thing, it is a measurement of something else, and admitting it at any tier would put a fabricated number on the leaderboard. The tier is read from what was ESTABLISHED, so "the plumbing supports attestation" can never become an `attested` label. A non-positive or non-finite resource limit falls back to the default rather than being honored: failing open on a bound is the one direction that cannot be allowed. The documented limits table is generated from the same constants the runner uses, so the doc cannot drift from what is actually enforced. Part of #9264; running under genuine SNP remains blocked on #8535/#8536.
Contributor
|
Warning ⏸️ LoopOver review result - manual review recommendedReview updated: 2026-07-29 02:53:44 UTC
Review summary Blockers
Nits — 5 non-blocking
Concerns raised — review before merging
📋 Copy for AI agents — paste into your coding agentDecision drivers
Context & advisory signals — never blocks the verdict
Review context
Contributor next steps
Signal definitions
🧪 Chat with LoopOverAsk LoopOver a question about this PR directly in a comment — grounded only in the same cached, public-safe facts shown above, never a new claim.
Full command reference: https://loopover.ai/docs/loopover-commands 🧪 Experimental — new and may change. Decision record
🟩 Safe / merged · 🟦 Advisory · 🟨 Held for review · 🟥 Blocked / closed 💰 Earn for open-source contributions like this. Gittensor lets GitHub contributors earn for the work they already do — register to start earning →. Checked by LoopOver, a quiet PR intelligence layer for OSS maintainers.
|
Contributor
|
Superagent didn't find any vulnerabilities or security issues in this PR. |
Deploying with
|
| Status | Name | Latest Commit | Preview URL | Updated (UTC) |
|---|---|---|---|---|
| ✅ Deployment successful! View logs |
loopover-ui | d25ea2e | Commit Preview URL Branch Preview URL |
Jul 28 2026, 11:49 PM |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #9606 +/- ##
==========================================
+ Coverage 89.89% 89.90% +0.01%
==========================================
Files 883 884 +1
Lines 111692 111864 +172
Branches 26579 26587 +8
==========================================
+ Hits 100408 100576 +168
Misses 9992 9992
- Partials 1292 1296 +4
Flags with carried forward coverage won't be shown. Click here to find out more.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Scope note
#9264 reads as blocked on the CoCo/SNP fleet (#8535/#8536), but its own requirement 2 is explicit that the sandbox properties hold with or without SNP hardware — and those are enforced by the container runtime, which exists today. This PR builds that half in full. What genuinely remains blocked is running under real SNP, which is hardware procurement, not code.
Two facts drive the design
Attestation is not a sandbox. Attestation proves what ran; it constrains nothing about what that code can reach while running. So no property here may be skipped because attestation is present — and there's a test asserting a fully-unisolated run is refused even with
genuineAttestation: true.The network is the whole ballgame. A candidate that can reach the network can simply fetch the realized future — the outcomes it's being asked to predict — and score perfectly having learned nothing. #9259 leak-proofs the data; this leak-proofs the runtime. Both are required: a perfect snapshot handed to a networked process is not a benchmark.
What's here
candidateSandboxArgsreturns an argv array, never a shell string. A snapshot path is data we don't fully control; as a string it would be a command-injection seam. Asserted with a path containing'; rm -rf /; echo '. Exactly two mounts exist — task read-only, one output file writable — so no host filesystem of ours is reachable.decideCandidateRunVerdictis fail-closed, matching the attested-run path's existing posture. A run that couldn't establish its claimed isolation doesn't score at all — not scored-and-flagged, not scored at a lower tier. A networked agent's score isn't a weaker measurement of the same thing, it's a measurement of something else, and admitting it at any tier would put a fabricated number on the leaderboard. All failures are named at once so an operator fixes the runner in one pass.attestedrequires genuine hardware; everything else isreproducible. There is deliberately no path by which "the plumbing supports attestation" becomes anattestedlabel — benchmark: leaderboard artifacts as spec-conformant EvalScoreRecords #9265's emitter and this agree by construction.memoryMb: 0tightens nothing instead of removing the cap.describeCandidateLimitsgenerates the documentation from the same constants the runner uses, so the documented limits can't drift from what's actually enforced — asserted by describing a tightened policy and checking the numbers followed.Tests (7 node:test cases, 100% of branches)
Every claimed control present in the args; the read-only task mount and single writable output; the shell-metacharacter path staying inert as one argv element; limits failing closed on all four bad inputs plus honored overrides; the fail-closed verdict broken one property at a time (each naming exactly one failure) and all five at once with attestation present; the tier never assumed from plumbing; and the generated-documentation drift guard.
Part of #9264 — running under genuine SNP remains blocked on #8535/#8536, so I've left the issue open rather than closing it.