feat(auditing): commit-then-reveal scoring, encrypted test-data keys, resampling policy - #63
Open
Abidoyesimze wants to merge 1 commit into
Conversation
… resampling policy Part 2 of task_210726_6 (issue InfiniteZeroFoundation#40). Builds on InfiniteZeroFoundation#62 (Part 1, scoring) -- opened as a separate PR since they touch some of the same functions but aren't dependent on each other. Commit-then-reveal (§2a): auditors currently call setAuditScorenEligibility in plaintext, in one transaction -- a later auditor can just copy an earlier one's score. Replaced it with commitAuditScore (store keccak256(score, vote, salt)) and revealAuditScore (validate against the stored hash, then run the same scoring/eligibility logic the old function did). The two phases are gated by a new GIstate, LMSevaluationRevealStarted, so no reveal can happen until the model owner explicitly closes the commit window -- that's what actually prevents the copying. One decision worth flagging: I appended the new state to the end of GIstates instead of inserting it where it chronologically belongs (right after LMSevaluationStarted). dincli/cli/utils.py keeps a hand-written positional array of state names indexed by the enum's raw int value, and a compare in dincli/cli/context.py does ordinal `<` checks against a couple of states too. Inserting would've silently shifted every later state's number and broken every state name dincli displays from that point on. Appending isn't perfectly order-preserving for those `<` checks either, but I checked -- none of them use a threshold in the affected range, and the ones that would be are just read-only "show batches" queries anyway. Wrote this reasoning into the enum comment directly so it doesn't get relitigated later. An auditor who commits but never reveals just never sets hasAuditedLM, so they're excluded from quorum/median counting exactly like a non-participant, and slashAuditors' existing "missed vote" check already catches them -- didn't need to add anything for that case. Encrypted test-data keys (§2b): assignAuditTestDataset now takes a bytes[] of per-auditor encrypted keys alongside the CID, stored in a new encryptedTestDataKey mapping. On-chain plumbing only, no dincli/Python decryption change. Resampling policy (§2c): found the actual dataset-assembly function (create_audit_testDataCIDs in cache_model_0/services/modelowner.py) and wired the real policy into it instead of just writing a design doc -- a fixed ~40% reserved pool, resampled to a fresh half each round, full pool only on the round the model owner marks as final. Added is_final_round as a defaulted kwarg so the existing dincli call site (which doesn't pass it yet) keeps working unchanged; documented that gap plainly in Developer/design/test-set-resampling-policy.md rather than pretending it's fully wired end to end. 12 new Foundry tests, 14 new pytest tests. Existing suite passes.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part 2 of task_210726_6 (issue #40). Builds on #62 (Part 1, scoring) -- opened as a separate PR since they touch some of the same functions but aren't dependent on each other, and I didn't want to bundle unrelated review surface together.
Commit-then-reveal (§2a)
setAuditScorenEligibilitycurrently takes a plaintext score in one transaction -- a later auditor can just look at an earlier one's submission and copy it. Replaced it with two functions:commitAuditScorestoreskeccak256(score, vote, salt),revealAuditScorechecks the reveal matches that hash before running the same scoring/eligibility logic the old function did. The two phases are gated by a new GI state,LMSevaluationRevealStarted, so no reveal is accepted until the model owner explicitly closes the commit window -- that gate is what actually prevents the copying, not the hash by itself.One decision worth flagging directly: I appended the new state to the end of
GIstatesrather than inserting it where it chronologically belongs (right afterLMSevaluationStarted). Reason:dincli/cli/utils.pykeeps a hand-written array of state names indexed by the enum's raw integer value, anddincli/cli/context.pydoes a couple of ordinal<comparisons against specific states too. Inserting a new member in the middle would've silently shifted every later state's number and broken every state name dincli displays from that point on. Appending isn't perfectly order-preserving for those<checks either -- I checked, and none of them use a threshold in the affected range, and the ones that would be are read-only "show batches" queries whose data doesn't exist yet during this window regardless. Wrote the full reasoning into the enum's comment so it's there if anyone revisits this later.An auditor who commits but never reveals just never sets
hasAuditedLM, so they fall out of quorum/median counting the same way a total non-participant already does, andslashAuditors' existing "missed vote" check already catches them without any changes -- didn't need special-casing for the non-reveal case.Encrypted test-data keys (§2b)
assignAuditTestDatasetnow takes abytes[]of per-auditor encrypted keys alongside the CID, stored in a newencryptedTestDataKeymapping. On-chain plumbing only, no dincli/Python decryption change (matches the task's scope boundary).Resampling policy (§2c)
Found the actual dataset-assembly function (
create_audit_testDataCIDsincache_model_0/services/modelowner.py) and wired the real policy into it instead of just writing a design doc: a fixed ~40% reserved pool, resampled to a fresh half each round, full pool only on whatever round the model owner marks as final. Addedis_final_roundas a defaulted kwarg so the existing dincli call site (which doesn't pass it yet) keeps working unchanged. Documented that gap directly inDeveloper/design/test-set-resampling-policy.mdrather than implying it's fully wired end to end -- it isn't, dincli still needs a way to know which round is the last one.Tests
12 new Foundry tests (commit-reveal correctness + anti-copying properties, encrypted key assignment), 14 new pytest tests (reserved-pool/round-resample policy logic + one real end-to-end integration test through the actual function). Existing suite passes.