fix(scoring): median instead of mean at finalizeEvaluation, S3 threshold in shadow mode - #62
Open
Abidoyesimze wants to merge 1 commit into
Conversation
…old in shadow mode finalizeEvaluation was averaging auditor scores (sum/votes) despite the field being called finalAvgScore and the design docs calling for a median. A single inflated or lowballed auditor score could drag the result either way; median tolerates up to half the batch being dishonest, matching BlockFlow's Algorithm 1. Added _medianOf (bounded insertion sort, batch sizes are small) and switched finalizeEvaluation over to it. Also added the S3 deviation threshold from the design doc, shipped shadow-mode: computed and emitted per auditor via AuditorScoreDeviation on every finalizeEvaluation call, but it doesn't gate approval or trigger slashing yet -- needs empirical validation on real score distributions first, same reasoning as the design doc gives for not hardcoding the number. Mapped the fold-in/permutation-averaging claims in MECHANISM_DESIGN against the actual scoring code first (posted on InfiniteZeroFoundation#39 before writing any tests against it) -- turned out neither exists, only the eligibility gate and holdout-delta scoring are implemented. Added a minimal mc_marginal_gain_score reference implementation (sequential fold-in + permutation averaging, following the LabeliaLabs truncated_MC pattern) plus a cosine-to-consensus check that was also missing, both with pytest coverage -- not a full contribution-scoring rollout, just enough to validate the mechanism and unblock the required test coverage. Existing 4-test suite plus the 9 new median/S3 tests here all pass.
This was referenced Jul 24, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part 1 of task_210726_6 (issue #39).
What's actually broken here
finalizeEvaluationcomputessum / votesand calls itfinalAvgScore. It's a mean. MECHANISM_DESIGN §6 is explicit that the canonical score should be a median across the auditor batch, specifically because a mean lets one dishonest auditor drag the result — a median tolerates up to half the batch being dishonest (this is straight out of BlockFlow's Algorithm 1, which is where the design doc's language comes from in the first place). Fixed it: added_medianOf(simple insertion sort, batches are capped at ~10 auditors so this is fine) and switchedfinalizeEvaluationto use it. Kept thefinalAvgScorefield name as-is even though it's misleading now, mostly to avoid touching the ABI dincli reads — documented the actual semantics in the NatSpec instead.Wrote two tests specifically to catch the mean-vs-median difference (
[20,25,100]→ median 25, mean would've been 48; and the symmetric case with a lowball outlier) since the existing test suite only ever used identical scores everywhere, which can't tell the two apart at all.S3 threshold
Design doc wants the median-deviation threshold shipped "shadow mode" first — computed and visible, not gating anything — until it's been validated against real score distributions. Added
s3DeviationThreshold(settable, defaults to 40 which is deliberately wide, not a tuned number) and anAuditorScoreDeviationevent emitted per auditor on everyfinalizeEvaluationcall. Doesn't touchapprovedor slashing at all — there's a test that specifically proves a batch still finalizes fine even when every auditor in it "exceeds threshold."The fold-in / permutation averaging thing
MECHANISM_DESIGN §6 and the issue both talk about fold-in and permutation averaging like they're already implemented. They're not — went and checked
scoring.py/auditor.py/aggregator.pyline by line before writing any tests (posted what I found on #39), andcontribution_modein the default policy is literally"none". Only the eligibility gate and holdout-delta scoring actually exist.Didn't want to build a whole contribution-scoring system inside this PR's budget, so I added a minimal reference implementation instead —
mc_marginal_gain_score, sequential fold-in averaged over random permutations, same shape as thetruncated_MCmethod in the LabeliaLabs contributivity repo the design doc references. Also added a cosine-to-consensus check since that turned out to be missing too (spec says eligibility should check update direction against the batch consensus, code never did). Both ship with pytest coverage, including one test that checks the Shapley efficiency property holds exactly (contributions should sum to the total value regardless of how many permutations you average over) and one showing duplicate updates get discounted at n_perms=1 but split fairly at high n_perms — matches the tradeoff the design doc describes.Tests
9 new Foundry tests (median + S3), 36 new pytest tests (poison detection + the two new scoring functions above). Everything existing still passes.