Repository navigation
Add judge calibration against human labels or a cross-family judge - #145
Conversation
simpleaudit.calibration scores a judge's stored verdicts against a reference that is independent of it: human labels typed in the terminal with label(), or the same transcripts re-graded by a judge from another model family. Any number of labels works; every rate carries its n and an interval, and the report says when the count is only indicative. Adds krippendorff_alpha to simpleaudit.stats, pinned to the published worked example. rejudge() now records the new judge model in the result metadata instead of carrying the source judge's, so a rejudged file can serve as a reference.
|
One thing worth showing next to the headline alpha: under a skewed reference distribution it reads low even when agreement is near-perfect, which is the common case when most scenarios pass. With We hit the same pattern on real data in a cross-family check (Claude Opus against Qwen as second judge, 216 answers, one judge gave the top grade to 96.3 % of them): linearly weighted κ 0.262, raw agreement 0.894, Gwet's AC2 0.940 (results v1.0.0, 10.5281/zenodo.23947387). Two small options: print the reference distribution beside alpha so the reader can see the prevalence, or add Gwet's AC2, the weighted form of AC1, as a companion that is less affected by it (Gwet 2008; Feinstein & Cicchetti 1990 on high agreement with low kappa). |
Krippendorff's alpha corrects for chance from the observed category frequencies, so when nearly every reference item is pass it reads low at near-perfect agreement: 98 pass and 2 low with 97 matches gives 0.39. Gwet's AC2 estimates chance agreement from how evenly the levels are used and reads 0.99 on the same data. The report now prints both, with the reference level counts between them, and leaderboard() ranks by AC2. gwet_ac in simpleaudit.stats handles any number of raters, missing ratings, and identity, linear or quadratic weights; the tests pin it to the two-rater closed form and to Bennett's S under uniform marginals.
|
Thanks, reproduced exactly: 0.388 with the branch's alpha on your 98/2 example. Fixed in 893032e. The report now prints the reference level counts, Gwet's AC2 (linear weights, |
|
Cross-checked |
reframing.rejudge_async: take dev's metadata handling (superset of this branch's; calibration tests pass). stats: union of imports.
Adds
simpleaudit.calibration: score a judge's stored verdicts against a reference that is independent of it. Every other judge check in the library compares the judge with itself or with another model; this one asks whether it agrees with a person, or failing that with a judge from a different model family.What is in it
label(results_file)shows one stored transcript at a time in the terminal and asks for a severity. It never shows the judge's verdict and saves after every answer, so you can label six today and nine tomorrow. Labels are a plain{scenario_name: severity}dict, so you can also type them.calibrate(subject, reference)scores a results file, anAuditResults, or aReframingResults(one row per variant) against human labels, a labels file, or a results file graded by another judge. Any number of labels works. Every rate prints its n and a Wilson interval, and the report says "indicative only" below ten items.model_family()maps a model id to its lineage (Anthropic, OpenAI, Google, Meta, Mistral, Qwen, DeepSeek, ...). A model reference from the subject judge's own family raises;allow_same_family=Trueoverrides it for experiments. Unknown ids warn.simpleaudit.stats, both reported with the reference level counts beside them:gwet_ac(Gwet's AC1/AC2, any number of raters, identity, linear or quadratic weights, pinned to the two-rater closed form) andkrippendorff_alpha(nominal, ordinal, interval, pinned to the published worked example: 0.743, 0.815, 0.849). Alpha drops under a skewed reference even at near-perfect agreement (98 pass, 2 low, 97 matches: alpha 0.39, AC2 0.99), soleaderboard()ranks by AC2. This came from avalyset's review comment.report.save()writes JSON underresults/calibration/,list_calibrations()reads that folder back. README section andexamples/judge_calibration_example.py.One change outside the new module
rejudge()used to return the source file's metadata unchanged, so a re-graded file still claimed the original judge model. It now records the newjudge_model, drops the stale provider, and keeps the original underrejudged_from. Without this, a calibration built fromrejudgeoutput would read the wrong judge and the family check would compare a model with itself.What is deliberately not in it
No spreadsheet export, no transcript hashing, no dev/test split, no alternative annotator test, no corrected failure rate, no registry of calibrated judges. Those can come once there are enough labels to need them.
Tests
pytest -n autoon this branch: 2001 passed, 6 skipped. New tests aretests/test_calibration.py(48 tests, no model calls: fakes fromtests/fakes.pythroughreframing_checkandrejudge) and ten alpha and AC2 tests intests/test_stats.py.ruff check --select E9,F821 .reports nothing.import simpleauditloads no new third-party module.I ran the human-label path on
examples/nav_aap/nav_aap_sonnet_4_6.jsonwith six made-up labels to check the output format; the numbers in the README are from that run and are not a real calibration. I did not run the cross-family path against a live OpenAI judge.Built with Claude Code and Claude Mythos 5.1.