Repository navigation
feat: benchmark and latency numbers in EVALS (D9) - #11
Merged
Merged
Conversation
- internal/bench: a seeded generator written independently of the rules (own NRIC checksum and Luhn, own templates) producing 5,000 samples with 0-3 pieces of personal data in realistic styles, some the rules don't support, plus look-alike decoys. Scored per entity and per writing style; "caught" means fully covered by the same kind. - cmd/benchmark (make bench) prints Markdown: 76.5% recall overall, 100% on every supported style, 92.3% precision (482 false alarms, all bare 8-digit numbers as PHONE, the trade-off in sg-pii-rules ADR 0007). - Latency vs a baseline without Discreet: about 3 ms added at p50, of which 2.9 ms is the audit record flushed to disk and 0.08 ms detection. - Deliberate break (seed ignored) failed the determinism test; restored. Closes makoydev/vetted#39 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
makoydev
added
ai-drafted
This was referenced Oct 5, 2026
makoydev
added a commit
that referenced
this pull request
Oct 5, 2026
- internal/bench: a seeded generator written independently of the rules (own NRIC checksum and Luhn, own templates) producing 5,000 samples with 0-3 pieces of personal data in realistic styles, some the rules don't support, plus look-alike decoys. Scored per entity and per writing style; "caught" means fully covered by the same kind. - cmd/benchmark (make bench) prints Markdown: 76.5% recall overall, 100% on every supported style, 92.3% precision (482 false alarms, all bare 8-digit numbers as PHONE, the trade-off in sg-pii-rules ADR 0007). - Latency vs a baseline without Discreet: about 3 ms added at p50, of which 2.9 ms is the audit record flushed to disk and 0.08 ms detection. - Deliberate break (seed ignored) failed the determinism test; restored. Closes makoydev/vetted#39 Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes makoydev/vetted#39 (D9).
Results (all in
EVALS.md, reproduce withmake bench)#, DOB after "Birthday" or with a 2-digit yearPHONE(the accepted trade-off, sg-pii-rules ADR 0007); other entities had noneMethod notes
If asked in an interview
I measured detection on 5,000 made-up samples from a generator I wrote separately from the rules, including ways of writing personal data I expected the rules to miss, so the numbers can't simply confirm themselves. The result is a per-style picture: 100% on the formats the rules cover, and a clear list of formats they don't. Most of the 3 ms Discreet adds is writing each audit record safely to disk, which I measured rather than guessed. I rejected reusing the shared test cases as the benchmark, because rules always pass the examples they were written for.
🤖 Generated with Claude Code