Skip to content

Add reproducible 90-sample security benchmark#1

Closed
tiltom wants to merge 1 commit into
mainfrom
agent/reproducible-security-benchmark
Closed

Add reproducible 90-sample security benchmark#1
tiltom wants to merge 1 commit into
mainfrom
agent/reproducible-security-benchmark

Conversation

@tiltom

@tiltom tiltom commented Jul 13, 2026

Copy link
Copy Markdown
Member

What changed

  • adds a pinned 90-sample benchmark corpus and deterministic runner
  • adds 25 metadata-only OpenSSF OSV malicious-package samples
  • records canonical pre-fix and post-fix results
  • adds manual review notes for every severe reference finding
  • refreshes the bundled signature snapshot to 2026.07.12.003
  • adds benchmark pin/run scripts and regression coverage

Why

Flagrix needed reproducible evidence for malicious recall and false-positive launch gates. The runner never clones, installs, or executes benchmark samples.

Results

  • overall: 37/90 → 90/90 passed
  • critical fixtures: 9/10 → 10/10
  • malicious metadata: 5/25 → 25/25
  • reference High verdicts: 27/55 → 0/55
  • reference Critical findings: 16 → 0

Rollout dependency

The benchmark uses scanner-core from the adjacent local checkout. Before this PR is made ready or the CLI is released, update the npm dependency and lockfile from scanner-core 0.2.0 to the published 0.2.2 from flagrix-scanner-core#1.

Validation

  • 15 CLI tests passed
  • TypeScript type-check passed
  • all four benchmark gates pass

@tiltom tiltom closed this Jul 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant