Skip to content

Audit and remediation: 26 passes of fixes and regression tests - #143

Merged
SushantGautam merged 6 commits into
devfrom
audit/system-remediation
Nov 6, 2026
Merged

Audit and remediation: 26 passes of fixes and regression tests#143
SushantGautam merged 6 commits into
devfrom
audit/system-remediation

Conversation

@SushantGautam

@SushantGautam SushantGautam commented Nov 4, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Summary

Audit and remediation of SimpleAudit: 26 review passes over the grading core, judges, targets, experiment caching, tracing and docs. Every confirmed finding is fixed with a regression test (2420 passed, 2 skipped; ruff check --select E9,F821 clean).

Highlights

  • Security: credentials redacted across span attributes, span names, header text, non-JSON text (reprs, truncated JSON), provider-key formats (sk-, ghp_, AKIA…), tool events and URLs, tuned so the judge still sees the evidence it grades. OTLP auth fails closed. The auditor no longer inherits the judge's key across providers. Hard-coded keys removed from history.
  • Grading: verdict parsing and severity normalisation hardened (stray keys, think tags, off-ladder values, contradictions, duplicate checklist entries). SingleTurnAuditor and ModelAuditor grade the same input the same way.
  • Experiment cache: content-based fingerprints (judge logic, scenario text, dates, attachment bytes); stale runs quarantined without overwriting; resume re-runs only errored scenarios; combined results kept consistent with what is on disk.
  • Tracing: 16 MiB body cap on HTTP and gRPC receivers; lowercase OTLP ids; corrected exporter env-var docs.
  • Docs/examples: corrected README claims and the inverted custom-judge rubric; examples no longer crash on ERROR results.

Behaviour changes to note

  • Single-turn transcripts carry the documents' text on the user turn (snapshots regenerated).
  • Result JSONs add error_phase / parse_error / scenario_metadata; ModelStabilityReport.to_dict() adds n_ungraded.
  • Adaptive-settings sidecar renamed adaptive-settings.json; saved files now use the normal umask mode.

Known limits

  • The audit stopped before convergence (two consecutive clean passes) at the maintainer's request; later passes found mostly low/medium edge cases.
  • The shared OTLP receiver caps span count, not bytes.
  • vision_integrity and context_grounding packs don't yet meet the pack-checker conventions.

Test plan

  • .venv/bin/pytest -q -n auto -p no:cacheprovider --no-cov: 2420 passed, 2 skipped
  • .venv/bin/ruff check --select E9,F821 .: clean
  • CI green

@SushantGautam
fix: exhaustive system audit and remediation (24 passes)
584f120 
Twenty-four audit passes over the grading core, judges, targets, experiment
caching, tracing and docs. Every confirmed finding is fixed at its root with
a regression test; AUDIT.md records each finding, fix, verification result,
the pass log and 22 documented remaining risks.

Highlights:
- Security: remove a committed Open WebUI API key default from
  examples/audit_openwebui_rag.py (rotation is still required); redact
  credentials in span attributes, span names, header text, tool events
  and URLs without hiding evidence the judge grades; fail-closed OTLP auth;
  auditor no longer inherits judge credentials across providers.
- Grading: verdict parsing and severity normalisation hardened (stray keys,
  think tags, off-ladder values, contradictions); SingleTurnAuditor and
  ModelAuditor now grade the same input the same way (documents, default
  prompts, validation order, scenario_meta, empty ASR transcripts).
- Experiment cache: content-based fingerprints (judge logic, scenario
  text, dates, attachment bytes), safe quarantine of stale runs, resume
  re-runs only errored scenarios, case-insensitive label collisions.
- Tracing: 16 MiB body cap on HTTP and gRPC receivers, lowercase OTLP ids,
  corrected exporter env-var docs.
- Docs/examples: corrected README claims and the inverted custom-judge
  rubric; examples no longer crash on ERROR results.

Verification: 2396 passed, 2 skipped; `ruff check --select E9,F821` clean.
Copilot AI balanced review requested due to automatic review settings November 4, 2026 12:21
Copilot AI reviewed Nov 4, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing.

@SushantGautam
fix: percent-decode file:// attachments in cache fingerprints (pass 25)
20fc968 
Pass 25 (partial, Haiku review of pass-24 changes): file:// URIs with
percent-escapes were fingerprinted by path, so a regenerated clip could reuse
stale cached runs. Decode the path as the ASR target does. Records pass 25 in
AUDIT.md; the two-clean-pass criterion is still open.
@SushantGautam
chore: stop tracking local AUDIT.md
3d4ff80 
AUDIT.md is an internal working log and must not be published.
@SushantGautam
SushantGautam force-pushed the audit/system-remediation branch from 8807346 to 3d4ff80 Compare November 4, 2026 12:39
@SushantGautam
fix: pass 25 audit findings
4ed0918 
- Redact secrets in span/tool text that is not strict JSON (Python repr,
  truncated or trailing JSON), env assignments (*_PASSWORD=) and curl -u.
- Redact URL passwords containing / or @; treat pwd/pass/jwt/sid/session_id
  query parameters as secrets.
- Resume with adaptive_reruns off sets aside saved adaptive reruns instead of
  silently dropping them from the combined results.
- Inspect severity_counts counts between-step values toward the more severe.
- Local attachments with # or ? in the name are typed by their real extension.
- An off-ladder default-judge severity records parse_error and the reason.
@SushantGautam
fix: pass 26 audit findings
c4799a6 
- Checklist: duplicate entries that agree on status keep the one whose quote
  verifies, so the grade no longer depends on entry order; violated optional
  items are not listed as issues.
- Saved results get the usual umask mode instead of mkstemp's 0600.
- Redact provider API keys (sk-, ghp_, AKIA, xox, AIza, hf_) in free span and
  tool text; config.json redaction covers connection strings, pwd, dsn and
  signing keys.
- A fully cached experiment is not reported cancelled; a combined results file
  listing models this run omits is set aside instead of overwritten.
@SushantGautam SushantGautam changed the title Exhaustive system audit and remediation (24 passes) Audit and remediation: 26 passes of fixes and regression tests Nov 4, 2026
@SushantGautam
perf: parallel reps, close HTTP clients per run, retry blank probes
700aea8 
Found while testing PR #143 live against an OpenAI-compatible endpoint:
- AuditExperiment(max_parallel_reps=N) runs independent repetitions of
  run_scenario_reps concurrently (default 1 keeps the old behaviour); live,
  4 reps went from 26 s to 7 s with results in rep order.
- ModelAuditor.aclose() closes the HTTP clients it created; sync run() and
  each experiment repetition call it, so clients no longer close on a dead
  event loop ('Event loop is closed' tracebacks, sockets held open).
- An auditor probe that is empty only after <think> stripping is retried
  under max_retries instead of failing the scenario at once.
@SushantGautam

SushantGautam commented Nov 4, 2026

Copy link
Copy Markdown
Collaborator Author

Live end-to-end test

Ran this branch against a real OpenAI-compatible endpoint (Qwen3.8-27B-FP8), with that model acting as the target, the judge and the auditor. 31/31 checks passed. The API key was supplied through an environment variable only and does not appear in any saved file.

Area What ran Result
Multi-turn safety audit 4 built-in safety scenarios × 2 turns All graded; token counts recorded
Checklist judge Same scenario through SingleTurnAuditor and ModelAuditor Both pass; quotes verified; "May not …" counted as required
Groundedness with documents 2 context_grounding scenarios Graded; stored turn carries document text without the marks; correctness half present
binary_abstention Must-refuse, must-answer, misspelt expected_outcome Refusal and answer both pass; the typo is ERROR before any API call
rejudge / reframing_check Saved transcripts re-graded with harm, and safety vs harm with k=2 Every conversation graded, no ERROR verdicts
Secret redaction Tool events carrying keys in a URL, a header, a Python repr and an env assignment No secret in judge prompts or saved results; ordinary tool arguments still visible to the judge
Experiment cache 2 reps, resume, then edit a scenario Resume took 0.01 s; the edit re-ran instead of reusing stale runs; old runs kept as .stale; no API key in config.json or any saved file
run_scenario_reps (Studio path) 3 reps, then resume All graded; on_rep_done fired per rep; resume served from cache

Fixes from live testing (commit 700aea8)

Change Measured effect
AuditExperiment(max_parallel_reps=N): independent reps of run_scenario_reps run concurrently. The default of 1 keeps the old behaviour 4 reps: 26 s → 7 s, results in rep order
ModelAuditor.aclose(): owned HTTP clients are closed after each sync run() and each experiment rep Removes "Event loop is closed" tracebacks and lingering sockets
An auditor probe that is empty only after <think> stripping is retried under max_retries It previously failed the scenario even with retries left

Efficiency notes

  • Calls per scenario are minimal: one target call and one judge call per single-turn scenario, plus one auditor call per extra turn.
  • max_workers=6 made a 6-scenario × 2-rep experiment about 3× faster (32.7 s → 10.3 s). It still defaults to 1 to respect provider rate limits.

Unit suite: 2424 passed, 2 skipped; ruff check --select E9,F821 clean.

@SushantGautam

SushantGautam commented Nov 4, 2026

Copy link
Copy Markdown
Collaborator Author

@kelkalot could you take a look at this PR? The live test results are in the comment above, and the PR description lists the behaviour changes to check before merging.

@kelkalot
kelkalot changed the base branch from main to dev November 5, 2026 10:28
@SushantGautam
SushantGautam merged commit 74265d8 into dev Nov 6, 2026
8 checks passed
This was referenced Nov 6, 2026
Merged
Merged
This was referenced Nov 6, 2026
Open
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot
Copilot review effort
Applies to this pull request for everyone.Learn more about Copilot code review.
Copilot left review comments
@kelkalot kelkalot Awaiting requested review from kelkalot kelkalot is a code owner

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@SushantGautam