Repository navigation
Conversation
A single shared /tmp/evalrx-v2-outbox is owned by the first user who creates it; on a multi-user host every other user's RunLoggerV2 then fails with 'sqlite3.OperationalError: unable to open database file' (115 test failures on the shared server).
BUILTIN_DESCRIPTIONS, plain_description, judge_description and code_description now live in stages/fix_descriptions.py. reporting and contract import a public constant from a small module instead of a private name from the 4000-line agent.
…onts The dated Gemini agentic report joins the Gemma one in docs/validation/. docs/site-src kept a second copy of every font; build.py now reads the fonts in place under docs/assets/fonts instead of copying them there.
FixCandidate, FixValidation, FixOutcome, FixContext and truncated_count move verbatim. Internal callers import them from fix_types; fix_agent still exposes them for existing user code.
verdict, ebh_survivors, refine_signal, min_failures_for_power, no_fix_recommendation, recommend and tier_available become pure functions with explicit inputs instead of FixAgent methods reading self. FixAgent exposes alpha and a bound no_fix_recommendation(), which the loop now uses in place of the private _ebh_survivors/_refine_signal/_no_fix_recommendation.
FixRecorder holds the run logger and absorbs _emit, the three _log_*_exchange methods, _emit_codegen, and the duplicated InstrumentedModel wrapping. FixAgent.run_logger stays a public attribute (the loop swaps it between ladder stations) and now forwards to the recorder.
PipelineCoder writes (CLI coder first, judge fallback), runs, repairs and frozen-model-controls agent-written pipelines, and owns the coder's scratch state (sandbox, last raw stream/usage/repair prompt). The code guards become public functions there, and the text-only catalog note moves next to catalog_text in fix_tools. FixAgent builds one PipelineCoder and passes it max_tier and the max_tokens floor per call.
…lidation.py CandidateValidator measures the baseline, runs every candidate kind and pairs it against the baseline. The baseline is now a Baseline value (scores, rates, counts, unstable) passed to validation explicitly instead of rates stashed on the agent; alpha and the decode floor are passed per call. applies() becomes a free function. The unused _baseline_fresh is dropped.
FixProposer owns the proposal settings and builds candidates for every tier (runtime, L1-L4, built-in defaults, the coded pipeline via PipelineCoder). The per-call tier window and decode floor are passed in; resolve/enforce of the generation floor become proposer method / free function. FixAgent keeps max_repair_rounds and max_judge_candidates as public properties forwarding to their owners.
BUILTIN_SPECS (name -> spec builder) replaces the if-chain in _default_spec, and CONDITIONAL_DEFAULTS (name, batch gate, case predicate, spec, opt-in) replaces four near-identical hand-written blocks in propose(). Adding a built-in or conditional repair is now one entry. Verified against a 53-scenario snapshot of propose() output: identical before and after.
CandidateValidator.strategy() and scores() were if-chains over candidate.kind. Each kind is now its own method behind a kind->method map (per-case strategies vs batch executors), and the two copy-pasted visual search strategies become one, parameterised by kind. Adding a candidate kind is one method plus one map entry.
_confirm_from_explore looked up agent._recommend by string; once that moved to fix_verdict the lookup returned None and the recommendation silently fell back to 'stay at the ceiling'. FixAgent.recommend() is now the public, config-bound entry point and the loop uses it. Regression test included (fails with the old lookup: 'L2' instead of 'L4').
A BUILTIN_SPECS entry covers the floor and pre-registered allowlists, not the judge-less fallback list; reader-facing sentences live in BUILTIN_DESCRIPTIONS. Note the loop-facing changes for custom fix agents (alpha, recommend, no_fix_recommendation) in the changelog.
The thirteen fix_*.py / repair_catalog.py files sat flat in stages/ next to the probe, diagnosis and surgery stages. They now form one subpackage, evalrx.eval_agent.stages.fix, with the fix_ prefix dropped (stages.fix_agent -> stages.fix.agent, stages.repair_catalog -> stages.fix.repair_catalog). Imports, docstring cross-references, docs and example comments follow; `from evalrx.eval_agent import FixAgent` is unchanged. No aliases are kept at the old module paths. The architecture tests that scan stages/*.py now recurse so they keep covering the moved modules.
run_fix reconfigured its FixAgent between ladder rungs by assigning max_tier, min_tier, run_logger and max_validation_cases, and restored them differently on each path (the no-split ladder and the explicit-ceiling path left max_tier changed for the next call). The window and log suppression are now arguments: propose_and_validate(..., max_tier=, min_tier=, record=False), and recommend()/no_fix_recommendation() take the run's ceiling. FixRecorder.muted() scopes the suppression inside the agent. validate_candidate always confirms on the whole batch, as every caller already arranged. A 40-scenario snapshot of run_fix (split x escalation x ceiling x fixing tier, plus skipped stages) through a real FixAgent shows identical proposals, validations, run-log events and outcomes; only the agent's attributes after return differ, now unchanged. A tier ladder on a custom agent that cannot take the window raises instead of mislabelling rungs.
run_fix was 347 lines with three near-copies of the tier ladder logic inline. It now reads as the partition choice plus three short branches: _fix_hypotheses (refuted filtering, unverified leads and their note), _skip_fix (the evidence-gate skip), _climb_ladder (one tier per rung, failed attempts fed forward, stop at the first fix) shared by both escalation modes, _merge_rungs (e-BH over the union when there is no holdout) and _log_fix. The held-out and no-holdout ladders keep their own post-processing: confirm-once versus merged union. The 40-scenario run_fix snapshot is byte-identical. One edge changes: an opaque custom-agent result inside a no-holdout ladder is now returned as the held-out ladder already did, instead of raising AttributeError.
evalrx.benchmark.tasks.llm loaded its slice definitions by file path from examples/dataset_selection, so the nine LLM tasks failed outside a repo checkout. The nine specs, the adapters/graders they use, the row fetcher and the catalog now live in evalrx.benchmark.tasks.llm (specs, fetch, catalog). requests and HF datasets are imported on use, so the task registry imports without them. band_locate.py keeps the other survey candidates and reuses the library specs; datasets.py is reduced to the catalog CLI and the 2B probe records. The benchmark image no longer copies examples/dataset_selection.
extract_run_data is what the dynamic report, the Langfuse bundle export and the legacy static page all read a run through, but it lived inside html_report.py next to 1,100 lines of HTML templating. It now lives in evalrx/reporting/run_data.py with the run-layout helpers and analyzer glossary it uses; html_report.py keeps only the static renderer.
The fix-stage helpers behind VLDiagnoseLoop.run_fix / run_m5 (target selection, the tier ladder, EXPLORE->CONFIRM) move unchanged to loop_fix.py, next to loop_reports.py and run_metadata.py. loop_fix does not import loop. VLDiagnoseLoop is unchanged; loop.py goes from 2,141 to 1,706 lines.
The import test now loads evalrx before blocking requests, so it checks the task registry rather than every third-party import evalrx pulls in. The CLI docs say the LLM slice definitions ship with the package and their rows still need the data extra.
Vitest covers reportAccess and buildBrief (snapshotted on a real run's payload); a Playwright smoke test opens every view of that run from the production build and fails on any console error; ESLint runs the TypeScript and hook rules. All three run in a new frontend CI job. The tsbuildinfo caches are no longer tracked.
Adds Prettier (format / format:check scripts, checked in CI) and applies it to the frontend sources. The rendered DOM of every view is unchanged on six real runs; the built CSS differs only in whitespace inside custom property values.
ReportData.stage_detail gets a contract model (evalrx.contract.stage_detail, one model per stage), exported to JSON Schema and contract.d.ts. The compiler builds it one function per stage with unchanged output, and a test validates real and synthetic reports against the models. The UI reads it through stageDetail() as typed partial views, which removes the any annotations it caused in views, brief and reportAccess.
views.tsx held every stage's full record, the case studio, the debug log and the formatting helpers in one file. Each stage now has its own module (stages/m1..m5.tsx) holding its full-record view beside its brief builder; the stage layout pieces, M2's shared evidence charts, the formatters, the detail shell, and the cases, debug and evidence pages are separate modules. brief.tsx keeps the shared brief rendering. Code moved unchanged; the unused BriefStatusIcon is dropped.
The view on screen was a string in App state, parsed by hand
("cases:split=..."), so a refresh dropped the reader back on the
overview and no view could be linked to. Routes are now typed and kept in
the hash (#/evidence/m3/full, #/cases/<id>?split=..., #/debug), which also
works in a portable export opened from disk. Moving between views adds a
history entry; selecting within a view replaces it. Every navigation call
passes a Route instead of a string.
transformers 5.23 expands grouped query/key heads inside torch_chunk_gated_delta_rule, so key can have fewer heads than the values, g and beta. mixing_matrix built its identity values from key's head count and failed with a shape mismatch on Qwen3-Next and Qwen3.5; it now uses beta's (value) head count, which equals key's on earlier versions.
The server held one ReportSession: opening another run from the runs
panel or an upload swapped it for every open tab. A RunRegistry now keeps
the runs the server itself registered (its start-up run, a listing, an
upload) by id, compiled on first use, and every report route names its
run: /api/runs/{run_id}/report, stages, cases, debug, media and
artifact. /api/session names the start-up run. The UI keeps the id of
the run on screen and builds every request and media URL from it
through api.ts.
Run reports now have one UI, the React report. evalrx dashboard and evalrx.analysis.dashboard (its http.server and the loaders only its tests used) are removed, with case_studio, run_view and stage_views, which nothing else used. Explore outputs are not M1-M5 runs and the React report cannot render them, so explore / run-codebase keep writing the static report.html; --serve-report now opens that file instead of starting a server, and the --dashboard alias and --port go. The deco_hallu scripts open each phase's run with evalrx serve, and the example docs point at serve.
Both only re-exported evalrx.agent_runtime under their old paths. The tests import from agent_runtime directly, and the architecture test now checks that no such facade comes back.
AutoDiagnoseLoop and SelfEvolveLoop (legacy.py) predate VLDiagnoseLoop; EvalOrchestrator sat on top of SelfEvolveLoop, and ab_runner, preregister and the eval_agent DiagnosticReport were used only by it. _tools had one live function, which SurgeryAgent's parameter sweep now calls directly. The tests whose subject is gone go with them; the Docker fallback of ProbeAgent, previously covered only through the legacy loop, gets a direct test. The docs' examples move to VLDiagnoseLoop.
checkpoint, evolution (EvolutionStore) and git_manager (ExperimentGitManager) served only AutoDiagnoseLoop's run_dir; nothing left in the package or the examples reaches them. Their tests and the docs sections describing them go too.
run() was one 198-line method; its cycle body is now _run_cycle over a small _LoopState, and the partition logging, held-out target and cycle summary are their own helpers. run_confirm's stats regeneration and M4 choice are _confirm_stats and _m4_verdicts. Starting and ending the run log (_start_run_log / _end_run) was written out four times, including in AgenticDiagnoseLoop.run; all four now share it. A 38- scenario behaviour snapshot of run, run_analysis, run_confirm and the agentic run (helper calls, logger calls, report) is identical before and after.
With the run id held only in UI state, reloading after opening another run from the runs panel showed the server's start-up run under the other run's view. The hash now carries the run id (?run=<id>); startup opens that run and falls back to the start-up run when the server does not know the id. The report route returns the run's id, label and root so the runs panel still marks the run on screen after a reload, and the stages / debug routes get a test.
Nanboy-Ronan
changed the title
jiaq-liu
approved these changes
Nov 13, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
A refactor of the fix stage, the report stack and
eval_agent, plus onecompatibility fix. Behaviour is unchanged except where noted under
"Breaking changes"; every such change is in
CHANGELOG.mdunder[Unreleased].Fix stage (M5).
fix_agent.pyis split into theevalrx/eval_agent/stages/fix/package (types, proposal, validation, codegen, pipeline, verdict, recorder, …).
Built-in repair candidates are data, candidate execution dispatches by kind,
and the loop passes the tier window per call instead of mutating
FixAgent.run_fixorchestration moved toloop_fix.py.VLDiagnoseLoop.runandrun_confirmare split into their steps(
_run_cycle,_confirm_stats,_m4_verdicts, …); starting and ending therun log is shared with
AgenticDiagnoseLoop.Report UI.
and Prettier, all in a new
frontendCI job.ReportData.stage_detailhas a contract type (evalrx.contract.stage_detail,exported to JSON Schema and
contract.d.ts), so the views read typed data.views.tsxis split into one module per stage (src/stages/m1..m5.tsx),each next to its brief builder.
shared links and Back work, including in a portable export opened from disk.
/api/runs/{run_id}/....reporting/run_data.py.Packaging. The nine text-LLM benchmark slices ship in the package
(
evalrx.benchmark.tasks.llm), so the LLM tasks work from an installedpackage instead of loading
examples/dataset_selectionat run time.Fix. The delta-rule mixing matrix is sized from the value heads:
transformers 5.23 expands grouped query/key heads inside
torch_chunk_gated_delta_rule, which broke the Qwen3-Next / Qwen3.5 hybridanalyzer tests on Python 3.11/3.12.
Breaking changes
evalrx dashboardandevalrx.analysis.dashboard. Useevalrx servefor a run.
explore/run-codebase/evalrx-explorestill write a staticreport.html(the React report cannot render an explore-only output);--serve-reportnow opens that file, and--dashboard/--portare gone./api/report,/api/cases, … routes are replacedby run-scoped
/api/runs/{run_id}/...;/api/sessionnames the start-up run.AutoDiagnoseLoop/SelfEvolveLoop(legacy.py),EvalOrchestratorand what only it used (
ABRunner,DataSplit,PreregisteredHypothesis,the
eval_agentDiagnosticReport),_tools, andAutoDiagnoseLoop'srun-directory infrastructure (
checkpoint,EvolutionStore,ExperimentGitManager).resume(), heartbeat, lesson recording and per-rungit branches go with it.
AutoDiagnoseReport.final_resultsis now always empty.eval_agent.cli_agent/cli_skillsre-export modules; importfrom
evalrx.agent_runtime.llm.catalog.getraisesKeyErrorfor an unknown name;llm.catalog()andEVALRX_DATASET_SELECTION_DIRare removed.How this was checked
six real runs (byte-identical), the rendered DOM of 125 report views, a
38-scenario snapshot of the loop entry points and a 64-scenario
run_fixsnapshot.
pytest -m "not gpu and not network",ruff check ., the contract exportcheck,
mkdocs build --strict, and the frontend lint / tests / build / smoketest (also in a clean
node:22container).image with a real download, deco_hallu, llm_band_probe).
Type of change
Checklist
ruff check .passesmypy evalrxpassespytest -m "not gpu"passes (all fast unit tests green)FakeModelCHANGELOG.mdupdated under[Unreleased]