Refactor the fix stage, report stack and eval_agent; retire legacy surfaces

.github/workflows/ci.yml

@@ -16,6 +16,27 @@ jobs:
       - run: pip install ruff
       - run: ruff check .
 
+  frontend:
+    runs-on: ubuntu-latest
+    defaults:
+      run:
+        working-directory: evalrx/reporting/web
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-node@v4
+        with:
+          node-version: "22"
+          cache: npm
+          cache-dependency-path: evalrx/reporting/web/package-lock.json
+      - run: npm ci
+      - run: npm run format:check
+      - run: npm run lint
+      - run: npm test
+      - run: npm run build
+      - run: npx playwright install --with-deps chromium
+      - name: Smoke-test every view of a real run
+        run: npm run test:e2e
+
   test:
     runs-on: ubuntu-latest
     strategy:
@@ -72,3 +93,13 @@ jobs:
         with:
           files: coverage.xml
           fail_ci_if_error: false
+
+  docs:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.11"
+      - run: pip install mkdocs-material
+      - run: mkdocs build --strict

.gitignore

@@ -256,6 +256,9 @@ examples/diagnosis_loops/deco_hallu/outputs/m1_state.pkl
 # Dynamic report UI: source and lockfile are versioned; these are local/build outputs.
 evalrx/reporting/web/node_modules/
 evalrx/reporting/web_dist/
+evalrx/reporting/web/*.tsbuildinfo
+evalrx/reporting/web/test-results/
+evalrx/reporting/web/playwright-report/
 
 # Stray pre-rename package snapshot left by the 2026-09-03 evalvitals/ -> evalrx/
 # rename (196M of it is reporting/web node_modules); local only, safe to delete.
@@ -278,3 +281,13 @@ docker-compose.override.yml
 # Dynamic-report serve cache (report_data.json / report_spec.json), dropped
 # next to wherever the report UI was served from; regenerated on every serve.
 examples/benchmark/**/report/
+
+# Local experiment artifacts stay at their original paths. See
+# docs/local_artifacts.md for the source/output boundary and inventory command.
+/.local-artifacts/
+/examples/m1_m4/**/data/
+/examples/m1_m5/mmau_qwen2_audio/demo_page/index.html
+# These two GPU notebooks contain incomplete local executions. Publish them
+# explicitly after validation; keep the TPU tutorial versioned.
+/examples/colab/gpu/code_reasoning_repair.ipynb
+/examples/colab/gpu/word_sorting_repair.ipynb

CHANGELOG.md

@@ -6,6 +6,221 @@ Versions follow [Semantic Versioning](https://semver.org/).
 
 ## [Unreleased]
 
+### Fixed — delta-rule relevance on transformers 5.23
+
+transformers 5.23 moved the grouped query/key head expansion of Qwen3-Next /
+Qwen3.5's `torch_chunk_gated_delta_rule` inside the kernel, so its `key` can
+now carry fewer heads than its values. The delta-rule mixing matrix was sized
+from `key` and failed with a shape mismatch (generic attention relevance and
+the delta-rule tap on hybrid models); it is now sized from the value heads.
+Verified on transformers 4.57.3, 5.20.2 and 5.23.0.
+
+### Removed — the legacy loop's run-directory infrastructure (breaking)
+
+Only `AutoDiagnoseLoop` used these, and nothing else reaches them:
+`evalrx.eval_agent.checkpoint` (checkpoint/heartbeat files),
+`evalrx.eval_agent.evolution` (`EvolutionStore`, `LessonEntry`,
+`extract_lessons`) and `evalrx.eval_agent.git_manager`
+(`ExperimentGitManager`). A run's output directory is `RunContext`'s.
+
+### Removed — the legacy `AutoDiagnoseLoop` / `SelfEvolveLoop` and the A/B orchestrator (breaking)
+
+`VLDiagnoseLoop` (and `AgenticDiagnoseLoop`) are the diagnosis loops. Removed,
+with everything only they used:
+
+- `evalrx.eval_agent.legacy`: `AutoDiagnoseLoop` (the pre-2026-06 M1→M2→M3→M5
+  sweep) and `SelfEvolveLoop`. Lost with `AutoDiagnoseLoop`: `run_dir`
+  checkpoints, heartbeat and `AutoDiagnoseLoop.resume()`, automatic lesson
+  recording, and the per-run git branch. Use `VLDiagnoseLoop(model, protocol,
+  ...)`; inject a `SurgeryAgent` with `surgery_agent=` and run it after the
+  loop with `loop.run_m5(report, cases)`.
+- `evalrx.eval_agent.orchestrator` (`EvalOrchestrator`) and what only it used:
+  `ab_runner` (`ABRunner`, `ABResult`), `preregister` (`DataSplit`, `Split`,
+  `PreregisteredHypothesis`, `PreregistrationLog`) and `report`
+  (`DiagnosticReport` — `evalrx.reporting.DiagnosticReport` is a different,
+  kept class). For a paired comparison use `evalrx.stats.compare`.
+- `evalrx.eval_agent._tools`, whose one live function `SurgeryAgent`'s
+  parameter sweep now calls directly.
+
+`AutoDiagnoseReport` stays: both loops return it. Its `final_results` field is
+now always empty (only `AutoDiagnoseLoop` filled it). The docs' examples use
+`VLDiagnoseLoop`, and the quickstart's pre-registered A/B section is gone.
+
+### Removed — `evalrx.eval_agent.cli_agent` and `.cli_skills` (breaking)
+
+The two modules only re-exported `evalrx.agent_runtime`. Import from there:
+`CliAgentConfig`, `CliAgentResult`, `create_cli_agent`, `AgyModel`,
+`ClaudeModel`, `CodexModel` from `evalrx.agent_runtime`; the provider classes
+from `evalrx.agent_runtime.providers.<name>`; `SkillInstaller` /
+`CodexSkillInstaller` from `evalrx.agent_runtime.skills.installer`.
+`evalrx.eval_agent` still exports `CliAgentConfig`, `CliAgentResult` and
+`create_cli_agent`.
+
+### Removed — the `dashboard` report server (breaking)
+
+Run reports have one UI: the React report (`evalrx serve` / `evalrx report` /
+`evalrx publish-report`).
+
+- `evalrx dashboard` and `evalrx.analysis.dashboard` are removed: the
+  standard-library server that served the legacy static `report.html`, and the
+  `load_run` / `load_case_studies` / `load_loop_story` loaders, which nothing
+  outside their tests used. Use `evalrx serve <run dir>` for a run.
+- `evalrx explore`, `evalrx run-codebase` and `evalrx-explore`: an explore
+  output is not an M1–M5 run, and the React report cannot render one (all five
+  stages read "not run"), so these keep writing the legacy static
+  `report.html` (`evalrx.reporting.html_report`, now used only for this).
+  `--serve-report` opens that file in a browser instead of starting a server;
+  the deprecated `--dashboard` alias and `--port` are removed (`evalrx-explore`
+  had `--dashboard`, now `--serve-report`). The `run_explore` /
+  `run_codebase_cli` keywords `dashboard` / `dashboard_port` become
+  `open_report`.
+- Removed with them, as nothing else used them: `evalrx.analysis.case_studio`,
+  and the old workbench view models `evalrx.analysis.run_view` and
+  `evalrx.analysis.stage_views`.
+- The `deco_hallu` example's scripts open each phase's run with `evalrx serve`
+  (`--dashboard` → `--serve`; `run_fused.py --serve-report` writes and opens the
+  static report of its explore-shaped output). The example READMEs no longer
+  claim the report merges several log directories — it never did.
+
+### Changed — the report server's API is scoped by run (breaking)
+
+`evalrx serve` kept one global "current run": opening another run from the
+runs panel or an upload replaced it for every tab. The server now keeps no
+current run; every report request names one:
+`/api/runs/{run_id}/report`, `/stages/{stage_id}`, `/cases`,
+`/cases/{case_id}`, `/debug`, `/media/{media_id}` and `/artifact?path=…`.
+`/api/report`, `/api/stages/…`, `/api/cases…`, `/api/debug`, `/api/media/…`
+and `/api/artifact` are removed. `/api/session` now returns
+`{"run": {"id", "label", "root"} | null, "accepts_upload"}` (the run the
+server was started on), and `POST /api/runs/{run_id}/open` and
+`POST /api/upload` return the run's `id`, `label` and `root` beside its
+report, as does `GET /api/runs/{run_id}/report`. Run ids are issued only by
+the server itself (its start-up run, a runs listing, an upload), as before.
+`ReportSession` is replaced by `RunRegistry`. The UI keeps the id of the run
+on screen in the URL hash (`?run=<id>`), so a reload or a shared link reopens
+that run; an id the server does not know (after a restart) falls back to its
+start-up run.
+
+### Changed — the report UI keeps its place in the URL
+
+The view on screen is now the URL hash (`#/evidence/m3`, `#/evidence/m3/full`,
+`#/cases/<id>?split=…`, `#/debug`), so a refresh keeps the reader on the same
+stage or case, a copied link opens it, and Back returns to the previous view.
+This works the same in a portable export opened from disk. Moving between
+stages, cases or partitions within a view replaces the history entry rather
+than adding one.
+
+### Added — `ReportData.stage_detail` has a contract type
+
+The per-stage record the report UI's full-record views read is now declared
+in `evalrx.contract.stage_detail` (`StageDetail` and one model per stage),
+exported as `docs/contract/report.StageDetail.schema.json` and generated into
+`contract.d.ts`, so the UI reads it through typed views instead of `any`.
+The report compiler builds it as before, now one function per stage, and its
+output is unchanged; a test validates real and synthetic reports against the
+models, so a key the compiler adds without declaring it fails there.
+
+### Fixed — the text-LLM benchmark tasks work from an installed package
+
+`evalrx.benchmark.tasks.llm` used to load its slice definitions at run time
+from `examples/dataset_selection` (by file path, or
+`EVALRX_DATASET_SELECTION_DIR`), so the nine LLM tasks failed with
+`FileNotFoundError` outside a repo checkout. The definitions now ship in the
+package, as `evalrx.benchmark.tasks.llm`:
+
+- `specs` — `Spec`, the nine slice specs (`SPECS`, keyed by task name), and
+  the adapters/graders they use (`plain_adapter`, `mc_adapter`,
+  `bbh_adapter`, `cruxeval_adapter`, `grade_latex`, ...).
+- `catalog` — `Entry`, `CATALOG`, `BY_NAME`, `get`, `acquisition`. `get`
+  raises `KeyError` for an unknown name (it used to raise `SystemExit`).
+- `fetch` — `fetch_rows`, `fetch_rows_hub`, `parse_where`. `requests` and HF
+  `datasets` are imported on use, so importing the task registry needs
+  neither.
+
+`llm.catalog()`, `llm.dataset_selection_dir()` and
+`EVALRX_DATASET_SELECTION_DIR` are removed. Manifests, grades and protocols
+are unchanged. In `examples/dataset_selection`, `band_locate.py` keeps the
+survey's other candidates and reuses the library specs; `datasets.py` is now
+only the catalog CLI plus the 2B probe records; the benchmark image no longer
+copies `examples/dataset_selection`.
+
+### Changed — `run_fix` orchestration moved to `evalrx.eval_agent.loop_fix`
+
+The fix-stage helpers behind `VLDiagnoseLoop.run_fix` / `run_m5` (target
+selection, tier ladder, EXPLORE→CONFIRM) moved from `loop.py` to
+`loop_fix.py`, unchanged. `VLDiagnoseLoop`'s methods are unchanged. Their
+log lines now come from the `evalrx.eval_agent.loop_fix` logger.
+
+### Changed — the run reader moved to `evalrx.reporting.run_data`
+
+`extract_run_data` (and the run-layout helpers and analyzer glossary it uses)
+moved from `evalrx.reporting.html_report` to `evalrx.reporting.run_data`. It
+is the reader behind the dynamic report, the Langfuse bundle export and the
+legacy static page, and no longer lives inside the legacy renderer.
+`html_report` keeps the static page (`build_html_report`,
+`generate_html_report`, `embed_figures`, `embed_media`). Import
+`extract_run_data` from `evalrx.reporting.run_data`.
+
+### Changed — FixAgent split into the `stages/fix/` package
+
+`stages/fix_agent.py` (3,952 lines, one 3,200-line class) is now an
+orchestrator over modules that each own one job and its state, and all fix
+modules live in one subpackage, `evalrx.eval_agent.stages.fix`:
+`agent` (`FixAgent`), `types` (data model), `proposal` (`FixProposer`;
+conditional and floor/pre-registered built-ins as the
+`CONDITIONAL_DEFAULTS` / `BUILTIN_SPECS` tables), `codegen`
+(`PipelineCoder`), `validation` (`CandidateValidator`, `Baseline`;
+execution dispatched by candidate kind), `verdict` (verdicts, e-BH,
+recommendations), `recorder` (run-log writes), `descriptions`, plus the
+existing `tiers`, `tools`, `pipeline`, `internals` and `repair_catalog`.
+
+Module paths changed: `evalrx.eval_agent.stages.fix_<name>` is now
+`evalrx.eval_agent.stages.fix.<name>`, and `stages.repair_catalog` is
+`stages.fix.repair_catalog`. No aliases are kept at the old paths;
+`from evalrx.eval_agent import FixAgent, FixTier, ...` is unchanged.
+
+The `FixAgent` constructor, `propose_and_validate`, `validate_candidate` and
+its public attributes are unchanged; `FixAgent.alpha`,
+`FixAgent.recommend()` and `FixAgent.no_fix_recommendation()` are now public,
+and the loop no longer reaches into `FixAgent` private methods. A custom fix
+agent passed to the loop now needs an `alpha` attribute, and its
+`recommend`/`no_fix_recommendation` (no leading underscore) are what the loop
+calls; `_ebh_survivors`/`_refine_signal` overrides are no longer used. Code
+that imported private helpers from `fix_agent` must import them from their
+new modules.
+
+### Changed — the fix loop no longer writes to the fix agent
+
+`VLDiagnoseLoop.run_fix` used to reconfigure its `FixAgent` between tier
+rungs by assigning `max_tier`, `min_tier`, `run_logger` and
+`max_validation_cases`, restoring them inconsistently afterwards. It now
+passes everything per call: `propose_and_validate(..., max_tier=, min_tier=,
+record=False)`, and `recommend()` / `no_fix_recommendation()` take the run's
+ceiling as `max_tier=`. `validate_candidate` always scores the whole batch
+(`max_validation_cases` is a search budget; every in-repo caller already
+confirmed uncapped). Proposals, validations, run-log events and outcomes are
+unchanged. Visible differences:
+
+- After `run_fix(max_tier=...)` or an `auto_escalate` run, the agent keeps
+  its configured `max_tier`; it used to keep the requested or last-tried
+  tier, which leaked into the next `run_fix` call.
+- A custom fix agent whose `propose_and_validate` cannot take
+  `max_tier`/`min_tier` now raises `TypeError` under `auto_escalate` or an
+  explicit `max_tier`, instead of running every rung at its own tier.
+- A custom fix agent whose `propose_and_validate` does not take `record`
+  is no longer silenced during the search (the loop used to set its
+  `run_logger` to `None`); it logs whatever it logs per round.
+- Under `auto_escalate` without a held-out split, a custom agent's opaque
+  (non-`FixOutcome`) result is returned as-is, as the held-out ladder
+  already did, instead of raising `AttributeError`.
+
+### Fixed — RunLoggerV2 outbox on multi-user hosts
+
+The Langfuse outbox lived in a single shared `/tmp/evalrx-v2-outbox`, owned by
+whichever user created it first; every other user's run then failed with
+`sqlite3.OperationalError: unable to open database file`. The directory is
+now per user.
+
 ### Fixed — hybrid analyzer compatibility with Transformers 5
 
 Recognize Qwen3.5 / Qwen3-Next linear-attention mixers when Transformers calls

CONTRIBUTING.md

@@ -5,17 +5,46 @@
 ```bash
 git clone https://github.com/evalvitals/evalrx.git
 cd evalrx
-pip install -e ".[dev]"
 pip install torch --index-url https://download.pytorch.org/whl/cpu  # CPU torch for tests
+pip install -e ".[dev,local,api,data,finetune,viz,ui,contract]"
 ```
 
 ## Running tests
 
 ```bash
-pytest -m "not gpu"   # fast unit tests, no GPU needed (~165 tests)
+pytest -m "not gpu and not network"  # same selection as CI
 pytest --run-gpu      # GPU integration tests (requires CUDA + cached weights)
 ```
 
+The report tests need the built frontend. Use Node.js 22 and install from its
+lockfile before building, including after a branch change updates dependencies:
+
+```bash
+cd evalrx/reporting/web
+npm ci
+npm run typecheck
+npm run format:check  # `npm run format` rewrites
+npm run lint
+npm test               # unit tests (vitest)
+npm run build
+npx playwright install chromium   # once
+npm run test:e2e       # smoke test of every view on a real run's payload
+```
+
+`fixtures/README.md` says how the smoke test's payload was made.
+
+From the repository root, validate the generated contracts and documentation:
+
+```bash
+python -m evalrx.contract.export --out docs/contract --check
+pip install mkdocs-material
+mkdocs build --strict
+```
+
+Keep run data, local notebooks, and standalone report exports out of commits.
+The [local artifact policy](docs/local_artifacts.md) lists retained paths and
+provides an inventory command. Generated files are preserved locally.
+
 ## Linting
 
 ```bash
@@ -48,6 +77,8 @@ The new analyzer is instantly discoverable via `registry.analyzers.names_compati
 ## Pull requests
 
 - Keep PRs focused: one analyzer, one model, or one infrastructure change per PR.
-- All CI checks (ruff, mypy, pytest fast suite) must pass.
+- All configured CI checks must pass: Ruff, the frontend format/lint/tests/build, contract export,
+  documentation build, and the non-GPU/non-network test suite. Mypy is an
+  additional local check, not currently a CI gate.
 - Add or update the relevant entry in `CHANGELOG.md` under `[Unreleased]`.
 - Include paper citations (arXiv link + BibTeX key) for any method from the literature.

README.md

@@ -50,7 +50,7 @@ supply the question and the ceiling — everything between is unattended.
 </picture>
 
 The held-out split is taken *before* Explore runs, so Verify always scores on
-rows the analysis never touched. [Full-loop quickstart →](docs/quickstart.md#vldiagnoseloop--automated-failure-attribution-current) · [Intervention guide →](docs/intervention.md)
+rows the analysis never touched. [Full-loop quickstart →](docs/quickstart.md#diagnosis-loop) · [Intervention guide →](docs/intervention.md)
 
 ## Quickstart
 
@@ -203,7 +203,7 @@ raising it rather than climbing on its own.
 | **L2** | Scaffolds around an unchanged model — multi-call, tools, aggregation | ✅ |
 | **L3a** | Read internals — attention-guided cropping, contrastive decoding | ✅ |
 | **L3b** | Write internals — attention reweighting, activation steering | ✅ |
-| **L4** | **Parameter space — build a dataset, fine-tune, re-test** | ✅ LoRA on the LLM ([`fix_internals.py`](evalrx/eval_agent/stages/fix_internals.py)); other recipe shapes recorded, not yet executed |
+| **L4** | **Parameter space — build a dataset, fine-tune, re-test** | ✅ LoRA on the LLM ([`fix_internals.py`](evalrx/eval_agent/stages/fix/internals.py)); other recipe shapes recorded, not yet executed |
 
 **L3b and L4 only exist for open weights** — you cannot modify a forward pass
 or fine-tune through somebody's API.