build(deps-dev): update pytest-asyncio requirement from >=0.21.0 to >=1.4.0

.github/workflows/ci.yml

@@ -77,6 +77,13 @@ jobs:
           key: hf-${{ runner.os }}-${{ hashFiles('pyproject.toml') }}
           restore-keys: hf-${{ runner.os }}-
 
+      - name: Metric regression gate
+        # 359 lines of metric arithmetic had no test that would notice a change. This
+        # scores a frozen fixture (no models, no network) and fails on any movement
+        # beyond one percentage point. It is a blocking gate, not advisory: an advisory
+        # metric check reads as enforced and is worse than none (audit P3-6).
+        run: python scripts/check_metric_regression.py --tolerance 0.01
+
       - name: Run pytest with coverage
         run: |
           pytest --cov=docureason --cov-report=xml --cov-report=term-missing tests/

.gitignore

@@ -1,8 +1,31 @@
+# Private LLM context & local documents (never commit)
+cv.md
+agent.md
+update.md
+interview.md
+audit.md
+Documentations/audit.md
+project_ppt.pptx
+scratch/
 plan/
 artifacts/
 notebooks/
 .ipynb_checkpoints/
 
+# Allow only the Kaggle evaluation notebook, ignore other scratch notebooks
+notebooks/*
+!notebooks/docureason_full_eval_kaggle.ipynb
+
+# Evaluation corpus build output (scripts/build_eval_corpus.py)
+# Allowlist: keep dataset definitions, ignore gigabyte-sized corpus build artifacts
+data/*
+!data/eval*/
+data/eval*/*
+!data/eval*/queries.json
+!data/eval*/filings/
+data/eval*/filings/*
+!data/eval*/filings/provenance.json
+
 
 # Byte-compiled / optimized / DLL files
 __pycache__/
@@ -223,3 +246,25 @@ __marimo__/
 
 # Streamlit
 .streamlit/secrets.toml
+
+# ---------------------------------------------------------------------------
+# Evaluation corpus build output (scripts/build_eval_corpus.py)
+# ---------------------------------------------------------------------------
+# Ingested corpora and raw filings are large (1.2 GB for eight 10-Ks) and fully
+# regenerable: provenance.json carries every filing URL plus its SHA-256, and the
+# corpus is a deterministic function of those filings and the pipeline config.
+#
+# queries.json and provenance.json are NOT ignored — together they are the dataset
+# definition, and a query set that lives only on one machine cannot be evaluated
+# against by anyone else.
+# This is an allowlist, not a denylist, on purpose: the previous rule was
+# `data/eval*/corpus/`, which matched data/eval/corpus/ but not the second corpus at
+# data/eval/corpus_no_layout/, so 3.3 GB of build output was committed and GitHub
+# rejected the push. Naming what may be committed cannot miss a new directory name.
+data/*
+!data/eval*/
+data/eval*/*
+!data/eval*/queries.json
+!data/eval*/filings/
+data/eval*/filings/*
+!data/eval*/filings/provenance.json

CHANGELOG.md

@@ -4,7 +4,54 @@ All notable changes to the **DocuReason** framework (`docureason-framework`) wil
 
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
-## [Unreleased]
+## [1.2.0] - 2026-09-02
+
+### Measurability Release — audit remediation (Sprint 2)
+
+Continues [`Documentations/audit.md`](Documentations/audit.md) §13, Sprint 2. Sprint 1 made the architecture real; this sprint makes it **measurable** — a real corpus, a baseline to compare against, executed ablations, and intervals around every number.
+
+#### Added — evaluation infrastructure
+- **A real evaluation corpus from SEC EDGAR** (`docureason/tripath/evaluation/edgar.py`, `scripts/build_eval_corpus.py`). Everything scoreable before came from eight fixture files totalling ~3 KB with invented answers (`benchmark_dataset.py` asserted `"answer": "2.4M"` against a table containing 100M and 125M), so no retrieval number was falsifiable. The fetcher pulls 10-K primary documents with the SEC-required `User-Agent` (`DOCUREASON_SEC_USER_AGENT`, no default — requests are refused rather than sent under a fabricated identity), rate-limits itself well under the 10 req/s fair-access limit, caches by accession number, and writes `provenance.json` with each filing's URL, accession, filing date and SHA-256.
+- **Stratified gold queries whose labels are derivable** (`docureason/tripath/evaluation/gold_queries.py`). Table-cell queries know their gold region by construction (row label × column header); prose and figure queries are anchored on a phrase **verified unique across the whole corpus**, and a candidate whose anchor is not unique is discarded rather than emitted — scoring a correct region as a miss would measure the label format, not retrieval. Rejections are counted by reason, and every record carries `provenance: "programmatic"`: these are derived labels, not human annotation, which over-represents lookup and cannot express multi-hop reasoning.
+- **The naive-RAG baseline** (`docureason/tripath/evaluation/baselines.py`). The docs quoted improvements over "naive RAG" with no naive RAG in the repository, so nothing achieved could be attributed to the tri-path architecture rather than the corpus or the generator. Recursive character splitting, one flat exact index, top-5, **the same generator and the same pinned embedding model** — only the retrieval architecture differs. Chunks record every region they overlap, so a splitter that cuts across gold units is not penalised for the label format.
+- **A real ablation runner** (`docureason/tripath/evaluation/ablation.py`). The module it replaces subtracted one pre-computed `average_recall_at_5` from another and had **zero callers**, while `Documentations/eval_method.md` §21 published seven deltas to one decimal place that nothing in the codebase could have produced. Each ablation now re-runs every query with the component disabled and reports a **paired** comparison: mean delta, bootstrap CI and a sign-flip p-value. Each row carries its published claim beside the measurement, and an ablation that could not be executed is reported `executed: false` with a reason — never as a delta of zero.
+- **Five ablation switches that did not exist.** `retrieval.enable_sparse_retrieval`, `retrieval.enable_table_sql`, `reranker.enable_cross_encoder`, `reranker.enable_heading_penalty` and `ingestion.enable_layout_parser`. Four of the seven documented ablations were not operations the system supported: there was no way to turn those components off, so "remove X and re-measure" could only ever have been asserted. Each switch is also reported through the `degradation` channel, because an ablated response still means something different from a healthy one.
+- **Bootstrap intervals and paired significance testing** (`docureason/tripath/evaluation/statistics.py`). Percentile bootstrap for a metric's mean, paired bootstrap for the difference between two systems, and an exact sign-flip permutation test (enumerated for n ≤ 14, sampled with an add-one correction above it, so a sampled p-value is never reported as 0). Paired because query-to-query variance dwarfs between-system variance. Pure Python against a seeded `random` — the CI gate needs determinism more than speed.
+- **`confidence_intervals`, `stage_latency_ms` and `per_stratum` in every suite report.** A corpus-wide mean can hold steady while tabular lookup collapses and prose improves; the strata are the only place that shows.
+- **Per-stage latency instrumentation** (`docureason/tripath/utils/timing.py`). Reported latency was one number measured *around* the call (audit P0-5), so an embedding reload, a cross-encoder cold start and a DuckDB retry were indistinguishable. `StageTimer` accumulates `route`, `dense_search`, `sparse_search`, `index_merge`, `corpus_scan`, `table_sql`, `fuse`, `chart_enrich`, `parent_expand`, `rerank`, `generate` and `attribute`, and reports a **signed residual** — total minus the sum of the stages — so a decomposition with a hole in it says so instead of quietly describing 60% of the request. A negative residual is the parallel dispatch overlapping, and is reported rather than clamped.
+- **A metric regression gate in CI** (`scripts/check_metric_regression.py`, `tests/fixtures/`). 359 lines of metric arithmetic had no test that would notice a change. A frozen fixture — 20 cases across four strata, fixed retrieval output, no models, no network — is scored and compared against a committed baseline of 99 values; anything moving more than 0.01 fails the build. Latency and throughput are excluded because they measure the runner. It is a **blocking** gate: an advisory metric check reads as enforced and is worse than none (P3-6).
+- **DuckDB tables registered at ingestion** (`docureason/tripath/indexing/table_store.py`). `TableSQLRetriever` re-created every table it touched on **every query** — one `CREATE TABLE` plus one `INSERT` per row into a throwaway `:memory:` database — and re-inferred column types each time, so the same column could be `DECIMAL` for one question and `VARCHAR` for the next. Tables are now registered once into `tables.duckdb` with `tables_index.json` beside the corpus, and queries run against a read-only connection. On the 8-filing corpus that is **1,952 tables / 17,310 rows** prepared once instead of per query. A region absent from the index falls back to the in-memory path, so a partially built store degrades rather than failing.
+- **Parallel modality dispatch** (`serving.enable_parallel_paths`). The three retrieval paths are independent and now run concurrently, with results collected in a fixed task order so ranking is unchanged. A failing path no longer takes the query down: it returns empty, is logged, and surfaces as `mode: "failed"` in that modality's stats. The dense and sparse index builders gained load locks and a metadata cache — without them three threads could each load the embedding model, and `search()` re-read the whole metadata sidecar from disk on every query.
+- **A bounded response cache** (`serving.response_cache_size`, default 64). Keyed on the query **and** a fingerprint of the effective config, so a per-query override or an ablation switch can never be served a response computed under the opposite setting. A hit is marked `cached: true` and its `latency_breakdown` describes the **cache lookup**, with the original run preserved under `latency_breakdown_first_run` — replaying the original timings would credit the pipeline with work it did not do and drag every reported percentile toward the cache's.
+- **`ServingConfig`.** `configs/serving.yaml` existed but was never loaded (`load_from_yaml` reads `config.yaml` only), so both settings above were unreachable configuration for behaviour the service did not have.
+
+#### Fixed
+- **Per-query overrides mutated the shared service.** `apply_query_overrides` mutated `self` and returned it, so one request carrying `{"final_top_k": 1}` permanently narrowed the `QueryService` every later request used, and an ablation run left its own switches behind for the next one. It now returns a deep copy. Overrides are also driven by a single table shared with the REST layer and the ablation definitions, and an unknown key is recorded in `source_map["unknown_overrides"]` instead of silently doing nothing.
+- **SEC filings were parsed without a layout parser.** `_build_regions` routed only `.pdf/.docx/.pptx/.xlsx` to Docling, so an HTML filing reached the rule-based segmenter, which saw raw markup and produced **no table regions at all**. `.html`/`.htm` are now layout-parsed: one Apple 10-K yields 62 tables and 1,107 body regions where it previously yielded neither.
+- **Figures were captioned from the wrong image, or from a blank one** (P1-3, P1-4). The captioner returned `page.images[0]` for *every* figure on a page, so three charts on one page were all described from whichever image came first; when a page had no embedded image it built a **blank white canvas**, cropped it and captioned that, and the result was indexed as a figure. A new `PageRasterizer` (`pypdfium2`) renders the page and crops each figure at its own bbox, and a crop with no visual variation is refused and counted as `blank_crop_skipped`.
+- **OCR read only the first embedded image per page** (P2-13), so a scan split across two images OCR'd to half its text — and a page with no embedded image had a blank canvas handed to EasyOCR, costing seconds per page to read nothing. Pages are now rendered and blank renders are skipped before OCR runs.
+- **`image_rendering_dpi` had no consumer.** It was in the config, in `config.yaml` and in the startup banner, and reached no code (§6). It is now the rasterization DPI for both figure crops and OCR.
+- **The scanned-page detector was off by one page** (P1-2). Character counts were accumulated under Docling's 1-based `page_no` and looked up under a 0-based page loop index, so page *N*'s text was compared against page *N+1*'s content. Conversion now happens in one place (`page_index_from_region`), and OCR regions are emitted 1-based to match Docling.
+- **A pinned model revision turned a network hiccup into an empty index.** Pinning makes the loader resolve that commit against the hub; when the hub was unreachable but the commit was already cached, the load failed outright and the dense index silently degraded. Both loaders now retry with `local_files_only=True` before giving up.
+- **Windows console logging lost every message containing an arrow or em dash.** `UnicodeEncodeError` inside `logging` printed a traceback instead of the line, so ingestion logs were unreadable exactly where they mattered. The stream handler is now reconfigured to UTF-8.
+- **`bootstrap_ci` iteration count is a parameter.** The ablation runner computes intervals for eight systems and every stratum; at the default 10,000 resamples that dominated the sweep.
+
+#### Measured, on 8 real 10-K filings
+| | |
+| :--- | ---: |
+| filings (Apple, Microsoft, JPMorgan, J&J, Exxon, Walmart, Caterpillar, Pfizer) | 8 |
+| regions | 20,966 |
+| chunks | 24,919 |
+| DuckDB tables registered / rows | 1,952 / 17,310 |
+| gold queries (tabular_lookup / numeric / prose) | 150 (50/50/50) |
+| gold-id and anchor-uniqueness validation | PASS |
+| hermetic unit tests | 202 (was 95) |
+
+#### Known gaps after this sprint
+- **The `figure` stratum is empty**: Docling's HTML pipeline emits no figure regions for these filings, so the corpus cannot support figure queries. Recorded in the generation report as `figure:no_regions_in_corpus` rather than as a silent zero.
+- **The corpus is 8 filings, not the 20–50 the roadmap calls for.** Ingestion is ~2.4 min/filing on CPU and gold ids only exist after it; `scripts/build_eval_corpus.py --filings 24` scales without code changes.
+- **`no_layout_parser` cannot be measured from a single corpus.** It is ingestion-scoped, so it needs a second corpus built with `enable_layout_parser: false`; the runner reports it as not executed until one is supplied.
+- Gold labels are **programmatically derived, not annotated**. Real NLI entailment (P0-2), authentication (S1) and the remaining P1 findings are unchanged.
 
 ### Architecture Release — audit remediation (Sprint 1)
 

Documentations/eval_method.md

@@ -595,6 +595,25 @@ Quantifies the exact contribution of each architectural block by measuring perfo
 
 $$\Delta \text{Metric} = \text{Metric}_{\text{Full\_Pipeline}} - \text{Metric}_{\text{Ablated\_Pipeline}}$$
 
+> **Status of the numbers in this table.** The deltas below were published before any
+> runner existed that could produce them, and four of the seven components had no switch
+> to turn them off, so "remove X and re-measure" was not an operation this system
+> supported. Treat every figure in the last two columns as an **unverified expectation**.
+>
+> `docureason/tripath/evaluation/ablation.py` now executes the matrix for real:
+>
+> ```bash
+> python scripts/run_ablations.py --queries data/eval/queries.json --limit 20 --stratified
+> ```
+>
+> It re-runs every query with the component disabled, reports a paired delta with a
+> bootstrap confidence interval and a sign-flip p-value, and prints each measured delta
+> **beside the claim below** with a verdict (`consistent_in_magnitude`, `same_direction_different_magnitude`,
+> `contradicted`, `not_significant`, `not_measurable`). `no_layout_parser` is
+> ingestion-scoped and is reported as *not executed* until a corpus ingested with
+> `ingestion.enable_layout_parser: false` is supplied. Where the measurement and this
+> table disagree, the measurement is the result.
+
 #### Master Ablation Test Matrix Table
 
 | Ablated Component | Primary Metric Impact | Expected Performance Drop ($\Delta$) | Diagnostic Conclusion |
@@ -666,6 +685,7 @@ All evaluation modules are located in `docureason/tripath/evaluation/` and `docu
 # Execute automated evaluation harness
 python scripts/evaluate_system.py
 
-# Run ablation studies
-python -c "from docureason.tripath.evaluation.ablation import AblationStudy; AblationStudy().run()"
+# Run ablation studies (the old AblationStudy().run() had no callers and computed
+# nothing; it now raises and points here)
+python scripts/run_ablations.py --queries data/eval/queries.json --limit 20 --stratified
 ```