Validate numeric evidence, report completeness and metric bindings

docs/numeric-evidence-completion.md

@@ -0,0 +1,96 @@
+# Numeric evidence at structured completion
+
+The first native reliability campaign found a reserved-case evidence regression:
+the candidate reported the correct profit in major currency units, but both cited SQL
+queries returned only minor units. The conversion appeared in the prose, not executable
+output. The independent evidence evaluator correctly rejected the answer; its score and
+the original candidate remain unchanged.
+
+Previously, runtime finalization re-executed every evidence item and rejected execution
+errors, but it did not check the reported number against the output. The revised numeric
+completion check collects native query cells and successful Python stdout, and requires
+the scalar answer to appear in that output. It accepts scientific notation, grouped
+numeric text and the existing display-rounding allowance. Booleans and nonfinite values
+do not establish a numeric answer. A number in a note or unprinted program source is not
+output. Every cited program still runs and any execution failure rejects finalization.
+
+The existing Pydantic/ToolStrategy retry returns a specific error to the analyst, asking
+it to put conversion or rounding into the executable evidence and output the final
+number. The host does not guess a currency conversion or rewrite the answer. The check
+uses no question-specific expected value, oracle, grading reference or hidden data.
+Independent correctness, report, refusal and evidence scoring remain unchanged. Numeric
+presence is not proof of correctness or full data lineage.
+
+The same reserved batch exposed an answered report with unavailable required metrics:
+its current-period mean and change were null even though the method explained that the
+comparison could not be calculated. Runtime report finalization now rejects null metric
+values and asks the analyst to decline with the missing-information reason. It does not
+infer answerability from hidden references or fabricate a complete report. Historical
+records remain parseable by the permissive persisted contract.
+
+The initial implementation's offline verification included a real installed
+graph that rejects an omitted conversion and then accepts corrected SQL. Additional
+tests cover SQL and Python mismatches, printed versus unprinted values, display rounding
+and notation, plus partial-report rejection followed by a proper refusal. Ruff, Vulture,
+diff and Gitleaks checks passed. This is implementation
+evidence, not live qualification of the revised candidate.
+
+The consumed acceptance-v1 reserved worlds cannot qualify a revision as unseen. The
+evaluator builder therefore supports an explicit `reserved_generation=2` for the
+acceptance split only. Generation 1 remains the default and reproduces its original
+identities/data. Generation 2 uses distinct world IDs, periods, financial values,
+report distributions and price/delivery inputs. Thirteen answerable references are
+independently checked by SQL; four cases require refusal. Its 17 worlds are controlled
+variants, not new customer domains. Old manifests and database files remain intact.
+
+The initial prepared follow-up used 63 new responses: 12 candidate original regressions, 17
+candidate development responses, and 34 fresh reserved baseline/candidate responses.
+The 29 original/development baseline records retain their actual unchanged source and
+data/evaluator identities; they are reused rather than relabeled or regenerated.
+Models, scoring rules and tolerances remained fixed. Its initial proposed additional
+allocation was $2.15, subject to the cumulative allowance. Later staged execution used
+the existing allowance as recorded below; no additional spending is approved here.
+
+Local freeze: `data/output/reliability-native-refresh-20261114-r2/proposal.json` in the
+canonical operator checkout. It retains source/manifest/policy/data/reference hashes and
+the proposed caps. It superseded the first zero-call preparation, which preceded the
+second observed failure; both remain retained. The following checkpoint supersedes
+the preparation-only state. No v0.0.1 release is claimed.
+
+## Generation-2 result and report-binding revision
+
+The d0d476e04fc7f378 candidate passed 12/12 original and 17/17 development responses,
+then 16/17 fresh reserved candidate responses. The generation-2 baseline arm did not
+run. All 46 new records are valid; one mandatory reserved result failed evidence
+reproduction even though its reported facts were correct. The status label was in the
+SQL unit column while the explicitly named metric's value was numeric. The original
+data, score and rejected result remain unchanged.
+
+Runtime finalization now also checks explicit report metric/value associations in SQL
+and structured Python JSON/JSON-lines output. A label in a unit column or a matching
+value under another metric does not repair a contradictory binding. A second query
+with the correct value cannot erase a contradictory cited result. Single wide rows
+are recognized; several wide population rows are not arbitrarily mapped to one global
+metric. Unstructured Python output still has independent evidence grading. No runtime
+code imports oracle answers or the benchmark scorer.
+
+The existing finalization feedback asks the analyst to repair its executable output.
+Real installed-graph regression checks exercise this repair with unrelated local data.
+The current code passes 277 tests with 15 optional skips, plus Ruff/Vulture/secret checks.
+Code review also preserved SQLite's 0/1 representation for Boolean report values.
+The scalar-output and unavailable-metric guards remain in place. The frozen independent
+evaluator and tolerances are unchanged; output consistency is not full data lineage or
+an independent correctness proof.
+
+Generation 2 is consumed regression evidence. Explicit generation 3 supplies 17 distinct
+reserved worlds for the next frozen candidate, preserving both earlier generations and
+all old manifests. Thirteen numerical references are independently checked by SQL;
+four scenarios require refusal. No generation-3 model inference has run.
+
+The completed revised inference used $1.413655 in conservative estimates. Cumulative
+goal usage and retained reservations is $4.92681681225 of the existing $5 authorization.
+The next full qualification requires 63 new responses and can reuse 29 unchanged
+baseline records. A proposed shared $2.05 cap would keep the total below $7, subject
+to owner approval; it supersedes the earlier $6 request because another candidate
+revision and fresh reserved generation are now required. No new allowance is implied
+by this document, and no release acceptance is claimed.

src/backintel/answer_validation.py

@@ -9,6 +9,7 @@
 
 import json
 import math
+import re
 from collections.abc import Sequence
 
 from langchain.agents.middleware import AgentMiddleware, ModelRequest
@@ -21,6 +22,7 @@
 from backintel.stores.multi import DIRECTIVE, MultiStore
 
 DOMAIN_READS = frozenset({"list_tables", "describe_table", "run_sql", "run_query", "run_python"})
+NUMERIC_OUTPUT = re.compile(r"(?<![\w.])[+-]?(?:\d+(?:,\d{3})*(?:\.\d*)?|\.\d+)(?:[eE][+-]?\d+)?(?![\w.])")
 FINISH_REPEATED_READ = (
     "The last three reads used identical arguments and returned the same successful result. "
     "Further identical reads add no information. Finalize using the structured answer tool now: "
@@ -111,20 +113,92 @@ def validate_answer_shape(answer: AnalystAnswer) -> None:
             metric = row.get("metric")
             if not isinstance(metric, str) or not metric.strip() or "value" not in row or "unit" not in row:
                 raise ValueError("Each report row requires a nonempty metric identifier, value, and unit.")
+            if row["value"] is None:
+                raise ValueError(
+                    "An answered report cannot contain an unavailable metric value. If a required "
+                    "metric cannot be calculated, return status='cannot_answer' and explain the "
+                    "missing information instead of returning a partial answered report."
+                )
             names.append(" ".join(metric.split()).casefold())
         if not names or len(names) != len(set(names)):
             raise ValueError("Report metric identifiers must be unique: return exactly one row per metric.")
     if not answer.evidence:
         raise ValueError("An answered response must cite the exact executable evidence.")
 
 
+def _number_matches(target: float, cell) -> bool:
+    if cell is None or isinstance(cell, bool):
+        return False
+    try:
+        value = float(cell.replace(",", "") if isinstance(cell, str) else cell)
+    except (ValueError, TypeError, OverflowError):
+        return False
+    return math.isfinite(value) and math.isclose(value, target, rel_tol=1e-4, abs_tol=0.011)
+
+
+def _report_value_matches(value, cell) -> bool:
+    if isinstance(value, str):
+        if not isinstance(cell, str):
+            return False
+        target, observed = (" ".join(v.split()).casefold() for v in (value, cell))
+        return observed == target or target in observed or (len(observed) >= 3 and observed in target)
+    if isinstance(value, bool):
+        # SQLite represents Boolean query results as integer 0/1.
+        return isinstance(cell, (bool, int, float)) and value == cell
+    return value is not None and _number_matches(float(value), cell)
+
+
+def _collect_report_bindings(bindings: dict[str, list], columns, rows) -> None:
+    names = [" ".join(name.split()).casefold() for name in columns]
+    tall = "metric" in names and "value" in names
+    # A wide table with several rows can describe several populations; its column
+    # names alone do not bind any one value to the global report metric.
+    if not tall and len(rows) != 1:
+        return
+    for row in rows:
+        pairs = (
+            [(row[names.index("metric")], row[names.index("value")])]
+            if tall
+            else zip(names, row, strict=True)
+        )
+        for key, cell in pairs:
+            if isinstance(key, str) and key.strip():
+                bindings.setdefault(" ".join(key.split()).casefold(), []).append(cell)
+
+
+def _collect_python_report_bindings(bindings: dict[str, list], text: str) -> None:
+    documents = []
+    try:
+        documents.append(json.loads(text))
+    except (ValueError, RecursionError):
+        for line in text.splitlines():
+            try:
+                documents.append(json.loads(line))
+            except (ValueError, RecursionError):
+                continue
+    rows = [
+        row
+        for document in documents
+        for row in (document if isinstance(document, list) else [document])
+        if isinstance(row, dict)
+    ]
+    for row in rows:
+        if len(rows) > 1 and not {"metric", "value"}.issubset(" ".join(k.split()).casefold() for k in row):
+            continue
+        _collect_report_bindings(bindings, list(row), [list(row.values())])
+
+
 def validate_answer_evidence(answer: AnalystAnswer, store: Store, sandbox: PythonSandbox) -> None:
     """Execute every cited item under the same read-only bindings used by analysis tools.
 
-    This is an execution-integrity check, not a correctness/lineage judge. Independent
-    evaluators still check values, missing/extra requested metrics, and unsupported claims.
+    Numeric answers must also appear in executed output, allowing display rounding.
+    Explicit SQL/JSON report bindings may not contradict the reported metric values.
+    This does not establish correctness or lineage: independent evaluators still check
+    reference values, missing/extra requested metrics, and unsupported claims.
     """
     problems = []
+    numeric_output = []
+    report_bindings: dict[str, list] = {}
     for index, item in enumerate(answer.evidence, 1):
         try:
             if item.kind == "python":
@@ -136,6 +210,10 @@ def validate_answer_evidence(answer: AnalystAnswer, store: Store, sandbox: Pytho
                 )
                 if result.exit_code != 0:
                     problems.append(f"Evidence {index}: program failed: {result.stderr[-300:]}")
+                else:
+                    numeric_output.extend(NUMERIC_OUTPUT.findall(result.stdout))
+                    if answer.answer_type == "report":
+                        _collect_python_report_bindings(report_bindings, result.stdout)
             else:
                 code = item.code
                 if isinstance(store, MultiStore):
@@ -149,18 +227,39 @@ def validate_answer_evidence(answer: AnalystAnswer, store: Store, sandbox: Pytho
                         code = f"-- store: {item.store}\n{code}"
                 elif item.store is not None and item.store != store.id:
                     raise QueryRefused("Evidence names a store that is not connected.")
-                store.query(code)
+                result = store.query(code)
+                numeric_output.extend(cell for row in result.rows for cell in row)
+                if answer.answer_type == "report":
+                    _collect_report_bindings(report_bindings, result.columns, result.rows)
         except (QueryRefused, NotImplementedError) as error:
             problems.append(f"Evidence {index}: query failed: {str(error)[:300]}")
     if problems:
         raise ValueError("; ".join(problems))
+    if answer.status == "answered" and answer.answer_type == "report":
+        for row in answer.rows:
+            observed = report_bindings.get(" ".join(row["metric"].split()).casefold(), [])
+            if observed and not all(_report_value_matches(row["value"], cell) for cell in observed):
+                raise ValueError(
+                    f"Executable output contradicts reported metric {row['metric']!r}. "
+                    "Return that metric's reported value in the value column, with units or "
+                    "formatting in a separate column. A label in the unit column, another "
+                    "metric, or a second query cannot repair a contradictory value binding."
+                )
+    if answer.status == "answered" and answer.answer_type == "number":
+        if any(_number_matches(answer.value, cell) for cell in numeric_output):
+            return
+        raise ValueError(
+            "Cited evidence executes but does not output the reported numeric value in its "
+            "reported units. Include any unit conversion or rounding in the executable "
+            "evidence and output the final value."
+        )
 
 
 def answer_schema(store: Store, sandbox: PythonSandbox) -> type[AnalystAnswer]:
     """Bind runtime-only validation to the native structured-output retry mechanism."""
 
     class VerifiedAnalystAnswer(AnalystAnswer):
-        """Return an answer with unique metrics and evidence that successfully re-executes."""
+        """Return unique metrics, executable evidence and numeric values present in its output."""
 
         @model_validator(mode="after")
         def verify_before_returning(self):