feat: add bounded diagnostic evaluation and daily admission

docs/skill-eval-comparison.md

@@ -259,3 +259,47 @@ python -m tests.fixtures.skill_eval_compare_demo --mode update --output <new-off
 The fixture creates real temporary Git snapshots and produces one fix, one
 regression, one inconclusive pair and one missing pair. It never constructs the
 real client. Its fixed answers are pipeline controls, not model measurements.
+
+### Synthetic KV admission diagnostic adapter
+
+The explicit comparison configuration `adapter: "kv-admission-diagnostic-v1"`
+selects a different typed input/answer contract, prompt and trusted verifier.
+Omitting `adapter` preserves the performance-report audit path. This adapter
+requires `mode: "update"`: freeze the PR's exact base and head as the baseline
+and candidate, and export the same selected skill with its dependencies in both
+arms. A no-skill baseline measures the whole bundle, not the PR's incremental
+contribution, and is rejected for this adapter. Custom schema/grader overrides
+are rejected. Retained answers are regraded with the trusted diagnostic verifier.
+
+Inputs contain synthetic observations at one decision boundary: cache lookup,
+receive state, KV identity match, and allocator result, each with a stable event
+ID. Missing observations remain absent, not guessed. Output contains a diagnosis,
+event citations, one read-only next check, and an explicit reuse authorization
+boolean. Citations must match the decisive observation set exactly (all available
+receive/identity/allocator observations for an insufficient-evidence answer,
+falling back to lookup only when no other event is available), so citing everything cannot
+pass a specific diagnosis. The finite-state oracle distinguishes readiness, capacity, invalid
+reuse and insufficient evidence. Failure/mismatch prohibits reuse; pending
+receive is not capacity proof; capacity needs completed matching KV and allocator
+denial. None of these blocked-request cases authorizes executing reuse.
+
+`tests/fixtures/skill_eval_diagnostic_cases.json` contains development and holdout
+counterfactual candidates, without expected answers. Expected responses and the
+oracle stay outside the model-visible workspace. Freeze the chosen cases before
+candidate trials, and use paired observations with unchanged model/settings.
+The fixture list is not a paid execution plan; choose a feasible frozen subset
+only after accounting for full bundle and request budget, without truncating the
+knowledge. Preserve raw responses, per-assertion results, and failures. Passing
+these synthetic cases establishes only small-case diagnostic utility, not skill
+coverage, engine correctness, production throughput or service efficacy.
+
+The reuse check is intentionally one-sided: every case is a blocked request.
+Always refusing reuse alone therefore earns no diagnostic success and does not
+establish balanced safe/unsafe execution discrimination.
+
+Selected snapshot closure follows existing KnowledgeReader semantics: the complete
+selected skill directory plus declared `metadata.references`. It does not recursively
+follow arbitrary cross-skill Markdown navigation links. The retained manifest records
+original Git paths, modes and blobs, including original index provenance and the
+projected index hash. Unsafe entries inside the selected closure are rejected rather
+than omitted; the existing whole-library snapshot guard is unchanged.

docs/skill-eval-daily-admission.md

@@ -0,0 +1,106 @@
+# Global daily admission rollout (offline implementation)
+
+The controller requires **both** the permanent USD 200 lifetime PR ledger and
+one repository-wide USD 1,000 / Asia/Shanghai-day ledger. All PRs compete against
+one immutable repository identity and one global ref, not separate per-day refs.
+This code change does not activate paid execution or initialize any ledger.
+
+## Public configuration migration
+
+Schema 1 remains readable for historical evidence and free preparation. It is
+ineligible for paid execution. Schema 2 preserves every existing field and adds
+exactly this public, non-secret field (replace placeholders only after audit):
+
+```json
+{
+  "daily_budget": {
+    "currency": "USD",
+    "amount": "1000",
+    "timezone": "Asia/Shanghai",
+    "anchor": {
+      "ref": "refs/heads/skill-eval-budget/<repository_id>/daily",
+      "genesis_commit": "<audited_genesis_commit>",
+      "checkpoint_commit": "<independent_checkpoint_commit>"
+    }
+  }
+}
+```
+
+The checked-in schema-1 configuration is deliberately unchanged. Missing daily
+configuration/genesis, unknown obligations, corrupt history, expired rates or
+unavailable live identity block paid execution, never imply zero spend. Free
+planning and retained evidence do not require a new paid grant.
+
+An authorized rollout must first quiesce all older paid writers and reconcile
+**all** historical charges, existing PR reservations and uncertain/in-flight
+work, including historical PR 159 and 160 run records. Preserve their existing
+PR anchors, history and evidence byte-for-byte; do not reset/rebase their budgets.
+Import conservative historical daily intervals and outstanding upper bounds into
+an explicitly audited global genesis. Missing provider bills are not proof of
+zero cost. Unknown upper exposure prevents activation. Pin the audited genesis
+and independent checkpoint in reviewed configuration; no initializer, zero
+bootstrap, ref write or config activation is supplied here.
+
+## Durable admission and failure boundaries
+
+The order is: verify both histories; commit a **new** daily reservation; commit a
+**new** PR reservation; obtain a one-shot daily claim on the admission day; obtain
+a one-shot PR claim; recheck all live authority; launch once in this process.
+Any duplicate, conflict, lost response or crash stops that invocation. Partial
+admission burns capacity. Observing an earlier committed reservation/claim never
+recovers a launch grant. A genuinely new attempt needs a new full frozen binding
+and new reservations; a workflow rerun or label toggle cannot reset old ledgers.
+
+Before launch and every provider request, the existing ephemeral broker invokes
+the controller to re-read current policy/label/configuration and both exact
+claimed bindings, and verify the global ledger is not quarantined. The trusted
+wall clock is sampled for admission, again for daily claim, and after live reads
+for each grant. A changed Shanghai day denies all subsequent requests; no
+cross-midnight continuation or automatic day rollover exists. A request already
+in flight cannot be recalled. Its full unresolved bound remains counted on the
+next day, so midnight does not free its capacity.
+
+The existing trusted provider client enforces the frozen request count, payload
+and output bounds and requires a fresh broker grant immediately before every
+HTTP request; transport retries do not get a replay grant. The reservation covers
+all calls using the full verified model-context ceiling, highest applicable
+input/cache rate, maximum output and request fees, rounded upward. If those
+upper-bound assumptions cannot be established, do not run.
+
+## Conservative terminal policy and quarantine
+
+The controller performs **no automatic terminal settlement or refunds**, even
+when its worker returns successfully. Worker failure, cancellation, crashes and
+missing/partial actual usage retain the entire PR and global bound. Unresolved
+global reservations remain charged on every later day; exhaustion stops future
+work instead of resetting at midnight. This deliberately trades availability for
+safety in the initial bounded pilot.
+
+A future settlement path must prove provider work and every retry are terminal,
+sample a fresh trusted closing day **after** that proof, and retain unknown-cost
+bounds. Never use a cached admission day as a settlement day. Do not interpret a
+successful subprocess return or artifact's claimed usage as that proof.
+
+Evidence of an exceeded upper bound or unknowable exposure requires persistent
+global quarantine before further activity; ordinary unavailable actual billing
+under a still-valid conservative bound is different. Quarantine is monotonic:
+no automatic clearing/reset. The controller checks the persisted quarantine, but
+does not itself ingest provider billing or invent evidence of terminal usage.
+Activation therefore still requires the reviewed pricing/bounds, audited
+history, compatible storage permissions and a trusted operational path to
+record any externally discovered over-bound exposure. Git administrator rollback
+limitations remain as documented in [the accounting core](skill-eval-budget-core.md).
+
+## Offline validation
+
+```sh
+PYTHONPATH=src:. python -m unittest tests.test_skill_eval_daily_budget tests.test_skill_eval_auto_daily -v
+```
+
+The controller matrix covers all four durable admission writes under conflicts,
+uncertain successful writes and crashes; replay after partial progress; rollover
+before daily claim, after both claims and between provider requests; claim
+revocation and worker failure after launch. Tests use only local in-memory state
+and fake credentials, not provider requests, GitHub writes or paid experiments.
+Full pytest validation requires the development dependencies; these unit tests
+do not establish provider billing or evaluation efficacy.

docs/skill-eval-pr.md

@@ -361,3 +361,17 @@ model, no provider selection and no executable comparison freezes. It never
 silently falls back to another model/provider. These blocked plans are rejected
 by PR authorization and execution as well as the automatic controller. Direct
 standalone comparison-harness behavior is unchanged.
+
+A trusted mapping item may explicitly set `adapter: "kv-admission-diagnostic-v1"`
+to use the synthetic diagnostic contract documented in
+[skill comparisons](skill-eval-comparison.md#synthetic-kv-admission-diagnostic-adapter).
+This is not inferred from PR content. It requires an existing baseline bundle
+and preserves exact PR base/head comparisons; unknown adapters or an addition
+remain unsupported. Report regeneration rechecks the frozen built-in adapter,
+not a performance-audit grader applied to diagnostic answers.
+
+Mapping a skill's changed files selects the comparison; it does not establish
+that a small case covers every mechanism in those files. Unmapped generated
+projections, navigation documents and tests remain visible coverage gaps.
+Source-only review dispositions must be explicit and must not be represented as
+model-evaluated passes. The maintained mapping is not expanded automatically.

docs/skill-eval-quarantine.md

@@ -0,0 +1,58 @@
+# Persisting an incident quarantine
+
+This is a trusted operator/controller stop operation, not paid admission or a
+billing ingestion interface. Run only from the current reviewed implementation
+with its trusted schema-2 daily anchor configuration and the existing GitHub
+credential injected by the authorized environment:
+
+```sh
+PYTHONPATH=src:. python -m tools.skill_eval_quarantine \
+  --repository <owner/repository> \
+  --evidence https://github.com/<owner>/<repository>/issues/<incident-number>
+```
+
+The evidence must be a sanitized public GitHub issue/PR reference, optionally a
+comment/review permalink. Do not pass raw bills, provider errors, credentials or
+private payloads. No provider credential is needed or read. No model, dispatch,
+ledger initialization, reset, settlement, clearing, or repository-setting API
+is called. This CLI itself performs real Git ledger writes when invoked against
+a live client; tests and delivery of its source do not authorize invocation.
+
+Use this operation when a trusted incident establishes exposure above the
+reserved upper bound or makes that upper bound unknowable. A missing actual
+bill under a still-valid conservative bound does not by itself establish either.
+No expiry/pricing/label check can prevent this safety-only stop: it loads only
+the trusted repository identity and daily anchor from root configuration, then
+the Git adapter verifies repository identity and full pinned ledger lineage.
+
+The writer retries at most three verified read/compare-and-swap cycles, with a
+final read-only verification. A concurrent admission is preserved and the stop
+is recomputed against the newer history. An unknown/lost write receipt is
+reconciled **only into a verified stop**, never into a launch grant. Existing
+quarantine is sufficient; repeated calls preserve its first evidence and do not
+rewrite history. Subsequent admission and per-request checks reject quarantine.
+Already-issued or in-flight provider requests cannot be recalled by this write.
+
+Exit zero and `{"state":"quarantine_verified",...}` mean fresh authenticated
+readback established the durable stop. Merely receiving a successful CAS is not
+enough. Nonzero exit / `Global quarantine NOT verified` means the operation is
+incomplete: keep paid execution stopped, resolve storage/identity/anchor access,
+and rerun this monotonic stop until verified. Do not restart paid work because
+retries were exhausted, and do not change/reset anchors to bypass failure.
+If storage is inaccessible, this CLI cannot claim it stopped other processes;
+that requires the separately authorized operational response.
+
+There is no unquarantine command. Resuming after an incident requires separately
+reviewed reconciliation; deleting a ledger/ref or lowering a checkpoint is not
+reconciliation. This operation does not prove provider billing, undo spend or
+provide administrator-resistant rollback protection.
+
+Offline tests:
+
+```sh
+PYTHONPATH=src:. python -m unittest tests.test_skill_eval_quarantine -v
+```
+
+They cover concurrent admission, lost successful write receipt, repeat-stop
+idempotence, conflict exhaustion, failed readback, expired pricing, rejection of
+raw/private evidence inputs, and nonzero CLI failure without a success report.

src/inferrsi/harness/skill_eval_diagnostic.py

@@ -0,0 +1,131 @@
+"""Offline, synthetic KV admission diagnosis contract; not an engine simulator.
+
+Only observations cross the task boundary. The oracle stays in the validator,
+not in the task or selected knowledge bundle. No provider or execution hooks.
+"""
+import json
+import os
+import re
+import sys
+from pathlib import Path
+
+ADAPTER = "kv-admission-diagnostic-v1"
+PROMPT = (
+    "Read task.json and diagnose why the synthetic request cannot run. Knowledge is fallible "
+    "guidance; observations are authoritative. Cite only decisive event IDs. For insufficient "
+    "evidence cite all observed receive, identity and allocator events, or the lookup event "
+    "only if none of those exists. Choose one next read-only "
+    "check, and decide whether these observations authorize reusing KV now. Return only the "
+    "required JSON. Inspect only this workspace; do not modify inputs, use network, execute "
+    "repository code, run another agent, commit or publish. No service-performance claim is asked."
+)
+DIAGNOSES = ("readiness", "capacity", "invalid_reuse", "insufficient_evidence")
+CHECKS = ("check_receive_completion", "inspect_allocator_accounting",
+          "check_transfer_integrity", "collect_missing_events")
+SCHEMA = {
+    "type": "object", "additionalProperties": False,
+    "required": ["diagnosis", "evidence", "next_check", "reuse_authorized"],
+    "properties": {
+        "diagnosis": {"type": "string", "enum": list(DIAGNOSES)},
+        "evidence": {"type": "array", "items": {"type": "string"}},
+        "next_check": {"type": "string", "enum": list(CHECKS)},
+        "reuse_authorized": {"type": "boolean"},
+    },
+}
+VALUES = {
+    "cache_lookup": {"hit", "miss"},
+    "receive": {"pending", "complete", "failed"},
+    "identity": {"match", "mismatch"},
+    "allocator": {"denied", "granted", "not_attempted"},
+}
+
+
+def validate_task(task):
+    if not isinstance(task, dict) or set(task) != {
+        "id", "split", "request", "provenance", "events"
+    }:
+        raise ValueError("Diagnostic task requires only id/split/request/provenance/events")
+    if any(type(task[k]) is not str or not task[k].strip()
+           for k in ("id", "split", "request", "provenance")):
+        raise ValueError("Diagnostic task metadata must be nonempty strings")
+    if not re.fullmatch(r"[a-zA-Z0-9][a-zA-Z0-9._-]*", task["id"]):
+        raise ValueError("Diagnostic task ID must be a safe path component")
+    if task["split"] not in ("development", "holdout", "discovery"):
+        raise ValueError("Unknown case split")
+    if not isinstance(task["events"], list) or not task["events"]:
+        raise ValueError("Diagnostic observations must be a nonempty list")
+    ids, kinds = set(), set()
+    for event in task["events"]:
+        if not isinstance(event, dict) or set(event) != {"id", "kind", "value"}:
+            raise ValueError("Diagnostic event requires only id/kind/value")
+        if any(type(value) is not str for value in event.values()):
+            raise ValueError("Diagnostic event fields must be strings")
+        if not re.fullmatch(r"E[0-9]+", event["id"]) or event["id"] in ids:
+            raise ValueError("Diagnostic event IDs must be unique E-number identifiers")
+        if event["kind"] not in VALUES or event["value"] not in VALUES[event["kind"]]:
+            raise ValueError("Unknown diagnostic observation")
+        if event["kind"] in kinds:
+            raise ValueError("Use a single observation per kind at the decision boundary")
+        ids.add(event["id"])
+        kinds.add(event["kind"])
+
+
+def expected(task):
+    """Finite-state decision table with conservative unknown-state handling.
+
+    An explicitly failed transfer or identity mismatch forbids reuse regardless
+    of allocation. A pending receive is a readiness barrier, not capacity proof.
+    Capacity attribution needs completed, matching KV and explicit denial.
+    Anything else is insufficient to identify why admission is blocked.
+    """
+    validate_task(task)
+    states = {e["kind"]: e["value"] for e in task["events"]}
+    ids = {e["kind"]: e["id"] for e in task["events"]}
+    invalid = {k for k, v in (("receive", "failed"), ("identity", "mismatch"))
+               if states.get(k) == v}
+    if invalid:
+        return "invalid_reuse", {ids[k] for k in invalid}, "check_transfer_integrity"
+    if states.get("receive") == "pending":
+        return "readiness", {ids["receive"]}, "check_receive_completion"
+    if (states.get("receive"), states.get("identity"), states.get("allocator")) == (
+        "complete", "match", "denied"
+    ):
+        return "capacity", {ids[k] for k in ("receive", "identity", "allocator")}, (
+            "inspect_allocator_accounting")
+    relevant = {event_id for kind, event_id in ids.items() if kind != "cache_lookup"}
+    return "insufficient_evidence", relevant or set(ids.values()), "collect_missing_events"
+
+
+def verify(task, answer):
+    diagnosis, required, next_check = expected(task)
+    valid = (isinstance(answer, dict) and set(answer) == set(SCHEMA["required"])
+             and type(answer["diagnosis"]) is str and answer["diagnosis"] in DIAGNOSES
+             and type(answer["next_check"]) is str and answer["next_check"] in CHECKS
+             and type(answer["reuse_authorized"]) is bool
+             and isinstance(answer["evidence"], list)
+             and all(type(v) is str for v in answer["evidence"]))
+    if not valid:
+        return {"passed": False, "assertions": {"schema": False}}
+    cited = set(answer["evidence"])
+    checks = {
+        "schema": True,
+        "diagnosis": answer["diagnosis"] == diagnosis,
+        "citation": cited == required
+        and len(cited) == len(answer["evidence"]),
+        "safe_next_check": answer["next_check"] == next_check,
+        # These blocked-request observations never authorize executing reuse.
+        "no_invariant_violation": answer["reuse_authorized"] is False,
+    }
+    return {"passed": all(checks.values()), "assertions": checks}
+
+
+def main():
+    task = json.loads(Path(sys.argv[1]).read_text(encoding="utf-8"))
+    answer = json.loads(Path(sys.argv[2]).read_text(encoding="utf-8"))
+    result = verify(task, answer)
+    Path(os.environ["INFERRSI_RESULT_PATH"]).write_text(json.dumps(result), encoding="utf-8")
+    raise SystemExit(0 if result["passed"] else 1)
+
+
+if __name__ == "__main__":
+    main()