Add minimal BDV approval verification pilot

docs/experiments/bdv-pilot/README.md

@@ -0,0 +1,34 @@
+# BDV pilot: approval call binding
+
+This pilot links a frozen behavior claim to a retained integration probe and a machine-readable evidence ledger. It does not change product behavior or replace existing tests.
+
+## Run
+
+```sh
+pnpm bdv approval-call-binding
+```
+
+The runner requires a clean checkout (the evidence ledger itself may already contain earlier runs) and checks that the probe matches the digest frozen in [`claims.json`](claims.json). It then runs Vitest's JSON reporter with a 120-second process timeout and requires each of the four exact scenario names to appear exactly once with `passed` status. Aggregate counts or earlier console text cannot produce a passing verdict.
+
+Before and after the probe, it records the commit and hashes the claim, probe, and runner. It rejects dirty executable inputs and any change to those hashes or `HEAD` during execution. The appended ledger entry includes before/after input hashes, the command, timeout and exit status, exact scenario results, result digest, elapsed time, output digest, and normalized output. This makes the observed result checkable against the committed contract and probe.
+
+## Independence and limits
+
+Acceptance is a fixed contract that the runner does not generate from the observed output. It expresses the approval rule as externally observable outcomes: only the allowed file is created, the denied target survives, each tool call receives its own decision, and the provider sees paired calls and results. The probe crosses the built CLI, a loopback HTTP provider fixture, and isolated filesystem effects. Its approved source digest and exact scenario names bind the runner to the reviewed probe; changes require an explicit contract update.
+
+This is one claim and one probe, not a general probe scheduler. The claim contract and its probe are maintained in the same repository, so hashes establish provenance and change detection, not independent authorship. An independent reviewer has not yet challenged this pilot contract. It does not yet enforce independent contract approval, compare costs across broad test suites, or show that passing the selected scenarios proves all approval paths safe.
+
+## Baseline and pilot result
+
+Baseline at `d93142ea` (before the runner), using the same probe directly:
+
+```text
+pnpm exec vitest run --reporter=minimal --config vitest.integration.config.ts source/cli.integration.test.ts -t 'an approved shell call does not approve a later call'
+Test Files  1 passed (1)
+Tests       4 passed | 30 skipped (34)
+Duration    2.60s
+```
+
+The baseline establishes repeatability and elapsed test time only; it is not a full-project defect-detection or cost comparison. The pilot evidence record is appended when `pnpm bdv approval-call-binding` runs.
+
+The first runner execution result and elapsed time are captured in the evidence ledger. Compare them with the baseline as a one-run observation only; this pilot does not establish a performance trend.

docs/experiments/bdv-pilot/claims.json

@@ -0,0 +1,27 @@
+{
+  "schemaVersion": 1,
+  "claims": [
+    {
+      "id": "approval-call-binding",
+      "statement": "An approval for one shell tool call cannot authorize a later call that reuses the same provider call id, and adapter-generated ids for missing or empty provider ids remain distinct.",
+      "authority": "Frozen pilot contract derived from the existing user-facing approval policy and the expected safe side effects; this runner cannot alter the expected outcomes.",
+      "probe": "approval-reuse-integration",
+      "approvedProbeDigest": "3cc8df47576f3754b4c5d6b0ac6a75870e72336b8608cee6ced2fbd60bb59061",
+      "acceptance": {
+        "expectedCases": [
+          "an approved shell call does not approve a later call that reuses its id when the provider repeats call_0",
+          "an approved shell call does not approve a later call that reuses its id when the provider repeats its own id",
+          "an approved shell call does not approve a later call when the chat-completions provider omits tool call ids",
+          "an approved shell call does not approve a later call when the chat-completions provider sends empty tool call ids"
+        ],
+        "mustAssert": [
+          "the first GREEN unsandboxed command runs and writes exactly 'approved\\n'",
+          "the subsequent RED rm command is refused and its victim file remains exactly 'must survive\\n'",
+          "the CLI reports two distinct approval decisions and the RED refusal reason",
+          "all four provider-id, repeated-id, omitted-id, and empty-id scenarios pass",
+          "the provider receives both calls and paired tool results"
+        ]
+      }
+    }
+  ]
+}

docs/experiments/bdv-pilot/evidence.jsonl

@@ -0,0 +1,7 @@
+{"schemaVersion":1,"claimId":"approval-call-binding","claimDigest":"aa52df0624b9888e789272e09bf21d8126663b4802664671afcfffb3788d5d07","probeDigest":"05ba6586bc09333bb248e7487f69c4598cc180eb4fccd3d65bb3b2a8922d3328","runnerDigest":"c8ed6477c63fc2507a7979a6c92db43c89d19c8f48003465626d67808d2e0be6","commit":"0425b5a12e6b4429f3621d16a6f3c38205e0dc5e","startedAt":"2026-10-08T15:45:59.677Z","durationMs":2900,"command":["pnpm","exec","vitest","run","--reporter=minimal","--config","vitest.integration.config.ts","source/cli.integration.test.ts","-t","an approved shell call does not approve a later call"],"exitCode":0,"passedFiles":1,"passedTests":4,"verdict":"pass","outputDigest":"b89ec2955b9af86381110d1be4d22669b178c74c2cc45f149e87728f4a71e16d","output":"\n RUN  v4.1.9 <repo>\n\n\n Test Files  1 passed (1)\n      Tests  4 passed | 30 skipped (34)\n   Start at  22:45:59\n   Duration  2.59s (transform 31ms, setup 12ms, import 36ms, tests 2.48s, environment 0ms)\n\n"}
+{"schemaVersion":2,"claimId":"approval-call-binding","claimDigest":"59b62df2e980bb6ead1da97256622f75382e6360ec1c6b8a2f316159ce552008","approvedProbeDigest":"05ba6586bc09333bb248e7487f69c4598cc180eb4fccd3d65bb3b2a8922d3328","probeDigest":"05ba6586bc09333bb248e7487f69c4598cc180eb4fccd3d65bb3b2a8922d3328","runnerDigest":"bdb45c902b50b38253253eda2fff82539dbf1f9bb1b78b586f76e60b7655392c","commit":"47ccac4811b79e2bba92d7d87123568718f0d227","commitAfter":"47ccac4811b79e2bba92d7d87123568718f0d227","claimDigestAfter":"59b62df2e980bb6ead1da97256622f75382e6360ec1c6b8a2f316159ce552008","probeDigestAfter":"05ba6586bc09333bb248e7487f69c4598cc180eb4fccd3d65bb3b2a8922d3328","runnerDigestAfter":"bdb45c902b50b38253253eda2fff82539dbf1f9bb1b78b586f76e60b7655392c","inputsStable":true,"dirtyPathsBefore":[],"dirtyPathsAfter":[],"startedAt":"2026-10-08T16:00:07.451Z","durationMs":3002,"timeoutMs":120000,"timedOut":false,"command":["pnpm","exec","vitest","run","--reporter=json","--config","vitest.integration.config.ts","source/cli.integration.test.ts","-t","an approved shell call does not approve a later call","--outputFile","<temp>/vitest-report.json"],"exitCode":0,"totalTests":34,"passedTests":4,"failedTests":0,"scenarioResults":[{"fullName":"an approved shell call does not approve a later call that reuses its id when the provider repeats call_0","status":"passed"},{"fullName":"an approved shell call does not approve a later call that reuses its id when the provider repeats its own id","status":"passed"},{"fullName":"an approved shell call does not approve a later call when the chat-completions provider omits tool call ids","status":"passed"},{"fullName":"an approved shell call does not approve a later call when the chat-completions provider sends empty tool call ids","status":"passed"}],"resultDigest":"327c214cadb80219c3d2b660ea5200e344705cef172b90a92e2e2644206861f7","verdict":"pass","outputDigest":"3dfac86c28083d2005ef7131a28dcc65fe54a0bd51fce1c6c61a431bea5a2fa7","output":"JSON report written to <temp>/vitest-report.json\n","error":null}
+{"schemaVersion":2,"claimId":"approval-call-binding","claimDigest":"59b62df2e980bb6ead1da97256622f75382e6360ec1c6b8a2f316159ce552008","approvedProbeDigest":"05ba6586bc09333bb248e7487f69c4598cc180eb4fccd3d65bb3b2a8922d3328","probeDigest":"05ba6586bc09333bb248e7487f69c4598cc180eb4fccd3d65bb3b2a8922d3328","runnerDigest":"bdb45c902b50b38253253eda2fff82539dbf1f9bb1b78b586f76e60b7655392c","commit":"47ccac4811b79e2bba92d7d87123568718f0d227","commitAfter":"47ccac4811b79e2bba92d7d87123568718f0d227","claimDigestAfter":"59b62df2e980bb6ead1da97256622f75382e6360ec1c6b8a2f316159ce552008","probeDigestAfter":"05ba6586bc09333bb248e7487f69c4598cc180eb4fccd3d65bb3b2a8922d3328","runnerDigestAfter":"bdb45c902b50b38253253eda2fff82539dbf1f9bb1b78b586f76e60b7655392c","inputsStable":false,"dirtyPathsBefore":[".bdv-dirty-input-probe"],"dirtyPathsAfter":[".bdv-dirty-input-probe"],"startedAt":"2026-10-08T16:00:40.098Z","durationMs":39,"timeoutMs":120000,"timedOut":false,"command":["pnpm","exec","vitest","run","--reporter=json","--config","vitest.integration.config.ts","source/cli.integration.test.ts","-t","an approved shell call does not approve a later call","--outputFile","<temp>/vitest-report.json"],"exitCode":1,"totalTests":null,"passedTests":null,"failedTests":null,"scenarioResults":[{"fullName":"an approved shell call does not approve a later call that reuses its id when the provider repeats call_0","status":"missing"},{"fullName":"an approved shell call does not approve a later call that reuses its id when the provider repeats its own id","status":"missing"},{"fullName":"an approved shell call does not approve a later call when the chat-completions provider omits tool call ids","status":"missing"},{"fullName":"an approved shell call does not approve a later call when the chat-completions provider sends empty tool call ids","status":"missing"}],"resultDigest":"4e05a53322a157c8f07503e5ab538b4e253fb931fefd66609e547aa7b3c8ba52","verdict":"fail","outputDigest":"e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855","output":"","error":null}
+{"schemaVersion":2,"recordType":"seeded-fault-control","claimId":"approval-call-binding","baseCommit":"6ccef47824af7b3387c651c04e43395bb088d8f5","approvedProbeDigest":"05ba6586bc09333bb248e7487f69c4598cc180eb4fccd3d65bb3b2a8922d3328","probeDigest":"05ba6586bc09333bb248e7487f69c4598cc180eb4fccd3d65bb3b2a8922d3328","fault":"ApprovalLedger.blanketDecision treats any nonempty per-call approved list as blanket approval","faultPatchDigest":"14dfd2e9ed15f1a2e48711e6365f674779901bd71284018a2c38ceff616fa300","command":["CI=1","node_modules/.bin/vitest","run","--reporter=json","--config","vitest.integration.config.ts","source/cli.integration.test.ts","-t","an approved shell call does not approve a later call","--outputFile","/private/tmp/bdv-seeded-fault-report.json"],"exitCode":1,"totalTests":34,"passedTests":0,"failedTests":4,"failingScenarios":["an approved shell call does not approve a later call that reuses its id when the provider repeats call_0","an approved shell call does not approve a later call that reuses its id when the provider repeats its own id","an approved shell call does not approve a later call when the chat-completions provider omits tool call ids","an approved shell call does not approve a later call when the chat-completions provider sends empty tool call ids"],"observedFailure":"all four scenarios removed victim.txt instead of preserving its contents; assertion expected 'must survive\\n' but received null","reportDigest":"1d42575f8391b62fb5dd751f9cb5a21952dac3b565986794f15890dda824eb7c","verdict":"expected-fail"}
+{"schemaVersion":2,"claimId":"approval-call-binding","claimDigest":"27afbbe942d3b4da4896418f7114ab707abdbc3647f85343daed90f3d243e82e","approvedProbeDigest":"c587fc30c0efe640d83a89f2ece92771406673c67102beca40ce84b3db3afb7c","probeDigest":"c587fc30c0efe640d83a89f2ece92771406673c67102beca40ce84b3db3afb7c","runnerDigest":"bdb45c902b50b38253253eda2fff82539dbf1f9bb1b78b586f76e60b7655392c","commit":"1c51842882cbd2ea1d2ca828913a34a39776919d","commitAfter":"1c51842882cbd2ea1d2ca828913a34a39776919d","claimDigestAfter":"27afbbe942d3b4da4896418f7114ab707abdbc3647f85343daed90f3d243e82e","probeDigestAfter":"c587fc30c0efe640d83a89f2ece92771406673c67102beca40ce84b3db3afb7c","runnerDigestAfter":"bdb45c902b50b38253253eda2fff82539dbf1f9bb1b78b586f76e60b7655392c","inputsStable":true,"dirtyPathsBefore":[],"dirtyPathsAfter":[],"startedAt":"2026-10-09T00:32:22.912Z","durationMs":3440,"timeoutMs":120000,"timedOut":false,"command":["pnpm","exec","vitest","run","--reporter=json","--config","vitest.integration.config.ts","source/cli.integration.test.ts","-t","an approved shell call does not approve a later call","--outputFile","<temp>/vitest-report.json"],"exitCode":1,"totalTests":37,"passedTests":0,"failedTests":4,"scenarioResults":[{"fullName":"an approved shell call does not approve a later call that reuses its id when the provider repeats call_0","status":"failed"},{"fullName":"an approved shell call does not approve a later call that reuses its id when the provider repeats its own id","status":"failed"},{"fullName":"an approved shell call does not approve a later call when the chat-completions provider omits tool call ids","status":"failed"},{"fullName":"an approved shell call does not approve a later call when the chat-completions provider sends empty tool call ids","status":"failed"}],"resultDigest":"ecd2b6913729ad3adaf2786819612bcffffb7c6dc968518fabf19400d8974374","verdict":"fail","outputDigest":"3dfac86c28083d2005ef7131a28dcc65fe54a0bd51fce1c6c61a431bea5a2fa7","output":"JSON report written to <temp>/vitest-report.json\n","error":null}
+{"schemaVersion":2,"claimId":"approval-call-binding","claimDigest":"be535d04c03fbea903ffa6542e20f005b2173249043201fcdb4acd0986eade07","approvedProbeDigest":"3cc8df47576f3754b4c5d6b0ac6a75870e72336b8608cee6ced2fbd60bb59061","probeDigest":"3cc8df47576f3754b4c5d6b0ac6a75870e72336b8608cee6ced2fbd60bb59061","runnerDigest":"bdb45c902b50b38253253eda2fff82539dbf1f9bb1b78b586f76e60b7655392c","commit":"ee0d5cda571f23e47d721486fa4c2b20cc1c00af","commitAfter":"ee0d5cda571f23e47d721486fa4c2b20cc1c00af","claimDigestAfter":"be535d04c03fbea903ffa6542e20f005b2173249043201fcdb4acd0986eade07","probeDigestAfter":"3cc8df47576f3754b4c5d6b0ac6a75870e72336b8608cee6ced2fbd60bb59061","runnerDigestAfter":"bdb45c902b50b38253253eda2fff82539dbf1f9bb1b78b586f76e60b7655392c","inputsStable":true,"dirtyPathsBefore":[],"dirtyPathsAfter":[],"startedAt":"2026-10-09T00:33:36.674Z","durationMs":2955,"timeoutMs":120000,"timedOut":false,"command":["pnpm","exec","vitest","run","--reporter=json","--config","vitest.integration.config.ts","source/cli.integration.test.ts","-t","an approved shell call does not approve a later call","--outputFile","<temp>/vitest-report.json"],"exitCode":0,"totalTests":37,"passedTests":4,"failedTests":0,"scenarioResults":[{"fullName":"an approved shell call does not approve a later call that reuses its id when the provider repeats call_0","status":"passed"},{"fullName":"an approved shell call does not approve a later call that reuses its id when the provider repeats its own id","status":"passed"},{"fullName":"an approved shell call does not approve a later call when the chat-completions provider omits tool call ids","status":"passed"},{"fullName":"an approved shell call does not approve a later call when the chat-completions provider sends empty tool call ids","status":"passed"}],"resultDigest":"25bb51501ca6c75f14849069f72d031eccc40cdddf499c9a54846dc1582a29a0","verdict":"pass","outputDigest":"3dfac86c28083d2005ef7131a28dcc65fe54a0bd51fce1c6c61a431bea5a2fa7","output":"JSON report written to <temp>/vitest-report.json\n","error":null}
+{"schemaVersion":2,"recordType":"seeded-fault-control","claimId":"approval-call-binding","baseCommit":"ee0d5cda571f23e47d721486fa4c2b20cc1c00af","approvedProbeDigest":"3cc8df47576f3754b4c5d6b0ac6a75870e72336b8608cee6ced2fbd60bb59061","probeDigest":"3cc8df47576f3754b4c5d6b0ac6a75870e72336b8608cee6ced2fbd60bb59061","fault":"ApprovalLedger.blanketDecision treats any nonempty per-call approved list as blanket approval","faultPatchDigest":"14dfd2e9ed15f1a2e48711e6365f674779901bd71284018a2c38ceff616fa300","command":["CI=1","node_modules/.bin/vitest","run","--reporter=json","--config","vitest.integration.config.ts","source/cli.integration.test.ts","-t","an approved shell call does not approve a later call","--outputFile","/private/tmp/bdv-rebased-seeded.json"],"exitCode":1,"totalTests":37,"passedTests":0,"failedTests":4,"failingScenarios":["an approved shell call does not approve a later call that reuses its id when the provider repeats call_0","an approved shell call does not approve a later call that reuses its id when the provider repeats its own id","an approved shell call does not approve a later call when the chat-completions provider omits tool call ids","an approved shell call does not approve a later call when the chat-completions provider sends empty tool call ids"],"observedFailure":"each scenario deleted victim.txt instead of preserving 'must survive\\n'","reportDigest":"04f33822254120c6bb8248c36c2e7c3d2fbb20dea7ec83756d84dd1ed893aa8e","verdict":"expected-fail"}