product-validation.md

Product Validation

Tabellio product validation answers a stronger question than “did tests exit zero?”: did the exact committed candidate satisfy its declared user outcomes while staying inside semantic, visual, operational, cost, and safety boundaries?

Contracts

tabellio-validation/v0.2 binds three layers in one committed manifest:

1. acceptance records the source request, risk, required outcomes, invariants, forbidden outcomes, and required validator types.
2. validators run shell-free argument arrays in an exact-head detached worktree under an external temporary root, never under the repository or Git common directory.
3. policy applies metric thresholds and cost requirements to a bounded evidence report.

Validator types are static, schema, semantic, workflow, visual, operational, and security. Static validators can use command exit status alone. Every other type must emit tabellio-validator-evidence/v0.1 at its declared relative path.

Evidence

An evidence report contains:

• status: the adapter’s passed, failed, or blocked observation;
• numeric metrics with explicit units;
• cost telemetry state, USD cost, model calls, and tool calls;
• immutable artifact metadata for screenshots, traces, reports, or source snapshots.

The runner validates the evidence shape and validator identity, embeds the bounded report in the durable result, and evaluates policy itself. Adapter-reported passed cannot override a failed metric threshold or exceeded cost cap.

Decision Semantics

Decision	Meaning
passed	Every required validator completed with trustworthy evidence inside all thresholds.
failed	Trustworthy evidence proves a required product or budget threshold was missed.
blocked	Required proof is missing or untrustworthy because of invalid evidence, missing metrics or cost telemetry, command errors, or timeouts.

Fail closed. Never translate blocked into passed, and never treat missing cost telemetry as zero.

Repository Adapters

Keep product knowledge in each repository. The adapter runs the existing eval, browser, design, cost, or security harness and writes the common evidence contract.

Use tabellio-validator for simple adapters. Commit .tabellio/validators.json with tabellio-adapter/v0.1 profiles containing shell-free command arrays, evidence metrics, cost telemetry, and a concise summary. Metrics may map command pass/fail to numeric values or extract a numeric capture group from bounded command output.

• AI systems: stable golden cases in pull requests; larger live suites on schedules or release gates.
• Web applications: critical journeys, desktop/mobile screenshot comparisons, loaded-font assertions, overflow, and accessibility metrics.
• Operational systems: projected and observed provider usage, worker counts, polling frequency, rate limits, latency, and spend.
• Structured outputs: strict schema validation plus semantic oracles that catch plausible but wrong values.

Use fixtures and isolated namespaces in pull requests. Live external mutation remains separately approval-gated.

Review Integration

Review sync reads only the newest validation result stored under the exact pull-request head. passed becomes a successful local check; failed or blocked prevents merge readiness. A new head requires new validation.

Run contract checks with:

node scripts/check-tabellio-validation.mjs --manifest examples/tabellio-validation/product-manifest.json
node scripts/check-tabellio-validation.mjs --evidence examples/tabellio-validation/minimal-validator-evidence.json

Run the exact-head suite with:

node scripts/tabellio-validate.mjs run \
  --repo . \
  --repo-id example/project \
  --base main \
  --commit HEAD \
  --manifest tabellio.validation.json

Use the fail-closed command in CI:

tabellio-validate gate \
  --repo . \
  --repo-id example/project \
  --base main \
  --commit HEAD \
  --manifest tabellio.validation.json

After a squash merge, validate the landed commit while binding checkpoint evidence to the pre-merge pull-request head:

tabellio-validate gate \
  --repo . \
  --repo-id example/project \
  --base HEAD^ \
  --commit HEAD \
  --checkpoint-base HEAD^ \
  --checkpoint-head refs/remotes/origin/pull/123/head \
  --manifest tabellio.validation.json

GitHub Actions resolves the merged pull request from the exact push commit, fetches GitHub's durable pull-request head ref, and supplies the separate checkpoint range. Direct pushes keep using the landed commit range and must carry their own checkpoint evidence.

The runner creates a private TabellioValidation-* session under the system temporary directory and removes the detached worktree plus isolated HOME after every run. Use --workspace-root /absolute/external/path to choose another external parent. Repository-internal and .git/** roots are rejected.

gate still records the result. It exits non-zero when the decision is failed or blocked.

Artifact upload and path-based automatic validator selection remain scheduler or repository-policy responsibilities. The first contract keeps evaluation portable without creating a second CI or deployment authority.

Private metadata-only native checkpoint transport

With pinned Entire 0.7.7 and supported Git, export only the checkpoint metadata needed by a candidate, while preserving original native commit/tree/object IDs:

npm run tabellio:checkpoint:export -- \
  --repo . --repo-id github.com/OWNER/REPOSITORY \
  --base origin/main --commit HEAD --out /private/operator/checkpoints.bundle

The exporter invokes the genuine native metadata reader and existing scope, completeness and Git trailer checks. It creates a native filtered Git bundle containing selected root/session metadata blobs, original commits and trees. Transcript and prompt blobs are excluded. The output is private: metadata may contain session IDs, file paths, summaries and token usage. Export does not publish the bundle or confer permission to transfer it.

Supply the approved private file through TABELLIO_CHECKPOINT_BUNDLE on a trusted worker. checkpoint-evidence.sh imports filtered bundles through Git's promisor-aware unbundle path, retains the original advertised native ref, and rejects divergent or concurrently changed history. Existing complete bundles remain supported. Validators still use Entire's native reader and exact Git trailer associations; standalone JSON exports are not accepted.

This does not add authenticated producer attestation: Entire 0.7.7 still does not detect a copied checkpoint ID on another commit. The operator supplying native evidence and the worker consuming it must remain trusted. No checkpoint record is synthesized or reconstructed. No private IntelIP remote is required.

The repository's one-time GitHub proof uses a reviewer-protected temporary environment and two encrypted secret chunks. The private envelope binds the repository, candidate, merge base, bundle digest and native tip. The loader refuses mismatches and pre-existing native/validation refs. It never prints the payload or native metadata. The normal gate remains unchanged.

The final cleanup step removes the temporary bundle and newly imported Git objects and private refs, preserving the original checkout objects. The operator must delete the temporary environment and secrets after the proof. Public logs contain only allowlisted gate status and cleanup outcome; no artifacts or caches receive private evidence. Subsequent candidates require their own genuine customer-owned evidence; deleted proof secrets cannot be reused.