docs: clarify validation and verification terminology - #14
Conversation
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
There was a problem hiding this comment.
Review of PR #14 — docs: clarify validation and verification terminology
Overview
This PR establishes a rigorous shared vocabulary across 19 documentation files, replacing the overloaded term "proof" with "evidence"/"verification method" and replacing "approve/ratify" with "validate". The new docs/concepts/terminology.md is well-structured and the V/E/O/P distinction is clearly motivated. Renamed how-to files (validate-the-graph.md, set-verification-requirements.md) are consistently cross-referenced within the docs tree. No code, schema, or CLI contract changes.
Findings
MEDIUM — Agent-skill vocabulary not updated to match new terminology
The PR introduces a clear vocabulary split that docs now enforce, but agent-skills/quality/SKILL.md and all of its references/ files still use the old vocabulary throughout. Searching the branch shows:
- 115 non-formal-proof uses of "proof" in
agent-skills/ - 36 uses of "ratif*" (ratify, ratification, ratified, unratified) in
agent-skills/ - Only 14 uses of "validate" in
agent-skills/
Specific examples of stale vocabulary agents will execute verbatim:
agent-skills/quality/SKILL.md machine-readable description field (line 3):
"Guides new-repository setup, project and feature mapping, saved assessment scopes, proof mapping, runtime assessment, and score-directed improvement while preserving independent scoring and human ratification."
agent-skills/quality/SKILL.md four-scores improvement table:
- Coverage: "Does every declared check have proof?" / "Create or map missing proof"
- Quality: "Is current proving evidence passing?"
- Structure confidence: "Ask a human to correct or ratify proposed structure"
The updated docs/concepts/the-four-scores.md now reads "Does every expected behavior have a mapped verification method?" and "whether a person validated them" — but an agent following SKILL.md will use the old wording.
agent-skills/quality/SKILL.md quality graph model still reads:
proof definitions: what could prove each check
runtime observations: did that proof pass?
The updated docs/concepts/quality-graph.md uses "verification methods" here.
agent-skills/quality/references/_shared/independence.md uses "ratification gates" as a section heading and "never self-advance a gate or ratify on the owner's behalf" across 36 occurrences. agent-skills/quality/references/map-feature/index.md repeats "proof definitions," "proof artifacts," "proof gaps," and "ratification gates" throughout.
Per the review guidance: agent-skills are instructions an agent executes verbatim and "a stale skill list, command set, or install ref there is a real defect, not a nit." The new docs/concepts/terminology.md (in this PR) explicitly says in the Human decisions section: "Avoid using ratification as a general synonym for validation." Publishing the docs change without the skill update locks in that contradiction.
The practical risk: a user reading the new terminology.md is told to "validate intent" and refer to "evidence gaps." But an agent executing SKILL.md will ask the user to "ratify" and discuss "proof gaps." Users get inconsistent signals about what action they are being asked to take.
Requested fix: Update agent-skills/quality/SKILL.md and its references/ files to replace non-formal-proof uses of "proof" with "evidence"/"verification method"/"mapped method" and "ratify/ratification" with "validate/validation", consistent with the vocabulary established in docs/concepts/terminology.md.
LOW — set-verification-requirements.md has no "Next steps" cross-links
The new docs/how-to/set-verification-requirements.md ends at the Troubleshooting section without a "Next steps" pointer. Other how-to files (e.g. map-a-feature.md, set-up-quality.md) all link to natural follow-ons. Adding links to validate-the-graph.md and act-on-a-weak-score.md would complete the navigability pattern. Not a broken-link issue, just a navigability gap.
LOW — Gate-4 description in independence.md uses "approved"
agent-skills/quality/references/_shared/independence.md gate-4 table row reads: "a human reviewed and approved the whole check list." The preferred post-PR verb is "validated." This is covered by the broader MEDIUM finding but worth calling out separately because independence.md is the most-referenced file in the agent skill and guards human-gated fields.
What the PR does well
docs/concepts/terminology.mdis clear and well-structured. The V/E/O/P hierarchy is well-motivated and the disambiguation of "schema validation" vs. "intent validation" preempts a known confusion with the existingvalidateCLI command.- The
proof_gapcompatibility-name acknowledgment ("Documentation may show that literal name when explaining the file format, but prose should use evidence gap") is exactly right. - All intra-docs cross-references were updated correctly. A grep for the old file names (
review-and-ratify,require-stronger-proof) finds zero remaining occurrences in any Markdown file. - No code changes, no schema version bumps, no CLI contract changes — scope is well-contained to vocabulary and documentation.
- The preferred-wording table at the end of
terminology.mdis a practical quick-reference that will help contributors stay consistent. - The
docs/verification/quality-prompt-sweep.mdresult table was correctly updated to use the new terms.
Summary
One MEDIUM issue blocks approval: agent-skills/quality/SKILL.md and its references/ files were not updated alongside the docs, creating a split between what users read ("validate", "evidence gap") and what agents say to users ("ratify", "proof gap"). Since the purpose of this PR is to establish consistent shared terminology and agent-skills are executed verbatim, the skill files need to be part of this change.
|
Addressed the review feedback in 95173f5:
Validation:
|
There was a problem hiding this comment.
Review: docs: clarify validation and verification terminology
Overview
PR #14 is a documentation-only terminology refactor across 46 files (+745/-526 lines). It replaces the overloaded word "proof" and the formal-sounding "ratification" with a precise, layered vocabulary:
- Validation: human confirmation that features/checks represent product intent
- Verification: evaluation of whether implementation/behavior satisfies a claim
- Verification method: a specific technique (test, static analysis, formal proof, etc.)
- Evidence: information (from reasoning or observation) that supports a claim
- Proof: reserved for deductive/formal proof only
A new docs/concepts/terminology.md anchors the whole vocabulary. Two how-to files are renamed (review-and-ratify.md to validate-the-graph.md, require-stronger-proof.md to set-verification-requirements.md) with cross-references updated throughout.
Project-invariant checks
| Invariant | Finding |
|---|---|
| Independence of scoring | No code changed; the deterministic engine path is untouched. Pass |
| Human-gated fields | checks_reviewed, structure_provenance, priority_provenance, and accepted_gaps all remain explicitly human-gated; agent restrictions are preserved verbatim under the new vocabulary. Pass |
| Determinism | No runtime code changed. Pass |
| Dependency direction | No package imports changed. Pass |
| Schema/saved-artifact compatibility | proof_gap field name is preserved; the PR explicitly calls it a "legacy literal field name" and instructs prose to use "evidence gap" instead. No schema_version bump, no migration required. Pass |
| Published CLI surface | Command names, flags, and @shiplightai/quality-tools exports are unchanged. Pass |
| Agent-skill safety | No new powers granted to agents. The instructions still forbid self-advancing gates, self-ratifying structure, or accepting risk on the owner's behalf — now expressed as "self-validate" / "validate intent on the owner's behalf." The command set (start, status, map-project, map-feature, assess, improve, help) is unchanged. Pass |
| Docs match reality | README, docs/README.md, how-to guides, and agent-skill references all consistently point to the renamed files. Pass |
| Agent-skill destructive ops | No guidance added for force-push, reset, delete branches, merge --admin, or bypassing review. Pass |
Findings
LOW — Minor wording inconsistency in validate-the-graph.md
"The validated quality map records that the complete list of checks was reviewed."
The word "reviewed" intentionally echoes the YAML field checks_reviewed, but after the wholesale shift to "validated" throughout, a reader who has not internalized the field name may find this jarring. A parenthetical "(the checks_reviewed field)" would eliminate the ambiguity. Not a defect, but worth a one-word clarification on a follow-up pass.
LOW — Pre-existing response defect documented in sweep (not introduced by this PR)
docs/verification/quality-prompt-sweep.md records a known agent behaviour defect ("it added a reviewer email not supplied in the prompt") under the newly renamed "Intent validation" row. This is carry-over documentation of a pre-existing issue, not introduced by this PR.
Strengths
- The new
docs/concepts/terminology.mdis clear and well-structured; the "Preferred wording" table at the end is especially useful. - Acknowledging that
proof_gapis a compatibility field name (not a conceptual term) prevents the old vocabulary from leaking back in through the schema. - Cross-references across all renamed files are consistently updated.
- The
.quality/evidence/YAML anchor/label strings match the revised AGENTS.md text, keeping traceability intact.
Verdict
No CRITICAL, HIGH, or MEDIUM issues. The refactor is internally consistent, preserves every behavioral constraint, does not touch any scoring path or schema contract, and the new terminology guide meaningfully improves the conceptual clarity of the project.
There was a problem hiding this comment.
PR #14 — docs: clarify validation and verification terminology
This PR is a large-scale but low-risk terminology normalization. It replaces "proof/ratification" vocabulary with "evidence/verification/validation" across documentation, agent skill instructions, quality-map YAML evidence files, templates, and test specs. It also renames two how-to guides and adds a new docs/concepts/terminology.md reference page. No source code, scoring engine, schema versions, or CLI contracts are changed.
Findings by severity
CRITICAL — None.
HIGH — None.
MEDIUM — None.
LOW — L1: Git-delete rather than rename for require-stronger-proof.md
docs/how-to/review-and-ratify.md is properly git-renamed to validate-the-graph.md (history preserved). docs/how-to/require-stronger-proof.md is deleted and docs/how-to/set-verification-requirements.md is a new file — so the history of the original is not reachable through the new file. All in-repo links have been updated within this PR (confirmed via grep across the diff), so there is no broken-link regression inside the repository. External bookmarks or off-repo links to the old filename will 404. Acceptable for a documentation refactor, but worth noting.
LOW — L2: Anchor strings in .quality/evidence/*.yaml updated to match renamed headings
Files .quality/evidence/002, 003, 004, 006 use label: / anchor: strings such as "Agents must not set human ratification fields..." and "Proof remains independent and scores are deterministic". These were updated to match the renamed source headings in AGENTS.md and docs/concepts/trust-boundaries.md. The changes are internally self-consistent within this PR: source headings and anchoring references land together. No breakage risk provided the PR merges atomically, which is the normal case.
Project invariant checklist
| Invariant | Status |
|---|---|
| Independence — no agent/LLM writes or infers a score | ✅ No source code changed; scoring engine untouched |
Human-gated fields — structure_provenance, checks_reviewed remain human-gated |
✅ Enum values and gating prohibition unchanged; AGENTS.md rule preserved (wording only) |
Determinism — no Date.now(), Math.random(), or unstable iteration |
✅ No code changes |
Dependency direction — quality-map ← core ← quality-tools |
✅ No import changes |
| Explorer read-only | ✅ No Explorer changes |
| Project-root confinement / path traversal | ✅ No path handling changes |
schema_version compatibility |
✅ No schema_version fields changed anywhere |
Published surface of @shiplightai/quality-tools |
✅ CLI command names (validate, analyze, fix-prompts, observations, schema) and skill sub-commands (start, status, map-project, map-feature, assess, improve, help) are unchanged |
| Agent-skill safety — no destructive git operations | ✅ No new destructive instructions added |
| Docs match reality | ✅ README, commands.md, and all how-to cross-links updated consistently |
Positive observations
docs/concepts/terminology.mdis a clear, well-structured reference that correctly preserves the human-gating invariant ("Quality records these decisions but does not make them") and disambiguates schema validation from intent validation — a genuine documentation gap that was missing.- The new README diagram explicitly labels the "intent validation" step and distinguishes reasoning-based from empirical verification paths. This is more accurate than the previous linear "proof" chain.
docs/verification/quality-prompt-sweep.mdcorrectly renames "Review and ratification" → "Intent validation" in the test result table, maintaining parity between test descriptions and the new terminology.- The distinction between
proof_gap(legacy machine-readable field) and "evidence gap" (preferred prose) is correctly documented interminology.md, avoiding a confusing schema rename.
Approve: all substantive invariants are preserved; the two LOW findings are cosmetic documentation concerns with no functional or security impact.
1 participant
Summary
Validation
git diff --check