Repository navigation
feat(evolution): add deterministic evidence ranking - #103
Merged
Pepps233 merged 3 commits intoAug 23, 2026
Merged
Conversation
- Prevents unverifiable or weakly recurring proposals from reaching mutation planning - Makes deterministic selection auditable through versioned score components and dispositions
- Keeps accepted but ineligible proposals as a durable no-op instead of forcing a write - Verifies validation artifacts, citation closure, and the five-selection invariant at pipeline boundaries
- Allows concurrently admitted children to reach the provider in either order - Continues to verify FIFO admission for queued children as permits become available
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implements deterministic evidence validation and proposal ranking over the structured deltas emitted by the Curator in #102 without enabling later persistence, Writer, activation, rollback, or Counter behavior.
The validator now closes every proposal over frozen lessons, evidence, traces, materializations, memory state, and applicable instructions, then the scorer assigns an auditable 0-100 evidence-ranking score, removes duplicates and conflicts, and selects no more than five proposal groups for later writing.
Reasoning
Read below for how deterministic scoring works
Evolution writes need a reproducible gate between model-generated proposals and repository mutation.
Validation requires support from at least two independent task lineages, or three for broad-scope guidance, with a narrow exception for direct user instructions supported by one lineage.
It rejects equal or stronger counterevidence, stale or invented closure fields, invalid artifact digests, oversized deltas, duplicate instructions, and scope that does not exactly match the cited evidence.
Evidence from incompatible or invalid materializations is quarantined instead of being allowed to influence ranking.
Each accepted proposal receives up to 100 positive points: 30 for recurrence, 20 for provenance integrity, 15 for source-signal strength, 15 for scope coverage, 10 for Curator confidence, and 10 for proposal completeness.
Recurrence contributes 10 points per independent supporting lineage up to 30.
Provenance uses the average integrity of supporting evidence, awarding 20 for complete invocations, 12 for evidence inside a verified partial prefix, and 0 for incompatible or unverified evidence.
Source-signal strength uses the strongest independent support signal: 15 for direct user evidence, 12 for deterministic outcomes, 9 for tool outcomes, 6 for observable events, and 3 for model self-assessment.
Scope coverage rewards distinct materializations and source revisions, while Curator confidence maps its 0-1 value to 0-10 points.
Structurally complete proposals receive the final 10 points because incomplete proposals are rejected before scoring.
Contradicting evidence subtracts up to 40 points in proportion to the ratio of independent contradiction strength to support strength.
Only proposals scoring at least 40 are eligible by default.
Eligible proposals are ordered by score, independent support count, earliest supporting evidence, and proposal ID, then exact operation fingerprints are collapsed, conflicts with higher-ranked proposals are removed, and the first five survivors are selected.
Versioned score components, validation results, explicit dispositions, and reason codes make every decision reproducible and auditable without asking a model to judge its own output.
Changes
evidence-ranking-v1deterministic validator and scorer with persisted component scores, final scores, and validation assurance.Testing
npm run check./test.shnode ../../node_modules/vitest/dist/cli.js --run test/proposal-ranking.test.ts test/control-plane.test.tstest/suite/subagent-startup.test.ts20 times after correcting its concurrent-start assertion.