Skip to content

feat(evolution): add deterministic evidence ranking - #103

Merged
Pepps233 merged 3 commits into
feat/structured-delta-curatorfrom
feat/evidence-ranking-validator
Aug 23, 2026
Merged

Pepps233 merged 3 commits into
feat/structured-delta-curatorfrom
feat/evidence-ranking-validator

Conversation

@Pepps233

@Pepps233 Pepps233 commented Aug 20, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Implements deterministic evidence validation and proposal ranking over the structured deltas emitted by the Curator in #102 without enabling later persistence, Writer, activation, rollback, or Counter behavior.

The validator now closes every proposal over frozen lessons, evidence, traces, materializations, memory state, and applicable instructions, then the scorer assigns an auditable 0-100 evidence-ranking score, removes duplicates and conflicts, and selects no more than five proposal groups for later writing.

Reasoning

Read below for how deterministic scoring works

Evolution writes need a reproducible gate between model-generated proposals and repository mutation.

Validation requires support from at least two independent task lineages, or three for broad-scope guidance, with a narrow exception for direct user instructions supported by one lineage.
It rejects equal or stronger counterevidence, stale or invented closure fields, invalid artifact digests, oversized deltas, duplicate instructions, and scope that does not exactly match the cited evidence.
Evidence from incompatible or invalid materializations is quarantined instead of being allowed to influence ranking.

Each accepted proposal receives up to 100 positive points: 30 for recurrence, 20 for provenance integrity, 15 for source-signal strength, 15 for scope coverage, 10 for Curator confidence, and 10 for proposal completeness.

Recurrence contributes 10 points per independent supporting lineage up to 30.

Provenance uses the average integrity of supporting evidence, awarding 20 for complete invocations, 12 for evidence inside a verified partial prefix, and 0 for incompatible or unverified evidence.

Source-signal strength uses the strongest independent support signal: 15 for direct user evidence, 12 for deterministic outcomes, 9 for tool outcomes, 6 for observable events, and 3 for model self-assessment.

Scope coverage rewards distinct materializations and source revisions, while Curator confidence maps its 0-1 value to 0-10 points.

Structurally complete proposals receive the final 10 points because incomplete proposals are rejected before scoring.

Contradicting evidence subtracts up to 40 points in proportion to the ratio of independent contradiction strength to support strength.

Only proposals scoring at least 40 are eligible by default.

Eligible proposals are ordered by score, independent support count, earliest supporting evidence, and proposal ID, then exact operation fingerprints are collapsed, conflicts with higher-ranked proposals are removed, and the first five survivors are selected.

Versioned score components, validation results, explicit dispositions, and reason codes make every decision reproducible and auditable without asking a model to judge its own output.

Changes

  • Add an evidence-ranking-v1 deterministic validator and scorer with persisted component scores, final scores, and validation assurance.
  • Verify frozen input, lesson, proposal, evidence, trace, instruction, memory, and scope closure before allowing a proposal into scoring.
  • Score recurrence, provenance integrity, source-signal strength, scope coverage, Curator confidence, and structural completeness, then apply a proportional contradiction penalty.
  • Collapse exact operation fingerprints, reject conflicts with higher-ranked survivors, apply a default eligibility threshold of 40, and enforce a hard five-selection limit.
  • Persist accepted, deferred, quarantined, rejected, selected, superseded, and unselected dispositions with stable reason codes.
  • Recompute validation during scoring and reject altered validation reports or scorer output that violates pipeline invariants.
  • Complete as a durable no-op when no proposal clears the threshold instead of forcing a later write.
  • Add focused coverage for evidence closure, provenance, recurrence, contradiction handling, instruction duplication, deterministic ordering, report tampering, and pipeline integration.
  • Correct a flaky concurrency assertion found by the full suite while retaining FIFO checks for queued subagents.

Testing

  • npm run check
  • ./test.sh
  • node ../../node_modules/vitest/dist/cli.js --run test/proposal-ranking.test.ts test/control-plane.test.ts
  • Repeated test/suite/subagent-startup.test.ts 20 times after correcting its concurrent-start assertion.

- Prevents unverifiable or weakly recurring proposals from reaching mutation planning
- Makes deterministic selection auditable through versioned score components and dispositions
- Keeps accepted but ineligible proposals as a durable no-op instead of forcing a write
- Verifies validation artifacts, citation closure, and the five-selection invariant at pipeline boundaries
- Allows concurrently admitted children to reach the provider in either order
- Continues to verify FIFO admission for queued children as permits become available
@Pepps233 Pepps233 self-assigned this Aug 20, 2026
@Pepps233
Pepps233 requested a review from Issacboy123 August 20, 2026 02:16
@Pepps233
Pepps233 marked this pull request as ready for review August 23, 2026 16:37
@Pepps233
Pepps233 merged commit f222033 into main Aug 23, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant