Skip to content

Research lane: stop failing digests that are right (quotes its page, honest none, empty page) - #858

Merged
dmmdea merged 1 commit into
mainfrom
fix/research-acceptance-false-rejections
Nov 12, 2026
Merged

dmmdea merged 1 commit into
mainfrom
fix/research-acceptance-false-rejections

Conversation

@dmmdea

@dmmdea dmmdea commented Nov 12, 2026

Copy link
Copy Markdown
Owner

Intent

The failure register's largest class with no cause was failed_verification (357 rows in 30 days). Reading the saved pages and answers showed that most of those rows are the research lane's own checks failing digests that were right. This change stops the three checks from doing that.

Scope

In: internal/delegate (the fingerprint's excuses), internal/research (the derived check, the empty-page refusal), one new test, two tests that pinned the old check, docs/systems/mcp-server.md, a changelog fragment.
Out: the fingerprint regex itself, the ledger classes (a companion pull request records the cause), any caller-written acceptance.

What changed

  • A digest that quotes its page is excused from the page-word tripwire. delegate.DigestGrounded: the decoded digest holds one string of at least 60 characters, or two different strings of at least 15, found in the contract's own documents and not in the goal. It lives beside the honest-absence excuse, so the merge gate, the quarantine and the retry rule read one answer. It only excuses: the node is neither struck nor counted as passing.
  • The derived check is any-of, not an item count. For a caller's schema that lists an array in required, research.Build writes nonempty:<f1>|<f2>|... over the required fields in place of min_items:<first required array>:1. A digest silent in every required field still fails. A caller who needs items writes min_items: themselves.
  • A page with no content is not dispatched. An HTML page under 150 characters of text is a skipped source with error_kind empty_page. JSON, plain text, feeds and subtitle tracks are never refused by length.

The real run and its output

An offline replay of 30 days of saved pages and answers through research.Build and delegate.EvalAcceptanceGraded, read-only. The original code reproduces every saved verdict (356 fail, 4,622 pass).

the 356 failed rows                         rows  now pass  refused  still fail
  digest holds strings of its page           187       125        0          62
  digest says the page lacks it, and claims   67         0        0          67
  seat could not read                          3         0        0           3
  digest all empty                             2         0        0           2
  residual                                    23         0        0          23
  item count, answer says the page lacks it   45        39        0           6
  shell page under 150 characters             16         0       16           0
  item count, other                            3         3        0           0
  other doors (not research)                  10         0        0          10
  total                                      356       167       16         173

the 4,622 passed rows: 4,513 still pass, 109 refused as pages with no content, 0 fail

Planted negatives made from real rows, old code against new, at the tripwire:

planted kind                                      n   old accepts   new accepts
digest of another page, other site              200           115           115
digest of another page, same site               100            83            83
quotes altered so they are not on the page      176            20            20
silent empty digest, no prose                   200             6             6
silent empty digest, with a closing word        200             0             0
goal echoed back                                100             0             0
false "none" for a page that holds the items    200           200           200

The grounding rule newly accepts none of them. Across 14,904 pairs of a digest with an unrelated page of another site, 7 are grounded (0.05%); across 11,186 same-site pairs, 1,281 (11%: shared footers, price tables, one page under two addresses).

Tests, each run once: go build ./... and go vet clean; go test ./internal/delegate/ ./internal/research/ pass; the root Changelog|Fragment|Leak|Docs subset passes. The new test fails with the grounding excuse reverted (the quotes case) and with the item count restored (the "none" case).

Blast radius

Every research call. Results that used to come back failed_verification for the three reasons above now come back as digests, or as a skipped source for a page with no content. No ledger class changes; a skipped source writes no ledger row, as before.

Risk

  • A seat that says "none" for a page that holds the items is believed. That was already so under the default schema (176 of 200 planted); under a caller's schema the item count used to fail 24 of 200, and no longer does.
  • The grounding excuse is no proof of a page against its own site (11% of same-site pairs). The old tripwire already accepted 83 of 100 same-site wrong-page digests, so nothing that was rejected there is newly accepted.
  • Of 109 passing pages now refused, about 105 are cookie gates, bot checks, titles and login walls; a few are short real text (a batch file, an abstract, a heading).

🤖 Generated with Claude Code

…s page, an honest "none", a page with no content

In the 30 days to 2026-11-12, 346 of the 356 failed_verification rows with a job id were
research pages, and three checks the lane writes itself were failing answers that were right:

- The page-word tripwire (docanchor) asks the final prose to restate one of twenty-four page
  words that the goal does not contain. A digest of a narrow goal restates the goal's words and
  the page's figures, so 282 digests failed it, and 187 of them hold strings found on the page.
- A caller's schema that lists an array in required got min_items:<first required array>:1,
  which failed the digest the goal asks for when the page has no such content (an empty list and
  a sentence that says so): 70 digests, 64 of which failed nothing else.
- A page with no content (a title, a cookie gate, a login wall) was dispatched as a full run:
  16 of the failures and 109 of the passes were HTML pages under 150 characters.

Changes:

- delegate.DigestGrounded excuses a failed fingerprint, in the one place the honest-absence
  excuse already lives (excusesFingerprint), when the decoded digest holds one string of 60+
  characters or two different strings of 15+ that are found in the contract's context documents
  (quotecheck.Locate) and are not in the goal. Thresholds come from a grid scored on the failures
  and on 26,000 digest-and-page pairs: an unrelated page of another site is grounded in 7 of 14,904
  pairs (0.05 %); same-site pairs 11 % (shared footers, one page under two addresses).
- research.Build derives nonempty:<f1>|<f2>|... over the caller's required fields instead of the
  item count, for a schema that lists an array in required and for no other. It can only fail
  what the item count failed, and a digest silent in every required field still fails.
- research.Build refuses an HTML page under research.MinPageRunes (150) as a skipped source with
  error_kind empty_page; JSON, plain text, feeds and subtitles are never refused by length.

Check: offline replay of the saved pages and answers through research.Build and
delegate.EvalAcceptanceGraded. The original code reproduces every saved verdict (356 fail, 4,622
pass). With the change: of the 356, 167 pass, 16 are refused as empty pages, 173 still fail; of
the 4,622 passes, 4,513 still pass, 109 are refused as empty pages, none fails. Planted negatives
from real rows (another site's page, altered quotes, silent empty digest, goal echoed back) gain
no acceptance from the grounding rule. Known limit, now for caller schemas too: a seat that says
"none" for a page that holds the items is believed.

One test drives the public path (Build, then EvalAcceptance) with a page and a digest per case.
Two tests that pinned the item-count check now pin the any-of check. Docs: docs/systems/mcp-server.md.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@dmmdea
dmmdea merged commit b550423 into main Nov 12, 2026
6 checks passed
@dmmdea
dmmdea deleted the fix/research-acceptance-false-rejections branch November 12, 2026 18:00
@dmmdea dmmdea mentioned this pull request Nov 12, 2026
dmmdea added a commit that referenced this pull request Nov 12, 2026
Two changes, each merged to main through its own pull request (#858, #859)
on green CI:

- The research lane stops failing digests that are right. A digest that
  quotes its page is excused from the page-word tripwire; the check derived
  from a caller's schema is "a value in any required field", not an item
  count on the first required array; an HTML page under 150 characters of
  text is a skipped source and is not dispatched. An offline replay of 30
  days of saved pages and answers: of 356 failed rows 167 now pass, 16 are
  refused as pages with no content and 173 still fail; of 4,622 passes
  4,513 still pass, 109 are refused as pages with no content, none fails.
- The failure register files the misread rows under the class that names
  them and records the cause of the four classes it had not investigated
  (failed_verification, other, poll_deadline, stall_cold_load). On the real
  ledger, 30 days: other 30 to 3, poll_deadline 3 to 0, with the totals
  unchanged; no class is left marked not investigated.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

Deletes: changes/unreleased/fix-register-rules-and-causes.md, changes/unreleased/fix-research-acceptance-false-rejections.md

Co-authored-by: dmmdea <dmmdea@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant