Skip to content

test: add human gold review gate and trusted scorer - #50

Merged
zyk1172 merged 21 commits into
mainfrom
codex/translation-gold-review
Sep 12, 2026
Merged

test: add human gold review gate and trusted scorer#50
zyk1172 merged 21 commits into
mainfrom
codex/translation-gold-review

Conversation

@zyk1172

@zyk1172 zyk1172 commented Sep 12, 2026

Copy link
Copy Markdown
Owner

Summary

  • add an explicit human-review receipt to page-level Gold annotations
  • require resolved page state, structural validity and a real review receipt before page Gold can become reportable
  • bind trusted scoring to the SHA-1 of the exact reviewed source image; a mismatched image is rejected
  • add a trusted baseline scorer for detection recall, mean region CER, reading order and bubble grouping
  • make the scorer throw instead of emitting metrics for candidate/non-reportable annotations
  • add and refresh a numbered Shirohage review overlay
  • correct the original 9-region machine candidate into 10 visually reconstructed vertical text columns based on the user-provided review screenshot
  • remove face/newspaper false positives, rebuild fragmented right/lower-left narration, and restore the missed middle column in the left-top speech balloon
  • keep the Shirohage fixture candidate, expectedPageState: unknown, review: null, and manifest status pending

Corrected candidate

The current candidate reconstructs these three visible text blocks:

  • 「有名人」の基準は、「ウィキペディアに記事があるか」です。
  • ウィキペディアにのってたら有名人です。
  • ウィキペディアにのってないのに有名人のようにふるまうのは自惚れです。

The three left-top columns share one physical speech-balloon ID. The right and lower-left rectangular narration boxes deliberately have no bubbleID. Decorative face/newspaper marks are excluded.

Human-review boundary

This PR still deliberately does not promote the Shirohage page to humanVerified and does not claim OCR accuracy from a model-created/corrected candidate. The user-provided screenshot was used as review evidence to fix obvious false positives, fragmentation and missing columns, but final character-by-character/rectangle confirmation against the pinned source image is still required before a review receipt can be recorded.

Pinned Shirohage source image SHA-1:
8c828fc750e946ce94038a00782cbe115537c15e

Review aid:

  • docs/shirohage-gold-review.md
  • docs/shirohage-gold-review-overlay.png

Regression coverage

  • candidate Gold may be structurally valid but remains non-reportable
  • humanVerified without a review receipt is rejected
  • malformed regions/order/reference mappings are rejected
  • source-image SHA mismatch is rejected by trusted scoring
  • trusted scorer rejects candidates
  • a synthetic matching human-reviewed Gold sample produces deterministic perfect detection/CER/order/grouping metrics
  • pending Shirohage fixture is explicitly asserted to have no review receipt

Scope note

An unrelated local fix for a one-page reader startup crash briefly landed on this branch. It has been removed from this PR's net diff and preserved separately in PR #51.

Next handoff

After final human confirmation of the corrected 10-region candidate, a small follow-up can record the receipt, resolve page state, promote the fixture to ready, and compute the first reportable real-page benchmark numbers.

@zyk1172
zyk1172 merged commit b5d2185 into main Sep 12, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant