Skip to content

fix: make visual OCR review text-first - #52

Merged
zyk1172 merged 43 commits into
mainfrom
codex/visual-ocr-reliability
Sep 13, 2026
Merged

fix: make visual OCR review text-first#52
zyk1172 merged 43 commits into
mainfrom
codex/visual-ocr-reliability

Conversation

@zyk1172

@zyk1172 zyk1172 commented Sep 12, 2026

Copy link
Copy Markdown
Owner

Summary

  • change block-scoped visual OCR review from full structured page OCR to text-first transcription
  • preserve the local OCR geometry, bubble geometry, safe region, orientation and role during visual text correction
  • accept coordinate-free JSON and plain-text fallback for region review
  • reject common no-image/refusal responses instead of treating them as OCR text
  • align the visual prompt field contract by including layoutSafeRegion
  • give full-page vision recognition its own JSON Schema when the provider supports structured output
  • align the full-page prompt and strict schema on optional geometry: required keys use null / [] when geometry is unavailable instead of contradicting each other about omitted fields
  • keep the existing response-format fallback for providers that reject structured formats
  • replace the old fake vision connection check (gray image + Return OK) with a real image-ingestion challenge: a random six-character code exists only inside the generated image and the model must read it back correctly
  • when an unknown model passes the real image challenge, mark it as vision-capable in the current provider editor state

Why

Real-device testing shows that local OCR works, while visual recognition has repeatedly ended with no usable text. The previous block-review path already knew the crop geometry but still required the vision model to return precise textBox/bubbleBox data; a model could read the text correctly and still have the entire result discarded for missing/malformed geometry.

A second issue was found during transport review: the settings-page vision test did not prove vision at all. It sent a blank/gray image and asked for OK, so a text-only model or gateway that ignored the image could still be reported as a successful vision connection.

A third contract issue existed in full-page recognition: the prompt told the model to omit some unknown geometry while strict structured output required those same keys. The contract now consistently uses null for unavailable rectangle fields and empty arrays for unavailable polygons.

Transport review

The app-side image payloads are protocol-specific:

  • OpenAI Chat Completions: image_url
  • OpenAI Responses: input_image
  • Anthropic Messages: base64 image

The remaining provider/model uncertainty is therefore now tested with the image-only challenge rather than an HTTP-only success check.

Regression coverage

  • coordinate-free region JSON is accepted
  • plain-text fallback is accepted
  • refusal text such as 无法返回文本 is rejected
  • visual text correction preserves the local OCR geometry
  • recognition prompt contract requests layoutSafeRegion and normalized text geometry
  • recognition prompt explicitly matches the strict schema's null / [] optional-geometry representation
  • vision capability probe accepts the correct image code, including harmless formatting/prose around it
  • vision capability probe rejects OK, refusal text and wrong codes

Validation boundary

CI proves the app-side request/response contract, parser behavior and challenge verifier. The real external provider/model still has to pass the in-app vision connection probe, because CI has no user API key. After that succeeds, retest the same manga page. If the probe fails, its returned excerpt now distinguishes “endpoint reachable” from “model actually read the image” and should be used to correct the selected visual model/API protocol before further OCR tuning.

@zyk1172
zyk1172 merged commit f5c8ddb into main Sep 13, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant