Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
43 commits
Select commit Hold shift + click to select a range
fa83c9a
chore: apply visual OCR reliability patch
zyk1172 Sep 12, 2026
63750df
chore: reset visual OCR patch workflow
zyk1172 Sep 12, 2026
0caaaec
chore: apply visual OCR reliability patch
zyk1172 Sep 12, 2026
1cbf809
chore: reset visual OCR patch workflow v2
zyk1172 Sep 12, 2026
ba126a2
chore: apply visual OCR reliability patch v3
zyk1172 Sep 12, 2026
1690947
chore: reset visual OCR patch workflow v3
zyk1172 Sep 12, 2026
18ec0ad
chore: apply visual OCR reliability patch v4
zyk1172 Sep 12, 2026
cf69ad5
chore: reset visual OCR patch workflow v4
zyk1172 Sep 12, 2026
82e3c79
chore: apply visual OCR reliability patch v5
zyk1172 Sep 12, 2026
a1d59bc
fix: make visual OCR review text-first
github-actions[bot] Sep 12, 2026
7fe5746
test: cover text-first visual OCR review
zyk1172 Sep 12, 2026
9f9dedb
docs: explain visual OCR reliability fix
zyk1172 Sep 12, 2026
83aefa4
test: assert visual OCR contract fields
zyk1172 Sep 12, 2026
da63362
docs: add visual OCR real-model retest checklist
zyk1172 Sep 12, 2026
fecbee4
docs: record visual OCR validation boundary
zyk1172 Sep 12, 2026
74536e5
docs: add provider debug handoff
zyk1172 Sep 12, 2026
a57581d
docs: consolidate visual OCR notes
zyk1172 Sep 12, 2026
78c716a
docs: consolidate visual OCR notes
zyk1172 Sep 12, 2026
974a19f
docs: consolidate visual OCR notes
zyk1172 Sep 12, 2026
0f8996f
docs: add concise visual OCR test plan
zyk1172 Sep 12, 2026
c3a26fb
docs: keep visual OCR notes consolidated
zyk1172 Sep 12, 2026
6d4b5a2
docs: note visual OCR status
zyk1172 Sep 12, 2026
cc2deb0
docs: remove redundant visual OCR status note
zyk1172 Sep 12, 2026
2982e7a
docs: summarize visual OCR PR
zyk1172 Sep 12, 2026
15452b2
docs: remove redundant PR note
zyk1172 Sep 12, 2026
3b67bfb
docs: record CI boundary
zyk1172 Sep 12, 2026
5cde76d
docs: note CI scope
zyk1172 Sep 12, 2026
a4f59c0
docs: keep one visual OCR source of truth
zyk1172 Sep 12, 2026
86811db
docs: keep visual OCR note consolidated
zyk1172 Sep 12, 2026
2d5fa99
docs: consolidate visual OCR notes
zyk1172 Sep 12, 2026
b450039
docs: remove redundant visual OCR notes
zyk1172 Sep 12, 2026
3e513a5
docs: remove redundant visual OCR notes
zyk1172 Sep 12, 2026
8eab36a
docs: remove redundant visual OCR notes
zyk1172 Sep 12, 2026
5722973
docs: remove redundant visual OCR notes
zyk1172 Sep 12, 2026
0f715fc
docs: remove redundant visual OCR notes
zyk1172 Sep 12, 2026
fe93c03
chore: apply real vision connection probe
zyk1172 Sep 13, 2026
2ce3eb4
fix: make vision connection test prove image input
github-actions[bot] Sep 13, 2026
35120c0
chore: remove temporary vision probe workflow
github-actions[bot] Sep 13, 2026
b26b148
docs: document real vision capability probe
zyk1172 Sep 13, 2026
0245fad
chore: align vision recognition prompt with schema
zyk1172 Sep 13, 2026
918c111
fix: align vision recognition prompt with schema
github-actions[bot] Sep 13, 2026
2f92a63
chore: remove temporary vision contract workflow
github-actions[bot] Sep 13, 2026
c78e180
docs: align full-page vision contract
zyk1172 Sep 13, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 27 additions & 0 deletions docs/visual-ocr-reliability.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
# Visual OCR reliability

This change separates two jobs that previously shared one overly strict response contract.

## Region review

When local OCR has already located a suspicious text region, the vision model is now asked only to transcribe the cropped region and return a confidence value. The local OCR geometry, bubble geometry, layout-safe region, orientation and role remain authoritative for that review pass.

A region review therefore no longer fails merely because a model omitted or reformatted `textBox`/`bubbleBox` coordinates.

The parser accepts the preferred coordinate-free JSON response and a plain-text fallback, while rejecting common refusal/no-image messages instead of storing them as OCR text.

## Full-page vision recognition

Full-page recognition still needs normalized geometry. Its system prompt and user contract now agree on `layoutSafeRegion`, and capable transports receive a dedicated recognition JSON Schema rather than only a generic JSON-object request. Existing fallback behavior remains available for providers that reject structured response formats.

The prompt and strict schema now use the same optional-geometry representation: every item includes the geometry keys; `bubbleBox` / `layoutSafeRegion` use `null` when unavailable, while `textPolygon` / `bubblePolygon` use empty arrays when they cannot be determined reliably. This avoids telling the model to omit fields that structured output simultaneously requires.

## Vision connection probe

The settings-page vision test now proves image ingestion instead of only proving HTTP connectivity. It renders a random six-character challenge code into an image, keeps the code out of the text prompt, and only reports success when the model reads the same code back. A text-only model that ignores the image can no longer pass by replying `OK`.

A successful probe also upgrades an `unknown` model descriptor to `supportsVision = true` in the current editor state. A failed probe does not automatically mark the model unsupported because provider outages and temporary model failures can produce false negatives.

## Validation boundary

Unit/CI coverage can verify parsing, geometry preservation, request-contract behavior and the challenge-response verifier. It still cannot prove that a specific external provider/model works until the user runs the visual connection probe and then retests a real manga page in the app.
77 changes: 68 additions & 9 deletions mreader/AIProviderSettingsView.swift
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,28 @@ nonisolated enum AIProviderModelSelectionPolicy {
}
}

nonisolated enum AIVisionConnectionProbe {
static let prompt = "读取图片中央的 6 位大写字母/数字验证码。答案只存在于图片中。只返回你看到的验证码,不要解释。"

static func makeChallengeCode(length: Int = 6) -> String {
let alphabet = Array("ABCDEFGHJKLMNPQRSTUVWXYZ23456789")
return String((0..<max(length, 1)).compactMap { _ in alphabet.randomElement() })
}

static func response(_ response: String, contains challenge: String) -> Bool {
let expected = normalizedASCIIAlphanumerics(challenge)
guard !expected.isEmpty else { return false }
return normalizedASCIIAlphanumerics(response).contains(expected)
}

private static func normalizedASCIIAlphanumerics(_ value: String) -> String {
value.uppercased().unicodeScalars
.filter { $0.value < 128 && CharacterSet.alphanumerics.contains($0) }
.map(String.init)
.joined()
}
}

struct AIProviderSettingsView: View {
@Environment(\.dismiss) private var dismiss
@State private var profiles: [AIProviderProfile] = []
Expand Down Expand Up @@ -490,19 +512,21 @@ private struct AIProviderEditorView: View {
defer { testingKind = nil }
do {
let request: AITransportRequest
let visionChallenge = kind == .vision ? AIVisionConnectionProbe.makeChallengeCode() : nil
let expectedItems = [
AIPageTranslationItem(id: "b0", sourceText: "Hello!", order: 0),
AIPageTranslationItem(id: "b1", sourceText: "Where are you going?", order: 1)
]
if kind == .vision {
guard let imageURL = tinyPNGDataURL() else {
guard let challenge = visionChallenge,
let imageURL = visionProbePNGDataURL(code: challenge) else {
throw AITranslationRequestError.invalidConfiguration("settings.imageEncodingFailed".localized)
}
request = AITransportRequest(
model: modelDescriptor,
userPrompt: "Return OK.",
userPrompt: AIVisionConnectionProbe.prompt,
imageDataURL: imageURL,
maxTokens: 8,
maxTokens: 32,
timeout: AITranslationRequestPolicy.connectionTestTimeout,
kind: .connectionTest
)
Expand All @@ -525,7 +549,30 @@ private struct AIProviderEditorView: View {
)
}
let data = try await AITranslationClient(apiKey: apiKey, baseURL: baseURL).send(request)
if kind == .text {
if kind == .vision {
let decoded = AIChatResponseDecoder.decode(data)
guard let challenge = visionChallenge,
let content = decoded.content else {
testFailed = true
testMessage = "视觉请求已返回,但没有可验证的文本响应。请检查视觉模型和 API 协议。"
HapticManager.shared.play(.error)
return
}
guard AIVisionConnectionProbe.response(content, contains: challenge) else {
testFailed = true
let excerpt = String(content.prefix(160)).replacingOccurrences(of: "\n", with: " ")
testMessage = "视觉接口可连接,但模型没有读出测试图片中的验证码。请检查视觉模型和 API 协议。返回:\(excerpt)"
HapticManager.shared.play(.error)
return
}
if modelDescriptor.supportsVision != true {
modelDescriptors[model] = AIModelDescriptor(
id: modelDescriptor.id,
apiProtocol: modelDescriptor.apiProtocol,
supportsVision: true
)
}
} else {
let decoded = AIChatResponseDecoder.decode(data)
guard let content = decoded.content else {
testFailed = true
Expand Down Expand Up @@ -564,12 +611,24 @@ private struct AIProviderEditorView: View {
}
}

private func tinyPNGDataURL() -> String? {
let size = 32
let renderer = UIGraphicsImageRenderer(size: CGSize(width: size, height: size))
private func visionProbePNGDataURL(code: String) -> String? {
let size = CGSize(width: 360, height: 180)
let renderer = UIGraphicsImageRenderer(size: size)
let image = renderer.image { context in
UIColor.gray.setFill()
context.fill(CGRect(x: 0, y: 0, width: size, height: size))
UIColor.white.setFill()
context.fill(CGRect(origin: .zero, size: size))

let paragraph = NSMutableParagraphStyle()
paragraph.alignment = .center
let attributes: [NSAttributedString.Key: Any] = [
.font: UIFont.monospacedSystemFont(ofSize: 56, weight: .bold),
.foregroundColor: UIColor.black,
.paragraphStyle: paragraph
]
(code as NSString).draw(
in: CGRect(x: 12, y: 52, width: size.width - 24, height: 76),
withAttributes: attributes
)
}
guard let data = image.pngData() else { return nil }
return "data:image/png;base64,\(data.base64EncodedString())"
Expand Down
Loading