feat(ink): add Azure OCR enrichment and settings - #234
Merged
Merged
Conversation
Enrich explicit Ink Query submissions with bounded Azure AI Vision OCR before canonical persistence, with image-only fallback and cancellation support. Add secure owner-only configuration and compact Settings controls, regression coverage, and architecture documentation. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot started reviewing on behalf of
Liangwei Wang (LiangweiOIO)
September 22, 2026 09:43
View session
Contributor
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
Two critical security/configuration issues and two follow-up UI/documentation issues remain unresolved.
Get a fresh assessment by requesting another Copilot review.
Review effort: Lite
Findings: 2
Open (2)
What changed in this PR
Adds optional Azure AI Vision OCR enrichment for explicit Ink Query submissions, with secure configuration, bounded processing, canonical replay, tests, and documentation.
Changes:
- Adds OCR contracts, selected-stroke rasterization, timeout/fallback handling, and prompt integration.
- Adds owner-only Azure endpoint/API-key settings with secure storage and environment fallbacks.
- Updates UI, translations, architecture documentation, regression tests, and
.env.example.
| File | Summary |
|---|---|
packages/shared/src/types/api/ink-ocr-config.ts |
OCR configuration contracts. Critical, 3 votes: HTTPS-only validation permits private or non-Azure hosts, enabling SSRF and key exfiltration. |
packages/shared/src/types/api/index.ts |
Exports OCR schemas. |
packages/shared/src/types/api/chat-envelope.ts |
Adds recognition envelope data. |
packages/shared/src/types/api/chat-envelope.test.ts |
Tests recognition bounds. |
packages/shared/src/types/agent/chat.ts |
Re-exports recognition types. |
docs/proposals/ink-lasso-question-flow.md |
Documents OCR behavior. Nit, 1 vote: Mark the shipped proposal as Shipped. |
docs/architecture/sketch-node.md |
Documents submission OCR behavior. |
docs/architecture/deployment-security.md |
Documents outbound OCR security. |
docs/architecture/credential-storage.md |
Documents credential persistence. |
docs/architecture/agent-context.md |
Documents OCR prompt context. |
apps/web/src/i18n/resources/zh-CN/common.json |
Adds Chinese OCR strings. |
apps/web/src/i18n/resources/en/common.json |
Adds English OCR strings. |
apps/web/src/components/Settings/SettingsModal.tsx |
Adds the OCR settings section. |
apps/web/src/components/Settings/sections/InkOcrSettings.tsx |
Implements the OCR settings editor. Nit, 1 vote: Preserve actionable API validation errors. |
apps/web/src/components/Settings/sections/InkOcrSettings.test.tsx |
Tests the settings UI. |
apps/web/src/api/inkOcr.ts |
Adds the settings API client. |
apps/web/src/api/_routes.ts |
Registers the OCR route. |
apps/server/src/security/secret-store.test.ts |
Tests encrypted OCR key storage. |
apps/server/src/security/secret-ids.ts |
Adds the OCR secret identifier. |
apps/server/src/security/environment-secret-store.ts |
Adds the environment key fallback. |
apps/server/src/modules/integrations/integrations.route.ts |
Mounts OCR configuration routes. |
apps/server/src/modules/integrations/ink-ocr-config.ts |
Resolves and persists OCR configuration. Critical, 1 vote: A failed endpoint write can leave a newly persisted key paired with the old endpoint; add rollback or transactional consistency. |
apps/server/src/modules/integrations/ink-ocr-config.route.ts |
Provides owner-only OCR configuration APIs. |
apps/server/src/modules/integrations/ink-ocr-config.route.test.ts |
Tests OCR configuration APIs. |
apps/server/src/modules/canvas/snapshot-nodes.ts |
Builds pure-Ink OCR rasters. |
apps/server/src/modules/canvas/snapshot-nodes.test.ts |
Tests raster generation and geometry. |
apps/server/src/modules/agent/conversation/transcript/history.test.ts |
Tests persisted recognition metadata. |
apps/server/src/modules/agent/conversation/prompt/ink-ocr.ts |
Escapes and renders OCR evidence. |
apps/server/src/modules/agent/conversation/prompt/ink-intent.test.ts |
Tests prompt and replay behavior. |
apps/server/src/modules/agent/conversation/prompt/build-prompt.ts |
Adds OCR evidence to canonical prompts. |
apps/server/src/modules/agent/conversation/ink-ocr.ts |
Implements the Azure OCR adapter and deadlines. |
apps/server/src/modules/agent/conversation/ink-ocr.test.ts |
Tests OCR outcomes, timeout, and cancellation. |
apps/server/src/modules/agent/conversation/envelope.ts |
Integrates OCR into envelope preparation. |
apps/server/src/modules/agent/conversation/envelope.test.ts |
Tests envelope OCR integration. |
apps/server/src/modules/agent/agent.route.ts |
Propagates preparation cancellation. |
.env.example |
Documents OCR environment variables. |
💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Copilot started reviewing on behalf of
Liangwei Wang (LiangweiOIO)
September 23, 2026 09:59
View session
Contributor
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
Three unresolved moderate findings concern cancellation propagation, unnecessary full-canvas cloning, and endpoint-specific validation guidance.
Get a fresh assessment by requesting another Copilot review.
Review effort: Lite
Findings: 1
Resolved since last review (2)
Comment on lines
+60
to
+64
| if (!parsed.success) { | ||
| return reply.code(400).send({ | ||
| message: 'Invalid Ink OCR configuration', | ||
| code: 'validation_failed', | ||
| }); |
Collaborator
Author
There was a problem hiding this comment.
Fixed in 1a5c12e (amended into the existing fix commit). The route now returns the first safe schema issue message with the generic fallback, so invalid endpoints surface the public-cloud/root-URL guidance in the existing settings UI. Unknown-key issues are excluded because their messages echo caller-controlled property names. Added regressions for actionable endpoint guidance, no credential/property-name reflection, and missing-update guidance. Validation passed: 55 route tests, 28 settings UI tests, repository formatting/lint/typecheck, and i18n parity. Please re-review the updated implementation.
Store endpoint and key overrides as one encrypted, serialized configuration record with lazy legacy migration. Validate public Azure resource roots before saving or fetching, require writable secure storage for all updates, and preserve actionable validation errors. Add regression coverage and update settings copy and documentation. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Liangwei Wang (LiangweiOIO)
force-pushed
the
feat/ink-query-ocr
branch
from
September 24, 2026 03:42
d3ae97c to
1a5c12e
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Behavior and boundaries
Validation
pnpm typecheckpnpm formatpnpm lint:fix— 0 errors, 291 existing warningsValidation used the existing local dependency workaround; dependency manifests and lockfile changes are not part of this PR. The full repository test suite was not rerun for this final pass; an earlier run had an unrelated
external-watcher.real.test.tsfailure while waiting for an empty snapshot after directory deletion. Live Azure recognition accuracy and latency have not been benchmarked as part of this validation.