diff --git a/CLAUDE.md b/CLAUDE.md index ce19e4e4..f6142b3c 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -416,7 +416,7 @@ Video/webcam -> facial morph + head + skeletal body animation, built on the exis - **LLMManager** (`src/LLMManager.h/cpp`): QML_SINGLETON wrapping llama.cpp for local inference. - **LLMWorker**: Runs inference in a worker thread. - **ModelDownloader**: Downloads GGUF/ONNX models from HuggingFace — the ONE download path behind all 21 model consumers. **#1029 (CWE-494) hardening:** `startDownload` **refuses any URL that is not `https://` or a host-less `file://`** (plain http is a byte-for-byte MITM injection point, and every base URL is user-overridable via env/QSettings; `file:///share/…` is refused too — on Windows that is a UNC/SMB fetch over the network wearing a local scheme; only an empty or `localhost` host is accepted, and the error names the reason) — refused BEFORE any filesystem side effect, so no `.part` or directory is created. An optional 4th argument `expectedSha256` (case-insensitive hex; empty = legacy no-check, so the 21 consumers + the two QML call sites in `AISettingsDialog.qml` work unchanged) is verified by a local streamed `QCryptographicHash` helper **on the finished `.part` on disk, before the rename** (deliberately NOT the updater's `UpdateVerifier::sha256HexOfFile`: `qtmesh_updater` is only built/linked under `ENABLE_AUTO_UPDATER`, while `ModelDownloader` compiles unconditionally — reusing it broke the `-DENABLE_AUTO_UPDATER=OFF` link, caught in review; six lines of Qt API beat coupling every model download to the optional updater + libsodium) — never as a running hash in `onReadyRead`, because a resumed download appends to bytes this process never saw. On mismatch the `.part` is **deleted** (a poisoned partial must not be resumed from or cached) and `downloadError` fires. **Two gotchas verified empirically:** (1) the scheme refusal is emitted **QUEUED** (`QMetaObject::invokeMethod` + `Qt::QueuedConnection`), never synchronously — consumers connect, call `startDownload`, then `loop.exec()`, so a synchronous `loop.quit()` fires before `exec()` and is lost, hanging them for their full timeout (the #1017 review race); queued delivery lands inside `exec()` for every consumer without touching any — proven on `PhotoDepth`, which has no `settled` guard, failing in 1s instead of 600s. (2) Both refusals log at **`qCritical`**, not `qWarning`: `cliMessageHandler` drops warnings unless `--verbose`, and every consumer discards `downloadError`'s text (#1037), so a warning would leave the user with "offline?" and no way to learn the cause. **Resume verification (#1036):** a resume sends `Range: bytes=N-`, but a server that ignores it (`file://` always; any proxy that strips the header) answers **200 with the whole body**, and appending that after the stale `.part` produced a corrupt model — reproduced: a 29-byte stale prefix yielded a 208,044,845-byte file that **still loaded and ran** (ORT parsed the garbage as an unknown protobuf field) while a 30-byte prefix failed with "Protobuf parsing failed"; load success proves nothing about integrity. So on the FIRST `readyRead` of a resumed request (`m_resumeUnverified`, armed at both Range-request sites) the downloader requires **206 + a `Content-Range` starting exactly at the resume offset**; anything else is treated as the full body — the `.part` is reopened `Truncate`, `m_resumeOffset`/`m_bytesReceived` reset to 0 (else progress adds a phantom offset), and the download continues from byte 0 with no error (a 200 is recoverable). The range unit is compared **case-insensitively** (RFC 9110 §14.1 — `Bytes 9-12/13` is a valid honoured resume; a case-sensitive match misread it as "ignored" and fell into the truncate path with a PARTIAL body, i.e. an incomplete file a no-digest caller would rename — caught in review). A 206 whose window is neither ours nor the whole resource is a genuinely partial body we did not ask for: writing it from byte 0 would yield an INCOMPLETE file, so that case **aborts** with `downloadError` and removes the `.part` (next attempt starts clean); only a 206 covering exactly `0..total-1` — a full body wearing a partial status — is truncated-and-taken like a 200. Checked once, not per chunk. `FakeNetworkReply` in the tests defaults to a plain 200 (what an ignoring server returns) and `withPartialContent(first,last,total)` models an honoured resume. Both guards mutation-verified with correct selectivity. `ModelDownloader::isAllowedDownloadUrl(url)` is the pure predicate for checking a base URL up front. Tests: `ModelDownloader_test.cpp` (both guards mutation-verified). **Review round on the resume fix (#1039):** (1) *honoured* now also requires the window to REACH THE END — `first == offset && last == total-1` — because `bytes 9-10/13` starts right yet leaves 11-12 missing, and appending it would have promoted an incomplete file (that finding had been marked addressed by the bot without the code changing; verify against the code, not the bot's annotation); (2) every 'this partial is unusable' exit goes through ONE path, `discardPartialAndFail` — remove-or-truncate the `.part`, reset `m_bytesReceived`/`m_resumeOffset` (a stale offset would make `resumeDownload` re-request the old range against a fresh file), abort the reply under `m_abortingInternally` so the synchronously-delivered `errorOccurred`/`finished` step aside, end the download, emit exactly ONE `downloadError` (the test's `FakeNetworkReply::signalOnAbort` models the synchronous delivery); (3) `onDownloadFinished` refuses to promote when `m_resumeUnverified` is still set (a resume reply that finished without ever delivering data — verification never ran) or when the `.part` size differs from the size the response committed to (`m_expectedTotalBytes`: the 206 total, else a full body's Content-Length) — the `.part` is KEPT as a valid prefix for the next resume, not discarded like a digest mismatch. A server that declares no size, with no digest configured, is accepted with a `qWarning` (chunked transfer; HF/GitHub and QNAM's `file://` backend always send Content-Length, so real downloads take the strict path). -- **AI agent harness** (`src/AIAgentManager.{h,cpp}`, `src/AIAgentTypes.{h,cpp}`, `src/AICapabilityRegistry.{h,cpp}`, #1000/#1001/#1002/#1003 + #1021 a–d): the orchestration layer above `AIChatManager`/`LLMManager` — `User → AIAgentManager → Planner → Capability router → Executor → Observer → Replan/Finish`. **All task state lives in `AIAgent::Plan`/`Step`/`Observation`, never in the prompt**; the prompt is rebuilt from that state on every planner call, so nothing scrolls out of a history window and the whole state machine (`Planning → Executing → Observing → Replanning/AwaitingConfirmation → Completed/Failed/Cancelled`) is driven headless in `AIAgentManager_test.cpp` by a scripted `AgentPlannerBackend` + `AgentToolExecutor` (no LLM, no Ogre). **Capabilities, not a catalog:** `AICapabilityRegistry` groups the ~170 MCP tools into 20 capabilities (scene, scene_io, view, materials, lighting, textures_ai, mesh_optimize, uv, rigging, segmentation, generation_3d, animation, motion_ai, morph_pose, node_animation, paint, mocap, cloud, ps1, other — `taxonomy()` + prefix rules; an unmapped tool lands in `other`, never dropped). The planner sees a ~20-line capability index plus the FULL per-tool docs only for the capabilities `routeByKeywords()` pre-selected; it may answer `{"need_capabilities":[...]}` to have more docs added before planning (dynamic discovery — the v1 loop's hard-coded 19-tool subset silently excluded rigging/segmentation/generation). Tool docs are generated from the live `buildToolsList()` schema (name, one-line description, params with type/required/enum), so they cannot drift. **Constrained protocol (#1003):** `validateArguments` checks every planned call against the schema BEFORE it reaches the server — required present, types checked, chatty-model output coerced (`"2"`→2, `"yes"`→true, `"2, 2, 2"`→array) with warnings, enums enforced; an invalid call becomes an `invalid_arguments` observation and a replan, never a tool call. **Observations** (`observationFromToolResult`) parse facts out of tool text (`Vertices: N`, JSON `boneCount`, `fallbackReason`→warning), collect file artifacts, keep `raw` for the transcript only — the replan prompt gets one compact line per observation. **Recovery:** a failing step is retried once (same call), then the planner is asked to repair the tail (`{"steps":[...]}` or `{"done":true,"summary"}`), bounded by `Limits` (12 steps, 2 replans, 2 planner retries); the same `Step::signature()` failing twice stops the task ("stuck, not unlucky"). **One undo group per task** (`QUndoStack::beginMacro(plan.title)` on the first non-read-only step, `endMacro` on every terminal path — `isReadOnly()` = get_*/list_*/toggle_*/screenshots/validators). **Safety rail (#1021d):** `destructiveReason()` names deletes/geometry rewrites, overwrites of an EXISTING file, and OUTBOUND/account actions (`cloud_upload` — the data leaves the machine —, `cloud_login`, `cloud_logout`; review finding: nothing local is destroyed, so the delete/overwrite rules let the upload through); unless `trustedMode` (QSettings `ai/agentTrustedMode`) the task pauses in `AwaitingConfirmation` and the panel shows Allow / Always allow / Skip step. **Scene context per turn (#1021c):** `setContextProvider` — `AIChatManager::sceneSummaryForAgent()` (scene info + user materials + recent files) is appended to every planner prompt. The final message is the deterministic `summarize()` (steps, statuses, parsed facts, artifacts, warnings), so a flaky model cannot misreport what happened; a `{"summary"}` reply with no steps answers a question without tools. **Facade:** `AIChatManager::agentMode` (QSettings `ai/agentMode`, default ON) routes `sendMessage` to `AIAgentManager::startTask`; while the agent drives, the v1 slots ignore `LLMManager` callbacks (`m_agentDriving`) and `LlmPlannerBackend` forwards generation signals only while it has a pending request. QML: `qml/AIChatPanel.qml` gained the agent/ask-trusted toggles, a live plan card, the confirmation bar, and a model tip (#1021e: Qwen 2.5 7B Q4_K_M is the recommended tool-calling model — its `ModelInfo` description says so). Sentry `ai.agent.plan|step|retry|replan|verify|confirm|done|fail|cancel`. **Field findings from the first real sessions (Qwen3 4B):** (1) ~25 MCP tools act on the CURRENT SELECTION (`auto_rig`, `compute_skin_weights`, `validate_mesh`, `generate_lods`, `auto_uv_unwrap`, `retopologize`, `remove_skeleton`, …) and the agent had no way to set it — every rig request died with "No mesh selected"; there is now a **`select_entity {name}`** MCP tool (node or entity name; empty clears), `get_scene_info` ends with a `Selected: …` line, `auto_rig`'s error names the fix, and planner rule 6 says to select first. (2) **Conversation memory:** the agent keeps the last 12 turns (`request → outcome first line [objects touched]`) and injects the last 6 plus an "objects from earlier turns" list into every planner prompt, so "now make it red" resolves against the previous task; `AIChatManager::clearHistory` clears it. (3) Colour NAMES where an `[R,G,B]` array is expected (`"diffuse": "red"`) are coerced with a warning; rule 7 spells out the create_material → apply_material recipe. (4) **Argument aliases:** the model wrote `material_name` and the validator rejected `apply_material` twice for a missing `material` although the MCP handler itself accepts `material_name` — the validator must never be stricter than the tool, so `normaliseAliases` maps camelCase and a small synonym table (`material_name→material`, `entity/entity_name/node→mesh`, `output/path→output_path`, `skeleton→template`, …) onto the schema's names BEFORE the required check, and a rejection lists the keys that were passed; the replan prompt shows the failing call's arguments and says "do not resend it unchanged". (5) **Chat dock focus:** clicking the QML input after another dock (a QQuickWidget) held focus re-focused the QML item but not the hosting widget, so keystrokes went elsewhere until a detour via the viewport; `ClickFocusFilter` (installed on the chat QQuickWidget) turns every press into `setFocus`. (6) **Context window:** switching to another model produced "Failed to decode prompt" — `LLMWorker`'s prompt-too-long pre-check compared against the *configured* context size while the context actually created is clamped to the model's training limit, so an oversized prompt reached `llama_decode`. The worker now checks against `llama_n_ctx` (with a clear message naming both numbers) and emits `contextReady(nCtx)` → `LLMManager::effectiveContextSize`; the agent budgets every planner prompt against it (`systemPromptWithinBudget`: drop history → keep only the most relevant capabilities → truncate the scene listing → hard cut, each step traced), and the default `contextSize` is 8192 (was 4096 — the tool docs alone need more). Changing the context size needs a model reload. (7) **Mesh fed to the image tool:** the 14B planner passed an `.obj` as `image_path` to `generate_mesh_from_image`; the schema now says "2D IMAGE … NOT a 3D mesh: use load_mesh", the handler refuses mesh extensions with the right tool named, the capability index says "not for existing meshes", and `rejectMeshAsImage` in the validator stops any image-typed parameter (`image_path`/`image`/`photo`/`texture`…) carrying a mesh path before the tool runs. (8) **LLM tab parity:** `LLMManager::deleteModelFile(fileName)` / `deleteAllModelFiles()` (models-directory only — paths are refused; unloads the active model first; removes the `.part` too) behind per-row Delete / Remove All / Open Folder in AI Model Settings, mirroring the QtMeshEditor Models tab; the chat header's model chip opens that dialog (`AIChatManager::openModelSettings` → `MainWindow::showAIModelSettings`). (10) **Invented image paths:** asked to "create a f22 raptor scene", the planner called `generate_mesh_from_image` with `~/Downloads/f22_raptor.png` (a file that never existed), the tool answered "image not found", and every repair round guessed another path (.jpg, .png again) — six failures, nothing made. The harness now repairs this itself BEFORE the call (`AIAgentManager::repairMissingImageInput`, in `executeNext` after coercion): a non-existent `image_path` is dropped and, when no `prompt` was given, the request's subject (`subjectFromGoal`: leading creation verbs/articles and a trailing "scene"/"model" stripped — "create a f22 raptor scene" → "f22 raptor") becomes the tool's text prompt (text → image → 3D), with a transcript note. Planner rule 8 says never to invent paths and when to use `prompt`; the replan prompt adds an explicit "that file does not exist — do NOT guess another path" hint whenever a step failed with not found / does not exist / no such file; the schema text for `image_path` says the same. Text → image still needs a stable-diffusion build + FLUX.2-klein (AI Model Settings) — without them the ONE remaining call fails with that actionable message instead of a path hunt. (11) **A heavy tool froze the whole UI.** The agent drives tools SYNCHRONOUSLY on the main thread, so `generate_mesh_from_image` (minutes of ONNX work) blocked the event loop: the window stopped painting and looked hung. The heavy work CANNOT simply move to a worker — `MeshGenBuilder::buildSceneNode` is Ogre and main-thread-only — so instead `MCPServer` gained a **`toolProgress(tool, stage, done, total)`** signal that the generation tool drives from `MeshGenPredictor::predict`'s existing `ProgressFn` (it already fires many times per stage and doubles as the cancel hook). The same callback pumps `processEvents(ExcludeUserInputEvents, 10)` at ~20 Hz — the window keeps painting, and excluding user input means no click can re-enter a tool mid-run. The image phase (FLUX via `generateSourceImageFromPrompt`) already spun nested `QEventLoop`s, so it never froze; its `SDManager::generationProgressChanged` ticks are relayed to the same signal so the bar moves there too. `McpToolExecutor` connects the signal to `AIAgentManager::reportToolProgress` → the `stepProgressLabel` / `stepProgress` properties (ignored when the agent is not busy; cleared around every `callTool`), and `qml/AIChatPanel.qml`'s plan card draws a labelled bar (indeterminate when `total <= 0`). (12) **Stop must reach a running heavy tool.** The first cut of (11) pumped with `ExcludeUserInputEvents`, which kept the window painting but made the Stop button unclickable — and the panel had no Stop button at all, so a multi-minute generation could not be aborted. The pump now delivers `AllEvents` (re-entry is prevented by the agent running one step at a time, not by dropping clicks), `MCPServer::requestToolCancel()` sets a flag the progress callback returns as `false` (→ `predict()` returns `cancelled`) and which is also checked between the image and 3D phases, `AgentToolExecutor::cancelRunningTool()` (overridden by `McpToolExecutor`) is called from `AIAgentManager::cancel()` while Executing, and `qml/AIChatPanel.qml` has a Stop button next to the thinking dots. The plan card also stays visible after the run (its `planCardPinned` guard was never set by anything, so the card vanished with the final result still unread). (13) **Window ACTIVATION gates more of QWidget than it looks — this test failed CI twice.** `ClickFocusFilter_test` first asserted a `FocusIn` count, then (the "fix") `hasFocus()`; both pass on a developer desktop and fail the Linux lane. Qt gates BOTH on the window being ACTIVE, and a headless CI display (Xvfb, no window manager) never activates one: `QWidget::hasFocus()` is `window()->focusWidget() == this && window()->isActiveWindow()`, and `QFocusEvent`s are not delivered at all. **`window.focusWidget()` is the one thing that tracks focus regardless** — assert that, and gate anything else behind `isActiveWindow()`. **`QT_QPA_PLATFORM=offscreen` does NOT reproduce this** (it reports `isActiveWindow() == true`, which is why both bad versions passed locally); a window that is never `show()`n does, on any platform — `ClickFocusFilter.FocusWidgetIsSetEvenWhenTheWindowIsNotActive` pins it that way. Rule for new widget tests: assert `window.focusWidget()`, never `hasFocus()` or focus-event counts, unless gated on activation. (13-old) **A focus-event assertion is not portable.** `ClickFocusFilter_test` asserted a fresh `FocusIn` after the filter re-asserts focus on an already-focused widget. Qt dispatches `QFocusEvent` only inside an **ACTIVE window**, and a headless CI display (Xvfb, no window manager) never activates one — the focus WIDGET still changes but no focus event is delivered, so the test passed on a developer desktop and failed the Linux lane (and the "5954/5955 discovered tests executed" coverage guard failed with it, i.e. one root cause counted as two suite failures). The test now asserts the `clearFocus()`+`setFocus()` PAIR via the focus-OUT count (only `clearFocus()` can produce it) and guards the event-count assertion behind `window.isActiveWindow()`. Rule for new widget tests: assert focus STATE (`hasFocus`, `window.focusWidget()`), not focus EVENT counts, unless the assertion is gated on window activation. (9) **Trace log:** every task overwrites `/ai_agent/last_task.log` (`AIAgentManager::traceLogPath()`) with the planner prompts, raw replies and raw tool results — read THAT when a task fails, the chat transcript only shows first lines. **Test gotcha:** fixture tests are `TEST_F(AgentFixture, …)` — a `--gtest_filter='AIAgent*'` does NOT run them; use `AgentFixture*`. Follow-ups (children of #1000): #1004 local VLM viewport inspection (mtmd is already linked for the captioner), #1005 model lifecycle/memory budget, #1006 verifier layer, #1007 e2e benchmark, #1008 batch factory. +- **AI agent harness** (`src/AIAgentManager.{h,cpp}`, `src/AIAgentTypes.{h,cpp}`, `src/AICapabilityRegistry.{h,cpp}`, #1000/#1001/#1002/#1003 + #1021 a–d): the orchestration layer above `AIChatManager`/`LLMManager` — `User → AIAgentManager → Planner → Capability router → Executor → Observer → Replan/Finish`. **All task state lives in `AIAgent::Plan`/`Step`/`Observation`, never in the prompt**; the prompt is rebuilt from that state on every planner call, so nothing scrolls out of a history window and the whole state machine (`Planning → Executing → Observing → Replanning/AwaitingConfirmation → Completed/Failed/Cancelled`) is driven headless in `AIAgentManager_test.cpp` by a scripted `AgentPlannerBackend` + `AgentToolExecutor` (no LLM, no Ogre). **Capabilities, not a catalog:** `AICapabilityRegistry` groups the ~170 MCP tools into 20 capabilities (scene, scene_io, view, materials, lighting, textures_ai, mesh_optimize, uv, rigging, segmentation, generation_3d, animation, motion_ai, morph_pose, node_animation, paint, mocap, cloud, ps1, other — `taxonomy()` + prefix rules; an unmapped tool lands in `other`, never dropped). The planner sees a ~20-line capability index plus the FULL per-tool docs only for the capabilities `routeByKeywords()` pre-selected; it may answer `{"need_capabilities":[...]}` to have more docs added before planning (dynamic discovery — the v1 loop's hard-coded 19-tool subset silently excluded rigging/segmentation/generation). Tool docs are generated from the live `buildToolsList()` schema (name, one-line description, params with type/required/enum), so they cannot drift. **Constrained protocol (#1003):** `validateArguments` checks every planned call against the schema BEFORE it reaches the server — required present, types checked, chatty-model output coerced (`"2"`→2, `"yes"`→true, `"2, 2, 2"`→array) with warnings, enums enforced; an invalid call becomes an `invalid_arguments` observation and a replan, never a tool call. **Observations** (`observationFromToolResult`) parse facts out of tool text (`Vertices: N`, JSON `boneCount`, `fallbackReason`→warning), collect file artifacts, keep `raw` for the transcript only — the replan prompt gets one compact line per observation. **Recovery:** a failing step is retried once (same call), then the planner is asked to repair the tail (`{"steps":[...]}` or `{"done":true,"summary"}`), bounded by `Limits` (12 steps, 2 replans, 2 planner retries); the same `Step::signature()` failing twice stops the task ("stuck, not unlucky"). **One undo group per task** (`QUndoStack::beginMacro(plan.title)` on the first non-read-only step, `endMacro` on every terminal path — `isReadOnly()` = get_*/list_*/toggle_*/screenshots/validators). **Safety rail (#1021d):** `destructiveReason()` names deletes/geometry rewrites, overwrites of an EXISTING file, and OUTBOUND/account actions (`cloud_upload` — the data leaves the machine —, `cloud_login`, `cloud_logout`; review finding: nothing local is destroyed, so the delete/overwrite rules let the upload through); unless `trustedMode` (QSettings `ai/agentTrustedMode`) the task pauses in `AwaitingConfirmation` and the panel shows Allow / Always allow / Skip step. **Scene context per turn (#1021c):** `setContextProvider` — `AIChatManager::sceneSummaryForAgent()` (scene info + user materials + recent files) is appended to every planner prompt. The final message is the deterministic `summarize()` (steps, statuses, parsed facts, artifacts, warnings), so a flaky model cannot misreport what happened; a `{"summary"}` reply with no steps answers a question without tools. **Facade:** `AIChatManager::agentMode` (QSettings `ai/agentMode`, default ON) routes `sendMessage` to `AIAgentManager::startTask`; while the agent drives, the v1 slots ignore `LLMManager` callbacks (`m_agentDriving`) and `LlmPlannerBackend` forwards generation signals only while it has a pending request. QML: `qml/AIChatPanel.qml` gained the agent/ask-trusted toggles, a live plan card, the confirmation bar, and a model tip (#1021e: Qwen 2.5 7B Q4_K_M is the recommended tool-calling model — its `ModelInfo` description says so). Sentry `ai.agent.plan|step|retry|replan|verify|confirm|done|fail|cancel`. **Field findings from the first real sessions (Qwen3 4B):** (1) ~25 MCP tools act on the CURRENT SELECTION (`auto_rig`, `compute_skin_weights`, `validate_mesh`, `generate_lods`, `auto_uv_unwrap`, `retopologize`, `remove_skeleton`, …) and the agent had no way to set it — every rig request died with "No mesh selected"; there is now a **`select_entity {name}`** MCP tool (node or entity name; empty clears), `get_scene_info` ends with a `Selected: …` line, `auto_rig`'s error names the fix, and planner rule 6 says to select first. (2) **Conversation memory:** the agent keeps the last 12 turns (`request → outcome first line [objects touched]`) and injects the last 6 plus an "objects from earlier turns" list into every planner prompt, so "now make it red" resolves against the previous task; `AIChatManager::clearHistory` clears it. (3) Colour NAMES where an `[R,G,B]` array is expected (`"diffuse": "red"`) are coerced with a warning; rule 7 spells out the create_material → apply_material recipe. (4) **Argument aliases:** the model wrote `material_name` and the validator rejected `apply_material` twice for a missing `material` although the MCP handler itself accepts `material_name` — the validator must never be stricter than the tool, so `normaliseAliases` maps camelCase and a small synonym table (`material_name→material`, `entity/entity_name/node→mesh`, `output/path→output_path`, `skeleton→template`, …) onto the schema's names BEFORE the required check, and a rejection lists the keys that were passed; the replan prompt shows the failing call's arguments and says "do not resend it unchanged". (5) **Chat dock focus:** clicking the QML input after another dock (a QQuickWidget) held focus re-focused the QML item but not the hosting widget, so keystrokes went elsewhere until a detour via the viewport; `ClickFocusFilter` (installed on the chat QQuickWidget) turns every press into `setFocus`. (6) **Context window:** switching to another model produced "Failed to decode prompt" — `LLMWorker`'s prompt-too-long pre-check compared against the *configured* context size while the context actually created is clamped to the model's training limit, so an oversized prompt reached `llama_decode`. The worker now checks against `llama_n_ctx` (with a clear message naming both numbers) and emits `contextReady(nCtx)` → `LLMManager::effectiveContextSize`; the agent budgets every planner prompt against it (`systemPromptWithinBudget`: drop history → keep only the most relevant capabilities → truncate the scene listing → hard cut, each step traced), and the default `contextSize` is 8192 (was 4096 — the tool docs alone need more). Changing the context size needs a model reload. (7) **Mesh fed to the image tool:** the 14B planner passed an `.obj` as `image_path` to `generate_mesh_from_image`; the schema now says "2D IMAGE … NOT a 3D mesh: use load_mesh", the handler refuses mesh extensions with the right tool named, the capability index says "not for existing meshes", and `rejectMeshAsImage` in the validator stops any image-typed parameter (`image_path`/`image`/`photo`/`texture`…) carrying a mesh path before the tool runs. (8) **LLM tab parity:** `LLMManager::deleteModelFile(fileName)` / `deleteAllModelFiles()` (models-directory only — paths are refused; unloads the active model first; removes the `.part` too) behind per-row Delete / Remove All / Open Folder in AI Model Settings, mirroring the QtMeshEditor Models tab; the chat header's model chip opens that dialog (`AIChatManager::openModelSettings` → `MainWindow::showAIModelSettings`). (10) **Invented image paths:** asked to "create a f22 raptor scene", the planner called `generate_mesh_from_image` with `~/Downloads/f22_raptor.png` (a file that never existed), the tool answered "image not found", and every repair round guessed another path (.jpg, .png again) — six failures, nothing made. The harness now repairs this itself BEFORE the call (`AIAgentManager::repairMissingImageInput`, in `executeNext` after coercion): a non-existent `image_path` is dropped and, when no `prompt` was given, the request's subject (`subjectFromGoal`: leading creation verbs/articles and a trailing "scene"/"model" stripped — "create a f22 raptor scene" → "f22 raptor") becomes the tool's text prompt (text → image → 3D), with a transcript note. Planner rule 8 says never to invent paths and when to use `prompt`; the replan prompt adds an explicit "that file does not exist — do NOT guess another path" hint whenever a step failed with not found / does not exist / no such file; the schema text for `image_path` says the same. Text → image still needs a stable-diffusion build + FLUX.2-klein (AI Model Settings) — without them the ONE remaining call fails with that actionable message instead of a path hunt. (11) **A heavy tool froze the whole UI.** The agent drives tools SYNCHRONOUSLY on the main thread, so `generate_mesh_from_image` (minutes of ONNX work) blocked the event loop: the window stopped painting and looked hung. The heavy work CANNOT simply move to a worker — `MeshGenBuilder::buildSceneNode` is Ogre and main-thread-only — so instead `MCPServer` gained a **`toolProgress(tool, stage, done, total)`** signal that the generation tool drives from `MeshGenPredictor::predict`'s existing `ProgressFn` (it already fires many times per stage and doubles as the cancel hook). The same callback pumps `processEvents(ExcludeUserInputEvents, 10)` at ~20 Hz — the window keeps painting, and excluding user input means no click can re-enter a tool mid-run. The image phase (FLUX via `generateSourceImageFromPrompt`) already spun nested `QEventLoop`s, so it never froze; its `SDManager::generationProgressChanged` ticks are relayed to the same signal so the bar moves there too. `McpToolExecutor` connects the signal to `AIAgentManager::reportToolProgress` → the `stepProgressLabel` / `stepProgress` properties (ignored when the agent is not busy; cleared around every `callTool`), and `qml/AIChatPanel.qml`'s plan card draws a labelled bar (indeterminate when `total <= 0`). (12) **Stop must reach a running heavy tool.** The first cut of (11) pumped with `ExcludeUserInputEvents`, which kept the window painting but made the Stop button unclickable — and the panel had no Stop button at all, so a multi-minute generation could not be aborted. The pump now delivers `AllEvents` (re-entry is prevented by the agent running one step at a time, not by dropping clicks), `MCPServer::requestToolCancel()` sets a flag the progress callback returns as `false` (→ `predict()` returns `cancelled`) and which is also checked between the image and 3D phases, `AgentToolExecutor::cancelRunningTool()` (overridden by `McpToolExecutor`) is called from `AIAgentManager::cancel()` while Executing, and `qml/AIChatPanel.qml` has a Stop button next to the thinking dots. The plan card also stays visible after the run (its `planCardPinned` guard was never set by anything, so the card vanished with the final result still unread). (13) **Window ACTIVATION gates more of QWidget than it looks — this test failed CI twice.** `ClickFocusFilter_test` first asserted a `FocusIn` count, then (the "fix") `hasFocus()`; both pass on a developer desktop and fail the Linux lane. Qt gates BOTH on the window being ACTIVE, and a headless CI display (Xvfb, no window manager) never activates one: `QWidget::hasFocus()` is `window()->focusWidget() == this && window()->isActiveWindow()`, and `QFocusEvent`s are not delivered at all. **`window.focusWidget()` is the one thing that tracks focus regardless** — assert that, and gate anything else behind `isActiveWindow()`. **`QT_QPA_PLATFORM=offscreen` does NOT reproduce this** (it reports `isActiveWindow() == true`, which is why both bad versions passed locally); a window that is never `show()`n does, on any platform — `ClickFocusFilter.FocusWidgetIsSetEvenWhenTheWindowIsNotActive` pins it that way. Rule for new widget tests: assert `window.focusWidget()`, never `hasFocus()` or focus-event counts, unless gated on activation. (13-old) **A focus-event assertion is not portable.** `ClickFocusFilter_test` asserted a fresh `FocusIn` after the filter re-asserts focus on an already-focused widget. Qt dispatches `QFocusEvent` only inside an **ACTIVE window**, and a headless CI display (Xvfb, no window manager) never activates one — the focus WIDGET still changes but no focus event is delivered, so the test passed on a developer desktop and failed the Linux lane (and the "5954/5955 discovered tests executed" coverage guard failed with it, i.e. one root cause counted as two suite failures). The test now asserts the `clearFocus()`+`setFocus()` PAIR via the focus-OUT count (only `clearFocus()` can produce it) and guards the event-count assertion behind `window.isActiveWindow()`. Rule for new widget tests: assert focus STATE (`hasFocus`, `window.focusWidget()`), not focus EVENT counts, unless the assertion is gated on window activation. (9) **Trace log:** every task overwrites `/ai_agent/last_task.log` (`AIAgentManager::traceLogPath()`) with the planner prompts, raw replies and raw tool results — read THAT when a task fails, the chat transcript only shows first lines. **Tool routing (`src/AIToolRouter.{h,cpp}`, the #1002 router v2):** the first router was a hand-written keyword table and "create a f22 raptor scene" fell through it (`I could not find tools for: generation_3d` — the planner asked for the capability by name, the table had no entry for it). Now the registry owns an **`AIToolRouter`**: BM25 (k1 1.2, b 0.75) over every tool's name + capability + description + parameter docs, fed by an **English intent lexicon** (`kIntents`: user words → tool-vocabulary terms, each with a weight — "green" → material/diffuse at 0.8, but generic verbs like "make"/"create" expand to generation at only 0.25–0.35, else "make it green" routes to TripoSR) plus one rule the lexicon cannot express: a creation verb next to a word NO tool doc mentions (f22, raptor, goblin) is a request to GENERATE that thing, so the unknown noun lifts the generation terms to 0.9. Both sides pass through the same `canonical()` stem (light suffix strip + trailing-e drop, so dance/dancing and image/images agree — it only has to be CONSISTENT, not linguistically right). `route()` returns ranked capabilities (scene always kept), a per-tool shortlist (`shortlist()` prunes a 26-tool capability to the ~10 relevant tools in the prompt — the real context-window win), the expanded terms for the trace, and `confident` (a raw request word hit a doc and the best score ≥ 1.0). **Non-English requests are NOT in the lexicon** (users are global, the docs are English): when the route is not confident the agent spends one 40-token LLM round (`Awaiting::Intent`, `requestIntentKeywords`) asking for English operation keywords and routes on those — any language, no extra model; a failed round falls back to the lexical route and plans anyway. `setIntentKeywordsEnabled(false)` in the fixture (the fake tool list has no vocabulary to be confident about). **`AIToolRouter_test.cpp` `RoutingBenchmark` is the regression bar** — every `need_capabilities` round the trace log records is a routing miss and belongs in its table (must hit every capability, ≥85 % top tool; a miss prints the top-3 scores and the expanded terms so the fix is a table edit, not a guess). An embedding router (e5-small / mmarco cross-encoder) was considered and deferred until the benchmark shows misses the lexicon cannot cover. **Test gotcha:** fixture tests are `TEST_F(AgentFixture, …)` — a `--gtest_filter='AIAgent*'` does NOT run them; use `AgentFixture*`. Follow-ups (children of #1000): #1004 local VLM viewport inspection (mtmd is already linked for the captioner), #1005 model lifecycle/memory budget, #1006 verifier layer, #1007 e2e benchmark, #1008 batch factory. - **ModelFetch** (`src/ModelFetch.{h,cpp}`, #1037): the ONE blocking "make sure this model file is on disk" primitive — `ModelFetch::ensureBlocking(Request{url,destination,label,timeoutMs,expectedSha256}) -> Outcome{ok,timedOut,path,error}`. Twenty consumers used to hand-roll the same nested-QEventLoop wait around `ModelDownloader`; the copies drifted: 18 DISCARDED the downloader's error text (so "refusing http://" / "SHA-256 mismatch" reached the user as "unavailable (offline?)"), and only 3 (`TextureInpaint`, `FaceRig/ArkitTemplate`, `FaceRig/FaceLandmarkDetector`) guarded the **synchronous-rejection race** (`startDownload` emits `downloadError` synchronously when busy → the handler's `loop.quit()` fires before `exec()` and is lost → the caller hangs for its full timeout). `ensureBlocking` owns only the wait and returns the downloader's own words; consumers keep what genuinely varies. **#1025 — an EXISTING file is verified when a digest is known:** with `Request::expectedSha256` set, a destination that already exists is hashed (`ModelDownloader::sha256HexOfFile`, cached per process by path+size+mtime so a 1.2 GB decoder is hashed once, not per rig) and a mismatch DELETES it (+ its `.part`) and re-fetches it, reporting `Outcome::replacedCorrupt`; without a digest "exists" still means ok. This is what caught the UniRig case below — a same-size corrupted download that loaded fine and produced NaN for months. Consumers with published digests (HF LFS oids) should pass them; UniRig does for its default hosting only (a mirror override may serve a different export) (base-URL env/QSettings/default resolution, the `*_NO_DOWNLOAD` guard, the timeout). **Convention for exposing the reason:** `ensureModelBlocking(QString* error = nullptr)` — done for `PhotoDepth` and `TextureInpaint` (their 4 CLI/MCP sites now print e.g. `…unavailable: Refusing to download …: scheme 'http' is not https://`, proven e2e); the other migrated consumers keep their signatures (no message site to enrich yet). Migrated (behaviour-preserving): every consumer with a blocking wait — `PhotoDepth`, `TextureInpaint` (with the `error` out-param), `AIAssistManager`, `ImageTo3D/ImageCaptioner`, `ImageTo3D/MeshGenPredictor`, `ImageTo3D/TripoSGPredictor`, `ImageTo3D/BackgroundRemover`, `MeshSegmenter`, `MotionInbetween`, `MotionGenerator`, `SkinTokensPredictor`, `UniRigPredictor`, `FaceRig/ArkitTemplate`, `FaceRig/FaceLandmarkDetector`, `Mocap/FaceCapPredictor`, `Mocap/PoseCapPredictor`, `Mocap/HandCapPredictor`. The two FaceRig consumers had their own `done` race guard — now `ModelFetch`'s `settled`; their `(timeout)` breadcrumb annotation comes from `Outcome::timedOut`. `MotionGenerator`'s old `guard` timer never cancelled the transfer on timeout; `ModelFetch` does. **Deliberately NOT migrated:** `MotionLibrary::ensureLibraryBlocking` (V1→V2 upgrade logic where a failed download must fall back to the local V1 file — `haveLocal ? dest : QString()` — not the canonical shape; the only hand-rolled **`ModelDownloader`** wait left — other `QEventLoop`s in the tree belong to the cloud client, HDR downloads and the updater, which have their own network paths). **Gotcha from this migration:** at least one source file is not valid UTF-8 — a Python `open(p).read()` sweep over `src/` raises `UnicodeDecodeError`; use `encoding='utf-8', errors='surrogateescape'` for read AND write so bytes round-trip exactly. `AIModelCatalog`/`LLMSettingsWidget` are GUI-async (no event loop) and already show the error text. Tests: `ModelFetch_test.cpp` drives the REAL singleton through QNAM's `file://` backend (existing-file short-circuit, real fetch, refusal text verbatim, missing-file network error, synchronous rejection returns in <1 s not after the timeout); the race guard and error capture are mutation-verified. ### AI Texture Generation diff --git a/src/AIAgentManager.cpp b/src/AIAgentManager.cpp index 0350061b..a9943dfd 100644 --- a/src/AIAgentManager.cpp +++ b/src/AIAgentManager.cpp @@ -340,7 +340,8 @@ bool AIAgentManager::startTask(const QString& request) m_plan.goal = goal; m_observations.clear(); m_failureCounts.clear(); - m_docCapabilities = m_registry.routeByKeywords(goal); + m_intentTerms.clear(); + const bool confident = routeGoal(); m_currentStep = -1; m_pendingIndex = -1; m_pendingReason.clear(); m_replans = 0; m_plannerRetries = 0; m_replanFailedIndex = -1; m_cancelRequested = false; m_lastSummary.clear(); m_lastError.clear(); @@ -349,11 +350,57 @@ bool AIAgentManager::startTask(const QString& request) emit planChanged(); emit confirmationChanged(); trace(QStringLiteral("task"), goal); - SentryReporter::addBreadcrumb("ai.agent.plan", QStringLiteral("task started (%1 routed capabilities)").arg(m_docCapabilities.size())); - requestPlan(); + SentryReporter::addBreadcrumb("ai.agent.plan", QStringLiteral("task started (%1 routed capabilities, %2)") + .arg(m_docCapabilities.size()).arg(confident ? "confident" : "asking intent")); + if (confident || !m_intentKeywordsEnabled) requestPlan(); + else requestIntentKeywords(); return true; } +bool AIAgentManager::routeGoal(const QStringList& extraTerms) +{ + m_route = m_registry.route(m_plan.goal, extraTerms); + m_docCapabilities = m_route.capabilities; + QStringList top; + for (int i = 0; i < m_route.scores.size() && i < 6; ++i) + top << QStringLiteral("%1(%2)").arg(m_route.scores[i].name).arg(m_route.scores[i].score, 0, 'f', 1); + trace(QStringLiteral("route"), QStringLiteral("capabilities: %1\nlexicon added: %2\nextra terms: %3\ntop tools: %4\nconfident: %5") + .arg(m_docCapabilities.join(", "), m_route.expandedTerms.join(' '), extraTerms.join(' '), + top.join(", "), m_route.confident ? "yes" : "no")); + return m_route.confident; +} + +// The tool docs are English; the request may be anything. One short +// generation turns "cria um dragão vermelho" into "generate mesh, prompt, +// material colour" and the lexical router does the rest. +void AIAgentManager::requestIntentKeywords() +{ + setState(State::Planning); + m_awaiting = Awaiting::Intent; + const QString sys = QStringLiteral( + "You translate a user's request for a 3D mesh editor into ENGLISH keywords naming the editor operations " + "and objects involved. Reply with 3 to 8 comma-separated English keywords and nothing else. " + "Examples: 'generate mesh, image prompt, vehicle' / 'material, colour, apply' / 'rig, skeleton, skin weights' / " + "'export, glb' / 'load mesh, file' / 'transform, scale' / 'animation, walk, motion' / 'scene info'."); + const QString user = QStringLiteral("Request: %1\nKeywords:").arg(m_plan.goal); + trace(QStringLiteral("intent request"), user); + m_planner->request(sys, user, 40); +} + +void AIAgentManager::handleIntentReply(const QString& text) +{ + QStringList terms; + static const QRegularExpression sep(R"([,;/\n]+)"); + for (const QString& t : text.split(sep, Qt::SkipEmptyParts)) { + const QString w = t.trimmed().toLower(); + if (!w.isEmpty() && w.size() < 40 && !terms.contains(w)) terms << w; + } + m_intentTerms = terms.mid(0, 8); + SentryReporter::addBreadcrumb("ai.agent.plan", QStringLiteral("intent keywords: %1").arg(m_intentTerms.join(", "))); + routeGoal(m_intentTerms); // confident or not, we plan now — the planner can still ask for more + requestPlan(); +} + void AIAgentManager::cancel() { if (!busy()) return; @@ -430,7 +477,7 @@ QString AIAgentManager::systemPrompt(const QStringList& capabilityIds, bool with "6. Tools described as acting on 'the selected mesh' (auto_rig, compute_skin_weights, validate_mesh, generate_lods, auto_uv_unwrap, retopologize, remove_skeleton, ...) use the CURRENT SELECTION: call select_entity {\"name\": ...} first unless the scene state already shows it selected.\n" "7. Colours are [R,G,B] arrays in 0..1 (red = [1,0,0]). To recolour an object: create_material with a diffuse colour, then apply_material to the object.\n" "8. Never invent file paths. generate_mesh_from_image takes EITHER the user's real image in image_path OR, when the user gave no image, a description of the object in prompt (text → image → 3D). To create something that does not exist yet, use prompt.\n") - .arg(m_registry.promptIndex(), capabilityIds.join(", "), m_registry.promptToolsFor(capabilityIds)) + .arg(m_registry.promptIndex(), capabilityIds.join(", "), m_registry.promptToolsFor(capabilityIds, m_route)) .arg(m_limits.maxSteps); const QString history = withHistory ? conversationContext() : QString(); if (!history.isEmpty()) s += QStringLiteral("\n%1\n").arg(history); @@ -666,9 +713,10 @@ void AIAgentManager::onPlannerCompleted(const QString& text) if (m_awaiting == Awaiting::None) return; const Awaiting what = m_awaiting; m_awaiting = Awaiting::None; - trace(what == Awaiting::Plan ? QStringLiteral("plan reply") : QStringLiteral("replan reply"), text); + trace(what == Awaiting::Plan ? QStringLiteral("plan reply") : (what == Awaiting::Intent ? QStringLiteral("intent reply") : QStringLiteral("replan reply")), text); if (m_cancelRequested) return; - if (what == Awaiting::Plan) handlePlanReply(text); + if (what == Awaiting::Intent) handleIntentReply(text); + else if (what == Awaiting::Plan) handlePlanReply(text); else handleReplanReply(text); } @@ -684,6 +732,10 @@ bool AIAgentManager::expandCapabilities(const QStringList& need, QStringList* al if (m_docCapabilities.contains(id)) { if (alreadyHad) *alreadyHad << id; continue; } m_docCapabilities << id; added << id; + // explicitly requested → show it whole (drop any partial scoring so + // shortlist() falls back to "all tools") + for (int i = m_route.scores.size() - 1; i >= 0; --i) + if (m_route.scores[i].capability == id) m_route.scores.removeAt(i); } if (added.isEmpty()) return false; SentryReporter::addBreadcrumb("ai.agent.plan", QStringLiteral("expanded capabilities: %1").arg(added.join(", "))); @@ -801,6 +853,13 @@ void AIAgentManager::handleReplanReply(const QString& text) void AIAgentManager::onPlannerFailed(const QString& error) { if (m_awaiting == Awaiting::None) return; + if (m_awaiting == Awaiting::Intent) { + // The keyword step is an optimisation: plan with the lexical route. + m_awaiting = Awaiting::None; + trace(QStringLiteral("intent failed"), error); + requestPlan(); + return; + } m_awaiting = Awaiting::None; m_lastError = QStringLiteral("planner error: %1").arg(error); say(QStringLiteral("The AI model failed: %1").arg(error)); diff --git a/src/AIAgentManager.h b/src/AIAgentManager.h index ded657ad..7bc760f4 100644 --- a/src/AIAgentManager.h +++ b/src/AIAgentManager.h @@ -116,6 +116,10 @@ class AIAgentManager : public QObject /// Takes ownership (parented). Production creates an LlmPlannerBackend lazily. void setPlanner(AgentPlannerBackend* planner); void setLimits(const AIAgent::Limits& limits) { m_limits = limits; } + /// When the lexical router is not confident, ask the LLM for English + /// operation keywords first (any-language requests). Default on; tests + /// with scripted planners switch it off unless they exercise it. + void setIntentKeywordsEnabled(bool on) { m_intentKeywordsEnabled = on; } /// Undo stack the task's mutating steps are grouped on (default: UndoManager's). void setUndoStack(QUndoStack* stack) { m_undoStack = stack; } /// Extra text appended to every planner prompt (the scene summary). The @@ -224,13 +228,21 @@ private slots: explicit AIAgentManager(QObject* parent = nullptr); ~AIAgentManager() override; - enum class Awaiting { None, Plan, Replan }; + enum class Awaiting { None, Intent, Plan, Replan }; void setState(AIAgent::State s); void ensurePlanner(); void requestPlan(const QString& extraInstruction = QString()); void requestReplan(int failedIndex); QString systemPrompt(const QStringList& capabilityIds, bool withHistory = true, int sceneChars = -1) const; + /// Route the goal (BM25 + lexicon, plus `extraTerms`) into + /// m_docCapabilities / m_route; returns the route's confidence. + bool routeGoal(const QStringList& extraTerms = {}); + /// Ask the LLM for a few English operation keywords (any-language + /// requests, odd paraphrases) before routing — only when routing is + /// not confident. + void requestIntentKeywords(); + void handleIntentReply(const QString& text); /// systemPrompt() shrunk until `system + user + reply` fits the planner's /// context window: history dropped first, then capabilities beyond the /// most relevant, then the scene state, then the tool docs themselves. @@ -262,6 +274,8 @@ private slots: AIAgent::Plan m_plan; QVector m_observations; QStringList m_docCapabilities; // capabilities whose docs the planner has seen + AIToolRouter::Route m_route; // per-request tool relevance (prompt pruning) + QStringList m_intentTerms; // English keywords the LLM added for routing QHash m_failureCounts; // step signature → failures int m_currentStep = -1; int m_pendingIndex = -1; @@ -272,6 +286,7 @@ private slots: bool m_macroOpen = false; bool m_cancelRequested = false; bool m_trustedMode = false; + bool m_intentKeywordsEnabled = true; QString m_lastSummary; QString m_lastError; diff --git a/src/AIAgentManager_test.cpp b/src/AIAgentManager_test.cpp index 69644742..6a45c1ef 100644 --- a/src/AIAgentManager_test.cpp +++ b/src/AIAgentManager_test.cpp @@ -93,7 +93,11 @@ class FakePlanner : public AgentPlannerBackend void request(const QString& sys, const QString& user, int) override { systemPrompts << sys; userPrompts << user; pendingFlag = true; - if (replies.isEmpty()) { QTimer::singleShot(0, this, [this]() { pendingFlag = false; emit failed("no scripted reply"); }); return; } + if (failNextRequest || replies.isEmpty()) { + failNextRequest = false; + QTimer::singleShot(0, this, [this]() { pendingFlag = false; emit failed("no scripted reply"); }); + return; + } const QString r = replies.takeFirst(); QTimer::singleShot(0, this, [this, r]() { if (!stoppedFlag) { pendingFlag = false; emit completed(r); } }); } @@ -101,6 +105,7 @@ class FakePlanner : public AgentPlannerBackend bool pending() const override { return pendingFlag; } int contextTokens() const override { return ctxTokens; } int ctxTokens = 0; + bool failNextRequest = false; bool isAvailable = true; bool stoppedFlag = false; bool pendingFlag = false; @@ -153,6 +158,7 @@ struct AgentFixture : public ::testing::Test { m->setPlanner(planner); m->setUndoStack(&undo); m->setTrustedMode(false); + m->setIntentKeywordsEnabled(false); // the fake tool list has little vocabulary; the intent test turns it on QObject::connect(m, &AIAgentManager::chatMessage, [this](const QString& role, const QString& text, bool) { transcript << role + ": " + text; }); @@ -459,6 +465,42 @@ TEST_F(AgentFixture, RepeatedRequestForAlreadyProvidedDocsIsNudgedNotFailed) EXPECT_TRUE(m->lastError().contains("kept asking")) << m->lastError().toStdString(); } +// Global users: a request in any language first gets an English-keyword +// round from the LLM, and the lexical router routes on those. English +// requests with lexical signal skip that round. +TEST_F(AgentFixture, NonEnglishRequestGetsAnIntentKeywordRoundBeforePlanning) +{ + m->setIntentKeywordsEnabled(true); + planner->replies << "generate mesh, image prompt, material colour"; // intent keywords + planner->replies << planJson({{"auto_rig", {{"template", "generic"}}}}); // then the plan + ASSERT_TRUE(m->startTask("cria um dragão vermelho")); // no English word → no lexical signal + ASSERT_TRUE(pumpToEnd(m)); + ASSERT_EQ(planner->userPrompts.size(), 2); + EXPECT_TRUE(planner->userPrompts[0].contains("Keywords:")) << "first round asks for English keywords"; + EXPECT_TRUE(planner->systemPrompts[0].contains("ENGLISH keywords")); + EXPECT_TRUE(planner->userPrompts[1].startsWith("Task:")) << "second round is the plan"; + EXPECT_EQ(m->state(), State::Completed); + + // an English request with lexical signal plans immediately + AIAgentManager::kill(); SetUp(); + m->setIntentKeywordsEnabled(true); + planner->replies << planJson({{"auto_rig", {{"template", "humanoid"}}}}); + ASSERT_TRUE(m->startTask("rig the wolf")); + ASSERT_TRUE(pumpToEnd(m)); + ASSERT_EQ(planner->userPrompts.size(), 1); + EXPECT_TRUE(planner->userPrompts[0].startsWith("Task:")); + EXPECT_TRUE(planner->systemPrompts[0].contains("- auto_rig:")); + + // if the keyword round fails, the lexical route is used and planning proceeds + AIAgentManager::kill(); SetUp(); + m->setIntentKeywordsEnabled(true); + planner->failNextRequest = true; // the intent round errors out + planner->replies << planJson({{"get_scene_info", {}}}); // the plan round still has its reply + ASSERT_TRUE(m->startTask("何がありますか")); + ASSERT_TRUE(pumpToEnd(m)); + EXPECT_EQ(m->state(), State::Completed) << m->lastSummary().toStdString(); +} + TEST_F(AgentFixture, QuestionIsAnsweredWithoutRunningTools) { planner->replies << "{\"summary\": \"The scene holds one entity, Floor.\"}"; diff --git a/src/AICapabilityRegistry.cpp b/src/AICapabilityRegistry.cpp index 985b7703..45df3987 100644 --- a/src/AICapabilityRegistry.cpp +++ b/src/AICapabilityRegistry.cpp @@ -94,45 +94,6 @@ QString capForPrefix(const QString& tool) return {}; } -struct Keyword { const char* word; const char* cap; }; -const std::vector kKeywords = { - {"rig", "rigging"}, {"skeleton", "rigging"}, {"skin", "rigging"}, {"bone", "rigging"}, {"weights", "rigging"}, - {"blendshape", "rigging"}, {"arkit", "rigging"}, {"unirig", "rigging"}, - {"segment", "segmentation"}, {"parts", "segmentation"}, {"split", "segmentation"}, {"explode", "segmentation"}, - {"join", "segmentation"}, - {"anim", "animation"}, {"keyframe", "animation"}, {"play", "animation"}, {"clip", "animation"}, - {"sprite", "animation"}, {"isometric", "animation"}, - {"walk", "motion_ai"}, {"run", "motion_ai"}, {"dance", "motion_ai"}, {"idle", "motion_ai"}, {"motion", "motion_ai"}, - {"wave", "motion_ai"}, {"jump", "motion_ai"}, {"in-between", "motion_ai"}, {"inbetween", "motion_ai"}, - {"material", "materials"}, {"color", "materials"}, {"colour", "materials"}, {"shiny", "materials"}, - {"metal", "materials"}, {"texture", "materials"}, {"glossy", "materials"}, {"matte", "materials"}, - {"pbr", "textures_ai"}, {"upscale", "textures_ai"}, {"inpaint", "textures_ai"}, {"normal map", "textures_ai"}, - {"roughness", "textures_ai"}, {"atlas", "textures_ai"}, {"depth", "textures_ai"}, - {"light", "lighting"}, {"hdr", "lighting"}, {"environment", "lighting"}, {"shadow", "lighting"}, {"tonemap", "lighting"}, - {"lod", "mesh_optimize"}, {"decimate", "mesh_optimize"}, {"simplify", "mesh_optimize"}, {"retopo", "mesh_optimize"}, - {"optimi", "mesh_optimize"}, {"weld", "mesh_optimize"}, {"triangle", "mesh_optimize"}, {"polycount", "mesh_optimize"}, - {"uv", "uv"}, {"unwrap", "uv"}, {"seam", "uv"}, - {"generate", "generation_3d"}, {"image to 3d", "generation_3d"}, {"from image", "generation_3d"}, - {"from photo", "generation_3d"}, {"prompt", "generation_3d"}, {"create a 3d", "generation_3d"}, {"model of", "generation_3d"}, - // "create a f22 raptor scene" / "make me a dragon": creating something that is not a primitive - {"create", "generation_3d"}, {"make", "generation_3d"}, {"build", "generation_3d"}, {"scene", "generation_3d"}, - {"a 3d", "generation_3d"}, {"character", "generation_3d"}, {"creature", "generation_3d"}, {"vehicle", "generation_3d"}, - {"load", "scene_io"}, {"import", "scene_io"}, {"open", "scene_io"}, {"export", "scene_io"}, {"save", "scene_io"}, - {"file", "scene_io"}, {".glb", "scene_io"}, {".fbx", "scene_io"}, {".obj", "scene_io"}, {"folder", "scene_io"}, - {"screenshot", "view"}, {"camera", "view"}, {"look at", "view"}, {"render", "view"}, {"show me", "view"}, - {"zoom", "view"}, {"normals", "view"}, - {"morph", "morph_pose"}, {"pose", "morph_pose"}, {"expression", "morph_pose"}, - {"paint", "paint"}, {"brush", "paint"}, {"layer", "paint"}, {"stencil", "paint"}, - {"mocap", "mocap"}, {"webcam", "mocap"}, {"video", "mocap"}, {"capture", "mocap"}, - {"cloud", "cloud"}, {"upload", "cloud"}, - {"ps1", "ps1"}, {"playstation", "ps1"}, - {"node anim", "node_animation"}, {"spin", "node_animation"}, {"rotate over time", "node_animation"}, - {"delete", "scene"}, {"remove", "scene"}, {"move", "scene"}, {"scale", "scene"}, {"rotate", "scene"}, - {"duplicate", "scene"}, {"copy", "scene"}, {"box", "scene"}, {"cube", "scene"}, {"sphere", "scene"}, - {"cylinder", "scene"}, {"plane", "scene"}, {"primitive", "scene"}, {"validate", "scene"}, {"check", "scene"}, - {"scene", "scene"}, {"select", "scene"}, {"bigger", "scene"}, {"smaller", "scene"}, {"larger", "scene"}, - {"twice", "scene"}, {"half", "scene"}, {"mesh", "scene"}, {"object", "scene"}, {"info", "scene"}, -}; } // namespace @@ -158,6 +119,20 @@ AICapabilityRegistry::AICapabilityRegistry(const QJsonArray& toolList) if (cap.isEmpty()) cap = QStringLiteral("other"); m_capabilities[m_capIndex.value(cap)].tools << name; } + // Lexical router over the same docs the planner reads (name + description + // + parameter docs), so routing and prompt vocabulary cannot drift. + QVector docs; + for (const Capability& c : m_capabilities) { + for (const QString& t : c.tools) { + const ToolInfo& info = m_tools[t]; + QString text = info.description; + const QJsonObject props = info.schema["properties"].toObject(); + for (auto p = props.begin(); p != props.end(); ++p) + text += ' ' + p.key() + ' ' + p.value().toObject()["description"].toString(); + docs.push_back({t, c.id, text}); + } + } + m_router = AIToolRouter(docs); } QStringList AICapabilityRegistry::capabilityIds() const @@ -250,37 +225,23 @@ QString AICapabilityRegistry::promptToolsFor(const QStringList& capabilityIds) c return s; } -namespace { -// Whole-word-ish match for short keys, plain substring for phrases/extensions. -bool keywordHits(const QString& word, const QString& request) +AIToolRouter::Route AICapabilityRegistry::route(const QString& request, const QStringList& extraTerms) const { - if (word.contains(' ') || word.startsWith('.')) return request.contains(word); - static QHash cache; - auto it = cache.find(word); - if (it == cache.end()) - it = cache.insert(word, QRegularExpression(QStringLiteral("\\b%1").arg(QRegularExpression::escape(word)))); - return it->match(request).hasMatch(); + QString q = request; + if (!extraTerms.isEmpty()) q += ' ' + extraTerms.join(' '); + AIToolRouter::Route r = m_router.route(q); + if (r.capabilities.isEmpty() && capability(QStringLiteral("scene"))) r.capabilities << QStringLiteral("scene"); + return r; } -} // namespace -QStringList AICapabilityRegistry::routeByKeywords(const QString& request) const +QString AICapabilityRegistry::promptToolsFor(const QStringList& capabilityIds, const AIToolRouter::Route& route) const { - const QString r = request.toLower(); - QHash score; - for (const Keyword& k : kKeywords) { - const QString w = QLatin1String(k.word); - if (keywordHits(w, r)) score[QLatin1String(k.cap)] += (w.size() >= 6 ? 2 : 1); + QString s; + for (const QString& id : capabilityIds) { + if (!capability(id)) continue; + for (const QString& t : m_router.shortlist(route, id)) s += toolDoc(t); } - QStringList caps; - for (const Capability& c : m_capabilities) - if (!c.tools.isEmpty() && score.value(c.id) > 0) caps << c.id; - std::stable_sort(caps.begin(), caps.end(), [&](const QString& a, const QString& b) { - return score.value(a) > score.value(b); - }); - if (caps.size() > 4) caps = caps.mid(0, 4); - if (!caps.contains(QStringLiteral("scene")) && capability(QStringLiteral("scene"))) - caps << QStringLiteral("scene"); // inspection tools are always useful - return caps; + return s; } namespace { diff --git a/src/AICapabilityRegistry.h b/src/AICapabilityRegistry.h index f1be745e..0238dee0 100644 --- a/src/AICapabilityRegistry.h +++ b/src/AICapabilityRegistry.h @@ -28,6 +28,8 @@ #include #include +#include "AIToolRouter.h" + class AICapabilityRegistry { public: @@ -58,15 +60,22 @@ class AICapabilityRegistry QString promptIndex() const; /// Full docs (description + params) for every tool of the given capabilities. QString promptToolsFor(const QStringList& capabilityIds) const; + /// Same, but a large capability contributes only the tools the router + /// found relevant to `route` (small ones and no-signal ones stay whole). + QString promptToolsFor(const QStringList& capabilityIds, const AIToolRouter::Route& route) const; /// One tool's doc block: "- name: description\n param: doc (type, required)". QString toolDoc(const QString& tool) const; // ---- routing ---- - /// Keyword heuristic: which capabilities a request most likely needs. - /// Used to pre-narrow the planner's docs and as the fallback when the - /// planner's own capability pick is unusable. Always non-empty for a - /// non-empty request (falls back to "scene"). - QStringList routeByKeywords(const QString& request) const; + /// BM25 + intent lexicon over the tool docs (AIToolRouter): ranked + /// capabilities, per-tool scores and a confidence flag. `extraTerms` + /// (e.g. English keywords the LLM produced for a non-English request) + /// are appended to the query. + AIToolRouter::Route route(const QString& request, const QStringList& extraTerms = {}) const; + /// Capabilities only — kept for callers that need just the list. Always + /// non-empty for a non-empty request (falls back to "scene"). + QStringList routeByKeywords(const QString& request) const { return route(request).capabilities; } + const AIToolRouter& router() const { return m_router; } // ---- constrained protocol (#1003) ---- /// Validate `args` against the tool's input schema. Missing required → @@ -101,6 +110,7 @@ class AICapabilityRegistry QHash m_tools; QVector m_capabilities; // stable display order QHash m_capIndex; + AIToolRouter m_router; }; #endif // AICAPABILITYREGISTRY_H diff --git a/src/AICapabilityRegistry_test.cpp b/src/AICapabilityRegistry_test.cpp index 1735664b..a14d301b 100644 --- a/src/AICapabilityRegistry_test.cpp +++ b/src/AICapabilityRegistry_test.cpp @@ -93,19 +93,24 @@ TEST(AICapabilityRegistry, KeywordRoutingPicksRelevantCapabilitiesAndAlwaysKeeps { AICapabilityRegistry reg(sampleTools()); const QStringList rig = reg.routeByKeywords("rig and skin the wolf then export it as glb"); - EXPECT_TRUE(rig.contains("rigging")); - EXPECT_TRUE(rig.contains("scene_io")); + EXPECT_TRUE(rig.contains("rigging")) << rig.join(",").toStdString(); + EXPECT_TRUE(rig.contains("scene_io")) << rig.join(",").toStdString(); EXPECT_TRUE(rig.contains("scene")); - EXPECT_LE(rig.size(), 5); + EXPECT_LE(rig.size(), 4); const QStringList simple = reg.routeByKeywords("make the box twice as large"); - EXPECT_EQ(simple.first(), "scene"); + EXPECT_TRUE(simple.contains("scene")); EXPECT_FALSE(simple.contains("rigging")); EXPECT_TRUE(reg.routeByKeywords("hello").contains("scene")) << "never empty"; // creating something that is not a primitive routes the (single-tool) generation capability up front EXPECT_TRUE(reg.routeByKeywords("create a f22 raptor scene").contains("generation_3d")); EXPECT_TRUE(reg.routeByKeywords("make me a dragon").contains("generation_3d")); + // a request in another language has no lexical signal → not confident (the agent asks the LLM for keywords) + EXPECT_FALSE(reg.route("cria um dragão vermelho").confident); + EXPECT_TRUE(reg.route("rig the wolf").confident); + // ...and English keywords supplied for it route it correctly + EXPECT_TRUE(reg.route("cria um dragão vermelho", {"generate mesh", "prompt", "material colour"}).capabilities.contains("generation_3d")); } TEST(AICapabilityRegistry, ValidateArgumentsEnforcesRequiredTypesAndEnums) diff --git a/src/AIToolRouter.cpp b/src/AIToolRouter.cpp new file mode 100644 index 00000000..491a2bd9 --- /dev/null +++ b/src/AIToolRouter.cpp @@ -0,0 +1,333 @@ +#include "AIToolRouter.h" + +#include +#include +#include +#include + +namespace { + +const QSet& stopwords() +{ + static const QSet s = { + "a", "an", "the", "and", "or", "of", "to", "in", "on", "for", "with", "it", "its", "is", "are", "be", + "this", "that", "as", "at", "by", "from", "into", "then", "than", "so", "me", "my", "please", "can", + "you", "we", "i", "do", "does", "not", "no", "yes", "if", "when", "use", "using", "used", "one", + "all", "any", "each", "every", "also", "just", "only", "new", "current", "given", "optional", + "default", "value", "values", "name", "names", "e", "g", "etc", + }; + return s; +} + +QString stem(QString w) +{ + // Light English stemmer: enough to make "rigging" meet "rig" and + // "materials" meet "material"; deliberately conservative. + if (w.size() > 5 && w.endsWith(QLatin1String("ation"))) return w.left(w.size() - 5) + QLatin1String("ate"); + if (w.size() > 4 && w.endsWith(QLatin1String("ing"))) { w.chop(3); if (w.size() > 2 && w[w.size()-1] == w[w.size()-2]) w.chop(1); return w; } + if (w.size() > 4 && w.endsWith(QLatin1String("ies"))) return w.left(w.size() - 3) + QLatin1String("y"); + if (w.size() > 3 && w.endsWith(QLatin1String("ed"))) { w.chop(2); return w; } + if (w.size() > 3 && w.endsWith(QLatin1String("es")) && !w.endsWith(QLatin1String("ses"))) { w.chop(2); return w; } + if (w.size() > 3 && w.endsWith(QLatin1String("s")) && !w.endsWith(QLatin1String("ss"))) { w.chop(1); } + return w; +} + +// Both queries and docs pass through this, so the stem only has to be +// CONSISTENT, not linguistically right: dropping a trailing 'e' makes +// dance/dancing, create/creating and image/images agree. +QString canonical(QString w) +{ + w = stem(w); + if (w.size() > 4 && w.endsWith('e')) w.chop(1); + return w; +} + +// intent word → tool-vocabulary terms, with a weight relative to the user's +// own words (1.0). Generic verbs expand WEAKLY: "make it green" must route +// to materials, not to "make → generate". +struct Intent { const char* trigger; const char* terms; double weight; }; +const Intent kIntents[] = { + // creating something that is not a primitive → generation (weak: many + // requests say "make"/"create" and mean something else) + {"create", "generate mesh image prompt", 0.35}, + {"make", "generate mesh image prompt", 0.25}, + {"build", "generate mesh image prompt", 0.35}, + {"design", "generate mesh image prompt", 0.5}, + {"generate", "generate mesh image prompt", 0.6}, + {"scene", "scene info", 0.4}, + {"character", "generate mesh image prompt", 0.5}, + {"creature", "generate mesh image prompt", 0.6}, + {"monster", "generate mesh image prompt", 0.6}, + {"vehicle", "generate mesh image prompt", 0.5}, + {"car", "generate mesh image prompt", 0.5}, + {"jet", "generate mesh image prompt", 0.6}, + {"plane", "generate mesh image prompt", 0.4}, + {"dragon", "generate mesh image prompt", 0.6}, + // appearance → materials + {"colour", "material diffuse apply create colour", 0.80}, + {"color", "material diffuse apply create colour", 0.80}, + {"red", "material diffuse apply create colour", 0.80}, + {"green", "material diffuse apply create colour", 0.80}, + {"blue", "material diffuse apply create colour", 0.80}, + {"yellow", "material diffuse apply create colour", 0.80}, + {"white", "material diffuse apply create colour", 0.80}, + {"black", "material diffuse apply create colour", 0.80}, + {"gold", "material diffuse specular shininess apply", 0.80}, + {"shiny", "material specular shininess apply", 0.80}, + {"glossy", "material specular shininess apply", 0.80}, + {"metal", "material specular shininess apply metallic", 0.80}, + {"metallic", "material specular shininess apply metallic", 0.80}, + {"matte", "material specular shininess apply", 0.80}, + {"paint", "material diffuse apply paint", 0.50}, + {"recolour", "material diffuse apply", 0.80}, + {"recolor", "material diffuse apply", 0.80}, + {"texture", "texture set material diffuse map", 0.50}, + // rigging + {"rig", "rig skeleton skin weight auto template", 0.80}, + {"skeleton", "bone skin rig", 0.40}, // weak: "remove the skeleton" must not become auto_rig + {"bone", "rig skeleton skin weight", 0.80}, + {"skin", "skin weight rig skeleton", 0.80}, + {"weight", "skin weight", 0.80}, + // transforms + {"bigger", "transform scale node", 0.80}, + {"larger", "transform scale node", 0.80}, + {"smaller", "transform scale node", 0.80}, + {"twice", "transform scale node", 0.80}, + {"half", "transform scale node", 0.80}, + {"scale", "transform scale node", 0.50}, + {"resize", "transform scale node", 0.80}, + {"move", "transform position node", 0.50}, + {"left", "transform position node", 0.35}, + {"right", "transform position node", 0.35}, + {"up", "transform position node", 0.35}, + {"down", "transform position node", 0.35}, + {"forward", "transform position node", 0.35}, + {"rotate", "transform rotation node", 0.50}, + {"turn", "transform rotation node", 0.50}, + {"spin", "node animation clip rotation", 0.80}, + // files + {"export", "export mesh output path file", 0.80}, + {"save", "export save scene output path file", 0.50}, + {"glb", "export mesh output path", 0.80}, + {"fbx", "export mesh output path", 0.80}, + {"obj", "load mesh file path", 0.80}, + {"load", "load mesh file path", 0.50}, + {"open", "load open mesh scene file path", 0.50}, + {"import", "load mesh file path", 0.80}, + {"file", "load export mesh file path", 0.50}, + {"folder", "list search files directory", 0.50}, + // view + {"screenshot","screenshot camera viewport", 0.80}, + {"show", "screenshot camera viewport", 0.50}, + {"look", "camera look viewport", 0.50}, + {"render", "screenshot camera viewport", 0.80}, + {"zoom", "camera viewport frame", 0.80}, + {"camera", "camera viewport frame", 0.80}, + // animation / motion + {"animate", "animation generate motion clip play", 0.80}, + {"animation", "animation clip play keyframe", 0.80}, + {"walk", "generate motion animation prompt", 0.80}, + {"run", "generate motion animation prompt", 0.80}, + {"dance", "generate motion animation prompt", 0.80}, + {"idle", "generate motion animation prompt", 0.80}, + {"jump", "generate motion animation prompt", 0.80}, + {"wave", "generate motion animation prompt", 0.80}, + {"play", "play animation", 0.50}, + // mesh ops + {"lod", "lod level detail generate", 0.80}, + {"decimate", "decimate reduce triangle", 0.80}, + {"simplify", "decimate reduce triangle simplify", 0.80}, + {"reduce", "decimate reduce triangle lod", 0.50}, + {"polygon", "decimate reduce triangle lod", 0.50}, + {"triangle", "mesh info count", 0.50}, + {"retopo", "retopology quad", 0.80}, + {"weld", "weld vertex", 0.80}, + {"optimize", "optimize vertex cache", 0.80}, + {"optimise", "optimize vertex cache", 0.80}, + {"uv", "uv unwrap", 0.80}, + {"unwrap", "uv unwrap", 0.80}, + {"segment", "segment part split", 0.80}, + {"part", "segment part split explode", 0.50}, + {"split", "segment part split", 0.50}, + {"explode", "segment part explode", 0.80}, + {"delete", "delete entity", 0.50}, + {"remove", "delete remove entity", 0.50}, + {"duplicate", "duplicate entity", 0.80}, + {"copy", "duplicate entity", 0.80}, + {"validate", "validate mesh", 0.80}, + {"check", "validate mesh info", 0.50}, + {"inspect", "scene info mesh info", 0.50}, + {"info", "scene info mesh info", 0.50}, + {"what", "scene info mesh info", 0.35}, + {"which", "scene info mesh info", 0.35}, + {"how", "scene info mesh info", 0.35}, + {"many", "mesh info count", 0.50}, + {"count", "mesh info count", 0.50}, + {"light", "light create lighting", 0.50}, + {"shadow", "light lighting", 0.50}, + {"hdr", "hdr environment", 0.80}, + {"pose", "pose library apply", 0.80}, + {"morph", "morph target weight", 0.80}, + {"face", "face blendshape arkit", 0.80}, +}; + +} // namespace + +QStringList AIToolRouter::tokenize(const QString& text) +{ + static const QRegularExpression sep(R"([^\p{L}\p{N}]+)"); + QStringList out; + for (const QString& raw : text.toLower().split(sep, Qt::SkipEmptyParts)) { + if (raw.size() < 2 && !raw[0].isDigit()) continue; + if (stopwords().contains(raw)) continue; + out << canonical(raw); + } + return out; +} + +QHash AIToolRouter::expandIntentWeighted(const QStringList& queryTokens) +{ + struct Entry { QStringList terms; double weight; }; + static QHash table; + if (table.isEmpty()) { + for (const Intent& i : kIntents) { + Entry e; e.weight = i.weight; + for (const QString& t : QString::fromLatin1(i.terms).split(' ', Qt::SkipEmptyParts)) e.terms << canonical(t); + table.insert(canonical(QString::fromLatin1(i.trigger)), e); + } + } + QHash extra; + for (const QString& q : queryTokens) { + const auto it = table.constFind(q); + if (it == table.constEnd()) continue; + for (const QString& t : it->terms) { + if (queryTokens.contains(t)) continue; + extra[t] = std::max(extra.value(t, 0.0), it->weight); + } + } + return extra; +} + +QStringList AIToolRouter::expandIntent(const QStringList& queryTokens) +{ + QStringList out = expandIntentWeighted(queryTokens).keys(); + std::sort(out.begin(), out.end()); + return out; +} + +AIToolRouter::AIToolRouter(const QVector& docs) +{ + long long total = 0; + for (const ToolDoc& d : docs) { + Indexed ix; + ix.doc = d; + // the name's words count twice: "auto_rig" should win "rig" queries + const QStringList toks = tokenize(d.name.split('_').join(' ') + ' ' + d.name.split('_').join(' ') + + ' ' + d.capability.split('_').join(' ') + ' ' + d.text); + for (const QString& t : toks) ix.tf[t]++; + ix.length = static_cast(toks.size()); + total += ix.length; + for (auto it = ix.tf.begin(); it != ix.tf.end(); ++it) m_df[it.key()]++; + m_docs.push_back(ix); + } + m_avgLen = m_docs.isEmpty() ? 1.0 : double(total) / double(m_docs.size()); +} + +double AIToolRouter::bm25(const Indexed& d, const QHash& weightedQuery) const +{ + constexpr double k1 = 1.2; + constexpr double b = 0.75; + const double N = double(m_docs.size()); + double score = 0.0; + for (auto it = weightedQuery.begin(); it != weightedQuery.end(); ++it) { + const int tf = d.tf.value(it.key(), 0); + if (tf == 0) continue; + const int df = m_df.value(it.key(), 0); + const double idf = std::log(1.0 + (N - df + 0.5) / (df + 0.5)); + const double norm = tf * (k1 + 1.0) / (tf + k1 * (1.0 - b + b * d.length / m_avgLen)); + score += it.value() * idf * norm; + } + return score; +} + +AIToolRouter::Route AIToolRouter::route(const QString& request, int maxCapabilities) const +{ + Route r; + const QStringList raw = tokenize(request); + const QHash expanded = expandIntentWeighted(raw); + r.expandedTerms = expanded.keys(); + std::sort(r.expandedTerms.begin(), r.expandedTerms.end()); + // The user's own words weigh 1.0; lexicon expansions carry their intent + // weight, so they hit where the raw words cannot without drowning them. + QHash query = expanded; + for (const QString& w : raw) query[w] = 1.0; + // "create/make/build " — an f22, a raptor, + // a goblin — is a request to GENERATE that thing: the unknown noun is + // the strongest generation signal there is, so it lifts the (otherwise + // weak) creation-verb expansion to full weight. + { + static const QStringList creationVerbs = {canonical("create"), canonical("make"), canonical("build"), canonical("design"), canonical("model")}; + static const QRegularExpression numberLike(QStringLiteral("^\\d+x?$")); + bool creation = false, unknownNoun = false; + for (const QString& w : raw) { + if (creationVerbs.contains(w)) creation = true; + else if (!m_df.contains(w) && !expanded.contains(w) && !numberLike.match(w).hasMatch() + && expandIntentWeighted({w}).isEmpty()) unknownNoun = true; + } + if (creation && unknownNoun) + for (const QString& t : {canonical("generate"), canonical("mesh"), canonical("image"), canonical("prompt")}) + if (!raw.contains(t)) query[t] = std::max(query.value(t, 0.0), 0.9); + } + + QHash capScore; + for (const Indexed& d : m_docs) { + const double s = bm25(d, query); + if (s <= 0.0) continue; + r.scores.push_back({d.doc.name, d.doc.capability, s}); + // capability = best tool + a little for breadth + capScore[d.doc.capability] = std::max(capScore.value(d.doc.capability), s) + 0.15 * s; + } + std::stable_sort(r.scores.begin(), r.scores.end(), [](const ScoredTool& a, const ScoredTool& b) { return a.score > b.score; }); + + QStringList caps = capScore.keys(); + std::stable_sort(caps.begin(), caps.end(), [&](const QString& a, const QString& b) { return capScore[a] > capScore[b]; }); + // keep only capabilities that are not far below the best (noise cut) + if (!caps.isEmpty()) { + const double best = capScore[caps.first()]; + QStringList kept; + for (const QString& c : caps) if (capScore[c] >= 0.25 * best || kept.size() < 2) kept << c; + caps = kept; + } + if (caps.size() > maxCapabilities) caps = caps.mid(0, maxCapabilities); + bool haveScene = false; + for (const Indexed& d : m_docs) if (d.doc.capability == QLatin1String("scene")) { haveScene = true; break; } + if (haveScene && !caps.contains(QStringLiteral("scene"))) { + if (caps.size() >= maxCapabilities) caps.removeLast(); + caps << QStringLiteral("scene"); + } + r.capabilities = caps; + for (const ScoredTool& t : r.scores) if (caps.contains(t.capability)) r.tools << t.name; + r.bestScore = r.scores.isEmpty() ? 0.0 : r.scores.first().score; + // Confident = a real request word (not only lexicon expansions) hit at + // least one tool with a decent score. A request in another language has + // no word in the English tool docs and lands here as not confident. + QStringList rawHits; + for (const Indexed& d : m_docs) for (const QString& w : raw) if (d.tf.contains(w) && !rawHits.contains(w)) rawHits << w; + r.confident = !rawHits.isEmpty() && r.bestScore >= 1.0; + return r; +} + +QStringList AIToolRouter::shortlist(const Route& route, const QString& capability, int keepAll, int maxTools) const +{ + QStringList all; + for (const Indexed& d : m_docs) if (d.doc.capability == capability) all << d.doc.name; + if (all.size() <= keepAll) return all; + QStringList scored; + for (const ScoredTool& t : route.scores) { + if (t.capability != capability) continue; + scored << t.name; + if (scored.size() >= maxTools) break; + } + // No signal inside this capability → show it whole rather than hide it. + return scored.isEmpty() ? all : scored; +} diff --git a/src/AIToolRouter.h b/src/AIToolRouter.h new file mode 100644 index 00000000..4992dc93 --- /dev/null +++ b/src/AIToolRouter.h @@ -0,0 +1,85 @@ +#ifndef AITOOLROUTER_H +#define AITOOLROUTER_H + +// Lexical tool router for the AI agent (#1002 follow-up): BM25 over the MCP +// tool docs, fed by an INTENT LEXICON that translates what the user means +// into the words the tool docs actually use. +// +// "create a f22 raptor scene" → +generate +mesh +image +prompt +3d +// "make it green" → +material +diffuse +colour +apply +// "rig and skin the wolf" → +rig +skeleton +skin +weights +// +// BM25 alone knows only the tool vocabulary ("diffuse colour [R,G,B]"); +// the lexicon bridges English user vocabulary ("green"). Other languages +// and odd paraphrases are handled one level up: when a route has little +// signal (`Route::confident` false), AIAgentManager asks the loaded LLM for +// a few English operation keywords and routes on those — any language, no +// extra model. Zero dependencies, microseconds per request; an embedding +// router stays a later option if the benchmark (AIToolRouter_test) shows +// misses this cannot cover. +// +// Output: ranked capabilities (for the planner's capability pick) and a +// per-request tool shortlist (so a 26-tool capability contributes only its +// relevant tools to the prompt — the actual context-window win). + +#include +#include +#include +#include + +class AIToolRouter +{ +public: + struct ToolDoc { + QString name; + QString capability; + QString text; // description + parameter docs + }; + struct ScoredTool { QString name; QString capability; double score; }; + struct Route { + QStringList capabilities; // best first; never empty for a non-empty request + QStringList tools; // all tools of the chosen capabilities that scored, best first + QVector scores; // every tool with a positive score, best first + QStringList expandedTerms; // what the lexicon added (for the trace) + double bestScore = 0.0; // top tool score + bool confident = false; // enough lexical signal to skip the LLM intent step + }; + + AIToolRouter() = default; + explicit AIToolRouter(const QVector& docs); + + bool isEmpty() const { return m_docs.isEmpty(); } + + /// Rank capabilities and tools for a request. `maxCapabilities` caps the + /// pick (the planner can still ask for more); "scene" is always kept + /// when present since its inspection tools are needed everywhere. + Route route(const QString& request, int maxCapabilities = 4) const; + + /// Tools of `capability` worth showing for this request: the ones that + /// scored, plus every tool when the capability is small (<= keepAll) or + /// nothing in it scored (no signal → show all rather than none). + QStringList shortlist(const Route& route, const QString& capability, int keepAll = 8, int maxTools = 10) const; + + // ---- pure helpers (unit-tested) ---- + /// Lowercase word tokens with a light stemmer (plural/-ing/-ed/-tion), + /// stopwords removed; "auto_rig" → {auto, rig}. + static QStringList tokenize(const QString& text); + /// Intent lexicon: extra English query terms implied by the request's + /// words, each with a weight (generic verbs like "make" expand weakly, + /// specific words like "green" strongly). Deterministic, table-driven. + static QHash expandIntentWeighted(const QStringList& queryTokens); + static QStringList expandIntent(const QStringList& queryTokens); + +private: + struct Indexed { + ToolDoc doc; + QHash tf; + int length = 0; + }; + QVector m_docs; + QHash m_df; // term → number of docs containing it + double m_avgLen = 1.0; + double bm25(const Indexed& d, const QHash& weightedQuery) const; +}; + +#endif // AITOOLROUTER_H diff --git a/src/AIToolRouter_test.cpp b/src/AIToolRouter_test.cpp new file mode 100644 index 00000000..94c7a3b2 --- /dev/null +++ b/src/AIToolRouter_test.cpp @@ -0,0 +1,182 @@ +// Routing benchmark + unit tests for the BM25 + intent-lexicon tool router. +// The benchmark requests are the ones typed during the first agent sessions +// (2026-09-17) plus paraphrases; every `need_capabilities` round the trace +// log records is a routing miss and belongs in this table. +#include + +#include "AIToolRouter.h" + +namespace { + +QVector corpus() +{ + // A slice of the real MCP surface: names + (abridged) real descriptions. + return { + {"get_scene_info", "scene", "Get a summary of the current scene: all scene nodes (with names), entities (with materials), and material count."}, + {"get_mesh_info", "scene", "Get detailed information about loaded meshes: vertex/index counts, submeshes, materials, bounding box, skeleton data."}, + {"create_primitive", "scene", "Create a procedural 3D primitive (box, sphere, cylinder, cone, plane) and add it to the scene. name: node name"}, + {"transform_mesh", "scene", "Set the position, rotation, and/or scale of a named scene node. position [x,y,z] scale [x,y,z] rotation"}, + {"delete_entity", "scene", "Delete an entity from the scene. entity_name"}, + {"duplicate_entity", "scene", "Duplicate an entity/node in the scene, creating a clone with the same mesh, materials, and transform."}, + {"select_entity", "scene", "Select a scene node/entity by name so that tools which act on the selected mesh target it."}, + {"validate_mesh", "scene", "Validate the selected mesh for common issues: degenerate triangles, non-finite UV coordinates."}, + {"load_mesh", "scene_io", "Load a 3D mesh file (obj, fbx, glb, gltf, dae, stl) into the scene. path"}, + {"export_mesh", "scene_io", "Export a mesh entity to a file (glb, gltf, fbx, obj). output_path"}, + {"save_scene", "scene_io", "Save the whole scene to a glTF file. path"}, + {"list_files", "scene_io", "List files in a directory, optionally filtered by extension."}, + {"take_screenshot", "view", "Capture a screenshot of the 3D viewport to a PNG file."}, + {"camera_control", "view", "Control the 3D viewport camera. Set position, look-at target, zoom, or frame the selection."}, + {"create_material", "materials", "Create a new Ogre3D material with optional colors. Colors are [R,G,B] arrays (0.0-1.0): ambient, diffuse, specular, emissive, shininess."}, + {"apply_material", "materials", "Apply a material to a mesh entity in the scene. material, mesh"}, + {"modify_material", "materials", "Modify an existing material's properties: ambient, diffuse, specular, emissive colors, shininess, texture."}, + {"set_texture", "materials", "Bind a texture image to a material's texture unit."}, + {"create_light", "lighting", "Create a scene light (point, directional, spot) with colour and intensity."}, + {"set_hdr_environment", "lighting", "Set the HDR environment (IBL) used for PBR lighting and the skybox."}, + {"generate_pbr_maps", "textures_ai", "AI PBR map synthesis: predict normal, roughness and height maps from a diffuse/albedo texture."}, + {"upscale_texture", "textures_ai", "AI super-resolution (Real-ESRGAN 2x/4x) of a texture image."}, + {"generate_lods", "mesh_optimize", "Generate LOD (Level of Detail) levels for the selected mesh, reducing polygon count at distance."}, + {"decimate_mesh", "mesh_optimize", "Reduce the triangle count of a mesh by a reduction ratio (0..1) or to a target triangle count."}, + {"retopologize", "mesh_optimize", "Quad-dominant retopology of the selected mesh via triangle pairing."}, + {"auto_uv_unwrap", "uv", "Auto UV-unwrap the selected entity via xatlas into UV0."}, + {"auto_rig", "rigging", "Auto-rig the currently selected STATIC (unrigged) mesh by embedding a skeleton template (humanoid, biped, quadruped, generic, vehicle) or UniRig; optionally skin."}, + {"compute_skin_weights", "rigging", "Compute and apply skin weights for the currently selected mesh against its skeleton (skintokens, geodesic-voxel, inverse-distance)."}, + {"remove_skeleton", "rigging", "Remove the ENTIRE skeleton from the currently selected mesh."}, + {"segment_mesh", "segmentation", "AI part segmentation: per-part vertex/face labels (head, torso, arms, legs; vehicle body, wheels)."}, + {"split_mesh_by_segments", "segmentation", "Split the segmented mesh into one named submesh per part."}, + {"generate_mesh_from_image", "generation_3d", "Generate a NEW 3D mesh from a 2D image file (png/jpg) or from a text prompt (TripoSR/TripoSG/TRELLIS.2). image_path, prompt, output"}, + {"list_skeletal_animations", "animation", "List the skeletal animations of an entity."}, + {"play_animation", "animation", "Play or stop a skeletal animation on an entity."}, + {"add_keyframe", "animation", "Add a keyframe to a bone track at a time."}, + {"merge_animations", "animation", "Merge animations from other files into the entity's skeleton."}, + {"resample_animation", "animation", "Resample an animation to N keyframes."}, + {"simplify_animation", "animation", "Remove redundant keyframes from an animation."}, + {"bake_animation_fps", "animation", "Re-grid every track to a uniform FPS."}, + {"generate_isometric_sprites", "animation", "Render an 8-direction isometric sprite atlas (optionally animated) to PNG."}, + {"generate_motion", "motion_ai", "Text-to-motion: generate a skeletal animation from a text prompt (walk, run, jump, dance, wave, idle...) on a humanoid rig."}, + {"motion_in_between", "motion_ai", "AI in-betweening: fill the gap between keyframes with plausible poses."}, + {"set_morph_weight", "morph_pose", "Set a morph target (blend shape) weight on the selected entity."}, + {"add_node_animation_clip", "node_animation", "Create a node transform (TRS) animation clip on a scene node (spinning props, doors)."}, + {"paint_set_enabled", "paint", "Enter or leave texture-paint mode on the selected mesh."}, + {"capture_face_from_video", "mocap", "Facial performance capture from a video file to blendshape keyframes."}, + {"cloud_upload", "cloud", "Upload a project to QtMesh Cloud."}, + }; +} + +// topTool: the tool expected on top; "a|b" accepts either (both are correct +// first moves — e.g. a colour change may create or modify a material). +struct Case { const char* request; const char* mustInclude; const char* topTool; }; + +} // namespace + +TEST(AIToolRouter, TokenizerStemsAndDropsStopwords) +{ + const QStringList t = AIToolRouter::tokenize("Rigging the Materials of auto_rig, please!"); + EXPECT_TRUE(t.contains("rig")) << t.join(",").toStdString(); + EXPECT_TRUE(t.contains("material")); + EXPECT_TRUE(t.contains("auto")); + EXPECT_FALSE(t.contains("rigging")) << "stemmed"; + EXPECT_FALSE(t.contains("the")); + EXPECT_FALSE(t.contains("please")); + EXPECT_EQ(AIToolRouter::tokenize("f22"), QStringList({"f22"})); +} + +TEST(AIToolRouter, IntentLexiconTranslatesUserWordsIntoToolVocabulary) +{ + const QStringList green = AIToolRouter::expandIntent(AIToolRouter::tokenize("make it green")); + EXPECT_TRUE(green.contains("material")) << green.join(",").toStdString(); + EXPECT_TRUE(green.contains(AIToolRouter::tokenize("diffuse").first())) << "terms are stored canonically stemmed"; + const QStringList create = AIToolRouter::expandIntent(AIToolRouter::tokenize("create a f22 raptor scene")); + EXPECT_TRUE(create.contains(AIToolRouter::tokenize("generate").first())); + EXPECT_TRUE(create.contains("prompt")); + EXPECT_TRUE(AIToolRouter::expandIntent(AIToolRouter::tokenize("xyzzy")).isEmpty()); + // generic verbs expand weakly, specific words strongly + const auto w = AIToolRouter::expandIntentWeighted(AIToolRouter::tokenize("make it green")); + EXPECT_LT(w.value(AIToolRouter::tokenize("generate").first()), w.value("material")) << "'make' must not outweigh 'green'"; + // stems are canonical on both sides: dance/dancing agree + EXPECT_EQ(AIToolRouter::tokenize("dancing"), AIToolRouter::tokenize("dance")); + EXPECT_EQ(AIToolRouter::tokenize("creating images"), AIToolRouter::tokenize("create image")); +} + +TEST(AIToolRouter, RoutingBenchmark) +{ + AIToolRouter router(corpus()); + // request → a capability the route MUST include, and the tool expected on top + const Case cases[] = { + {"create a f22 raptor scene", "generation_3d", "generate_mesh_from_image"}, + {"make me a dragon", "generation_3d", "generate_mesh_from_image"}, + {"generate a 3d model from this photo", "generation_3d", "generate_mesh_from_image"}, + {"make it green", "materials", "create_material|modify_material|apply_material"}, + {"change the material of luigi to red", "materials", nullptr}, + {"give the car a shiny metal look", "materials", nullptr}, + {"rig the wolf as a quadruped and skin it", "rigging", "auto_rig"}, + {"compute skin weights", "rigging", "compute_skin_weights"}, + {"remove the skeleton", "rigging", "remove_skeleton"}, + {"make the box twice as large", "scene", "transform_mesh"}, + {"move the crate 2 units to the left", "scene", "transform_mesh"}, + {"delete the cube", "scene", "delete_entity"}, + {"duplicate the sphere", "scene", "duplicate_entity"}, + {"what is in the scene?", "scene", "get_scene_info"}, + {"how many triangles does the mesh have", "scene", "get_mesh_info"}, + {"export luigi as glb", "scene_io", "export_mesh"}, + {"load the wolf obj", "scene_io", "load_mesh"}, + {"save the scene", "scene_io", "save_scene"}, + {"take a screenshot", "view", "take_screenshot"}, + {"show me the model from the front", "view", nullptr}, + {"make it walk", "motion_ai", "generate_motion"}, + {"animate the character dancing", "motion_ai", nullptr}, + {"reduce the polygon count by half", "mesh_optimize", nullptr}, + {"generate 3 lods", "mesh_optimize", "generate_lods"}, + {"unwrap the uvs", "uv", "auto_uv_unwrap"}, + {"segment the body into parts", "segmentation", nullptr}, + {"add a point light above the scene", "lighting", "create_light"}, + {"upscale the texture 4x", "textures_ai", "upscale_texture"}, + {"generate pbr maps from the albedo", "textures_ai", "generate_pbr_maps"}, + // non-English requests are the LLM intent round's job (AgentFixture), not the lexical router's + }; + int capHits = 0, toolHits = 0, toolCases = 0; + QStringList misses; + for (const Case& c : cases) { + const auto r = router.route(QString::fromUtf8(c.request)); + const bool cap = r.capabilities.contains(QLatin1String(c.mustInclude)); + if (cap) ++capHits; else misses << QStringLiteral("%1 → %2 (wanted %3)").arg(c.request, r.capabilities.join("+"), c.mustInclude); + if (c.topTool) { + ++toolCases; + const QStringList accepted = QString::fromLatin1(c.topTool).split('|'); + if (!r.scores.isEmpty() && accepted.contains(r.scores.first().name)) ++toolHits; + else { + QStringList top; + for (int i = 0; i < std::min(3, int(r.scores.size())); ++i) top << QStringLiteral("%1=%2").arg(r.scores[i].name).arg(r.scores[i].score, 0, 'f', 2); + misses << QStringLiteral("%1 → top tool %2 (wanted %3) [%4] +%5").arg(c.request, r.scores.isEmpty() ? "-" : r.scores.first().name, c.topTool, top.join(" "), r.expandedTerms.join(",")); + } + } + } + const int total = static_cast(sizeof(cases) / sizeof(cases[0])); + EXPECT_EQ(capHits, total) << "capability misses:\n " << misses.join("\n ").toStdString(); + EXPECT_GE(toolHits * 100 / std::max(1, toolCases), 85) << "top-tool misses:\n " << misses.join("\n ").toStdString(); + RecordProperty("capability_hits", capHits); + RecordProperty("top_tool_hits", toolHits); +} + +TEST(AIToolRouter, SceneIsAlwaysKeptAndCapabilitiesAreCapped) +{ + AIToolRouter router(corpus()); + const auto r = router.route("rig it, skin it, animate it walking, export it, light it and unwrap the uvs", 3); + EXPECT_EQ(r.capabilities.size(), 3); + EXPECT_TRUE(r.capabilities.contains("scene")) << r.capabilities.join(",").toStdString(); + EXPECT_TRUE(router.route("").capabilities.contains("scene")) << "an empty request still routes to scene"; +} + +TEST(AIToolRouter, ShortlistPrunesLargeCapabilitiesToTheRelevantTools) +{ + AIToolRouter router(corpus()); + const auto r = router.route("resample the walk animation to 30 keyframes"); + // animation has 8 tools in this corpus (> keepAll=4) → only the scoring ones, best first + const QStringList anim = router.shortlist(r, "animation", /*keepAll=*/4, /*maxTools=*/3); + ASSERT_FALSE(anim.isEmpty()); + EXPECT_LE(anim.size(), 3); + EXPECT_EQ(anim.first(), "resample_animation") << anim.join(",").toStdString(); + // a small capability is always shown whole + EXPECT_EQ(router.shortlist(r, "uv", 4, 3), QStringList({"auto_uv_unwrap"})); + // a large capability nothing matched is shown whole rather than hidden + EXPECT_EQ(router.shortlist(router.route("xyzzy"), "animation", 4, 3).size(), 8); +} diff --git a/src/CMakeLists.txt b/src/CMakeLists.txt index b686645b..bcf9ff39 100755 --- a/src/CMakeLists.txt +++ b/src/CMakeLists.txt @@ -241,6 +241,7 @@ AIChatManager.cpp AIAgentManager.cpp AIAgentTypes.cpp AICapabilityRegistry.cpp +AIToolRouter.cpp AIModelCatalog.cpp AppStorage.cpp WelcomeScreenController.cpp @@ -438,6 +439,7 @@ ClickFocusFilter.h AIAgentManager.h AIAgentTypes.h AICapabilityRegistry.h +AIToolRouter.h AIModelCatalog.h WelcomeScreenController.h WelcomeDialog.h diff --git a/tests/CMakeLists.txt b/tests/CMakeLists.txt index 393b6d91..0149101c 100644 --- a/tests/CMakeLists.txt +++ b/tests/CMakeLists.txt @@ -284,6 +284,7 @@ if(BUILD_TESTS) ${CMAKE_CURRENT_SOURCE_DIR}/../src/AIAgentManager.cpp ${CMAKE_CURRENT_SOURCE_DIR}/../src/AIAgentTypes.cpp ${CMAKE_CURRENT_SOURCE_DIR}/../src/AICapabilityRegistry.cpp + ${CMAKE_CURRENT_SOURCE_DIR}/../src/AIToolRouter.cpp ${CMAKE_CURRENT_SOURCE_DIR}/../src/AIModelCatalog.cpp ${CMAKE_CURRENT_SOURCE_DIR}/../src/AppStorage.cpp ${CMAKE_CURRENT_SOURCE_DIR}/../src/WelcomeScreenController.cpp