[aperolschpritz]

Summary

Sharpens the QuickStack MCP surface for agents and adds an evaluation loop.

 MCP client
   tools/list
-  search_operations
-  execute_read_operation
-  execute_operation
+  quickstack_search_operations
+  quickstack_execute_read_operation
+  quickstack_execute_operation

   quickstack_search_operations
+    AND token search, relevance sort, limit, total/returned/hint
+    detail "full" requires a search term
     └─ schema
        quickstack_execute_read_operation | quickstack_execute_operation
+          results >48k chars truncated with a narrowing hint

• Discovery: namespaced tools, server version 1.1.0, server instructions, search limits and relevance sorting.
• Arguments and errors: REST field validation is passed through to tool errors. List endpoints always return a pagination envelope (items, total, limit, nextCursor), defaulting to 50 items when limit is omitted. This breaks callers expecting a bare array; clients must read items and follow nextCursor.
• Writes: route descriptions now include examples, saveApp/saveAgent upsert semantics, destructive side effects, and deployApp/deployAgent polling guidance.
• Descriptions: routes define the public OpenAPI text in description and the agent text in x-mcp-description. MCP prefers the override and falls back to description; the generated OpenAPI spec keeps the override as a valid vendor extension.
• Access modes: the Settings card now offers Off, Read only and Read & write. Read only registers the search and read tools and filters write operations out of search results. The legacy mcpServerEnabled flag is migrated to the new mcpServerMode parameter.
• Secret handling: response models mark secret fields with .meta({ secret: true }): environment variables, build arguments, registry passwords, Git tokens, SSH private keys, basic-auth passwords, webhook IDs, health-check headers, LLM admin keys and temporary sandbox access URLs/tokens. MCP responses redact those fields by default, for read and write tools. The new Send secrets in MCP responses toggle restores the full REST payload. Secret redaction applies only to MCP responses.
• Protocol: replaced deprecated -32002/-32001; verified the 2026-07-28 rules (-32002 must not be emitted, -32020..-32099 reserved) and added app codes 1001/1002 outside the reserved range. Only POST is exported, so GET/DELETE return 405.
• Evaluation: scripts/mcp-eval.mjs plus docs/mcp/evaluation.md, with scripted and LLM modes covering success rate, tool calls, errors, tokens and latency.

Files:

scripts/mcp-eval.mjs                       # scripted + LLM evaluation loop
docs/mcp/evaluation.md                     # harness mechanics and Inspector smoke test
src/server/utils/mcp-response.utils.ts     # MCP error codes and response helpers
src/server/mcp/mcp-secret-redaction.ts     # schema-driven secret removal
src/server/mcp/mcp-server.factory.ts       # tools, modes, search, truncation, instructions
src/server/mcp/operation-registry.ts       # MCP-description override + fallback
src/server/utils/api-operation-detail.utils.ts # typed x-mcp-description helper
src/shared/model/mcp-settings.model.ts     # access-mode schema and labels
src/shared/utils/mcp.utils.ts              # mode helper
src/shared/utils/pagination.utils.ts       # limit/cursor envelope helpers
src/server/utils/api-response.utils.ts     # date mapping preserves `.meta()`

Evidence

Before: tool names were unprefixed, every detail: "full" call could dump every schema, and MCP responses always mirrored REST secrets.

After:

$ yarn vitest run src/server/mcp src/__tests__/integration/server/mcp \
    src/app/api/mcp src/shared/model/mcp-secret-metadata.unit.spec.ts \
    src/server/utils/api-response.utils.unit.spec.ts
Test Files  8 passed (8)
Tests       53 passed (53)

$ yarn lint
Done in 10.97s

Full suite:

Test Files  5 failed | 126 passed | 1 skipped (132)
Tests       1029 passed | 44 skipped (1073)

The remaining failing suites (git.service.unit.spec.ts and the k3s
integration suites) need a container runtime:
Could not find a working container runtime strategy. They are unrelated to
this change.

yarn tsc --noEmit still reports only the three pre-existing errors in
unrelated spec files (actions.unit.spec.ts, format.utils.unit.spec.ts,
agent-sandbox.utils.unit.spec.ts).

MCP Inspector 2.12.0 against the real handler:

initialize  -> serverInfo: quickstack 0.1.0 (at the time of this run; now 1.1.0)
            -> instructions present
tools/list  -> quickstack_search_operations
               quickstack_execute_read_operation
               quickstack_execute_operation
            -> schema portability: 0 errors, 11 warnings

The scripted evaluation harness previously passed 3/3 fixture-based tasks in
mcp-eval.integration.spec.ts. The list-projects fixture still returns an array and does not reflect the current paginated REST response; its result does not validate that task against a real instance. The LLM mode was not run because this sandbox has
no live QuickStack instance or LLM credentials; the command is documented.

Merge Danger

Door: two-way.

The commit can be reverted, but the tool rename, access-mode setting and secret
redaction are consumer-visible. Existing MCP clients must update
search_operations / execute_operation / execute_read_operation to the
quickstack_ names.

Pagination is a breaking REST API change: affected list endpoints always return an envelope, with a default limit of 50 even when limit and cursor are omitted. Existing clients must read items and follow nextCursor to retrieve all results. The OpenAPI document now also carries
the additive x-mcp-description vendor extension on operations that define an
MCP-specific text.

Secret redaction is on by default. Agents that need secret fields must be
explicitly enabled by an admin with Send secrets in MCP responses. Write
arguments are still sent to QuickStack even when response redaction is on, so
agents should not paste secrets into prompts that should stay private.

Blast Radius: REST and MCP clients.

Agents that relied on detail: "full" without a search term now get a
corrective error; they must search first. Read-only mode intentionally hides the
write tool and write operations from search. Write descriptions now warn about
destructive behavior explicitly, which may change how agents ask for
confirmation.