Skip to content

Record every tool call in an audit log - #1554

Open
midego1 wants to merge 1 commit into
UsefulSoftwareCo:mainfrom
midego1:claude/tool-call-audit-log
Open

Record every tool call in an audit log#1554
midego1 wants to merge 1 commit into
UsefulSoftwareCo:mainfrom
midego1:claude/tool-call-audit-log

Conversation

@midego1

@midego1 midego1 commented Aug 7, 2026

Copy link
Copy Markdown

The gap

Executor keeps no record of tool usage. A run that calls GitHub or Search Console leaves one HTTP line —

INFO http.span: Sent HTTP response { "http.method": "POST", "http.url": "/mcp", "http.status": 200 }

— and nothing about which integration, which tool, or what came back. There is no executions/audit table in the schema, and packages/core/analytics is anonymous by construction: its own header forbids tool addresses, connection names and arguments, which are exactly the fields an audit needs. The executor.tool.execute span carries the right data, but only where an OTel exporter is configured, and it can't answer "which connection did this agent use last week".

Two questions were therefore unanswerable after the fact: what did this agent touch, and what did my policies actually stop.

What this adds

execute in packages/core/sdk/src/executor.ts is the one place every call passes through — every plugin kind, every host, MCP and REST alike — so the row is written there, from an Effect.onExit wrapper that sees every way a call can end:

outcome meaning
ok reached the upstream and succeeded
fail the tool's own error result — which rides the success channel by design, so a channel-only reading records an upstream 404 as a healthy call
blocked a block policy stopped it
declined a human refused the approval
error tool/connection missing, plugin not loaded, transport broke

blocked and declined are the rows that make this worth having: both end before any request is made, so nothing upstream ever saw them and no HTTP-level observation can.

Each row carries the address as called, its integration/connection/tool, the governing policy (action + pattern), the duration, and the top-level argument names.

Readable three ways:

  • executor.toolCalls.list({ integration, connection, outcome, since, limit })
  • GET /api/tool-calls with the same filters
  • an Activity page in the console

Read-only by construction: a log a caller can edit is not evidence, so there is no write or delete endpoint. executor.toolCalls.prune({ before }) exists for retention, and nothing schedules it — an audit log that silently deletes itself on a default nobody chose seemed worse than one that grows.

What it deliberately does not store

Arguments, results, and any text that came from outside:

  • Messages are never persisted. A failed call keeps its code; plugins derive error.message from upstream response bodies (the OpenAPI plugin lifts it straight out), which routinely echo the request back — token included.
  • Codes must look like codes. ToolError.code is typed as any string, so a plugin can forward a body into it. Anything not matching an identifier shape is dropped; the outcome column already says what happened.
  • Argument names, never values — and only names that look like parameters. execute takes unknown args, so { "ghp_realtoken": null } is a reachable shape; names are length-bounded, identifier-shaped, and credential-shaped ones are dropped.

Failure behaviour

Writing a row can never change the outcome of the call it describes. The write is wrapped in catchCause, so an insert failure — or a defect — is logged and swallowed. It is awaited on purpose (a forked write would be interrupted when a per-request host tears the executor down, and a silently missing row is the one thing an audit log may not do), but awaited under a 2s cap, so a sick database costs a tool call that timeout rather than the driver's own.

Scope and follow-ups

  • Local/self-host/Cloudflare need no migration: ensureDrizzleRuntimeSchemaFromTables creates the table at boot from coreTables. Cloud gets 0016_nosy_expediter.sql.
  • apps/local/src/db/executor-schema.ts is left alone — it already predates artifact/subject, and it only drives a generate-time baseline.
  • Known limitation: a subject's view spans two partitions (its own rows plus the org's), so a newest-first read across both still sorts. Serving it takes a (tenant, created_at) index and the schema layer has no non-unique index API yet. Worth adding before this table gets large; I left a note at the table definition rather than adding a unique index whose leading columns wouldn't serve the query.

Verification

  • New suite packages/core/sdk/src/tool-call-log.test.ts — 18 tests: outcome classification per ending (including a decline raised inside a handler, which arrives wrapped in ToolInvocationError), the redaction rules with real-looking secrets, and executor-level assertions that a blocked call and a declined approval each leave a row.
  • format:check, lint, typecheck (44/44) clean; packages/core/sdk 613, packages/core/api 93, packages/react 320 tests pass. src/oauth-flow.test.ts flakes on its local OAuth test server under the parallel run (a different test each time, 3/3 green in isolation) — unrelated to this change.
  • Reviewed with codex review over two rounds; every finding from both is addressed in the branch.

🤖 Generated with Claude Code

Executor kept no record of tool usage. A run that called GitHub or Search
Console left one HTTP line (POST /mcp 200) and nothing about which integration,
which tool, or what came back — and the analytics catalog is anonymous by
construction, so it deliberately drops exactly those fields. Two questions were
therefore unanswerable after the fact: what did this agent touch, and what did
my policies actually stop.

`execute` is the one place every call passes through, whatever the plugin kind
and whatever the host, so the row is written there, from an `onExit` wrapper
that sees every way a call can end:

- ok / fail — reached the upstream. `fail` is a tool's own error result, which
  rides the SUCCESS channel by design and would otherwise be recorded as a
  healthy call.
- blocked / declined — never left the gateway. A policy stopped it, or a human
  refused the approval. These leave no other trace anywhere: they end before
  any request is made.
- error — the tool or connection did not exist, the plugin failed to load, the
  transport broke.

Arguments and results are never stored: an argument can be a credential. The
row keeps the top-level argument NAMES, which is what an audit needs without
the table becoming a place secrets accumulate.

Writing a row can never change the outcome of the call it describes — a failed
write is logged and swallowed, because an audit trail that can take the gateway
down with it is worse than one with a gap in it.

Readable three ways: `executor.toolCalls.list()`, `GET /api/tool-calls`
(filter by integration, connection, outcome, time), and an Activity page in the
console. Read-only by construction — a log a caller can edit is not evidence,
so there is no write or delete endpoint.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant