All commands accept --json for structured output.
Exit codes: 0 ok · 2 usage · 3 task/validation · 4 agent · 5 run failure · 8 regression threshold · 9 secrets detected · 130 interrupted.
| Command | Purpose |
|---|---|
doctor |
Environment, all four registries (tools, agents, scorers, sandboxes) and plugin discovery |
init [--agent A|ollama|llama_cpp|lm_studio|vllm|sglang] [--command CMD] [--dir D] [--task ID] [--ci github|gitlab|jenkins|circleci] [--no-ci] [--no-run] [--force] |
Set up this project against your agent: writes tooltrace.config.json and a CI workflow, then runs one real task and reports the outcome. Overwrites nothing without --force, and never writes a credential — openai_compat gets the name of an env var. A failing first run is still a successful init: that is a measurement, not a setup problem. --ci writes configuration for GitLab, Jenkins or CircleCI as well as GitHub, each at the path that system actually reads -- a correct pipeline in the wrong file is an inert file that looks like coverage |
tasks [--category] [--agent A] |
List bundled tasks. runnable_here accounts for missing toolchains on this machine; --agent additionally accounts for the adapter, because a task carrying an image is not runnable against one that cannot be sent an image and a listing that did not ask cannot know |
agents [--vision] |
List registered agent adapters, including any a plugin contributed. --vision says where an image goes in each adapter's request, or why it goes nowhere -- a task carrying an attachment is skipped against an adapter with nowhere to put one, never scored, because a text-only model asked what is in a screenshot answers fluently and wrongly. Sending an image and the model being able to read it are different facts, and only the first is checkable here |
tools [--dialect openai|anthropic|mcp|gemini|prompt] [--equivalence] [--seeds] |
What a model is actually told about the tools it may call: name, description and the declared argument schema. --dialect renders the catalogue as one provider receives it, which is the only way to see that a schema survives the translation -- Gemini rejects oneOf and half the other JSON Schema keywords, and narrowing a union there is reported in the description rather than done silently. The plain output ends with how many registered tools declare a schema at all, because "nothing is validated" and "everything passed validation" look identical from outside. --equivalence asks the other question: does declaring a tool once mean the same thing to all four? The comparison is on meaning rather than bytes -- a dialect that renames a key is equivalent, one that drops a constraint is not. --seeds says which adapters pass a seed to the model and what that guarantees, which is nothing: every provider that accepts one documents it as best-effort |
backends [--json] |
Local model servers this project knows -- Ollama, llama.cpp, LM Studio, vLLM, SGLang -- with their ports, model-name conventions and which is listening. All five speak the OpenAI chat API and run through the one openai_compat adapter; detection probes localhost only, and an open port is evidence something is listening rather than a positive identification of the server |
run --task ID --agent A [--agent-config JSON] [--out DIR] |
One deterministic task against one agent; writes a bundle |
benchmark --agent A [--runs N] [--task ids] [--limit N] [--shuffle] [--seed S] [--shard i/n] [--context-sweep] [--min-success-rate] [--budget X] |
Repeated reliability runs with pass@k/pass^k-aware summaries and trajectory metrics. --limit runs a recorded subset for CI; the policy, seed and selected ids are reported and a subset announces itself on stderr. --budget is a hard ceiling: the sweep stops rather than continuing, warns at 80%, and the summary reports what did not run. An unpriced run is not counted against it, and the budget says so. --shard i/n (zero-based, like a CI matrix index) runs one slice of the selection so a sweep can be split across machines; a shard announces on stderr that its numbers describe a slice, and an empty shard exits 0 because sharding a small task set is not an error |
fleet enqueue|work|status|collect --queue DIR ... |
Run one sweep across a fleet of workers. A file queue, not a broker: a shared directory is something a lab, a CI cache or an NFS mount already has, and a claim is os.replace, which is atomic everywhere this runs -- two workers racing for one job cannot both win. enqueue fills the queue with one job per task-run and skips what the agent cannot be given; work claims jobs and runs them with bounded concurrency, resuming from its own state file and isolating a failed item to that item; status reports pending/claimed/mergeable plus this machine's inventory; collect merges the workers' states and refuses on a conflict rather than picking a winner |
merge SHARD_DIRS... [--out DIR] |
Combine the shards of one sweep into a set that can be read as a whole. Reading a rate off a single shard reads the rate of whichever tasks landed in that bucket, and because the sharding is deterministic that number is stable and wrong in the same way every time -- which is worse than noisy. Refuses on a conflict rather than picking a winner: two shards reporting different results for the same run means the runs were not what they claim to be, and keeping one silently would hide that behind a clean-looking total |
power [--runs N | --detect D] [--baseline-rate R] |
What a planned sweep can detect, before you spend anything on it. Detecting a 10-point difference at a 50% baseline needs ~393 runs per arm; a 5-point difference ~1570 |
showdown --agents a,b [--runs N] [--limit N] [--shuffle] [--seed S] |
Fair multi-agent ranking on identical cohorts. Emits {standings, verdict, note, ranking_is_provisional}; the verdict is ranked only when the sample is large enough and the leader's confidence interval clears the runner-up's, otherwise not distinguishable at this sample size. Also carries a power block (what this sample could have detected, which is what makes that verdict readable), a Beta-Binomial bayesian posterior, and a variance split separating nondeterminism from task diversity |
compare --baseline B --current C [--metrics m1,m2] |
Metric-by-metric comparison of two single-run bundles |
pr-report --baseline DIR --current DIR [--out F] [--metrics m1,m2] |
Compare two sets of runs and render a pull-request comment. Five verdicts per metric: regressed, improved, no_change_detected, inconclusive, and not_measured. Only an established regression exits non-zero (8) — a change has to be both real (its interval excludes zero) and large enough to matter. Refuses to compare different task sets or artifact versions. Gates on token count as well, so a prompt change that leaves every score identical and doubles the spend is caught by the one metric that would notice. A run reporting no usage is not_measured rather than uncertain -- more runs will not fix that, an adapter that reports the number will. --metrics narrows the gate: on a shared CI runner wall_ms is wall-clock rather than a property of the agent, and a gate that fails because of a noisy neighbour gets switched off -- which costs the four metrics that were worth gating on |
baseline --name N --bundle PATH |
Record a named baseline |
regression --baseline B --current C --thresholds JSON |
CI gate for score/tool/latency regressions |
a2a-card CARD [--key-env KID=ENV_VAR] [--key KID=SECRET] |
Check an A2A Agent Card, and say what its signature does and does not prove. Conformance and provenance are separate: a card can be perfectly formed and unsigned, and most in the wild are. The verification key comes from you, never from the card -- a signature checked against a key the document itself names proves only that the document agrees with itself. An algorithm this package cannot verify is reported no_verifier rather than assumed valid. Prefer --key-env, which names an environment variable: every other credential path in this project takes the name of a variable rather than its value, and a secret in argv is readable by any process on the machine while the command runs and lands in shell history after. --key still works and says so on stderr |
mcp-conformance [--url URL] [-- COMMAND...] |
Check an MCP server against the protocol, over stdio or HTTP. --url reaches a hosted server; the checks do not change with the transport, which is the point of running them over each. Exits non-zero only on a required failure; a missing tool description is reported as recommended, not a violation. Defaults to the bundled fixture |
mcp-scan (--from FILE | --registry URL) [--markdown] |
Score many MCP servers at once: conformance, protocol versions and malformed input per server, worst first. A command may only come from a file on this machine. A registry fetched over the network contributes URL targets and nothing else -- executing a command string that arrived from a server on the internet is remote code execution, and a reputable registry changes how likely that is to be abused, not what the code does. A fetched entry naming a command is reported as skipped with the reason, because a silent drop reads as a pass |
mcp-versions [--markdown] [--url URL] [-- COMMAND...] |
Handshake once per published MCP revision and report what came back. Neither supported nor unsupported is the finding: answering a revision you do not implement with one you do is correct negotiation. The finding is a server that echoes back whatever it was sent, which is why a version that cannot exist is in the list -- such a server is agreeing rather than negotiating, and the mismatch surfaces later as a field missing for no visible reason |
mcp-fuzz [--markdown] [-- COMMAND...] |
Send malformed JSON-RPC at an MCP server and report what it did. Three severities: accepting a must_reject case is a violation, tolerating a should_reject one is sloppy and does not fail, and a may_reject case conforms either way. A crash is always a problem -- a server that dies on malformed input is a denial-of-service surface whatever the spec says about the input |
redaction --bundles DIRS [--out DIR] |
What a bundle had removed, what personal-data shapes remain, and what nobody can certify. Exits 9 only when a secret pattern still matches after sanitisation, which is a defect in the sanitiser; personal data is reported and never fatal, because whether an email address matters depends on whose it is and where the bundle is going. A clean scan is not proof of absence -- every detector matches a shape, and names have none. Not differential privacy, and says so: DP means noise, and a bundle exists to be reproduced byte for byte |
evidence --bundles DIRS [--out DIR] [--framework nist-ai-rmf|iso-42001] [--history] |
Assemble an evidence dossier for a regulated review: runs, verification status, and what each obligation is and is not evidenced by. Never a compliance determination. --framework re-files the same facts against NIST AI RMF or ISO/IEC 42001, and each mapping lists the controls no benchmark can evidence so a partial mapping is never read as coverage. --history says when each obligation first became evidenceable at all -- a dossier produced before a capability shipped is silent on that obligation for a reason unrelated to the agent |
verify BUNDLE [--no-schema] [--no-integrity] [--signature F] |
Check a bundle's checksums, schema conformance and anti-gaming integrity (dropped assertions, a task modified after publication, an expected answer visible in the prompt); read-only, exits 5 on any problem. --signature additionally verifies a cosign signature: checksums are tamper-evident (they detect a change), a signature establishes who produced the bundle |
sign --bundles DIRS [--signer TOOL] [--verify] |
Sign bundles with cosign, or say plainly that they are unsigned. Checksums are tamper-evident -- they detect that a bundle changed; a signature establishes who produced it, which is the question an independent reader has, and verify --signature could check one long before anything here could produce one. No custom cryptography: this shells out to the external tool and reports honestly when it is absent. Exits 0 with nothing signed, because a machine without cosign is a normal machine and an optional step that failed the build would be switched off |
reproduce BUNDLE [--out DIR] [--no-rerun] |
Verify hashes and optionally re-run |
attest BUNDLE [--attester WHO] [--signature S] [--promote] |
Reproduce a bundle and record who did it. A re-run on the machine that produced the bundle stays LOCAL -- it demonstrates determinism, not independent reproduction. An unsigned attestation from another machine reaches COMMUNITY_VALIDATED; a signed one reaches REPRODUCED. MAINTAINER_VERIFIED is never awarded by a machine |
card [--bundles DIR] [--agent A] [--out F] |
A system card generated from recorded runs. A task with fewer than 10 runs is reported as insufficiently measured rather than as a capability or a limitation, and the not measured section is generated too |
self-audit [--bundles DIR] |
Would the evidence you hold demonstrate anything? A checklist with named gaps, deliberately not a percentage. Exits 0 even with gaps: a gap is a finding, not a build failure |
counterfactual --task ID --agent A [--agent-config JSON] [--runs N] |
Would the task still pass with one tool taken away? Two agents can score identically and be doing entirely different things -- one has a plan and adapts, the other walks a path it has walked before -- and every other metric here reads a run in which everything worked. redundant does not mean useless: the agent found another way, possibly a worse one. unused on a security task usually means the tool is the attack surface and a resistant agent not touching it is the pass condition |
hardware [--bundles DIRS] |
The machine, where a run's wall time went, and four efficiency questions. Prefill overhead is exact and nobody measures it: the system prompt and tool catalogue are charged on every turn of every run. Cache hit rate comes from what the provider reported; a run that said nothing is excluded rather than counted as a miss. TTFT is unmeasured with the reason -- no adapter streams, and dividing total latency would measure the length of the reply. Energy is read from RAPL or nvidia-smi where they exist, and is never estimated from a datasheet TDP. Carbon is not computed at all: grid intensity varies by hour. A quantization curve is drawn only when nothing else varied -- two quantizations benchmarked on different machines produce a curve of the two machines |
perturb --task ID [--perturbation kind:tool] [--runs N] [--min-recovery-rate R] |
Inject safe faults and measure recovery rate; --out writes bundles |
drift --current DIR [--baseline DIR] [--objective R] |
Has behaviour moved between two windows of runs? Watches five metrics, not just accuracy -- an agent whose success rate held while its step count doubled has changed, and silent_decay names that case. --objective turns an SLO into an error budget: how many failures are left |
shadow --recorded TRACE (--candidate TRACE | --task ID --agent A) |
Did a candidate make the same decisions as a recorded production run? Not outcomes -- and the difference matters: the recorded run happened against real systems holding real state, and re-running a candidate in a temp workspace is a different task sharing an objective. A divergence is a place to look rather than a fault: the candidate may be doing something better, and a production log is not an oracle. Without --candidate, the candidate is run here, because producing a second trace by hand is where most people stop |
promote-trace TRACE --task-id ID [--out F] |
Turn a production trace into a draft regression task. Generates the trajectory assertions and refuses to invent a workspace or a correctness assertion, leaving a TODO for each -- a generated task that looked finished would run, pass, and test nothing |
trace BUNDLE [--filter SUBSTR] [--assertions] [--limit N] |
Inspect a bundle trace in the terminal (checksum-verified) |
| Command | Purpose |
|---|---|
report --bundles DIR... [--format json|csv|md|junit|html] [--output F] |
Aggregate bundles into a report |
badge (--summary F | --bundles DIR) [--out F] [--svg-url U] [--link U] |
Render an embeddable reliability badge. The sample size is always in the message, and the colour comes from the confidence interval's lower bound rather than the rate — 10 of 10 runs is 100% with a lower bound near 72%, so it renders amber, not green. Writes a shields.io endpoint JSON beside the SVG |
cost --bundles DIR [--forecast-tasks N --forecast-runs N] [--human-baseline X] |
Where the money went (by task, and by how the run ended), what a planned sweep would cost, and whether it beats a human baseline. --human-baseline has no default: a viability verdict against an invented baseline is an opinion, not a measurement |
platforms --bundles DIR --out DIR [--target langfuse|phoenix|datadog|honeycomb|grafana] [--base-url U] |
Convert a run set for an observability platform, and print the push command without sending anything. Most of them already ingest OTLP, so what ships for those is the endpoint, the auth header and the environment variable each reads its key from -- not five exporters posting the same bytes to five paths. W&B and MLflow are run-based rather than span-based and get real converters. Langfuse also gets its scores: it models an evaluation score as an object on a trace, not a span attribute, so a plain OTLP push arrives complete and unscored |
export --out DIR [--stdin] |
Run exporter plugins on a payload |
serve --dir web/dist [--host] [--port] |
Serve the built frontend locally |
--agent-config accepts inline JSON, or @path to read a JSON file:
tooltrace run --task file-editing/fix-config-typo --agent subprocess --agent-config @tooltrace.config.json@path exists so that the file tooltrace init writes is a file the other
commands can read. It also keeps a long config out of shell history and out of
whichever shell's quoting rules you are subject to.
| Command | Purpose |
|---|---|
owasp [--markdown] |
OWASP Agentic Top 10 coverage, computed from the installed packs. A category counts as covered only when a task declares it and that task runs here; declared, not runnable is a third state because nothing has been measured |
lint [--path PACK] |
Task-lint: ambiguous scoring, unreachable assertions, undeclared side effects, missing cleanup, unsafe network, and allowed_tools naming a tool that is not registered (every call to which fails as "unknown tool", so every agent appears to satisfy any assertion depending on it) |
dry-run --task ID |
Validate fixtures/assertions/sandbox lifecycle without any model |
self-test |
Harness self-test: sandbox cleanup, scoring determinism, monotonic timers, fixture/trace integrity |
snapshot --source DIR --output F [--changelog S] [--verify] |
Generate/verify hashed dataset snapshots |
govern provenance|index|duplicates|contamination |
Where the task data came from, and whether a model could already have seen it. provenance builds a manifest hashing every authored file a task carries and --verify checks one back, reporting content added since as uncovered rather than passing it -- checking only the manifest's own entries would let new, unvouched-for files through. index builds a versioned per-pack index and --requires checks a semver range against it, saying plainly when a range is one it cannot read instead of reporting the pack incompatible. duplicates groups tasks sharing a fingerprint across packs -- two tasks with one fingerprint measure the same thing twice. contamination reports public-exposure signals per task and flags any author declaration that sits below the evidence, which is the only direction that makes the benchmark look better than it is. Declared evidence, never proof: none means no signal matched |
validate --path PACK · task validate/test/scaffold |
Schema validation, pack tests, scaffolding |
| Command | Purpose |
|---|---|
import --format swe-bench|bfcl|tau-bench|agentbench --source FILE [--out DIR] |
Convert tasks from another benchmark into drafts. Every one of those measures something real and none measures what this project measures, so a converted task is not the original task -- what each conversion drops travels in the task's own metadata rather than being printed once. AgentBench environments graded by a model judge are refused rather than approximated: an approximated oracle is a task that scores something nobody chose |
online --source FILE --state DIR [--out F] [--policy P] [--rate R] [--seed S] |
One incremental pass over production traffic: process what is new since the last pass and leave a cursor behind. Nothing holds a connection open -- a benchmark that runs a daemon is one somebody has to operate. The cursor records the sampling policy, not just the position: a window sampled two ways is not comparable to itself, so a policy or harness change starts a new window rather than silently continuing the old one |
sample --source FILE [--out F] [--policy stratified|uniform|all] [--rate R] [--seed S] |
Choose which production traces to score. Uniform is the usual default and the wrong one: the thing worth finding is failure, failure is rare, and 1% of traffic yields 1% of the failures. Stratified keeps everything that already looks wrong -- which makes the sample deliberately biased, so the policy is written beside the traces and estimate_rate inverts the weights to recover the population figure. Selection is deterministic given the seed |
ingest --format F --in FILE [--out JSONL] [--task-id ID] [--agent A] [--score-against TASK_ID] |
Convert external traces (OTel GenAI spans / OpenAI steps) into ToolTrace events, and optionally score the trajectory against a task's trace assertions; workspace assertions are reported as skipped |
Ingested traces flow through tooltrace trace, replay and scoring unchanged — see
tooltrace.ingest. The pytest plugin (pytest11
entry point) exposes run_tooltrace / assert_tooltrace_pass fixtures so tasks run
as native pytest tests; see Plugins.
| Command | Purpose |
|---|---|
server [--host] [--port] |
Self-hosted API: REST + SSE + /metrics + /healthz + /readyz + /openapi.json (RBAC, quotas, audit) |
See Self-hosting for tokens, roles and policies.