A harness is the agent that is handed a prompt and a workspace and does the
work: Claude Code, GitHub Copilot CLI, Codex, and so on. It is a different
question from where the agent runs. --engine says where (on the host, or in
a container); the harness says what. The engines drive a harness as a
subprocess, so the contract between them is small, and this page is that
contract.
A harness defines the agent once, for both answers to where. On the host the
engine starts command(spec) as a subprocess: the executable resolved on the host,
and the runner's environment underneath what env(spec) sets. A sandbox leg starts
the same command inside the container, rather than delegating to a separate agent
definition such as an inspect_swe one. Otherwise every agent would exist twice in
this repo, with its own command line, environment, skill staging and event handling,
free to drift apart.
Two harnesses are registered, claude-code and copilot, and the contract exists
so that another is an adapter and a set of tests rather than a change to the
engines. --harness chooses one; see usage.md.
claude-code |
copilot |
|
|---|---|---|
| Binary | claude |
copilot |
| Skills directory | .claude/skills |
.claude/skills |
| A skill activates by | a Skill tool call |
a skill tool call |
| Session lists its skills | yes (system/init) |
no; copilot skill list --json does, without a model call |
| Authenticates with | ANTHROPIC_API_KEY or ANTHROPIC_AUTH_TOKEN |
COPILOT_GITHUB_TOKEN, GH_TOKEN or GITHUB_TOKEN |
| Config isolated by | CLAUDE_CONFIG_DIR |
COPILOT_HOME |
| Prompt | stdin | stdin |
| A run is billed in | dollars | premium requests |
Things to know about copilot before reading its numbers:
- Its room always contains built-in skills. Copilot ships a few of its own, so they are present beside the skills a run stages and can win a case. They are not reported as extras, since they are the harness and not a leftover on the runner.
- It reports its answer separately from its closing event. The harness remembers the last assistant message and attaches it to the end of the run.
- It reports tokens once, at the end, and no dollar cost, so a spend cap in dollars cannot bind.
- It runs unsandboxed with every permission granted (
--allow-all), as the host engine's Claude Code does, because a non-interactive run cannot answer a prompt. Use a throwaway runner. - It uses its own model names (
claude-sonnet-5.5, notsonnet), and models differ greatly in how many premium requests one run costs.
A harness is a subclass of skillscope.harness.Harness, registered in
skillscope/harness/__init__.py.
| Member | What it is |
|---|---|
name |
The name reports and --harness use. |
binary |
The executable's name. Whoever starts the process resolves it: on the host with shutil.which, in a container by the guest's PATH. |
skills_dir |
Where, relative to the workspace, the harness discovers skills (.claude/skills for Claude Code). The engine stages each skill there. |
has_skill_tool |
Whether activating a skill is a tool call in the stream. A harness that activates a skill by reading its SKILL.md sets this False, and routing reads activation from the file access instead. |
prompt_on_stdin |
Whether the prompt is written to stdin. A harness that takes it as an argument builds it into command and sets this False; stdin is closed either way, because some CLIs wait on it. |
require() |
Refuse to start on the host when the harness is not installed there. The default checks binary is on the runner's PATH. A leg that starts the agent in a container checks the image instead. |
default_model, resolve_model(model) |
The model used when the caller names none, and how the harness names the one they did. A harness with its own names maps the generic aliases (opus, sonnet, haiku) onto them. |
pin_model(model) |
The model a CI run is held to, so results stay comparable run to run. The default pins nothing. |
credentials, can_isolate_room(environ) |
The environment variables a credential can come from. A routing run keeps the runner's own config and skills out of the session only when the harness authenticates from one of them. |
supports_sandbox |
Whether a container leg exists (--engine sandbox). |
cost_flags(max_budget_usd) |
The CLI's own spend controls for a case. Empty when it has none, which the report then says rather than implying a cap. |
version() |
What <binary> --version prints, for the report. |
complete(prompt, model) |
Ask the model one question with no tools and return its answer. The judge uses it for a harness with no API of its own to call. The default runs the harness's own command with RunSpec.tool_free set, so command must grant nothing in that case, and reads the answer through normalizer(). |
room(workspace, env) |
The skills a session started in workspace would load, for a harness that does not announce them at session start; None if it cannot say. It must not cost a model call. Host only: it runs the harness's executable from the host's PATH, unlike command and env, which stay neutral about where the agent runs. It bounds its own run, stopping at the command's deadline and killing what it started; the engine puts no timeout around it. |
preflight(model) |
One trivial prompt through the harness's own command, to prove before any case runs that the credential, the model name and the network work. The default is enough for most harnesses. |
command(spec) |
The argv that runs one case, headless, streaming JSON events. The executable is named (binary), never an absolute path on this machine, so the same command can be started on the host or in a container. |
env(spec) |
What the harness sets in the process's environment, and nothing else: its own settings, and whatever points it at spec.config_dir so the runner's own settings and skills stay out of the session. Not the runner's environment (PATH, HOME, every key on the machine), which a container must not inherit. Whoever starts the process puts this on top of the environment it has. |
host_command(spec), host_env(spec) |
What a run started on the host executes: command(spec) with its executable found by shutil.which (on Windows an npm-installed CLI is a .cmd shim that only that finds), and os.environ with env(spec) on top. A sandbox leg uses command and env directly. |
normalize(event) |
Turns one event the harness printed into canonical events (below). Stateless. |
normalizer() |
A translator for one run's events, which may keep state. The engine asks for a fresh one per case. Override it when an event cannot be translated on its own, for instance when the closing event does not repeat the answer and it has to be remembered from an earlier one. The default returns normalize. |
spec is a RunSpec: the workspace, the prompt, the model and effort, an
optional throwaway config directory, and any spend or session flags the caller
built. The workspace and the config directory are strings naming a path wherever
the agent runs: a host path on the host, a path inside the container in a sandbox.
The engines read one event shape, and a harness translates into it. It is Claude Code's stream-json, reduced to what is read:
| Event | Carries |
|---|---|
{"type": "system", "subtype": "init", "skills": [...]} |
The skills the session registered. Optional: a harness that cannot say reports nothing, and the room check says it did not run rather than reporting an empty room. |
{"type": "assistant", "message": {"content": [{"type": "tool_use", "name": ..., "input": {...}}]}} |
A tool call. Activating a skill is a tool_use named Skill with input.skill set. |
{"type": "user", "message": {"content": [{"type": "tool_result", "content": ...}]}} |
A tool result. |
{"type": "result", "result": "..."} |
The end of the run, and its closing answer. A stream without one is a run that did not finish. |
A harness whose stream already has this shape keeps the default normalize,
which passes events through. Any other maps its events onto these and returns an
empty list for events that carry nothing the engines read. What a harness remembers about a stream belongs in the
translator normalizer() returns, never on the harness itself: one harness
instance serves every case in a run, and cases run concurrently. Every stop rule,
budget, score and judge sees canonical events only, never the raw ones, so each
is written once for every harness.
- Subclass
Harnessinskillscope/harness/<name>.py, fill in the members above, and register it inskillscope/harness/__init__.py. - Record two real runs of the CLI, headless, on a trivial skill: one where it
activates the skill and finishes, one where it finishes without activating
one. Scrub tokens and paths. Put them in
tests/fixtures/harness/<name>/trigger.jsonlandno_trigger.jsonl, with anexpected.jsonsaying what the engine should read out of each (see the Claude Code one) and the CLI version they were recorded with. - Run
python -m unittest tests.test_harness. The contract test runs once per registered harness and fails if there are no fixtures. It checks activation, completion, the closing answer, that the command carries the model and effort and no credential, that the config directory is isolated, that two runs translated at the same time do not share memory, and that a skill is staged where the harness looks and without itsevals/directory. It also checks that the harness refuses to build a tool-free command or builds one with none of the permission-granting flags it knows about. That is a list of known flags, not a guarantee: a harness with a differently named escape hatch would pass it, so a new harness'stool_freehandling (whatcompleteruns the judge with) needs a human look as well. - Say in the pull request how the harness differs from the others in ways a reader of a report should know: how it activates skills, what it cannot report, and what it needs to authenticate.
A report from one harness is its own result. Harnesses send different system prompts, offer different tools, and activate skills in different ways, so a score from one is not comparable with a score from another.
The contract covers staging, the command, the environment, the events, the model, the credential check, the spend controls, the preflight, the judge and the room check. Three things still assume Claude Code:
- Claude Code's own judge is its inspect model, not
complete; its command refusestool_freerather than run a question with permissions skipped; - the sandbox leg:
--engine claude-codestill runs Claude Code throughinspect_swe, with its own command line, environment, skill staging and event handling. It should start the harness'scommand(spec)in the container, so that each agent is defined once. Until it does, a harness other than Claude Code has no sandboxed leg; .claude/skillsis the path routing treats as a skills inspection, rather than a harness's ownskills_dir.