Skip to content

Latest commit

 

History

History
151 lines (128 loc) · 11.3 KB

harnesses.md

File metadata and controls

151 lines (128 loc) · 11.3 KB

Harnesses

A harness is the agent that is handed a prompt and a workspace and does the work: Claude Code, GitHub Copilot CLI, Codex, and so on. It is a different question from where the agent runs. --engine says where (on the host, or in a container); the harness says what. The engines drive a harness as a subprocess, so the contract between them is small, and this page is that contract.

A harness defines the agent once, for both answers to where. On the host the engine starts command(spec) as a subprocess: the executable resolved on the host, and the runner's environment underneath what env(spec) sets. A sandbox leg starts the same command inside the container, rather than delegating to a separate agent definition such as an inspect_swe one. Otherwise every agent would exist twice in this repo, with its own command line, environment, skill staging and event handling, free to drift apart.

Two harnesses are registered, claude-code and copilot, and the contract exists so that another is an adapter and a set of tests rather than a change to the engines. --harness chooses one; see usage.md.

The harnesses

claude-code copilot
Binary claude copilot
Skills directory .claude/skills .claude/skills
A skill activates by a Skill tool call a skill tool call
Session lists its skills yes (system/init) no; copilot skill list --json does, without a model call
Authenticates with ANTHROPIC_API_KEY or ANTHROPIC_AUTH_TOKEN COPILOT_GITHUB_TOKEN, GH_TOKEN or GITHUB_TOKEN
Config isolated by CLAUDE_CONFIG_DIR COPILOT_HOME
Prompt stdin stdin
A run is billed in dollars premium requests

Things to know about copilot before reading its numbers:

  • Its room always contains built-in skills. Copilot ships a few of its own, so they are present beside the skills a run stages and can win a case. They are not reported as extras, since they are the harness and not a leftover on the runner.
  • It reports its answer separately from its closing event. The harness remembers the last assistant message and attaches it to the end of the run.
  • It reports tokens once, at the end, and no dollar cost, so a spend cap in dollars cannot bind.
  • It runs unsandboxed with every permission granted (--allow-all), as the host engine's Claude Code does, because a non-interactive run cannot answer a prompt. Use a throwaway runner.
  • It uses its own model names (claude-sonnet-5.5, not sonnet), and models differ greatly in how many premium requests one run costs.

What a harness provides

A harness is a subclass of skillscope.harness.Harness, registered in skillscope/harness/__init__.py.

Member What it is
name The name reports and --harness use.
binary The executable's name. Whoever starts the process resolves it: on the host with shutil.which, in a container by the guest's PATH.
skills_dir Where, relative to the workspace, the harness discovers skills (.claude/skills for Claude Code). The engine stages each skill there.
has_skill_tool Whether activating a skill is a tool call in the stream. A harness that activates a skill by reading its SKILL.md sets this False, and routing reads activation from the file access instead.
prompt_on_stdin Whether the prompt is written to stdin. A harness that takes it as an argument builds it into command and sets this False; stdin is closed either way, because some CLIs wait on it.
require() Refuse to start on the host when the harness is not installed there. The default checks binary is on the runner's PATH. A leg that starts the agent in a container checks the image instead.
default_model, resolve_model(model) The model used when the caller names none, and how the harness names the one they did. A harness with its own names maps the generic aliases (opus, sonnet, haiku) onto them.
pin_model(model) The model a CI run is held to, so results stay comparable run to run. The default pins nothing.
credentials, can_isolate_room(environ) The environment variables a credential can come from. A routing run keeps the runner's own config and skills out of the session only when the harness authenticates from one of them.
supports_sandbox Whether a container leg exists (--engine sandbox).
cost_flags(max_budget_usd) The CLI's own spend controls for a case. Empty when it has none, which the report then says rather than implying a cap.
version() What <binary> --version prints, for the report.
complete(prompt, model) Ask the model one question with no tools and return its answer. The judge uses it for a harness with no API of its own to call. The default runs the harness's own command with RunSpec.tool_free set, so command must grant nothing in that case, and reads the answer through normalizer().
room(workspace, env) The skills a session started in workspace would load, for a harness that does not announce them at session start; None if it cannot say. It must not cost a model call. Host only: it runs the harness's executable from the host's PATH, unlike command and env, which stay neutral about where the agent runs. It bounds its own run, stopping at the command's deadline and killing what it started; the engine puts no timeout around it.
preflight(model) One trivial prompt through the harness's own command, to prove before any case runs that the credential, the model name and the network work. The default is enough for most harnesses.
command(spec) The argv that runs one case, headless, streaming JSON events. The executable is named (binary), never an absolute path on this machine, so the same command can be started on the host or in a container.
env(spec) What the harness sets in the process's environment, and nothing else: its own settings, and whatever points it at spec.config_dir so the runner's own settings and skills stay out of the session. Not the runner's environment (PATH, HOME, every key on the machine), which a container must not inherit. Whoever starts the process puts this on top of the environment it has.
host_command(spec), host_env(spec) What a run started on the host executes: command(spec) with its executable found by shutil.which (on Windows an npm-installed CLI is a .cmd shim that only that finds), and os.environ with env(spec) on top. A sandbox leg uses command and env directly.
normalize(event) Turns one event the harness printed into canonical events (below). Stateless.
normalizer() A translator for one run's events, which may keep state. The engine asks for a fresh one per case. Override it when an event cannot be translated on its own, for instance when the closing event does not repeat the answer and it has to be remembered from an earlier one. The default returns normalize.

spec is a RunSpec: the workspace, the prompt, the model and effort, an optional throwaway config directory, and any spend or session flags the caller built. The workspace and the config directory are strings naming a path wherever the agent runs: a host path on the host, a path inside the container in a sandbox.

The canonical event form

The engines read one event shape, and a harness translates into it. It is Claude Code's stream-json, reduced to what is read:

Event Carries
{"type": "system", "subtype": "init", "skills": [...]} The skills the session registered. Optional: a harness that cannot say reports nothing, and the room check says it did not run rather than reporting an empty room.
{"type": "assistant", "message": {"content": [{"type": "tool_use", "name": ..., "input": {...}}]}} A tool call. Activating a skill is a tool_use named Skill with input.skill set.
{"type": "user", "message": {"content": [{"type": "tool_result", "content": ...}]}} A tool result.
{"type": "result", "result": "..."} The end of the run, and its closing answer. A stream without one is a run that did not finish.

A harness whose stream already has this shape keeps the default normalize, which passes events through. Any other maps its events onto these and returns an empty list for events that carry nothing the engines read. What a harness remembers about a stream belongs in the translator normalizer() returns, never on the harness itself: one harness instance serves every case in a run, and cases run concurrently. Every stop rule, budget, score and judge sees canonical events only, never the raw ones, so each is written once for every harness.

Adding a harness

  1. Subclass Harness in skillscope/harness/<name>.py, fill in the members above, and register it in skillscope/harness/__init__.py.
  2. Record two real runs of the CLI, headless, on a trivial skill: one where it activates the skill and finishes, one where it finishes without activating one. Scrub tokens and paths. Put them in tests/fixtures/harness/<name>/trigger.jsonl and no_trigger.jsonl, with an expected.json saying what the engine should read out of each (see the Claude Code one) and the CLI version they were recorded with.
  3. Run python -m unittest tests.test_harness. The contract test runs once per registered harness and fails if there are no fixtures. It checks activation, completion, the closing answer, that the command carries the model and effort and no credential, that the config directory is isolated, that two runs translated at the same time do not share memory, and that a skill is staged where the harness looks and without its evals/ directory. It also checks that the harness refuses to build a tool-free command or builds one with none of the permission-granting flags it knows about. That is a list of known flags, not a guarantee: a harness with a differently named escape hatch would pass it, so a new harness's tool_free handling (what complete runs the judge with) needs a human look as well.
  4. Say in the pull request how the harness differs from the others in ways a reader of a report should know: how it activates skills, what it cannot report, and what it needs to authenticate.

A report from one harness is its own result. Harnesses send different system prompts, offer different tools, and activate skills in different ways, so a score from one is not comparable with a score from another.

What is not yet behind the contract

The contract covers staging, the command, the environment, the events, the model, the credential check, the spend controls, the preflight, the judge and the room check. Three things still assume Claude Code:

  • Claude Code's own judge is its inspect model, not complete; its command refuses tool_free rather than run a question with permissions skipped;
  • the sandbox leg: --engine claude-code still runs Claude Code through inspect_swe, with its own command line, environment, skill staging and event handling. It should start the harness's command(spec) in the container, so that each agent is defined once. Until it does, a harness other than Claude Code has no sandboxed leg;
  • .claude/skills is the path routing treats as a skills inspection, rather than a harness's own skills_dir.