feat(agent): configure terminal-bench eval prompts

evals/terminal-bench/ARTIFACTS.md

@@ -7,6 +7,15 @@ The backend waits up to 120 seconds for volume readiness and records its name, I
 The file retains a `mounts` list with session and environment names, so separate verifiers and restarted environments do not replace the agent's recovery location.
 Harbor's Docker-in-Docker backend binds the corresponding `/logs` subdirectories from the sandbox host into the task container.
 
+Sandbox creation allows an initial attempt plus three startup-timeout retries, with deadlines of 60, 60, 60, and 300 seconds.
+There is no retry backoff; each timed-out attempt first receives up to 20 seconds for sandbox lookup and deletion before the next create starts.
+The adapter uses one unique sandbox name across these attempts to recover accepted creates after lost responses and prevent duplicate allocation from late requests.
+`sandbox-startup.json` records attempt deadlines, outcomes, sandbox names, and cleanup results.
+Failed cleanup stops retries and retains any known sandbox reference for Harbor's final cleanup.
+Cancellation and non-timeout provider errors do not trigger creation retries.
+The fourth creation timeout raises `EnvironmentStartTimeoutError`; Harbor's overall environment deadline still applies, including volume preparation and cleanup overhead.
+These retries occur before agent execution and do not rerun evaluation attempts or change `--max-retries 0`.
+
 Agent home, installed task packages, caches, and mutable harness state live in a private `/tmp/diffex-tb-state-*` directory on sandbox disk.
 They remain available to task services through verification and disappear with sandbox deletion.
 The exported `state.json` records these paths without linking the state tree into the logs.

evals/terminal-bench/README.md

@@ -121,12 +121,13 @@ The unit checks cover installation pins, configuration validation, explicit cred
 They also cover outcome parsing, incomplete streams, transport failures, log-collection failures, and repeated cancellation during recovery.
 Converter checks cover deduplication, tool observations, nested child trajectories, auxiliary usage, unknown telemetry, images, and partial messages.
 The execution smoke installs the real npm release, then redirects a fixture model to a loopback faux provider with a test-only wrapper.
-It exercises shell and file tools, multiline instructions with shell metacharacters and Unicode, repeated trials, enabled child agents, and non-root execution without paid API calls.
+It exercises shell and file tools, multiline instructions with shell metacharacters and Unicode, repeated trials, and non-root execution without paid API calls.
 The fixture checks the actual request credentials without storing their values, verifies empty initial sessions and state, and scans retained artifacts for the fixture secrets.
-It also checks that task context and the exact release instructions reach the system prompt while project settings, extensions, and `SYSTEM.md` are ignored.
+It checks the supplied eval system prompt and the anti-cheating sentence in the user message against actual provider requests.
+Automatic project context, packaged release instructions, project settings, extensions, and `SYSTEM.md` are excluded from the system prompt.
 Only the agent log directory is mounted, and each invocation removes its container in a `finally` block.
 Generated evidence is retained under `results/diffex-execution-*/`.
-Each execution smoke validates ATIF with Harbor's validator and reconciles nonzero input, output, cached-token, and cost totals against every root and child response.
+Each execution smoke validates ATIF with Harbor's validator and reconciles nonzero input, output, cached-token, and cost totals against every response.
 The outcome smoke exercises success, authentication failure, exhausted retry, recovered retry, interrupted JSON output, and a Harbor-style execution timeout as both root and a non-root user.
 It uses the pinned npm release with a loopback provider and checks nonzero terminal-error exits and rejection of truncated output even when the process exits zero.
 Success, provider-failure, forced-CLI-death, and timeout cases create a TERM-resistant background service and verify that its PID and start time are unchanged across adapter cleanup.
@@ -256,7 +257,8 @@ uv run --locked harbor run --config execution.yaml \
 
 [execution.yaml](execution.yaml) records the default policy and allows 630 seconds for Harbor setup, encompassing the installer's 600-second deadline and transport overhead.
 The task's agent timeout remains in effect.
-Use `--ak thinking=medium`, `--ak tools=read,bash,edit,write`, or `--ak subagents=disabled` to change the corresponding policy explicitly.
+Use `--ak thinking=medium` or `--ak tools=read,bash,edit,write` to change the corresponding policy explicitly.
+Subagents must remain disabled; `--ak subagents=enabled` is rejected during preflight.
 Inspect accepted options with `uv run --locked harbor agent schema diffex_terminal_bench.adapter:DiffexAgent`.
 Unknown options, arbitrary native config, prompt templates, injected skills, and MCP servers are rejected.
 Model IDs must include a supported provider prefix; fuzzy patterns and thinking suffixes are rejected.
@@ -287,37 +289,39 @@ The adapter uploads the complete UTF-8 instruction and runs `diffex --mode json
 The CLI receives stdin EOF through `/dev/null` and the launcher's ignored stdin.
 Native `@file` transport adds Diffex's standard `<file name="...">` wrapper while retaining all instruction bytes, including leading/trailing whitespace and CRLF line endings.
 Instruction contents never become shell command text or command-line flags.
-The observer extension checks that the first user message contains exactly that wrapper and the original instruction.
-It registers only a `message_start` observer, preserving compatibility with child agents.
+The launcher appends `Do not cheat by using online solutions or hints specific to this task.` after the native file wrapper, separated by a blank line, in the same user message.
+The original instruction file and its hash remain unchanged.
+The observer extension checks that the first user message contains exactly the wrapper, original instruction, and anti-cheating suffix.
+It registers only a `message_start` observer and does not rewrite provider requests.
 
 Each invocation creates an exclusive `run-<uuid>/` under Harbor's agent logs.
 Its `DIFFEX_CODING_AGENT_DIR` is a fresh private `/tmp/diffex-tb-state-*` directory on sandbox disk, with a fresh home and global `settings.json` containing `{"evolution":{"enabled":false}}` before startup.
 Packages and caches remain outside the exported logs and survive until sandbox teardown so retained task services can use them during verification.
 No prior settings, credentials, sessions, or evolved harness selections are copied into it.
 Startup network updates are disabled with the supported offline setting so the pinned model catalog remains stable.
 
-The `context-only` project policy retains ancestor `AGENTS.md`/`CLAUDE.md` context and disables project trust, discovered extensions, skills, prompt templates, and themes.
-It does not pass `--no-context-files` or replace the system prompt.
-The pinned npm `0.3.1` release includes `dist/AGENTS.md`, which matches the adapter's checked-in release instructions.
-The adapter retains that packaged context without adding a global duplicate.
-The effective system prompt must contain the release instructions, and their SHA-256 is recorded.
+The eval adapter passes `--system-prompt` with the checked-in [eval prompt](src/diffex_terminal_bench/eval-system-prompt.txt) and `--no-context-files`.
+The entire automatic `<project_context>` section is omitted, including packaged release instructions and ancestor `AGENTS.md`/`CLAUDE.md` files and their paths.
+Task files remain on disk and can be read explicitly by the agent.
+The `context-only` project resource setting continues to disable project trust, discovered extensions, skills, prompt templates, and themes; eval launches additionally suppress automatic context files.
+The effective system prompt must match the checked-in prompt's SHA-256, and execution evidence must confirm context files are disabled.
+This uses supported flags in the pinned Diffex release and requires no product code or release changes.
 
-The checked-in `benchmark.toml` disables subagents, omitting `spawn_agent`, `agent_action`, and `agent_status` from the explicit tool allowlist.
-The separate controlled smoke configuration in `execution.yaml` enables subagents and includes those tools.
-Diffex decides whether to call those tools; enabling them does not require a child on every task.
-Children share their trial's sandbox, and the pinned CLI owns child scheduling and its default concurrency limit.
-The staged runner's `-n/--concurrency` controls simultaneous trials and sandboxes separately.
-An adapter invocation that omits the subagent policy still defaults to disabled, omitting coordination tools.
-Root and child sessions are retained together beneath `run-<uuid>/sessions/`, including the parent-owned child session directories.
+All eval configurations require disabled subagents and omit `spawn_agent`, `agent_action`, and `agent_status` from the tool allowlist.
+The pinned Diffex 0.3.1 CLI does not propagate `--system-prompt` or `--no-context-files` to child agents, so enabled subagents cannot satisfy this prompt policy.
+The adapter rejects attempts to enable them before setup or execution.
+The staged runner's `-n/--concurrency` still controls simultaneous independent trials and sandboxes.
+Native sessions remain beneath `run-<uuid>/sessions/`; trajectory conversion continues to support retained child sessions from earlier evaluations.
 
 ### Execution evidence
 
 Each run directory retains:
 
-- `requested.json`: pins, model, thinking, tools, subagent policy, project policy, credential variable names, and instruction hash.
+- `requested.json`: pins, model, thinking, tools, subagent policy, project policy, credential variable names, instruction hash, user-message suffix, and system prompt hash.
 - `instruction.txt`: the original instruction.
-- `effective.json`: resolved model, thinking, tools, working directory, session path, prompt hashes, and release-context verification.
-- `system-prompt.txt`: the effective system prompt, including release and task context.
+- `eval-system-prompt.txt`: the requested eval system prompt.
+- `effective.json`: resolved model, thinking, tools, working directory, session path, prompt hashes, user-message verification, and context-file suppression.
+- `system-prompt.txt`: the observed system prompt, without automatic project context.
 - `stdout.jsonl`, `stderr.log`, and `process.json`: native JSON events, diagnostics, and process status.
 - `launcher.json` and `cli.json`: launcher and root CLI PID/start-time identities.
 - `ownership/*.json`: durable shell ownership records published by the CLI.

evals/terminal-bench/RELEASE_GATE.md

@@ -29,8 +29,8 @@ The preparation command writes `results/release-gate/dataset.json` without creat
 Harbor 0.23.0 and Daytona SDK 0.210.0 are locked in the benchmark's private Python environment.
 Node 22.23.2, Diffex 0.3.1, npm integrity, registry, and Node archive checksums come from [pins.toml](src/diffex_terminal_bench/pins.toml).
 The default is `openai/gpt-5.6-luna`, `max` thinking, tools `read,bash,edit,write,update_plan`, disabled subagents, disabled evolution, and `context-only` project resources.
-Enabled subagents add `spawn_agent`, `agent_action`, and `agent_status` to the model's tools; Diffex chooses whether to use them.
-Children execute inside the parent trial's sandbox and their sessions and usage remain in the trial evidence.
+Enabled subagents are rejected because the pinned CLI does not propagate eval prompt and context-file flags to children.
+Every new eval uses the checked-in system prompt without automatic project context and appends the anti-cheating instruction to the user message.
 Change and commit the configuration before starting a new qualification sequence.
 Every subsequent stage must match the exact recorded identity, including the adapter commit and dependency lock.
 The earlier TB 2.0 / Diffex 0.2.2 oracle evidence cannot qualify this configuration.
@@ -237,10 +237,10 @@ Use `-n 1` throughout that sequence while capacity remains constrained; task dea
 Each stage writes `gate.json` before starting remote work and updates it after every completed trial, including failures.
 The report separates verified `trials` from `failures`, records `completed_trials` and `expected_trials`, and keeps `failure` as the first observed failure.
 Failure entries identify the task, trial index, evidence directory, error type, and phase; a cleanup error following an execution error is retained in `additional_errors`.
-It records dataset/archive and per-task digests, adapter commit, Harbor/Daytona/Python versions, dependency lock hash, installation pins, model, thinking, tools, subagent policy, and hashes of adapter/harness files and the release instructions.
+It records dataset/archive and per-task digests, adapter commit, Harbor/Daytona/Python versions, dependency lock hash, installation pins, model, thinking, tools, subagent policy, and hashes of adapter/harness files and the eval system prompt.
 Each Diffex trial additionally retains the effective system prompt and its SHA-256.
-Full system prompt hashes can differ between isolated trials because context includes trial-specific paths.
-The release instructions and harness source identity must remain consistent.
+The effective system prompt must exactly match the checked-in eval prompt across isolated trials.
+The user-message suffix, disabled automatic context files, and harness source identity are also verified.
 
 Trial directories contain Harbor `config.json`, `lock.json`, `result.json`, verifier reward and combined stdout/stderr, agent logs, native sessions, and sandbox lifecycle evidence.
 Diffex installation evidence includes the three CLI checks and verified npm/Node pins.

evals/terminal-bench/execution.yaml

@@ -11,6 +11,6 @@ agents:
     kwargs:
       thinking: high
       tools: read,bash,edit,write,update_plan
-      subagents: enabled
+      subagents: disabled
       project_resources: context-only
       background_process_lifetime: environment

evals/terminal-bench/full.yaml

@@ -14,7 +14,8 @@ agents:
     kwargs:
       thinking: max
       tools: read,bash,edit,write,update_plan
-      subagents: enabled
+      # Child agents in the pinned CLI do not inherit eval prompt flags.
+      subagents: disabled
       project_resources: context-only
       background_process_lifetime: environment
     env: