Skip to content

feat(agent): configure terminal-bench eval prompts - #247

Merged
Pepps233 merged 3 commits into
feat/terminal-bench-artifact-isolationfrom
feat/terminal-bench-eval-prompts
Sep 24, 2026
Merged

Pepps233 merged 3 commits into
feat/terminal-bench-artifact-isolationfrom
feat/terminal-bench-eval-prompts

Conversation

@Pepps233

@Pepps233 Pepps233 commented Sep 24, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Terminal-Bench evaluations now use the supplied fixed system prompt without the automatic <project_context> section.

The task user message ends with Do not cheat by using online solutions or hints specific to this task.

Reasoning

The pinned Diffex 0.3.1 CLI already supports --system-prompt and --no-context-files, so this change stays in the eval integration and requires no product release.

The pinned CLI does not propagate those flags to child agents; eval preflight therefore requires disabled subagents.

Changes

  • Preserve the original instruction bytes and append the anti-cheating sentence after the native file wrapper in the same user message.
  • Check in the exact eval prompt, suppress automatic context loading, and validate requested and observed prompt hashes.
  • Require disabled subagents and update both controlled and full eval configurations.
  • Update smoke tests, evidence checks, and documentation.

Testing

  • All 188 Python tests passed; the seven execution-contract tests passed again after adding full-config coverage.
  • node --test tests/launcher.test.cjs and all seven Linux cleanup regressions passed.
  • The pinned-release Linux arm64 execution smoke passed three trials and nine loopback provider requests, including repeated root runs and UID 65534.
  • Every captured provider request had the exact system prompt, no project context, and the anti-cheating sentence only in the user message.
  • npm run check and the repository's isolated ./test.sh passed.

Smoke evidence: results/diffex-execution-b57c6f5b84a4/smoke.json.
No paid model calls or full benchmark rerun were performed.

@Pepps233 Pepps233 added the enhancement New feature or request label Sep 24, 2026
@Pepps233 Pepps233 self-assigned this Sep 24, 2026
@Pepps233
Pepps233 added this pull request to stack #248 September 24, 2026 14:52
@Pepps233
Pepps233 merged commit 9d5f610 into main Sep 24, 2026
4 of 17 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant