Add prompt-rule regression tests to the agent eval suite - #3210
Merged
Conversation
Catches mechanical regressions in apps/cli/ai/system-prompt.ts and the installed skills at PR time via 10 single-turn tests that grade a specific rule each (wp_cli shell-syntax avoidance, theme.json button neutralization, block decompose, ABSPATH-eval post updates, etc.). The suite imports the live buildSystemPrompt from source so it can't fall out of date, runs on Sonnet 4.5 with deterministic assertions, and blocks merge on failure for PRs that touch the prompt files. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Drops the parallel setup added in 4335dc7 (apps/cli/ai/tests/promptfoo + .github/workflows/prompt-eval.yml). Studio already has an agent eval suite at eval/promptfoo.config.yaml that runs the real agent via startAiAgent() and a Studio-authed grader; the new tests will live there. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds 10 single-turn, prose-graded cases covering mechanical rules that have silently regressed before (wp_cli shell syntax, theme.json button neutralization, block decompose, ABSPATH-eval post updates, etc.). Each prompt asks the agent to narrate in prose so the assertions can grep `textSegments` without spinning up a real site. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
youknowriad
changed the title
promptfoo's exec provider tries to hash a file from the first token of the command to build a cache key. `exec:node ../apps/cli/dist/cli/eval-runner.mjs` starts with `node`, which isn't a file, so promptfoo logs "Could not hash any files" warnings on every run. Point it at eval/run-agent.sh instead. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Reverts the prose-regression tests added in 8199be3 and the shell wrapper added in c07ea03. Keeps a single new case: building a one-page site must keep every assistant turn under 40s of wall-clock time. eval-runner now emits `turnDurationsMs` (delta between successive assistant messages) so the assertion can flag the slowest turn with context. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The suite was getting stuck at 75% for three independent reasons, all of
which compounded into "one stuck test plus three errored":
1. promptfoo's default maxConcurrency (4) let two site-building cases run
at the same time, fighting over Studio state. Pin maxConcurrency to 1.
2. Studio tools and pi-tui spinners print to process.stdout throughout a
run. promptfoo's exec provider caps stdout at 1 MB (node's default
child_process buffer), so long builds died with
ERR_CHILD_PROCESS_STDIO_MAXBUFFER and the assertion saw a truncated
table instead of JSON. Redirect process.stdout.write to stderr for the
duration of the run; serialize the result payload to a tmp file instead
and print only an EVAL_RUNNER_RESULT_FILE=<path> marker via a raw
fs.writeSync(1, ...) that bypasses the stream wrapper.
3. The Claude Agent SDK keeps internal handles open after the conversation
ends (its `claude` subprocess, ipc pipes). We've already written the
result file, so bail out with process.exit() rather than letting the
event loop drain indefinitely.
On top of that, the existing JavaScript assertions all called
`require('fs')`, which broke under promptfoo 0.121 because assertions run
via eval() inside an ESM context where require is undefined. Rewrite
every javascript assertion to use `import('node:fs').then(...)` and
return a Promise.
Also adds an opt-in heartbeat (EVAL_RUNNER_HEARTBEAT + optional log file)
so "is it stuck or just slow" is answerable without ps-walking the
process tree.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two fixes so the suite now passes 4-for-4: 1. Identity test's llm-rubric was grading the agent's stdout, which under the new runner contract is just `EVAL_RUNNER_RESULT_FILE=/tmp/…json` (the marker, not the agent's prose). The rubric reasonably judged "that's not an identity claim" and failed. Replace the rubric with a deterministic check against the runner payload's `textSegments`: agent must mention "WordPress Studio" and must not claim to be a generic LLM. 2. The single-page-build turn-cadence threshold was 40s. In practice the first big `Write` after an initial `Bash` probe lands in the 40–50s band (agent generating multi-KB theme file content). 40s was too tight and made the test flaky. Bump to 60s — still well below anything a user would feel as stuck, but above observed p99. Verified locally: 4/4 pass, duration 2m 23s. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The single-page-build-turn-cadence test creates a site named "Eval Turn Timing". Without a cleanup step, a second run lands on a directory that already exists and `site_create` errors out, which triggers the agent to backtrack (site_list → site_info → re-read → …) and inflates the total turn count. The extra work also pushes the first substantive post-Skill turn past our 60s bar because the model has to reason through recovery. Mirror the pattern the `site-creation` test already uses — list, delete if present, then create — scoped explicitly to this site only. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The heartbeat was only ever used while diagnosing the "stuck at 75%" failures — it's not load-bearing. Drop it along with the EVAL_RUNNER_HEARTBEAT* env vars, the appendFileSync import, and the turn-by-turn log writes. Also collapse the verbose rationale comments in the runner and config (`Sink the runner's JSON ... two problems this solves ...`, `Each case drives the real Studio Code agent ...`, etc.) down to one sentence each. Net: 119 lines removed, 28 added. Suite still passes 4/4 in 6m 14s. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
youknowriad
marked this pull request as ready for review
April 24, 2026 13:08
Contributor
Author
|
I'm merging cause I need this on a separate PR. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Related issues
How AI was used in this PR
Claude Code drafted the test cases and assertions. Verified locally that the YAML validates cleanly (`npx promptfoo@0.121.4 validate -c eval/promptfoo.config.yaml`). The real agent runs have not yet been executed — they need `npm run eval` against an authed Studio session.
Proposed Changes
Adds ten prompt-rule regression cases to the existing agent eval suite at `eval/promptfoo.config.yaml`. Each pins a specific mechanical rule that has silently regressed in the past; together they guard against losing recent prompt evolution.
Each case asks the agent to narrate an answer in prose (no tool execution), so assertions can grep `d.textSegments.join('\n')` for the load-bearing substrings. That keeps the runs fast and self-contained: no real site, no permission loop, no filesystem side effects.
The ten rules under test
Why the existing harness
Studio already has an agent eval suite at `eval/` that runs the real agent via `startAiAgent()` and grades via Studio's WP.com-authed proxy. An earlier commit on this branch added a parallel standalone promptfoo setup — that was the wrong call, and it's reverted in `4804b783` in favor of this integration. The assertion style (parse `output` JSON, inspect `textSegments` / `toolCalls`) matches the existing cases in the suite so reviewers don't have to learn two patterns.
Scope boundaries
Prerequisite
The tests are designed against the rules from #3207 (two-phase build + `blockify` skill). If #3207 has not yet merged when this PR is evaluated, several cases (2, 4, 5, 8, 9, 10) will fail on this branch because their rules are not yet in `trunk`. Recommended order: merge #3207 first, rebase this PR, then run `npm run eval`.
Testing Instructions
Validate YAML:
```sh
npx promptfoo@0.121.4 validate -c eval/promptfoo.config.yaml
```
→ "Configuration is valid."
Full eval run (builds the CLI, then runs all 13 cases — 3 existing + 10 new):
```sh
studio auth login
npm run eval
npm run eval:view
```
Single case (fast iteration):
```sh
npm run eval -- --filter-pattern "wp_cli filter uses eval"
```
Deliberate regression sanity-check: temporarily remove the `Do NOT use --post_content-file` line from `apps/cli/ai/system-prompt.ts`, rebuild, rerun the suite — `apply-content-abspath-eval` should fail with a clear `not-contains` violation.
Pre-merge Checklist
🤖 Generated with Claude Code