diff --git a/.claude/harness-candidates.md b/.claude/harness-candidates.md index dd3912db..8bf1b8af 100644 --- a/.claude/harness-candidates.md +++ b/.claude/harness-candidates.md @@ -107,3 +107,23 @@ Deferred lint/test guardrails surfaced during reviews. Promote to a `CExxx` rule caught them. The cleanup plan explicitly deferred this as YAGNI for the one-time purge, but any future doc rename/deletion re-opens the same blind spot — caught in the 2026-07-03 open-source-docs-cleanup implementation run. + +## From PR #77 (command-executed shell-normalize) — CE030-to-criteria deferred + +- [ ] **Extend CE030 doc/schema-parity to the `SuccessCriterion` union** so a new + criterion (or field) can't ship undocumented. Attempted in PR #77 and reverted: + CI installs `--extra uipath`, and in that environment `coder_eval.models.criteria` + gains a `CliCalledCriterion` (fields `log`/`positional`) that is NOT present in a + plain checkout (it did not reproduce on macOS, whose lockfile resolution omits the + contributing linux-only component). It defeated every discriminator tried — union + membership, a `__module__` string filter (it is spoofed to `coder_eval.models.criteria`), + a genuine-module-attribute scan (it is `setattr` onto the module), and even an AST + parse of the `SuccessCriterion` union literal in `criteria.py` source (CI's imported + criteria module resolves to a file whose union literal already contains it). No + runtime OR source signal available in the lint could separate the injected criterion + from an in-tree one. Revisit only with a way to identify the in-tree criterion set that + is provably immune to the uipath integration — e.g. a hardcoded name allowlist of the + in-tree criteria (losing auto-coverage of new ones), or first understanding exactly how + that environment injects the criterion. Until then CE030 stays scoped to the four + top-level models; the `command_pattern`/`exclude_pattern` contract this PR changed is + documented in the Field descriptions and TASK_DEFINITION_GUIDE regardless. diff --git a/docs/TASK_DEFINITION_GUIDE.md b/docs/TASK_DEFINITION_GUIDE.md index 1e988237..f10a7ccb 100644 --- a/docs/TASK_DEFINITION_GUIDE.md +++ b/docs/TASK_DEFINITION_GUIDE.md @@ -796,6 +796,7 @@ Runs a command and checks the exit code, with optional stdout matching. **Binary | `expected_exit_code` | 0 | Expected exit code | | `expected_stdout` | `null` | When set, stdout is also checked | | `stdout_match` | `"exact"` | Match mode: `exact` (stripped), `contains` (substring), `regex` (pattern) | +| `score_from_stdout` | `false` | Read a float score (0.0–1.0) from the first stdout line (remaining lines become details); a non-zero exit code or a parse failure scores 0.0. Mutually exclusive with `expected_stdout`. | ### `file_matches_regex` @@ -823,6 +824,12 @@ Compares agent's code with a reference solution using similarity scoring. **Cont weight: 2.0 ``` +| Field | Default | Description | +|-------|---------|-------------| +| `agent_file` | *required* | Path to the agent's generated file (relative to the sandbox root). | +| `comparison_method` | `"ast"` | `ast` (structure), `token` (text), or `complexity` (metrics). | +| `similarity_threshold` | 0.8 | Minimum similarity score to pass (0.0–1.0). | + **Comparison methods:** - `ast` — Abstract Syntax Tree similarity (structure-based) - `token` — Token-based similarity (implementation details) @@ -841,6 +848,17 @@ Checks whether the agent executed specific tools/commands during evaluation. Ins description: "Agent must use curl to fetch weather" ``` +| Field | Default | Description | +|-------|---------|-------------| +| `tool_name` | `null` | Tool-name filter (e.g. `Bash`); `null` counts any tool. | +| `command_pattern` | `null` | Regex to match the command; `null` matches any command. Matched with shell normalization (see below). | +| `min_count` | 1 | Minimum matching commands required. `0` permits zero matches — combine with `max_count: 0` to assert a command must **NOT** run. | +| `max_count` | `null` | Optional inclusive upper bound. When set, the criterion passes iff `min_count <= matches <= max_count`. | +| `require_success` | `false` | Only count commands that completed successfully. | +| `exclude_pattern` | `null` | Regex that must NOT match; a command matching both `command_pattern` and `exclude_pattern` is skipped. Also matched with shell normalization (see below). | + +**Shell normalization.** For a Bash command, both `command_pattern` and `exclude_pattern` are matched against the raw command text **and** its shell-normalized form — the `bash`/`sh`/`zsh -lc "..."` wrapper stripped and shell quoting resolved with `shlex` — and a hit on *either* form counts. So a pattern like `curated_channels` matches whether the agent wrote the argument bare, `'single'`-quoted, `"double"`-quoted, or `\"escaped\"`; you do **not** hand-encode shell quoting. Because the same haystacks also feed `exclude_pattern` and the `max_count` gate, normalization is **not** purely additive: a quote-obfuscated call can now be caught by an exclusion or a `max_count: 0` gate that the raw text alone would have missed — and, conversely, an unedited `exclude_pattern` may now exclude a call it previously let through. Cross-repo suites that hand-encoded quote tolerance in their patterns should re-baseline. + **Codex limitation.** Codex agents map `Read`, `Grep`, and `Glob` tools to `shell` commands (they execute via bash), so `tool_name: "Read"` on Codex returns no matches. Use `tool_name: "Bash"` or `tool_name: null` (any tool) for Codex-compatible checks. This criterion works correctly on Claude Code agents, which emit separate `Read`/`Grep`/`Glob` telemetry. ### `cli_called` @@ -1026,6 +1044,8 @@ Have an LLM grade the task against a rubric written in the task YAML. **Continuo | `temperature` | `0.0` | Sampling temperature (0.0 = deterministic) | | `max_tokens` | `2000` | Maximum tokens in the judge's response | | `max_file_chars` | `20000` | Per-file (and agent_output) truncation applied before building the prompt | +| `capture_transcript` | `true` | Persist a `JudgeTranscript` (raw verdict + rendered prompts + token usage) to a sibling `judge-.yaml`. Set `false` to drop it when on-disk size matters (e.g. 1000-row datasets); the `findings` on the result persist regardless. | +| `max_transcript_chars` | `100000` | Aggregate cap on captured transcript text (verdict + prompt + system, split 60/30/10). Exceeding it marks the transcript `truncated=True`. | **Transport selection.** The judge call is routed by the active `API_BACKEND`: @@ -1099,6 +1119,8 @@ Spawn a full Claude Code SDK agent as the judge. Unlike `llm_judge` (a single LL | `max_turns` | `50` | Judge's inner-loop turn limit | | `turn_timeout` | `300` | Wall-clock timeout (seconds) | | `agent` | hardened judge defaults | Nested `AgentConfig` — `model`, `permission_mode`, `allowed_tools`, `disallowed_tools`, `ignore_patterns`, `sdk_options`. A partial block (e.g. only `model:`) still applies the judge security defaults for missing fields, and the security floor (`.claude` / `.mcp.json` / `_reference` ignore patterns, `setting_sources=[]`) is always enforced. | +| `capture_transcript` | `true` | Persist a `JudgeTranscript` (tool calls + token usage + raw verdict + rendered prompts) to a sibling `judge-.yaml`. Set `false` to drop the trajectory log when on-disk size matters; the `findings` on the result persist regardless. | +| `max_transcript_chars` | `100000` | Aggregate cap on captured transcript text (verdict + prompt + system + tool detail/result-preview lines, split 60/30/10 with tool calls prioritized). Exceeding it marks the transcript `truncated=True`. | **Security** diff --git a/src/coder_eval/criteria/command_executed.py b/src/coder_eval/criteria/command_executed.py index e5343ef2..69aad136 100644 --- a/src/coder_eval/criteria/command_executed.py +++ b/src/coder_eval/criteria/command_executed.py @@ -3,6 +3,8 @@ import json import logging import re +import shlex +from functools import lru_cache from typing import TYPE_CHECKING from coder_eval.criteria.base import BaseCriterion, CheckContext, LiveVerdict, register_criterion @@ -16,10 +18,127 @@ logger = logging.getLogger(__name__) -# Limit regex search input length to mitigate ReDoS on large command strings +# Limit regex search input length to mitigate ReDoS on large command strings. +# Normalization runs over this same truncated window (see _match_haystacks), so +# shlex never sees more than this many chars and needs no separate size guard. _MAX_PATTERN_SEARCH_LEN = 2000 +def _is_shell_program(arg0: str) -> bool: + """True if argv[0]'s basename looks like a POSIX shell. + + A predicate rather than an enumerated allowlist: the set of shells is open + (``bash``/``sh`` on Linux, ``zsh`` on macOS — Codex shells through the host's + login shell, see codex_agent.py — plus ``dash``/``ksh``/…), and every common + shell basename ends in ``sh``. Matching is additive (the raw text stays a + haystack), so favouring recall over a hand-maintained list is safe. + """ + return arg0.rsplit("/", 1)[-1].endswith("sh") + + +def _is_command_flag(tok: str) -> bool: + """True for a short-option cluster carrying a ``-c`` command string. + + Covers ``-c``, ``-lc``, ``-ic``, ``-lic`` (login/interactive + command) in + any order — a single ``-``-prefixed token whose option letters are all + alphabetic and include ``c``. ``--long`` options and ``-o=val`` forms are + rejected, so this is the first flag that actually introduces the command + string. The combined and split (``bash -l -c``) forms both work: a non-``c`` + short flag like ``-l`` simply isn't the command flag and the scan continues. + """ + return len(tok) >= 2 and tok[0] == "-" and tok[1] != "-" and tok[1:].isalpha() and "c" in tok[1:] + + +@lru_cache(maxsize=1024) +def _normalize_shell(cmd_text: str) -> str | None: + """Quote-resolved, wrapper-stripped form of a shell command, or None. + + ``command_pattern`` regexes are written against the *logical* command + (``uip is resources run list ``), but telemetry records the + raw ``bash -lc "..."`` wrapper — so whichever way the agent happened to quote + an argument (bare, ``"double"``, ``'single'``, ``\\"escaped\\"``) leaks into + the pattern. Authors then hand-model that escaping and get it subtly wrong + (e.g. allowing ``"`` but not ``'``), silently under-counting correct calls. + + This unwraps a ``bash``/``sh``/``zsh -c`` wrapper and resolves shell quoting + with ``shlex`` so a pattern can match argv semantics regardless of quoting. + Shell operators (``&&``, ``|``, ``>``) survive as their own tokens, so + patterns that reference them keep working, and embedded newlines collapse to + single spaces. Returns ``None`` when the text can't be parsed (an odd quote + count — NOT heredocs, which tokenize fine); the caller then keeps only the + raw text as a haystack. + + Adding a second (normalized) haystack is *not* purely additive: a + ``command_pattern`` can only gain matches, but the same haystacks feed + ``exclude_pattern`` and the ``max_count`` gate, so a normalized form can + newly satisfy an exclusion or trip a ``max_count`` cap — i.e. a command that + counted on the raw text alone can stop counting. See + ``CommandExecutedChecker._matching_commands``. + + Memoized (pure function of ``cmd_text``): the early-stop watcher re-scans the + whole accumulated trajectory on every tool-call event, so the same command is + normalized many times per run — the cache collapses that to once per distinct + (already-truncated) command string. + """ + try: + tokens = shlex.split(cmd_text, posix=True) + except ValueError: + return None + if not tokens: + return None + # Unwrap `bash -lc "