feat(sandbox): generate CLI recording shims via record_cli

docs/TASK_DEFINITION_GUIDE.md

@@ -17,6 +17,7 @@ Complete reference for defining evaluation tasks in Coder Eval.
 - [Agent Configuration](#agent-configuration)
 - [Run Limits](#run-limits)
 - [Sandbox Configuration](#sandbox-configuration)
+  - [Recording CLI Invocations](#recording-cli-invocations)
 - [Template Sources](#template-sources)
 - [Success Criteria](#success-criteria)
   - [Continuous Scoring](#continuous-scoring)
@@ -513,6 +514,33 @@ Under `driver: tempdir` only `timeout` is enforced — the agent can consume
 arbitrary host memory, CPU, and PIDs. Use `driver: docker` when you need the
 container limits above to actually bind.
 
+### Recording CLI Invocations
+
+`record_cli` shadows executables with generated recording shims, so a task can assert on **what the agent actually ran** without hand-writing a mock:
+
+```yaml
+sandbox:
+  record_cli:
+    - tool: uip
+      exit_code: 1
+      stderr: "uip: not connected to a tenant in this sandbox.\n"
+    - tool: curl                   # so a disobedient agent cannot reach the network
+```
+
+Each shim records the invocation, writes the configured `stdout`/`stderr`, and exits with `exit_code` — which **defaults to 1**, so a bare `- tool: curl` makes the shadowed tool look like it failed. Set `exit_code: 0` when the agent should see success. Values outside 0-255 are rejected, since `sys.exit` truncates mod 256.
+
+`tool` must be a bare executable name, and a small reserved set (`python`, `python3`, `env`, `sh`, `bash`, `node`, `git`, `uv`, `cmd`) is refused: shadowing those breaks the harness itself rather than the tool under test — the shim's own interpreter, or the shell that `run_command` criteria use.
+
+The sandbox writes the shims into `cli_mocks/` and PATH-prepends that directory, then appends one JSON record per invocation to `cli_mocks/calls.jsonl` — the log [`cli_called`](#cli_called) reads by default. Nothing else to wire: no `mock_path_dirs`, no `template_sources`, no `log:` on the criterion.
+
+Notes:
+
+- **A `.cmd` twin** is generated beside each shim so a bare `uip` also resolves through Windows PATHEXT lookup.
+- **The log is seeded empty**, so a correct run that legitimately calls nothing still satisfies a `max_count: 0` guard — while a *missing* log (mock never ran, or wrote elsewhere) still fails.
+- **stdin is never read** by the shim: reading it would block whenever the sandbox leaves stdin attached to an open pipe, hanging the task.
+- **Collisions are rejected.** If a `mock_path_dirs` entry already provides an executable of the same name, setup raises rather than letting directory order decide which one runs.
+- **It stubs a tool; it does not proxy one, and it does not serve per-invocation responses.** Recording a *real* executable on the way through, or returning different output per invocation, stays a hand-written mock under `mock_path_dirs` — both depend on state the harness cannot guarantee (the tool being installed, PATH order, live credentials, a fixture set).
+
 ## Template Sources
 
 Tasks can start with preset files instead of an empty sandbox. Multiple sources are applied sequentially (last wins for conflicts).
@@ -870,7 +898,7 @@ Use this instead of `command_executed` or `file_matches_regex` when a test shado
 ```yaml
 - type: "cli_called"
   description: "Switched the project to the capable model"
-  log: "mocks/calls.jsonl"            # Path to the JSON Lines invocation log (required)
+  log: "mocks/calls.jsonl"            # Invocation log; omit it to use the record_cli default
   verb: "ixp projects configure-model" # Ordered prefix of the non-flag arguments
   positional: ["my_invoices-ixp"]      # Non-flag arguments following the verb, in order
   flags:
@@ -881,6 +909,8 @@ Use this instead of `command_executed` or `file_matches_regex` when a test shado
   ignore_flags: ["output"]             # Flags dropped before matching (default: ["output"])
 ```
 
+`log` defaults to `cli_mocks/calls.jsonl`, where [`sandbox.record_cli`](#recording-cli-invocations) writes — so a task using generated recorders never sets it. Point it elsewhere only when supplying your own mock.
+
 **Log format.** One JSON object per line. Only `argv` is required; `tool` lets one log serve several shadowed executables, and `exit`/`ts` are recorded for reporting rather than matched. Unknown keys are ignored, so a mock may record more.
 
 ```json
@@ -925,7 +955,7 @@ Defaulting to "switch" is deliberate: `--yes` / `--force` / `-y` before the targ
 
 `ignore_flags` drops a flag from matching but does **not** make it value-bearing — an ignored flag that takes a value must also appear in `value_flags` (as `output` does by default). Otherwise `ignore_flags: ["verbose"]` on `delete --verbose proj-1` would let `--verbose` eat `proj-1`.
 
-**Limitation: bundled short flags are not split.** `-rf` parses as one flag named `rf`, so a predicate on `f` will not see it — including `absent: true`, which passes despite `-rf` being present. Assert on the long spelling, or add the bundled form via `aliases`. Likewise a bare negative number in flag position (`seek -1`) is read as a flag named `1`.
+**Clustered short flags are split, and declarations win.** `-rf` matches predicates on `r` and `f` — so a `-yf` cannot escape an `aliases: ["y"]` guard. If your CLI has a genuine multi-character short flag, naming it (in `flags`, `value_flags`, or `ignore_flags`) keeps it whole; and `-fvalue` binds when `f` is value-bearing. A bare negative number stays positional (`seek -1`), unless you declare a flag by that name (`head -1`).
 
 **Negative guards want the FEWEST facets that capture the forbidden act.** This is the opposite of a positive assertion, and it is easy to get backwards. `max_count: 0` passes when *nothing matches*, so every facet you add is another way for the real invocation to slip past the pattern and report a false PASS.
 

src/coder_eval/criteria/cli_called.py

@@ -1,12 +1,12 @@
 """CLI-called criterion checker — structured matching over an invocation log."""
 
-import json
 import logging
 import re
 import shlex
 from typing import TYPE_CHECKING, Any
 
 from coder_eval.criteria.base import BaseCriterion, CheckContext, register_criterion
+from coder_eval.invocation_log import parse_log
 from coder_eval.models import CliCalledCriterion, CriterionResult, FlagMatch
 
 
@@ -21,6 +21,7 @@ def _split_flags(
     argv: list[str],
     ignore: frozenset[str],
     value_flags: frozenset[str],
+    known_names: frozenset[str] = frozenset(),
 ) -> tuple[list[str], dict[str, list[str]]]:
     """Split ``argv`` into non-flag arguments and a flag map.
 
@@ -33,8 +34,9 @@ def _split_flags(
     ``ignore`` names are dropped with their values. ``--`` ends flag parsing and is
     itself dropped; a lone ``-`` is positional.
 
-    Known limitation: bundled short flags are not split, so ``-rf`` is one flag
-    named ``rf`` and a predicate on ``f`` will not see it.
+    ``known_names`` are the flag names the criterion mentions at all (including
+    presence predicates and aliases). A declared name is always taken whole, so a
+    genuine multi-char short flag still matches; undeclared ones are split.
     """
     positional: list[str] = []
     flags: dict[str, list[str]] = {}
@@ -63,6 +65,28 @@ def record(name: str, value: str) -> None:
             continue
 
         name = token.lstrip("-")
+        known = name in value_flags or name in known_names
+
+        # A bare negative number is a value, not a flag. Reading `-1` as a flag
+        # named `1` drops it from the positionals -- the same silent-disappearance
+        # that let `--yes proj-1` slip a delete past a guard.
+        if not known and _is_number(name):
+            positional.append(token)
+            continue
+
+        # Clustered short flags: `-rf` is `-r -f`. Declared names win, so a real
+        # multi-char short flag still matches, and `-fvalue` binds when `f` takes
+        # a value; otherwise each character is its own switch, which is what stops
+        # `-yf` escaping an `aliases: [y]` predicate.
+        if not known and not token.startswith("--") and len(name) > 1:
+            head, rest = name[0], name[1:]
+            if head in value_flags:
+                record(head, rest)
+            else:
+                for char in name:
+                    record(char, "")
+            continue
+
         if name in value_flags and index < len(argv):
             record(name, argv[index])
             index += 1
@@ -73,6 +97,14 @@ def record(name: str, value: str) -> None:
     return positional, flags
 
 
+def _is_number(text: str) -> bool:
+    try:
+        float(text)
+    except ValueError:
+        return False
+    return True
+
+
 def _flag_matches(predicate: FlagMatch, values: list[str] | None) -> bool:
     """Whether a recorded flag satisfies one :class:`FlagMatch` predicate.
 
@@ -100,19 +132,6 @@ def _flag_matches(predicate: FlagMatch, values: list[str] | None) -> bool:
     raise AssertionError(f"FlagMatch has no matcher arm: {predicate!r}")
 
 
-def _usable_argv(record: dict[str, Any]) -> list[str] | None:
-    """The record's ``argv`` when it is a list of strings, else None.
-
-    None means the record cannot be evaluated at all — a different thing from
-    "evaluated and did not match", which is why the caller reports it rather than
-    quietly treating it as a non-match.
-    """
-    argv = record.get("argv")
-    if isinstance(argv, list) and all(isinstance(item, str) for item in argv):
-        return argv
-    return None
-
-
 def _record_matches(criterion: CliCalledCriterion, argv: list[str], record: dict[str, Any]) -> bool:
     """Whether one log record satisfies every configured facet of the criterion."""
     if criterion.tool is not None and record.get("tool") != criterion.tool:
@@ -126,6 +145,8 @@ def _record_matches(criterion: CliCalledCriterion, argv: list[str], record: dict
         frozenset(criterion.ignore_flags),
         frozenset(n for name, p in (criterion.flags or {}).items() if p.needs_value for n in (name, *p.aliases))
         | frozenset(criterion.value_flags),
+        frozenset(n for name, p in (criterion.flags or {}).items() for n in (name, *p.aliases))
+        | frozenset(criterion.ignore_flags),
     )
 
     offset = 0
@@ -205,27 +226,24 @@ def _check_impl(
                 error=f"Invocation log '{criterion.log}' does not exist",
             )
 
+        # The recorder leaves this beside the log when a write failed, so a record
+        # it could not append does not read as "the agent never ran the command".
+        sentinel = f"{criterion.log}.error"
+        if sandbox.file_exists(sentinel):
+            detail = sandbox.get_file_content(sentinel).strip().splitlines()
+            return CriterionResult(
+                criterion_type=criterion.type,
+                description=criterion.description,
+                score=0.0,
+                error=(
+                    f"Recorder could not write to '{criterion.log}' ({len(detail)} dropped record(s)); "
+                    f"the log is incomplete so the verdict cannot be trusted. First: {detail[0] if detail else '?'}"
+                ),
+            )
+
         content = sandbox.get_file_content(criterion.log)
 
-        usable: list[tuple[list[str], dict[str, Any]]] = []
-        unusable = 0
-        for line in content.splitlines():
-            stripped = line.strip()
-            if not stripped:
-                continue
-            try:
-                parsed = json.loads(stripped)
-            except ValueError:
-                unusable += 1
-                continue
-            if not isinstance(parsed, dict):
-                unusable += 1
-                continue
-            argv = _usable_argv(parsed)
-            if argv is None:
-                unusable += 1
-                continue
-            usable.append((argv, parsed))
+        usable, unusable = parse_log(content)
 
         if unusable:
             # A record we cannot read might BE the call a max_count: 0 guard

src/coder_eval/invocation_log.py

@@ -0,0 +1,151 @@
+"""The structured invocation log: the recording shim that writes it, and the reader.
+
+Named for the artifact rather than the writer because both sides live here -- the
+shim template `SandboxConfig.record_cli` renders, and `parse_log`, which the
+`cli_called` criterion reads it back with.
+
+The rendered script runs INSIDE the sandbox, where ``coder_eval`` is not
+installed, so it imports nothing from this package: its configuration arrives as
+embedded literals and everything else comes from the standard library.
+
+Keeping the template here rather than inline in :mod:`coder_eval.sandbox` lets
+:func:`render_recorder` be exercised directly (render, execute, read the log)
+without standing up a sandbox.
+"""
+
+import json
+import sys
+
+from coder_eval.models import RecordedCli
+
+
+# Written beside the shims, inside the generated recorder directory, so the log
+# travels with them if the sandbox root moves.
+LOG_FILENAME = "calls.jsonl"
+
+_TEMPLATE = '''\
+#!{interpreter}
+"""Recording shim for `{tool}` - generated by coder_eval SandboxConfig.record_cli.
+
+Appends one JSON record per invocation to {log_filename} beside this script, in
+the format the `cli_called` success criterion reads. Do not edit: regenerated on
+every sandbox setup.
+"""
+
+import json
+import os
+import sys
+import time
+
+TOOL = {tool!r}
+EXIT_CODE = {exit_code!r}
+STDOUT_TEXT = {stdout!r}
+STDERR_TEXT = {stderr!r}
+
+SHIM_DIR = os.path.dirname(os.path.abspath(__file__))
+LOG_PATH = os.path.join(SHIM_DIR, {log_filename!r})
+LOG_ERROR_PATH = LOG_PATH + ".error"
+
+
+def record(argv, exit_code):
+    """Append this invocation to the log.
+
+    Best-effort: a logging failure must never break the command the agent ran,
+    which would turn an evidence problem into a behaviour problem.
+
+    argv is stored as a LIST. A space-joined string cannot distinguish
+    `--flag "two words"` from two arguments, which is the whole reason this log
+    exists instead of a flattened command line. stdin is deliberately never
+    read: it would block whenever the sandbox leaves it on an open pipe, and in
+    passthrough mode it would consume the payload the real tool needs.
+    """
+    entry = {{
+        "ts": round(time.time(), 3),
+        "tool": TOOL,
+        "argv": list(argv),
+        "exit": exit_code,
+    }}
+    try:
+        # ensure_ascii escapes non-ASCII and any stray surrogate from
+        # undecodable argv bytes, so an exotic argument cannot make this write
+        # raise and silently drop the record.
+        with open(LOG_PATH, "a", encoding="utf-8", newline="\\n") as handle:
+            handle.write(json.dumps(entry) + "\\n")
+    except OSError as exc:
+        # Keep the agent's command working, but never lose a record silently: a
+        # dropped record reads exactly like "the agent never ran it".
+        sys.stderr.write("coder_eval recorder: log write failed: %r\\n" % (exc,))
+        try:
+            with open(LOG_ERROR_PATH, "a", encoding="utf-8") as sentinel:
+                sentinel.write("%r %r\\n" % (exc, argv))
+        except OSError:
+            pass
+
+
+def main(argv):
+    """Record the invocation, then fail like the tool would with nothing behind it.
+
+    Nothing is executed: no network, no auth, no side effects. A test that needs
+    the real tool's behavior recorded instead should supply its own wrapper under
+    mock_path_dirs -- proxying a live executable is a different job from stubbing
+    one, and this shim deliberately does only the second.
+    """
+    record(argv[1:], EXIT_CODE)
+    if STDOUT_TEXT:
+        sys.stdout.write(STDOUT_TEXT)
+    if STDERR_TEXT:
+        sys.stderr.write(STDERR_TEXT)
+    return EXIT_CODE
+
+
+if __name__ == "__main__":
+    sys.exit(main(sys.argv))
+'''
+
+
+def render_recorder(spec: RecordedCli, interpreter: str | None = None) -> str:
+    """Render the shim source for one ``record_cli`` entry.
+
+    ``interpreter`` is baked into the shebang as an ABSOLUTE path (defaulting to
+    the running interpreter). A ``#!/usr/bin/env python3`` shebang resolves
+    through the same PATH the recorder dir is prepended to, so `tool: python3`
+    made the shim re-exec itself forever.
+    """
+    return _TEMPLATE.format(
+        interpreter=interpreter or sys.executable,
+        tool=spec.tool,
+        exit_code=spec.exit_code,
+        stdout=spec.stdout,
+        stderr=spec.stderr,
+        log_filename=LOG_FILENAME,
+    )
+
+
+def parse_log(text: str) -> tuple[list[tuple[list[str], dict[str, object]]], int]:
+    """Parse recorder-log text into ``(usable, unusable_count)``.
+
+    A usable entry pairs the record's ``argv`` with the whole record. Unusable
+    means unparseable, not an object, or an ``argv`` that is not a list of
+    strings — counted rather than dropped, because a record that cannot be read
+    might be the very call a negative guard forbids.
+    """
+    usable: list[tuple[list[str], dict[str, object]]] = []
+    unusable = 0
+    for line in text.splitlines():
+        stripped = line.strip()
+        if not stripped:
+            continue
+        try:
+            parsed = json.loads(stripped)
+        except ValueError:
+            unusable += 1
+            continue
+        if not isinstance(parsed, dict):
+            unusable += 1
+            continue
+        argv = parsed.get("argv")
+        if isinstance(argv, list) and all(isinstance(item, str) for item in argv):
+            usable.append((argv, parsed))
+        else:
+            unusable += 1
+    return usable, unusable

src/coder_eval/models/__init__.py

@@ -155,10 +155,13 @@
 
 # Sandbox
 from coder_eval.models.sandbox import (
+    RECORD_CLI_DIR,
+    RECORD_CLI_LOG,
     DockerBuildConfig,
     DockerDriverConfig,
     NodeEnvConfig,
     PythonEnvConfig,
+    RecordedCli,
     ResourceLimits,
     SandboxConfig,
     validate_template_sources_list,
@@ -271,6 +274,9 @@
     "NodeEnvConfig",
     "PythonEnvConfig",
     "SandboxConfig",
+    "RecordedCli",
+    "RECORD_CLI_DIR",
+    "RECORD_CLI_LOG",
     "ResourceLimits",
     "validate_template_sources_list",
     # Telemetry