Release v0.10.0

.gitattributes

@@ -1,6 +1,6 @@
 # Check every text file out with LF on all platforms. Generated artifacts embed the bytes of
 # descriptors, profiles, and docs verbatim, and the byte-exact tests compare against LF fixtures, so
-# a CRLF working tree on Windows silently changes program output.
+# line-ending conversion would silently change program output.
 * text=auto eol=lf
 
 # Golden fixtures are byte-exact — never EOL-normalize them.

.github/workflows/ci.yml

@@ -16,13 +16,7 @@ env:
 jobs:
   test:
     name: Test suite
-    strategy:
-      # Windows is the platform this matrix exists to cover, so a Linux failure
-      # must not cancel it — that is exactly when its result is worth having.
-      fail-fast: false
-      matrix:
-        os: [ubuntu-latest, windows-latest]
-    runs-on: ${{ matrix.os }}
+    runs-on: ubuntu-latest
     steps:
       - uses: actions/checkout@v4
       - name: Install Rust toolchain
@@ -31,33 +25,14 @@ jobs:
           components: rustfmt, clippy
       - name: Cache cargo build
         uses: Swatinem/rust-cache@v2
-      - name: Install jq
-        # Git for Windows supplies sh, xargs, tr, and wc, but not jq, and the
-        # judge-recipe tests execute the shipped pipeline text rather than a
-        # stand-in for it.
-        if: runner.os == 'Windows'
-        run: choco install jq --yes --no-progress
-      - name: Permit symlink creation
-        # Windows creates symlinks only under Developer Mode or elevation, and
-        # the core::fs round-trips need one. Asking for it explicitly beats
-        # depending on how the runner's token happens to be built.
-        if: runner.os == 'Windows'
-        run: >
-          reg add "HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\AppModelUnlock"
-          /t REG_DWORD /f /v AllowDevelopmentWithoutDevLicense /d 1
       - name: Format check
         run: cargo fmt --all -- --check
       - name: Clippy
         run: cargo clippy --all-targets --all-features -- -D warnings
       - name: Test
-        # Capability-gated tests (the POSIX recipe pipelines, symlink
-        # round-trips, long-path staging) skip with a printed reason on a host
-        # that lacks the capability. This turns every such skip into a failure,
-        # so neither runner can quietly stop covering them. Ubuntu ships the
-        # recipe tools; the steps above provide them on Windows. Long paths need
-        # no provisioning — the runner passes core.longpaths to git itself.
-        # EVAL_MAGIC_SH stays unset deliberately: discovering the shell from the
-        # Git install root is what a Windows user hits, so CI should run it too.
+        # Capability-gated tests skip with a printed reason on a host that lacks
+        # the capability. CI turns every such skip into a failure so coverage
+        # cannot shrink silently.
         env:
           EVAL_MAGIC_REQUIRE_POSIX_TOOLS: 1
         run: cargo test --all-targets

.github/workflows/release.yml

@@ -113,9 +113,6 @@ jobs:
       GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
       BUILD_MANIFEST_NAME: target/distrib/${{ join(matrix.targets, '-') }}-dist-manifest.json
     steps:
-      - name: enable windows longpaths
-        run: |
-          git config --global core.longpaths true
       - uses: actions/checkout@v6
         with:
           persist-credentials: false
@@ -146,9 +143,7 @@ jobs:
           echo "dist ran successfully"
       - id: cargo-dist
         name: Post-build
-        # We force bash here just because github makes it really hard to get values up
-        # to "real" actions without writing to env-vars, and writing to env-vars has
-        # inconsistent syntax between shell and powershell.
+        # Force bash so every release runner writes GitHub outputs with one syntax.
         shell: bash
         run: |
           # Parse out what we just built and upload it to scratch storage

AGENTS.md

@@ -5,12 +5,12 @@
 `eval-magic` is a Rust CLI crate. The binary entry point is `src/main.rs`; reusable logic lives in
 `src/lib.rs` and submodules such as `cli/`, `pipeline/`, `sandbox/`, `validation/`, and
 `workspace/`. JSON schemas are tracked in `schema/`, harness descriptors (plus embedded harness
-assets such as the OpenCode write-guard plugin template) in `harnesses/`, shared
-prompt profiles in `profiles/`, and docs in `docs/` — user-facing Markdown under `docs/guides/`
-ships embedded in the binary as `eval-magic docs <topic>`; the other files are internal development
-docs. `docs/developer_overview.md` maps the repository and holds the placement policy. Integration
-tests are split by surface area under `tests/cli/` and `tests/run/`; unit tests usually live beside
-the module they exercise.
+assets such as the OpenCode write-guard plugin template) in `harnesses/`, the `RUNBOOK.md` prose
+template in `profiles/`, and docs in `docs/` — user-facing Markdown under `docs/guides/` ships
+embedded in the binary as `eval-magic docs <topic>`; the other files are internal development docs.
+`docs/developer_overview.md` maps the repository and holds the placement policy. Integration tests
+are split by surface area under `tests/cli/` and `tests/run/`; unit tests usually live beside the
+module they exercise.
 
 ## Build, Test, and Development Commands
 
@@ -54,41 +54,35 @@ Extraction is a size decision, not a style preference; don't split a small inlin
 **Spawning a child process from a test.** Use the hidden `__fixture` subcommand, never `sh`, `true`,
 `printf`, or a `#!/bin/sh` stub. It exits with a chosen code, emits chosen bytes, writes a chosen
 file, or checks a file or variable — see `FixtureArgs` in `src/cli/args.rs`. One invocation parses
-the same under `sh -c` and `cmd /C`, which is what keeps `command_check` tests off per-OS command
-strings. Build the command with the `fixture` helper (`tests/run/helpers.rs` for integration tests,
-the one in `src/pipeline/grade/command_check/tests.rs` for unit tests). Because the fixture is the
-binary, `cargo test --lib` alone does not build it — run `cargo test`, or `cargo build` first.
-
-**Tests are gated on capabilities, not on the OS.** `#[cfg(unix)]` on a test hides it from
-compilation and clippy on the other host and hides the coverage gap. Instead, probe for what the
-test actually needs and call `report_skip` (`src/core/runtime.rs`), which prints the reason and
-returns `true`. Setting `EVAL_MAGIC_REQUIRE_POSIX_TOOLS=1` turns every skip into a failure; CI sets
-it on both runners, so neither can quietly stop covering something. Three capabilities are gated
-today: the recipe tools beyond the shell itself (`require_posix_toolchain` — in practice `jq`),
-symlink creation, which Windows allows only under Developer Mode, and creating a path past
-Windows' 259-character limit (`deep_task_root`, `src/cli/run/orchestrate/git.rs`). The Windows
-runner is provisioned for those rather than exempted from them, so a skip there is a red build. The
-shell is not one of them; it is a hard requirement, per the section below.
-`require_posix_toolchain` is not test-only either — the `run` preflight uses it to warn about the
-same gap. Where a genuine per-OS difference is the behavior under test — signals, path separators —
-branch on `cfg!(windows)` at runtime so both arms still compile everywhere.
+predictably under `sh -c`, which keeps `command_check` tests focused on runner behavior instead of
+the host's utility implementations. Build the command with the `fixture` helper
+(`tests/run/helpers.rs` for integration tests, the one in
+`src/pipeline/grade/command_check/tests.rs` for unit tests). Because the fixture is the binary,
+`cargo test --lib` alone does not build it — run `cargo test`, or `cargo build` first.
+
+**Tests are gated on capabilities, not on broad platform labels.** Probe for what the test actually
+needs and call `report_skip` (`src/core/runtime.rs`), which prints the reason and returns `true`.
+Setting `EVAL_MAGIC_REQUIRE_POSIX_TOOLS=1` turns every skip into a failure; CI sets it so coverage
+cannot quietly shrink. Symlink creation is capability-gated because the backing filesystem may
+forbid it. The shell is not capability-gated; it is a hard requirement, per the section below.
 
 **A POSIX shell is required, for use and for development.** Harness `exec_template`s are POSIX
-command lines, so the dispatch and probe paths spawn `sh` via `posix_shell()`
-(`src/core/runtime.rs`) rather than a hardcoded `/bin/sh`: it searches `PATH`, then a Git for
-Windows install. Set `EVAL_MAGIC_SH` to override it. `cargo test` inherits the requirement — the
-scripted-turn tests spawn a `#!/bin/sh` harness stub through the resolved shell and do not skip —
-so a host without `sh` fails the suite instead of quietly covering less. `jq` is required alongside
-it for the parallel-dispatch and judge recipes; Git for Windows supplies the shell, `xargs`, `tr`,
-and `wc`, but not `jq`. `POSIX_TOOLING_REQUIREMENT` (`src/core/runtime.rs`) is the one wording the
+command lines, so the dispatch and probe paths spawn `sh` through `run_in_posix_shell` /
+`posix_shell()` (`src/core/runtime.rs`) rather than a hardcoded `/bin/sh`: it searches `PATH`, then
+checks `/bin/sh`. Set `EVAL_MAGIC_SH` to override it. `cargo test` inherits the requirement — the
+dispatch tests spawn a `#!/bin/sh` harness stub through the resolved shell and do not skip — so a
+host without `sh` fails the suite instead of quietly covering less. The shell is the whole
+requirement: `jq` was needed only while operators pasted the generated dispatch and judge recipes,
+and `eval-magic dispatch` drives both itself.
+`POSIX_TOOLING_REQUIREMENT` (`src/core/runtime.rs`) is the one wording the
 Markdown-carrying surfaces reuse: the shell-discovery errors, the `run` preflight warnings,
 `RUNBOOK.md`, and `dispatch-manifest.md`. State the requirement from there rather than rephrasing
 it. `--help` is the one deliberate restatement (`AFTER_HELP` in `src/cli/help.rs`), hard-wrapped and
 backtick-free because clap renders into a terminal; keep the two in step by hand.
 
-Which platforms that requirement is honored on — and why preparing on Windows but dispatching from
-WSL is a correctness boundary rather than a preference — is stated once under "Platform support" in
-`docs/developer_overview.md`.
+eval-magic supports Linux and macOS. On Windows, use and develop eval-magic entirely inside WSL;
+native Windows is unsupported. The complete boundary and the portable-data exception are stated
+under "Platform support" in `docs/developer_overview.md`.
 
 **Where user-facing warnings come from.** Library modules (`pipeline`, `workspace`, `sandbox`,
 `adapters`) never print. They return warning strings on their result struct — `#[serde(skip)]` when

Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "eval-magic"
-version = "0.9.1"
+version = "0.10.0"
 edition = "2024"
 description = "One-stop CLI for running skill evals — measure whether an agent skill actually shifts behavior."
 license = "MIT"
@@ -38,20 +38,18 @@ clap = { version = "4.6.1", features = ["derive"] }
 # schemas are embedded (include_str!) and use only internal #/definitions refs,
 # so no resolver is needed at all.
 jsonschema = { version = "0.46.5", default-features = false }
-# Bash write-mutation patterns (sandbox policy). default-features off drops the
-# Unicode tables: the shell-command patterns are ASCII, and ASCII `\b`/`\s` is
-# the intended semantics, so `perf` (matching speed) is all we keep beyond `std`.
+# Eval validation, transcript checks, dispatch parsing, and staging patterns.
+# Default features are disabled to drop Unicode tables: these authored patterns
+# use ASCII syntax, so `perf` (matching speed) is all we keep beyond `std`.
 regex = { version = "1.12.3", default-features = false, features = ["std", "perf"] }
 serde = { version = "1.0.228", features = ["derive"] }
 serde_json = { version = "1.0.150", features = ["preserve_order"] }
-similar = { version = "3.1.1", default-features = false }
 tempfile = "3.27.0"
 thiserror = "2.0.18"
 # Harness descriptor files (harnesses/*.toml). `display` serializes the
 # resolved (layer-merged) descriptor back to authorable TOML for
 # `harness show`.
 toml = { version = "0.9", default-features = false, features = ["parse", "serde", "display"] }
-walkdir = "2.5.0"
 
 [dev-dependencies]
 assert_cmd = "2.2.2"