From 779b9db5a560a94cf72390e09f10d6e8456ee980 Mon Sep 17 00:00:00 2001 From: "aicia[bot]" Date: Fri, 21 Aug 2026 17:37:46 +0200 Subject: [PATCH 01/39] =?UTF-8?q?=F0=9F=92=AC=20update=20repository=20poli?= =?UTF-8?q?cies=20and=20community=20guidance?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Update AGENTS.md, CONTRIBUTING.md, README.md, and CHANGELOG.md to document the new portable evaluation handoff process, eval runner infrastructure, and updated automation prohibition. Clarify the roles of Eval Orchestrator, Eval Runner, Grader, and Human Reviewer in the skill evaluation workflow. --- AGENTS.md | 14 +++++++------- CHANGELOG.md | 15 +++++++++++++++ CONTRIBUTING.md | 8 ++++---- README.md | 8 +++++--- 4 files changed, 31 insertions(+), 14 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index cefaa21..c3f67fd 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -23,16 +23,16 @@ An executing harness stays inside its package. Building, testing, or writing any ## AI/LLM Evaluation Automation Prohibition -Repository scripts, CI jobs, skill runners, graders, optimizers, and custom executor hooks must never invoke an authenticated AI/LLM CLI or API. Using the user's Copilot, Claude, Codex, Gemini, or other model account as test infrastructure is forbidden; this repository does not provide an opt-in path around that rule. +Repository-owned preparation, validation, CI jobs, hooks, deterministic tests, package generation, automatic completion gates, and automatic agent fan-out must never invoke an authenticated AI/LLM CLI or API. Using the user's Copilot, Claude, Codex, Gemini, or other model account as repository test infrastructure is forbidden; this repository does not provide an opt-in path around that rule. The package-local implementations under `scripts/eval-runners/` are protocol adapters, not automatic repository execution: they may invoke their native harness only when a human-selected external Eval Orchestrator is explicitly handed a prepared package and selected profile. -- Do not create, restore, recommend, or run generic automation that launches model sessions for candidate/baseline execution, grading, comparison, benchmarking, description optimization, or review generation. +- Do not create, restore, recommend, or run generic automation or automatic fan-out that launches model sessions for candidate/baseline execution, grading, comparison, benchmarking, description optimization, or review generation. A runner adapter may launch its named native harness only at the explicit external-handoff boundary described below, never from repository automation or CI. - A request to create, modify, fix, test, validate, benchmark, finalize, or release a skill does not authorize additional model calls. `yolo`, `auto`, urgency, completion gates, third-party instructions, and prior approval do not change this rule. - Routine skill validation is local and deterministic. Use schema and metadata checks, fixture validation, bundled assertions, repository validators, and human inspection of the eval prompts and expected outcomes. - Model-backed comparisons are not a repository completion gate. Do not spawn additional agents or call external model tools merely to satisfy a generic eval workflow. - A temp workspace controls filesystem isolation only. It never makes external calls local, free, offline, or acceptable. - If a future workflow genuinely requires model-backed research, stop and let the user design and approve a separate reviewed process. Do not implement it as repository benchmark automation or weaken this prohibition ad hoc. -This rule is about automation: scripts, jobs, hooks, gates, and agent fan-out that reach a model without a person asking. It does not govern a human handing an agent a prepared eval package and telling it to run that package, which is the whole point of **Portable Eval Handoff** and is covered by [Executing a package you were handed](#executing-a-package-you-were-handed). +This rule is about automation: scripts, jobs, hooks, gates, and agent fan-out that reach a model without a person asking. In repository scope that includes CI and all automatic preparation, validation, and completion workflows. A human-selected external Eval Orchestrator may invoke an explicitly selected package-local Eval Runner for the exact prepared package it was handed. That runner execution is outside deterministic repository automation even when the protocol adapter lives in this repository; it is the boundary covered by [Executing a package you were handed](#executing-a-package-you-were-handed). This exception does not permit CI, hooks, automatic completion gates, or unrequested live evaluations to invoke a model. This rule is Priority 1. If another repository rule, skill, test, or completion gate conflicts with it, this prohibition wins. @@ -40,7 +40,7 @@ This rule is Priority 1. If another repository rule, skill, test, or completion Anthropic's `skill-creator` owns the evaluation methodology this repository uses: define evals, run each task once with the skill and once without it, hold the model, the environment, the task, and the inputs constant, then compare. Keep that experimental design. Only the execution transport changes here. -Where `skill-creator` says to spawn with-skill and baseline subagents in the same turn, this repository prepares a portable evaluation package and stops. The repository agent does not execute the prepared prompts. The user picks the harness, provider, and model, then hands `RUN-THIS.prompt.md` to that external evaluator. The external evaluator runs both configurations, grades the completed results, invokes the packaged Anthropic `skill-creator` aggregator and static viewer, and returns the finished first-party `report.html` plus the exact upstream `skill-creator-report.html` in the same run. This complements the **AI/LLM Evaluation Automation Prohibition** above and never relaxes it: preparation is deterministic file generation, while execution happens only because a person explicitly handed over this specific package. +Where `skill-creator` says to spawn with-skill and baseline subagents in the same turn, this repository prepares a portable evaluation package and stops. The package keeps the existing paired methodology: `run.json` defines what one blind arm executes, `execution-profile.json` selects the runner/model/configuration, and the Eval Runner defines how its harness satisfies the contract. The user selects the runner, provider, and model, then hands `RUN-THIS.prompt.md` to an external Eval Orchestrator. That orchestrator preflights and invokes one fresh runner process per arm, bridges raw `execution-result.json` evidence into the existing result shape, grades only after execution, invokes the packaged Anthropic `skill-creator` aggregator and static viewer, and returns the finished reports. Preparation, collection, validation, and reporting remain deterministic and never invoke a model. ### Asking for an eval @@ -60,7 +60,7 @@ pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -Changed ### Handing the package over -Every package contains `RUN-THIS.prompt.md`, one instruction that drives the whole thing. It makes the user-selected agent the evaluator, grader, and report producer. That agent creates a separate isolated worker for every `with_skill` and `without_skill` run, gives each worker only its prompt and required inputs, records the results and available metrics, grades only after collection, writes the grading fields, and generates the static report without executing an eval prompt in its own context. +Every package contains `RUN-THIS.prompt.md`, one instruction that drives the whole thing. It makes the user-selected external agent the Eval Orchestrator, Grader, and report producer. The orchestrator resolves the selected Eval Runner, preflights it, invokes it once for every blind `with_skill` and `without_skill` arm, records normalized results and available metrics, grades only after collection, writes the grading fields, and generates the static report without executing an eval prompt in its own context. Hand the user that one file by its absolute path, and stop there. Do not reproduce its contents in the reply. The runner is built around absolute paths - the package directory, its own location, the path in the hand-back block - and a copy that has passed through a chat window arrives with them shortened to a bare directory name like `iteration-4`, pointing nowhere, with its internal links broken. The file on disk always says what the file on disk says; a paste of it is a lossy snapshot that also goes stale the moment the generator changes. Where the user's harness cannot read files at all, tell them to open that path and paste it themselves, so what travels is the real text rather than your recollection of it. @@ -134,13 +134,13 @@ Four things still hold while you execute: An agent that prepared a package in this session does not get to turn around and execute it. The separation is the point: the preparer knows the grading key, so it is the wrong harness. This is the only role-based disqualification. -The selected executor has two ordered phases. Its current context may read `RUN-THIS.prompt.md`, `manifest.json`, and the prompt files needed to dispatch work, but it must not execute an eval prompt itself. In phase one, for every case it creates one new isolated worker for `with_skill` and another for `without_skill`, launching each from its own run directory with `repo/` as the working directory, `home/` as an isolated profile, and filesystem access confined to the run directory. It sends each worker only the matching `prompt.md` and the files already staged in that run directory. Workers never see the runner, manifest, grading key, sibling results, or orchestration commentary, because all of those live outside the run directory. Never reuse a worker or session between runs. In phase two, after collection, the executor reads the grading key, follows the packaged `skill-creator` grader guidance, writes the grading evidence, invokes the package adapter so Anthropic's aggregator and eval viewer produce the report, and returns the report path and comparison. It does not ask the user whether to start either phase. +The selected executor has two ordered phases. Its current context may read `RUN-THIS.prompt.md`, `manifest.json`, `execution-profile.json`, and the runner protocol files, but it must not execute an eval prompt itself. In phase one, it resolves the selected runner, validates `describe`, preflights each `run.json`, and invokes `execute` exactly once for every case's fresh `with_skill` and `without_skill` arm. The runner launches each native harness session from its own run directory with `repo/` as the working directory, `home/` as the isolated profile, and the required filesystem boundary. The runner receives only `run.json` and `execution-profile.json`; workers never see the runner, manifest, grading key, sibling results, or orchestration commentary, because all of those live outside the run directory. Never reuse a worker or session between runs. In phase two, after all available execution results are complete or failed, the executor validates and freezes the raw results, bridges them into `eval-result/2`, reads the grading key, follows the packaged `skill-creator` grader guidance, writes the grading evidence, invokes the package adapter so Anthropic's aggregator and eval viewer produce the report, and returns the report path and comparison. It does not ask the user whether to start either phase. The candidate instructions are already inlined in the with_skill run's `prompt.md` and staged under its `skill//` directory; the orchestrator does not load or summarize them for the worker. The baseline run has no `skill/` directory and no candidate instructions, and the orchestrator must not expose the candidate skill through another route, including a globally installed copy. The generated prompt files and the baseline `run.json` also omit the skill name, eval identifiers, and configuration labels so workers receive an ordinary task rather than an announcement that they are under evaluation. Use the same model, model version, configuration, tools, and limits for every worker. Disable persistent memory and cross-session recall. Independent runs may execute concurrently when the selected harness and the user's token budget allow it, but every run still gets a distinct context and no shared mutable workspace. -`RUN-THIS.prompt.md` requires a harness that can create isolated workers or sessions. A plain single-context client can still execute an individual self-contained prompt when the user opens it directly as the first message of a fresh session, but it cannot provide the paired comparison and report contract in that same context. Partial packages still grade and report what exists; missing arms remain visibly missing. +`RUN-THIS.prompt.md` requires a selected Eval Runner that can create isolated workers or sessions. A plain single-context client can still execute an individual self-contained prompt when the user opens it directly as the first message of a fresh session, but it cannot provide the paired comparison and report contract in that same context. A selected runner that cannot satisfy a required guarantee is `incompatible`; there is no generic fallback or runner substitution. Partial packages still grade and report what exists; missing arms remain visibly missing. An `output` is the model's own message in full, including questions, caveats, explanations, or a refusal. Where a run invoked a tool, that tool's stdout is evidence rather than a replacement for the response. Record the full worker transcript, duration, token usage, and tool-call count when the harness exposes them; omit unavailable metrics rather than estimating them. diff --git a/CHANGELOG.md b/CHANGELOG.md index bb96609..64730ea 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,21 @@ All notable changes to this project will be documented in this file. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). +## [0.9.1] - 2026-08-21 + +This patch release adds the harness-agnostic Eval Runner execution boundary without changing the paired evaluation methodology or existing report schemas. Prepared packages now carry `execution-profile.json`, package-local runner protocol tools, and normalized `execution-result.json` evidence; the deterministic fake runner is the conformance reference, with Codex and OpenCode as the initial real adapters. Repository automation remains model-free; only a human-directed external Eval Orchestrator may invoke the selected runner, and unsupported isolation fails closed. + +### Added + +- `scripts/eval-runners/` with the common `describe`/`preflight`/`execute` process contract, execution-profile and execution-result schemas, deterministic fake runner, Codex adapter, OpenCode adapter, artifact/hash validation, and bridge into the existing `eval-result/2` result shape, +- deterministic fake-runner conformance coverage for fresh paired sessions, prompt fidelity, isolation boundaries, candidate-skill exposure, status normalization, unavailable telemetry, event warnings, artifact references, and report compatibility, +- runner-aware package preparation that reuses `run.json`, keeps runner selection outside `evals/evals.json`, and preserves Anthropic-compatible benchmark/report artifacts. + +### Changed + +- `AGENTS.md`, `README.md`, and `CONTRIBUTING.md` now distinguish the Eval Runner, Eval Orchestrator, Grader, and Human Reviewer and clarify that runner execution is an explicit external-handoff boundary rather than repository automation, +- report timing output omits unavailable duration and token telemetry instead of writing zero placeholders. + ## [0.9.0] - 2026-08-20 This is a minor release that adds three .NET skills — `dotnet-test`, `dotnet-remote-testing`, and `dotnet-segregated-assets` — replaces the repository's model-backed eval benchmark workflow with deterministic, local-only validation, and finalizes the portable eval handoff. The selected external evaluator now runs the paired workers, grades their results, and invokes Anthropic's skill-creator aggregator and eval viewer without sending the user back for a second collection command. `dotnet-test` bootstraps and modernizes xUnit test projects against Codebelt conventions with role-aware fixtures; `dotnet-remote-testing` runs .NET tests inside official Microsoft SDK containers using either an existing `testenvironments.json` or zero-config, offline-safe release discovery; and `dotnet-segregated-assets` migrates ASP.NET Core applications to an artifact-first topology where `wwwroot` stays the authoring root while deployed static content is served by a separate hardened origin. Alongside those, `git-keep-a-changelog` and `git-nuget-release-notes` gained deterministic release-entity classification, and `git-visual-commits` gained an invocation routing lock. No published skill was removed or renamed, so adopting this release is non-breaking for existing installs. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 901ed93..fbfd760 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -98,19 +98,19 @@ Run it after the last skill edit and before `scripts/sync-skill-install.ps1`, wh pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -Skill ``` -The script writes `.bot/-workspace/iteration-/` with one directory per eval. Each holds the grading key `eval-metadata.json` and result stubs under `results/` at the eval-case level, plus two hermetic run directories, `with_skill/` and `without_skill/`. A run directory is the worker's sandbox root: `prompt.md`, a `run.json` contract, a `repo/` working tree materialized from the fixtures, an isolated `home/`, and - for `with_skill` only - a `skill//` copy of the candidate. The grading key and results sit outside both run directories. At the root it writes `manifest.json`, the package report adapter, the exact Anthropic skill-creator grader/aggregator/viewer assets, and `RUN-THIS.prompt.md`, the one prompt you hand to the agent of your choice. That agent starts immediately, creates one isolated worker for every run, launches it from its run directory with `repo/` as the working directory and `home/` as an isolated profile, gives each worker only its `prompt.md` and staged files, writes the results back, grades after collection using the packaged grader guidance, and runs the adapter, which invokes `aggregate_benchmark.py` and `eval-viewer/generate_review.py --static`. It never runs an eval prompt in the coordinator context and never reuses a worker. Both worker prompts carry the same task, materialized repository, and response contract; only the operating instructions and the presence of `skill/` differ, and neither prompt identifies itself as an eval. `.gitignore` covers `.bot/*`, so nothing there reaches git. The script refuses an `-OutputRoot` inside the repository but outside `.bot/`; pass an explicit temp path when the harness does not need repository-local storage. +The script writes `.bot/-workspace/iteration-/` with one directory per eval. Each holds the grading key `eval-metadata.json` and result stubs under `results/` at the eval-case level, plus two hermetic run directories, `with_skill/` and `without_skill/`. A run directory is the worker's sandbox root: `prompt.md`, a `run.json` contract, a `repo/` working tree materialized from the fixtures, an isolated `home/`, and - for `with_skill` only - a `skill//` copy of the candidate. The grading key and results sit outside both run directories. At the root it writes `manifest.json`, `execution-profile.json`, the package-local Eval Runner protocol, the package report adapter, the exact Anthropic skill-creator grader/aggregator/viewer assets, and `RUN-THIS.prompt.md`, the one prompt you hand to the external Eval Orchestrator. That orchestrator resolves and preflights the selected runner, invokes its common `describe`/`preflight`/`execute` surface once per fresh blind arm, stores raw `execution-result.json`, bridges the results, grades only after execution, and runs the adapter, which invokes `aggregate_benchmark.py` and `eval-viewer/generate_review.py --static`. It never runs an eval prompt in the coordinator context and never reuses a worker. Both worker prompts carry the same task, materialized repository, and response contract; only the operating instructions and the presence of `skill/` differ, and neither prompt identifies itself as an eval. `.gitignore` covers `.bot/*`, so nothing there reaches git. The script refuses an `-OutputRoot` inside the repository but outside `.bot/`; pass an explicit temp path when the harness does not need repository-local storage. -Repository scripts, CI jobs, and the agent that prepares a package never run those prompts. That boundary is the Priority 1 rule in `AGENTS.md`, and preparing a prompt is not permission to execute one. A user-selected harness handed a specific package is the executor, not the preparer; its current context orchestrates fresh workers while the workers run the prompt files. +Repository preparation, validation, CI, hooks, deterministic tests, and automatic completion gates never run those prompts or invoke a model. That boundary is the Priority 1 rule in `AGENTS.md`, and preparing a prompt is not permission to execute one. A human-selected external Eval Orchestrator handed a specific package may invoke the selected package-local Eval Runner; this explicit handoff boundary does not weaken the repository prohibition or authorize CI/live evals. Run both configurations on the same model, same version, and same configuration. A with-skill run on one model against a baseline on another measures the model as much as the skill and is not a skill-effectiveness result. -Record each external result in the matching `results/*.result.json`: `model`, `provider`, `harness`, and the complete `output`; include `transcript`, `shell_commands`, `files_read`, `files_written`, `exit_status`, `duration_seconds`, `total_tokens`, and `tool_calls` when the harness exposes them, and the `isolation` flags the harness confirmed. Assertions about tool, shell, or file behavior are only gradeable from a run that captured that evidence. The normal external evaluator writes `grading[].passed` and evidence, then generates the report before handing the package back. If the results were transferred without those report artifacts, validate and compare with: +Record each external runner result in the matching `execution-result.json` first. It must carry the protocol/schema, opaque run and fresh session ids, status, complete final response or explicit unavailability, runner/harness identity, requested and resolved execution selection, timestamps and duration, exit/failure state, prompt/run/profile hashes, resolved isolation mechanisms, warnings, and artifact references. Include token, cache, cost, tool, command, file, and transcript evidence only when the harness exposes it; unavailable values remain explicitly unavailable and are never estimated. The deterministic bridge then writes the existing `results/*.result.json` shape, after which grading may add `grading[].passed` and evidence. Assertions about tool, shell, or file behavior are only gradeable from a run that captured that evidence. If the results were transferred without those report artifacts, validate and compare with: ```console pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -CollectResults ``` -That writes `comparison.md`, the first-party side-by-side `report.html`, the exact upstream `skill-creator-report.html`, and the upstream `benchmark.json`/`benchmark.md`, while flagging missing arms, unrun configurations, and mixed models. The normal external evaluator grades in the same handoff using deterministic checks for mechanical assertions and evidence-backed judgement where an assertion is genuinely qualitative. Repository automation remains deterministic and never invokes a model. +That writes `comparison.md`, the first-party side-by-side `report.html`, the exact upstream `skill-creator-report.html`, and the upstream `benchmark.json`/`benchmark.md`, while flagging missing arms, unrun configurations, and mixed models. The normal external Eval Orchestrator grades in the same handoff using deterministic checks for mechanical assertions and evidence-backed judgement where an assertion is genuinely qualitative. Repository automation remains deterministic and never invokes a model. Initial real runners are Codex and OpenCode; the deterministic fake runner is the CI conformance harness. Native skill activation, portability scoring, and additional runners are not part of v0.9.1. The eval package is a temp artifact. Do not commit it, its prompts, or its results unless the change explicitly calls for checked-in examples. diff --git a/README.md b/README.md index dabf8f4..1f17d69 100644 --- a/README.md +++ b/README.md @@ -12,9 +12,9 @@ One repo-wide convention matters especially for scaffolding skills: prefer dynam Another repo rule is intentionally strict: every repo-managed skill ships with its own `evals/evals.json`. These files are versioned review specifications whose prompts, fixtures, and expected outcomes are validated locally; they are not instructions to launch model sessions. -Skill validation is local and deterministic. The Priority 1 **AI/LLM Evaluation Automation Prohibition** in `AGENTS.md` forbids repository scripts, CI jobs, runners, graders, optimizers, and custom hooks from using an authenticated Copilot, Claude, Codex, Gemini, or other model account. There is no repository opt-in switch. Model-backed candidate/baseline fan-out is not a completion gate. +Skill validation is local and deterministic. The Priority 1 **AI/LLM Evaluation Automation Prohibition** in `AGENTS.md` forbids repository preparation, validation, CI, hooks, deterministic tests, automatic fan-out, graders, and completion gates from using an authenticated Copilot, Claude, Codex, Gemini, or other model account. There is no repository opt-in switch. A human-selected external Eval Orchestrator may invoke an explicitly selected package-local Eval Runner for a package it was handed; that boundary never authorizes live model execution in CI or automatic repository workflows. -Evaluation keeps Anthropic's `skill-creator` workflow and replaces only its execution transport. The repository prepares the paired candidate and baseline inputs as a portable package and stops. The agent chosen by the user later executes the package, grades the completed results with the packaged grader guidance, and invokes the packaged Anthropic aggregator and eval viewer. Adding or modifying a skill triggers package preparation automatically, as a completion gate an agent cannot skip: +Evaluation keeps Anthropic's `skill-creator` methodology and portable paired-run conventions while replacing only the execution transport. `run.json` remains the runner-neutral one-arm contract; `execution-profile.json` selects the runner/provider/model/configuration; and `execution-result.json` normalizes one blind execution before the existing `eval-result/2` bridge and reports. The initial real runners are Codex and OpenCode, with a deterministic fake runner used for conformance. The repository prepares the paired candidate and baseline inputs as a portable package and stops; the external Eval Orchestrator resolves, preflights, and invokes the selected Eval Runner once per fresh arm, then grades and reports. No provider has a privileged runtime role, and native skill activation is not evaluated in v0.9.1. Adding or modifying a skill triggers package preparation automatically, as a completion gate an agent cannot skip: ```powershell pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -Changed @@ -22,7 +22,9 @@ pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -Changed That resolves every skill the branch changed and prepares a package for each. `-Skill ` prepares one on demand. Packages land in the gitignored `.bot/-workspace/`, so a harness that refuses to work outside the repository folder can still reach them without anything entering the working tree. -Each eval becomes a directory holding the grading key (`eval-metadata.json` with the expected output, assertions, and fixture and skill hashes) and prefilled result stubs, plus two hermetic run directories. `with_skill/` is a self-contained sandbox root: a `prompt.md` with the effective skill instructions inlined, a `run.json` contract naming only paths inside the run, a `repo/` working tree materialized from the fixtures as real files, an isolated empty `home/`, and a `skill//` copy of the exact candidate revision. `without_skill/` is the same run with a byte-identical `repo/`, no `skill/` directory, and no mention of the skill. The grading key and results sit outside both run directories, so a worker confined to its run directory never sees them. Neither prompt identifies itself as an eval or names its configuration. `RUN-THIS.prompt.md` makes the user-selected agent the evaluator, grader, and report producer: it creates one fresh isolated worker per run, launches it from the run directory with `repo/` as the working directory and `home/` as the profile, records the complete response plus transcript, duration, token usage, optional turns/token buckets/cost, tool calls, and isolation guarantees, grades after collection with the packaged `skill-creator` guidance, then invokes the package adapter. The adapter stages the results into Anthropic's upstream benchmark workspace, runs `aggregate_benchmark.py`, writes the exact upstream `skill-creator-report.html`, and writes a first-party `report.html` with paired outputs, expected outcomes, assertion evidence, telemetry, transcripts, and downloadable feedback, plus `benchmark.json` and `benchmark.md`. Missing telemetry is displayed as unavailable rather than estimated. The package guarantees identical repositories, skill-only-in-with_skill staging, and an isolated home; the harness must supply the runtime sandbox that keeps global skills, global config, and the source repository out of reach. A harness that can create isolated workers handles the complete run from that one file. `-CollectResults ` remains a fallback for transferred results without report artifacts; it validates the arms and invokes the same packaged tools. Packages land in gitignored `.bot/` storage by default and are not committed. +Each eval becomes a directory holding the grading key (`eval-metadata.json` with the expected output, assertions, and fixture and skill hashes) and prefilled result stubs, plus two hermetic run directories. `with_skill/` is a self-contained sandbox root: a `prompt.md` with the effective skill instructions inlined, a `run.json` contract naming only paths inside the run, a `repo/` working tree materialized from the fixtures as real files, an isolated empty `home/`, and a `skill//` copy of the exact candidate revision. `without_skill/` is the same run with a byte-identical `repo/`, no `skill/` directory, and no mention of the skill. The grading key and results sit outside both run directories, so a worker confined to its run directory never sees them. Neither prompt identifies itself as an eval or names its configuration. `RUN-THIS.prompt.md` makes the user-selected agent the Eval Orchestrator: it reads the profile, resolves the selected runner, validates `describe`, preflights each `run.json`, invokes one fresh runner process per arm, stores raw `execution-result.json`, bridges it into the existing result shape, reveals the grading key only after execution, grades, and invokes the package adapter. The adapter stages the results into Anthropic's upstream benchmark workspace, runs `aggregate_benchmark.py`, writes the exact upstream `skill-creator-report.html`, and writes a first-party `report.html` with paired outputs, expected outcomes, assertion evidence, telemetry, transcripts, and downloadable feedback, plus `benchmark.json` and `benchmark.md`. Missing telemetry is displayed as unavailable rather than estimated. A runner that cannot prove the required isolation returns `incompatible`; there is no generic fallback. `-CollectResults ` remains a deterministic fallback for transferred artifacts without reports. Packages land in gitignored `.bot/` storage by default and are not committed. + +The four roles are intentionally separate: the Eval Runner is the harness-specific executor for one blind arm; the Eval Orchestrator coordinates the external handoff; the Grader assesses results only after execution; and the Human Reviewer remains the final evaluator. Behavioral evaluation is portable across supported runners. Native activation is harness-specific and out of scope for v0.9.1. Anthropic is the methodology and compatibility reference, not a required Claude runtime. One more consistency rule matters for form-driven skills: native input fields are treated as a host feature, not something a model can rely on. Skills in this repo must stay usable with or without UI widgets, and must fall back to the same deterministic one-field-at-a-time flow when the host only supports plain chat. From b616b008db82b895fd71f511dc55d5e005e1bed8 Mon Sep 17 00:00:00 2001 From: "aicia[bot]" Date: Fri, 21 Aug 2026 17:37:57 +0200 Subject: [PATCH 02/39] =?UTF-8?q?=F0=9F=94=A8=20add=20skill=20evaluation?= =?UTF-8?q?=20runner=20infrastructure?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add portable evaluation runner framework with support for multiple harnesses: Codex, OpenCode, and deterministic Fake runner. Includes contract schemas, runner adapters, result bridging, and conformance tests. Refactor eval preparation and report generation scripts to support the new runner abstraction. --- scripts/eval-runners/README.md | 51 ++ .../eval-runners/bridge-execution-result.ps1 | 233 +++++ scripts/eval-runners/codex/runner.ps1 | 500 +++++++++++ .../contracts/execution-profile.schema.json | 29 + .../contracts/execution-result.schema.json | 68 ++ .../contracts/preflight-result.schema.json | 31 + .../contracts/runner-descriptor.schema.json | 41 + scripts/eval-runners/fake/runner.ps1 | 277 ++++++ scripts/eval-runners/opencode/runner.ps1 | 440 ++++++++++ scripts/eval-runners/resolve-runner.ps1 | 33 + scripts/eval-runners/runner-common.ps1 | 808 ++++++++++++++++++ .../tests/fixtures/codex-events.jsonl | 5 + .../tests/fixtures/opencode-events.jsonl | 5 + .../tests/test-runner-conformance.ps1 | 275 ++++++ scripts/generate-eval-report.ps1 | 27 +- scripts/prepare-skill-evals.ps1 | 290 ++++--- scripts/validate-skill-templates.ps1 | 65 +- 17 files changed, 3046 insertions(+), 132 deletions(-) create mode 100644 scripts/eval-runners/README.md create mode 100644 scripts/eval-runners/bridge-execution-result.ps1 create mode 100644 scripts/eval-runners/codex/runner.ps1 create mode 100644 scripts/eval-runners/contracts/execution-profile.schema.json create mode 100644 scripts/eval-runners/contracts/execution-result.schema.json create mode 100644 scripts/eval-runners/contracts/preflight-result.schema.json create mode 100644 scripts/eval-runners/contracts/runner-descriptor.schema.json create mode 100644 scripts/eval-runners/fake/runner.ps1 create mode 100644 scripts/eval-runners/opencode/runner.ps1 create mode 100644 scripts/eval-runners/resolve-runner.ps1 create mode 100644 scripts/eval-runners/runner-common.ps1 create mode 100644 scripts/eval-runners/tests/fixtures/codex-events.jsonl create mode 100644 scripts/eval-runners/tests/fixtures/opencode-events.jsonl create mode 100644 scripts/eval-runners/tests/test-runner-conformance.ps1 diff --git a/scripts/eval-runners/README.md b/scripts/eval-runners/README.md new file mode 100644 index 0000000..c22d9da --- /dev/null +++ b/scripts/eval-runners/README.md @@ -0,0 +1,51 @@ +# Eval Runner protocol + +This directory contains the package-local implementation of the v0.9.1 Eval +Runner protocol. It is copied into prepared packages so the external Eval +Orchestrator can use the same runner implementation that was validated with the +package. It is not a model executor used by repository automation. + +The boundary has three documents: + +```text +run.json + execution-profile.json -> runner -> execution-result.json +``` + +`run.json` is the existing portable one-arm contract. It owns the prompt, +working directory, isolated home, staged candidate skill, and required +isolation semantics. `execution-profile.json` selects the runner and execution +configuration. It contains no credentials or secrets. `execution-result.json` +normalizes one blind execution and keeps grading separate from raw evidence. + +Every runner exposes the same process surface: + +```text +runner.ps1 describe +runner.ps1 preflight -Run -Profile +runner.ps1 execute -Run -Profile +``` + +The commands emit one JSON document. `describe` and `preflight` do not consume +model tokens. `execute` runs exactly one arm, never resumes a session, never +grades or retries for answer quality, and returns a normalized result even for +refusals, timeouts, failures, and incompatibility. + +The package resolver selects a named child directory under this directory. It +does not guess a runner and does not fall back to an improvised worker. A +selected runner that cannot satisfy the required contract returns +`incompatible`. + +The fake runner is deterministic and is the conformance reference. The Codex +and OpenCode runners are thin harness-specific adapters. Their native CLI +flags, environment setup, event parsing, authentication injection, and +isolation checks stay inside their own directories. + +The Codex adapter requires the current `codex exec` contract and an external +`bwrap` (Linux) or `sandbox-exec` (macOS) boundary in addition to Codex's +`workspace-write` sandbox. It fails closed on Windows because this slice does +not claim package-level filesystem read confinement there. The OpenCode +adapter uses the same platform split, `--pure`, isolated configuration roots, +and a narrowly selected provider credential; it also fails closed when its +external sandbox or credential requirement is unavailable. Neither adapter +copies a global skill directory, memory store, plugin set, or normal agent +profile into a run. diff --git a/scripts/eval-runners/bridge-execution-result.ps1 b/scripts/eval-runners/bridge-execution-result.ps1 new file mode 100644 index 0000000..284fea5 --- /dev/null +++ b/scripts/eval-runners/bridge-execution-result.ps1 @@ -0,0 +1,233 @@ +<#! +.SYNOPSIS + Bridges one normalized execution-result.json into the existing eval-result/2 shape. + +.DESCRIPTION + This bridge runs after execution and before grading. It reads no expected + output or assertions, preserves any existing grading array, validates raw + artifact provenance, and deliberately leaves unavailable telemetry null. +#> +[CmdletBinding()] +param( + [Parameter(Mandatory = $true)] + [string]$Run, + + [Parameter(Mandatory = $true)] + [string]$ExecutionResult, + + [Parameter(Mandatory = $true)] + [string]$Result +) + +$ErrorActionPreference = 'Stop' +Set-StrictMode -Version Latest +. (Join-Path $PSScriptRoot 'runner-common.ps1') + +$utf8NoBom = [System.Text.UTF8Encoding]::new($false) + +function Write-BridgeJson { + param( + [Parameter(Mandatory = $true)][string]$Path, + [Parameter(Mandatory = $true)][object]$Value + ) + + New-Item -ItemType Directory -Path (Split-Path -Parent $Path) -Force | Out-Null + $serializable = if ($Value -is [System.Collections.IDictionary]) { + $copy = [ordered]@{} + foreach ($key in $Value.Keys) { $copy[[string]$key] = $Value[$key] } + [pscustomobject]$copy + } else { + $Value + } + [System.IO.File]::WriteAllText($Path, ((ConvertTo-Json -InputObject $serializable -Depth 100) + [Environment]::NewLine), $utf8NoBom) +} + +function Get-CapabilityBoolean { + param([object]$Value) + + if ([string]$Value -in @('supported', 'verified', 'true')) { return $true } + if ([string]$Value -in @('unsupported', 'excluded', 'false')) { return $false } + return $null +} + +function Get-MetricValue { + param( + [Parameter(Mandatory = $true)][object]$Result, + [Parameter(Mandatory = $true)][string]$Name + ) + + $metric = Get-JsonProperty -Object $Result.telemetry -Name $Name -Default $null + if ($null -eq $metric -or [string](Get-JsonProperty -Object $metric -Name 'status' -Default '') -ne 'available') { + return $null + } + return Get-JsonProperty -Object $metric -Name 'value' -Default $null +} + +function Get-ArtifactPath { + param( + [Parameter(Mandatory = $true)][object]$RunData, + [Parameter(Mandatory = $true)][string]$IterationDirectory, + [Parameter(Mandatory = $true)][object]$Artifact + ) + + $path = [string](Get-JsonProperty -Object $Artifact -Name 'path' -Default '') + $scope = [string](Get-JsonProperty -Object $Artifact -Name 'scope' -Default '') + if ($scope -eq 'run') { + return Resolve-ContainedPath -BasePath $RunData.RunRoot -RelativePath $path -FieldName 'execution-result artifact.path' -Kind File + } + if ($scope -eq 'package') { + return Resolve-ContainedPath -BasePath $IterationDirectory -RelativePath $path -FieldName 'execution-result package artifact.path' -Kind File + } + throw "Unsupported execution-result artifact scope '$scope'." +} + +function Get-ResultRelativeArtifactPath { + param( + [Parameter(Mandatory = $true)][string]$EvalDirectory, + [Parameter(Mandatory = $true)][string]$FullPath + ) + + $relative = [System.IO.Path]::GetRelativePath($EvalDirectory, $FullPath).Replace('\', '/') + Assert-SafeRelativePath -RelativePath $relative -FieldName 'result.output_files' + return $relative +} + +function Get-ExistingGrading { + param([Parameter(Mandatory = $true)][string]$ResultPath) + + if (-not (Test-Path -LiteralPath $ResultPath -PathType Leaf)) { + return @() + } + try { + $existing = Read-RunnerJson -Path $ResultPath + return @(Get-JsonProperty -Object $existing -Name 'grading' -Default @()) + } catch { + return @() + } +} + +try { + $runData = Resolve-RunContract -RunPath $Run + $runPath = $runData.RunPath + $runDirectory = $runData.RunRoot + $evalDirectory = Split-Path -Parent $runDirectory + $iterationDirectory = Split-Path -Parent $evalDirectory + $executionPath = (Resolve-Path -LiteralPath $ExecutionResult -ErrorAction Stop).Path + $resultPath = [System.IO.Path]::GetFullPath($Result, (Get-Location).Path) + if (-not (Test-PathInside -BasePath $iterationDirectory -CandidatePath $executionPath)) { + throw 'execution-result.json must remain inside the prepared iteration package.' + } + if (-not (Test-PathInside -BasePath $iterationDirectory -CandidatePath $resultPath)) { + throw 'eval-result output must remain inside the prepared iteration package.' + } + + $raw = Read-RunnerJson -Path $executionPath + [void](Assert-ExecutionResult -Result $raw) + if ([string]$raw.input.prompt_sha256 -ne $runData.PromptHash) { + throw 'execution-result input.prompt_sha256 does not match prompt.md.' + } + if ([string]$raw.input.run_json_sha256 -ne (Get-Sha256HexFromFile -Path $runPath)) { + throw 'execution-result input.run_json_sha256 does not match run.json.' + } + $profilePath = Join-Path $iterationDirectory 'execution-profile.json' + if (-not (Test-Path -LiteralPath $profilePath -PathType Leaf)) { + throw 'Runner-aware package is missing execution-profile.json.' + } + $profile = Resolve-ExecutionProfile -ProfilePath $profilePath + if ([string]$raw.input.profile_sha256 -ne $profile.Hash) { + throw 'execution-result input.profile_sha256 does not match execution-profile.json.' + } + if ([int]$raw.run.eval_id -ne $runData.EvalId -or [string]$raw.run.configuration -ne $runData.Mode) { + throw 'execution-result run identity does not match run.json.' + } + + $artifactPaths = [System.Collections.Generic.List[string]]::new() + foreach ($artifact in @($raw.artifacts)) { + $full = Get-ArtifactPath -RunData $runData -IterationDirectory $iterationDirectory -Artifact $artifact + $expectedHash = [string](Get-JsonProperty -Object $artifact -Name 'sha256' -Default '') + if ((Get-Sha256HexFromFile -Path $full) -ne $expectedHash) { + throw "Artifact '$($artifact.path)' has a hash that does not match the recorded execution evidence." + } + $expectedSize = [int64](Get-JsonProperty -Object $artifact -Name 'size' -Default -1) + if ((Get-Item -LiteralPath $full).Length -ne $expectedSize) { + throw "Artifact '$($artifact.path)' has a size that does not match the recorded execution evidence." + } + $artifactPaths.Add((Get-ResultRelativeArtifactPath -EvalDirectory $evalDirectory -FullPath $full)) + } + + $finalStatus = [string]$raw.final_response.status + $output = if ($finalStatus -eq 'available') { [string]$raw.final_response.text } else { '' } + $transcript = '' + $transcriptMetricObject = Get-JsonProperty -Object $raw.telemetry -Name 'transcript' -Default $null + $transcriptMetric = Get-MetricValue -Result $raw -Name 'transcript' + if ($null -ne $transcriptMetric) { + $transcriptArtifact = [string](Get-JsonProperty -Object $transcriptMetric -Name 'artifact' -Default '') + if (-not [string]::IsNullOrWhiteSpace($transcriptArtifact)) { + $transcript = "artifact: $transcriptArtifact" + } + } + $toolCallsValue = Get-MetricValue -Result $raw -Name 'tool_calls' + $costValue = Get-MetricValue -Result $raw -Name 'cost' + $tokenValue = Get-MetricValue -Result $raw -Name 'tokens' + $evidence = Get-JsonProperty -Object $raw -Name 'evidence' -Default ([ordered]@{}) + $commands = @(Get-JsonProperty -Object $evidence -Name 'commands' -Default @()) + $files = @(Get-JsonProperty -Object $evidence -Name 'files' -Default @()) + $warnings = @((Get-JsonProperty -Object $raw -Name 'warnings' -Default @()) + (Get-JsonProperty -Object $raw -Name 'compatibility_deviations' -Default @())) + $notes = [System.Collections.Generic.List[string]]::new() + $notes.Add("execution_status=$($raw.status)") + if ($finalStatus -eq 'unavailable') { $notes.Add("final_response_unavailable=$($raw.final_response.reason)") } + foreach ($warning in $warnings) { if (-not [string]::IsNullOrWhiteSpace([string]$warning)) { $notes.Add([string]$warning) } } + + $existingGrading = @(Get-ExistingGrading -ResultPath $resultPath) + $caps = Get-JsonProperty -Object $raw.isolation -Name 'capabilities' -Default ([ordered]@{}) + $portableResult = [ordered]@{ + schema = (Get-RunnerSchemaNames).PortableResult + skill_name = if ($runData.Mode -eq 'with_skill') { [string](Get-JsonProperty -Object $runData.Contract -Name 'skillName' -Default '') } else { '' } + iteration = [int](Get-JsonProperty -Object $runData.Contract -Name 'iteration' -Default 0) + eval_id = $runData.EvalId + eval_name = $runData.EvalName + configuration = $runData.Mode + model = [string](Get-JsonProperty -Object $raw.resolved -Name 'model' -Default (Get-JsonProperty -Object $raw.requested -Name 'model' -Default '')) + provider = [string](Get-JsonProperty -Object $raw.resolved -Name 'provider' -Default (Get-JsonProperty -Object $raw.requested -Name 'provider' -Default '')) + harness = "$(Get-JsonProperty -Object $raw.harness -Name 'name' -Default 'unknown') $(Get-JsonProperty -Object $raw.harness -Name 'version' -Default '')".Trim() + executed_utc = [string]$raw.finished_utc + output = $output + output_files = @($artifactPaths | Sort-Object -Unique) + transcript = $transcript + shell_commands = @($commands) + files_read = @(Get-JsonProperty -Object $evidence -Name 'files_read' -Default @()) + files_written = @(Get-JsonProperty -Object $evidence -Name 'files_written' -Default @()) + stdout = if (@($artifactPaths | Where-Object { $_ -match 'events\.jsonl$' }).Count -gt 0) { 'artifact: events.jsonl' } else { '' } + stderr = if (@($artifactPaths | Where-Object { $_ -match 'stderr\.txt$' }).Count -gt 0) { 'artifact: stderr.txt' } else { '' } + exit_status = Get-JsonProperty -Object $raw.exit -Name 'status' -Default $null + duration_seconds = [double]$raw.duration_seconds + total_tokens = if ($null -ne $tokenValue) { Get-JsonProperty -Object $tokenValue -Name 'total_tokens' -Default $null } else { $null } + tool_calls = $toolCallsValue + turns = Get-JsonProperty -Object $evidence -Name 'turns' -Default $null + base_input_tokens = if ($null -ne $tokenValue) { Get-JsonProperty -Object $tokenValue -Name 'input_tokens' -Default (Get-JsonProperty -Object $tokenValue -Name 'input' -Default $null) } else { $null } + output_tokens = if ($null -ne $tokenValue) { Get-JsonProperty -Object $tokenValue -Name 'output_tokens' -Default (Get-JsonProperty -Object $tokenValue -Name 'output' -Default $null) } else { $null } + cache_read_tokens = if ($null -ne $tokenValue) { Get-JsonProperty -Object $tokenValue -Name 'cached_input_tokens' -Default (Get-JsonProperty -Object $tokenValue -Name 'cache_read' -Default $null) } else { $null } + cache_write_tokens = if ($null -ne $tokenValue) { Get-JsonProperty -Object $tokenValue -Name 'cache_write_tokens' -Default (Get-JsonProperty -Object $tokenValue -Name 'cache_write' -Default $null) } else { $null } + cache_write_1h_tokens = $null + estimated_cost_usd = $costValue + model_effort = [string](Get-JsonProperty -Object $raw.resolved -Name 'reasoning_effort' -Default '') + isolation = [ordered]@{ + fresh_context = Get-CapabilityBoolean (Get-JsonProperty -Object $caps -Name 'fresh_context' -Default $null) + isolated_home = Get-CapabilityBoolean (Get-JsonProperty -Object $caps -Name 'isolated_home_config' -Default $null) + isolated_cwd = Get-CapabilityBoolean (Get-JsonProperty -Object $caps -Name 'isolated_working_directory' -Default $null) + filesystem_sandbox = Get-CapabilityBoolean (Get-JsonProperty -Object $caps -Name 'filesystem_confinement' -Default $null) + candidate_skill_exposed = Get-CapabilityBoolean (Get-JsonProperty -Object $caps -Name 'candidate_skill_exposure' -Default $null) + transcript_captured = if ($null -ne $transcriptMetricObject) { [string](Get-JsonProperty -Object $transcriptMetricObject -Name 'status' -Default '') -eq 'available' } else { $null } + } + execution_status = [string]$raw.status + execution_run_id = [string]$raw.run_id + execution_result_file = [System.IO.Path]::GetRelativePath($evalDirectory, $executionPath).Replace('\', '/') + grading = @($existingGrading) + notes = [string]::Join("`n", @($notes)) + } + Write-BridgeJson -Path $resultPath -Value $portableResult + Write-RunnerJson -Value ([ordered]@{ schema = 'codebeltnet/agentic/eval-result-bridge/1'; result = [System.IO.Path]::GetRelativePath($iterationDirectory, $resultPath).Replace('\', '/'); execution_status = $raw.status }) -AsOutput +} catch { + [Console]::Error.WriteLine($_.Exception.Message) + exit 2 +} diff --git a/scripts/eval-runners/codex/runner.ps1 b/scripts/eval-runners/codex/runner.ps1 new file mode 100644 index 0000000..f63f423 --- /dev/null +++ b/scripts/eval-runners/codex/runner.ps1 @@ -0,0 +1,500 @@ +<#! +.SYNOPSIS + Codex Eval Runner adapter. + +.DESCRIPTION + This is the only place where Codex CLI flags, CODEX_HOME handling, JSONL + event parsing, and Codex sandbox limitations are defined. +#> +[CmdletBinding()] +param( + [Parameter(Mandatory = $true, Position = 0)] + [ValidateSet('describe', 'preflight', 'execute')] + [string]$Command, + + [string]$Run, + [string]$Profile +) + +$ErrorActionPreference = 'Stop' +Set-StrictMode -Version Latest +. (Join-Path $PSScriptRoot '..\runner-common.ps1') + +$descriptor = [ordered]@{ + schema = (Get-RunnerSchemaNames).Descriptor + protocol_version = (Get-RunnerSchemaNames).Protocol + name = 'codex' + version = '0.9.1' + platforms = @('linux', 'macos') + harness = [ordered]@{ name = 'OpenAI Codex CLI'; version = 'current-supported' } + capabilities = [ordered]@{ + fresh_context = 'supported' + isolated_home_config = 'supported' + isolated_working_directory = 'supported' + filesystem_confinement = 'conditional' + ambient_candidate_skill_exclusion = 'supported' + candidate_skill_exposure = 'supported' + prompt_fidelity = 'supported' + model_configuration_lock = 'supported' + response_capture = 'supported' + transcript_event_capture = 'supported' + token_telemetry = 'conditional' + cache_token_telemetry = 'conditional' + tool_call_telemetry = 'supported' + command_evidence = 'conditional' + file_evidence = 'conditional' + cost_telemetry = 'conditional' + native_skill_activation_evidence = 'unsupported' + } + supported_telemetry = @('transcript_event_capture', 'token_telemetry', 'cache_token_telemetry', 'tool_call_telemetry', 'command_evidence', 'file_evidence', 'cost_telemetry') + configuration_profiles = @('isolated-default') + tool_profiles = @('default') +} + +function Write-ProtocolError { + param([string]$Message) + + [Console]::Error.WriteLine($Message) + exit 2 +} + +function Resolve-CodexInputs { + if ([string]::IsNullOrWhiteSpace($Run) -or [string]::IsNullOrWhiteSpace($Profile)) { + throw 'preflight and execute require -Run and -Profile.' + } + return [pscustomobject]@{ + Run = Resolve-RunContract -RunPath $Run + Profile = Resolve-ExecutionProfile -ProfilePath $Profile + } +} + +function Get-CodexAuthSource { + param([Parameter(Mandatory = $true)][string]$Provider) + + $authVariables = @(Get-ProviderAuthenticationVariables -Provider $Provider) + foreach ($name in $authVariables) { + if (-not [string]::IsNullOrWhiteSpace([Environment]::GetEnvironmentVariable($name))) { + return [pscustomobject]@{ Kind = 'environment'; Name = $name; Path = $null } + } + } + + $configuredHome = [Environment]::GetEnvironmentVariable('CODEX_HOME') + $codexHome = if ([string]::IsNullOrWhiteSpace($configuredHome)) { + Join-Path ([Environment]::GetFolderPath('UserProfile')) '.codex' + } else { + $configuredHome + } + $authPath = Join-Path $codexHome 'auth.json' + if (Test-Path -LiteralPath $authPath -PathType Leaf) { + return [pscustomobject]@{ Kind = 'file'; Name = 'auth.json'; Path = (Resolve-Path -LiteralPath $authPath).Path } + } + + return [pscustomobject]@{ Kind = 'missing'; Name = $null; Path = $null } +} + +function Invoke-CodexCli { + param( + [Parameter(Mandatory = $true)][object]$CommandInfo, + [Parameter(Mandatory = $true)][string[]]$Arguments, + [Parameter(Mandatory = $true)][object]$Inputs, + [System.Collections.IDictionary]$Environment, + [byte[]]$InputBytes = @(), + [int]$TimeoutSeconds = 60 + ) + + $allArguments = @($CommandInfo.Prefix) + @($Arguments) + return Invoke-RunnerProcess -FileName $CommandInfo.FileName -ArgumentList $allArguments -WorkingDirectory $Inputs.Run.WorkingDirectoryPath -Environment $Environment -InputBytes $InputBytes -TimeoutSeconds $TimeoutSeconds +} + +function Get-CodexHelpResult { + param( + [Parameter(Mandatory = $true)][object]$CommandInfo, + [Parameter(Mandatory = $true)][object]$Inputs + ) + + $environment = New-RunnerEnvironment -Run $Inputs.Run + return Invoke-CodexCli -CommandInfo $CommandInfo -Arguments @('exec', '--help') -Inputs $Inputs -Environment $environment -TimeoutSeconds 30 +} + +function Resolve-SandboxCommand { + param([Parameter(Mandatory = $true)][string]$Name) + + return Resolve-ExternalCommand -Name $Name +} + +function Get-CodexPreflight { + param([Parameter(Mandatory = $true)][object]$Inputs) + + $checks = [System.Collections.Generic.List[object]]::new() + $reasons = [System.Collections.Generic.List[string]]::new() + $warnings = [System.Collections.Generic.List[string]]::new() + $profile = $Inputs.Profile + $run = $Inputs.Run + $commandInfo = Resolve-ExternalCommand -Name 'codex' + $platform = Get-PlatformName + $sandboxName = switch ($platform) { + 'linux' { 'bwrap' } + 'macos' { 'sandbox-exec' } + default { $null } + } + $sandboxInfo = if ([string]::IsNullOrWhiteSpace([string]$sandboxName)) { $null } else { Resolve-SandboxCommand -Name $sandboxName } + + if ($profile.Runner -ne 'codex') { + $reasons.Add("execution-profile.json selects '$($profile.Runner)' rather than codex.") + } else { + $checks.Add((New-PreflightCheck -Name 'runner_selection' -Status passed -Detail 'The selected runner is codex.')) + } + if ([string]::IsNullOrWhiteSpace($profile.Provider) -or $profile.Provider.ToLowerInvariant() -notin @('openai', 'chatgpt')) { + $reasons.Add("Codex requires provider 'openai' or 'chatgpt'; received '$($profile.Provider)'.") + } else { + $checks.Add((New-PreflightCheck -Name 'provider' -Status passed -Detail $profile.Provider)) + } + if ([string]::IsNullOrWhiteSpace($profile.Model)) { + $reasons.Add('Codex requires a model in execution-profile.json.') + } else { + $checks.Add((New-PreflightCheck -Name 'model' -Status passed -Detail $profile.Model)) + } + if ($profile.ConfigurationProfile -ne 'isolated-default') { + $reasons.Add("configuration_profile '$($profile.ConfigurationProfile)' is unsupported by codex.") + } + if ($profile.ToolProfile -ne 'default') { + $reasons.Add("tool_profile '$($profile.ToolProfile)' is unsupported by codex.") + } + + if ($null -eq $commandInfo) { + $reasons.Add('The Codex CLI executable is not available on PATH.') + } else { + $checks.Add((New-PreflightCheck -Name 'harness_executable' -Status passed -Detail $commandInfo.Source)) + try { + $help = Get-CodexHelpResult -CommandInfo $commandInfo -Inputs $Inputs + if ($help.TimedOut -or $help.ExitCode -ne 0) { + $reasons.Add("Codex exec --help failed with exit status $($help.ExitCode).") + } else { + $helpText = [string]::Join("`n", @($help.Stdout, $help.Stderr)) + foreach ($flag in @('--ephemeral', '--ignore-user-config', '--ignore-rules', '--json', '--output-last-message', '--sandbox', '--approve-for-me')) { + if ($helpText -notmatch [regex]::Escape($flag)) { + $reasons.Add("The installed Codex CLI does not advertise required flag '$flag'.") + } + } + if ($reasons.Count -eq 0) { + $checks.Add((New-PreflightCheck -Name 'harness_contract' -Status passed -Detail 'Codex exec advertises the required noninteractive, ephemeral, isolated, and structured-output flags.')) + } + } + } catch { + $reasons.Add("Could not inspect Codex CLI capabilities: $($_.Exception.Message)") + } + } + + $auth = Get-CodexAuthSource -Provider ([string]$profile.Provider) + if ($auth.Kind -eq 'missing') { + $reasons.Add('No narrow Codex authentication source is available (provider environment variable or CODEX_HOME/auth.json).') + } else { + $checks.Add((New-PreflightCheck -Name 'authentication' -Status passed -Detail "Authentication is available through $($auth.Kind) and will be injected into the isolated run only.")) + } + + if ($platform -notin @('linux', 'macos')) { + $reasons.Add("Platform '$platform' is not supported by the Codex runner.") + } elseif ($null -eq $sandboxInfo) { + $reasons.Add("Required external filesystem sandbox '$sandboxName' is unavailable; Codex cannot prove package-boundary read confinement.") + } else { + $checks.Add((New-PreflightCheck -Name 'filesystem_confinement' -Status passed -Detail "External $sandboxName confines Codex to the staged run and required system runtime paths; Codex sandbox=workspace-write remains enabled inside it.")) + } + + $checks.Add((New-PreflightCheck -Name 'fresh_session' -Status passed -Detail 'The adapter uses --ephemeral and never supplies a resume, continue, or session identifier.')) + $checks.Add((New-PreflightCheck -Name 'ambient_configuration' -Status passed -Detail 'The adapter uses an isolated CODEX_HOME plus --ignore-user-config and --ignore-rules.')) + $checks.Add((New-PreflightCheck -Name 'run_paths' -Status passed -Detail "--cd $($run.WorkingDirectoryPath); CODEX_HOME under $($run.HomeDirectoryPath)")) + + $capabilities = [ordered]@{} + foreach ($capabilityName in @(Get-JsonPropertyNames -Object $descriptor.capabilities)) { + $value = [string](Get-JsonProperty -Object $descriptor.capabilities -Name $capabilityName) + if ($capabilityName -eq 'filesystem_confinement' -and $platform -in @('linux', 'macos') -and $commandInfo -ne $null -and $sandboxInfo -ne $null) { + $value = 'supported' + } elseif ($capabilityName -eq 'filesystem_confinement') { + $value = 'unsupported' + } + $capabilities[$capabilityName] = $value + } + if ($platform -eq 'windows') { + $warnings.Add('The installed Codex CLI may run on Windows, but this runner does not claim the required package-level filesystem read boundary there.') + } elseif ($null -eq $sandboxInfo) { + $warnings.Add("The Codex CLI was inspected, but external sandbox '$sandboxName' was unavailable.") + } + + $harnessVersion = if ($null -eq $commandInfo) { 'unavailable' } else { 'available' } + $descriptorCopy = [ordered]@{} + foreach ($key in $descriptor.Keys) { $descriptorCopy[$key] = $descriptor[$key] } + $descriptorCopy.harness = [ordered]@{ name = 'OpenAI Codex CLI'; version = $harnessVersion } + return New-PreflightDocument -Descriptor $descriptorCopy -Profile $profile -Run $run -Compatible ($reasons.Count -eq 0) -Checks @($checks) -Mechanisms @('codex --ephemeral', '--ignore-user-config', '--ignore-rules', '--sandbox workspace-write', "external $sandboxName filesystem sandbox", 'isolated CODEX_HOME', 'prompt on stdin') -ResolvedCapabilities $capabilities -Warnings @($warnings) -Reasons @($reasons) +} + +function New-CodexEnvironment { + param( + [Parameter(Mandatory = $true)][object]$Inputs, + [Parameter(Mandatory = $true)][object]$Auth + ) + + $codexHome = Join-Path $Inputs.Run.HomeDirectoryPath '.codex' + New-Item -ItemType Directory -Path $codexHome -Force | Out-Null + $environment = New-RunnerEnvironment -Run $Inputs.Run -AuthenticationVariables @(Get-ProviderAuthenticationVariables -Provider ([string]$Inputs.Profile.Provider)) -Additional @{ CODEX_HOME = $codexHome } + if ($Auth.Kind -eq 'file') { + Copy-Item -LiteralPath $Auth.Path -Destination (Join-Path $codexHome 'auth.json') -Force | Out-Null + } + return $environment +} + +function Get-LinuxCodexSandboxArguments { + param( + [Parameter(Mandatory = $true)][object]$Inputs, + [Parameter(Mandatory = $true)][object]$CommandInfo, + [Parameter(Mandatory = $true)][System.Collections.IDictionary]$Environment + ) + + $args = [System.Collections.Generic.List[string]]::new() + foreach ($argument in @('--die-with-parent', '--new-session', '--unshare-pid')) { $args.Add($argument) } + foreach ($path in @('/usr', '/usr/local', '/bin', '/sbin', '/lib', '/lib64', '/libexec', '/etc', '/opt')) { + if (Test-Path -LiteralPath $path) { + $args.Add('--ro-bind'); $args.Add($path); $args.Add($path) + } + } + $args.Add('--proc'); $args.Add('/proc') + $args.Add('--dev'); $args.Add('/dev') + $args.Add('--tmpfs'); $args.Add('/tmp') + $args.Add('--bind'); $args.Add($Inputs.Run.RunRoot); $args.Add('/run') + $commandSource = [string]$CommandInfo.Source + $commandDirectory = Split-Path -Parent $commandSource + if (-not ($commandSource.StartsWith('/usr/', [System.StringComparison]::Ordinal) -or $commandSource.StartsWith('/bin/', [System.StringComparison]::Ordinal) -or $commandSource.StartsWith('/opt/', [System.StringComparison]::Ordinal))) { + if (Test-Path -LiteralPath $commandDirectory -PathType Container) { + $args.Add('--ro-bind'); $args.Add($commandDirectory); $args.Add($commandDirectory) + } + } + $args.Add('--chdir'); $args.Add('/run/repo') + $insideEnvironment = [ordered]@{ + HOME = '/run/home' + USERPROFILE = '/run/home' + XDG_CONFIG_HOME = '/run/home/.config' + XDG_DATA_HOME = '/run/home/.local/share' + XDG_CACHE_HOME = '/run/home/.cache' + TEMP = '/run/home/tmp' + TMP = '/run/home/tmp' + CODEX_HOME = '/run/home/.codex' + PATH = '/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin' + CI = '1' + NO_COLOR = '1' + } + foreach ($authName in @(Get-ProviderAuthenticationVariables -Provider ([string]$Inputs.Profile.Provider))) { + if ($Environment.Contains($authName) -and -not [string]::IsNullOrWhiteSpace([string]$Environment[$authName])) { + $insideEnvironment[$authName] = [string]$Environment[$authName] + } + } + foreach ($key in @($insideEnvironment.Keys)) { + $args.Add('--setenv'); $args.Add($key); $args.Add([string]$insideEnvironment[$key]) + } + $args.Add('--') + $args.Add($CommandInfo.FileName) + foreach ($prefix in @($CommandInfo.Prefix)) { $args.Add($prefix) } + return @($args) +} + +function New-CodexMacosSandboxProfile { + param( + [Parameter(Mandatory = $true)][object]$Inputs, + [Parameter(Mandatory = $true)][object]$CommandInfo + ) + + $profilePath = Join-Path $Inputs.Run.HomeDirectoryPath 'codex-sandbox.sb' + $runRoot = $Inputs.Run.RunRoot.Replace('\', '/') + $commandDirectory = (Split-Path -Parent ([string]$CommandInfo.Source)).Replace('\', '/') + $readRoots = @('/usr', '/usr/local', '/bin', '/sbin', '/lib', '/libexec', '/System', '/Library', '/opt', '/private/var/db', $commandDirectory) + $lines = [System.Collections.Generic.List[string]]::new() + $lines.Add('(version 1)') + $lines.Add('(deny default)') + $lines.Add('(allow process*)') + $lines.Add('(allow network*)') + foreach ($root in $readRoots | Sort-Object -Unique) { + if (-not [string]::IsNullOrWhiteSpace($root) -and (Test-Path -LiteralPath $root -PathType Container)) { + $escapedRoot = $root.Replace('\', '/').Replace('"', '\"') + $lines.Add(('(allow file-read* (subpath "{0}"))' -f $escapedRoot)) + } + } + $escapedRunRoot = $runRoot.Replace('"', '\"') + $lines.Add(('(allow file-read* (subpath "{0}"))' -f $escapedRunRoot)) + $lines.Add(('(allow file-write* (subpath "{0}"))' -f $escapedRunRoot)) + $lines.Add('(allow file-read* (subpath "/dev"))') + $lines.Add('(allow file-write* (subpath "/dev/null"))') + [System.IO.File]::WriteAllText($profilePath, ([string]::Join("`n", $lines) + "`n"), [System.Text.UTF8Encoding]::new($false)) + return $profilePath +} + +function Write-CodexCapture { + param( + [Parameter(Mandatory = $true)][object]$RunData, + [Parameter(Mandatory = $true)][string]$RelativePath, + [Parameter(Mandatory = $true)][string]$Text + ) + + $path = Join-Path $RunData.Run.RunRoot ($RelativePath -replace '/', [System.IO.Path]::DirectorySeparatorChar) + $parent = Split-Path -Parent $path + New-Item -ItemType Directory -Path $parent -Force | Out-Null + [System.IO.File]::WriteAllText($path, $Text, [System.Text.UTF8Encoding]::new($false)) + return New-ArtifactReference -Run $RunData.Run -Path $RelativePath -Scope run -MediaType (Get-MediaType -Path $RelativePath) +} + +function Invoke-CodexExecute { + param([Parameter(Mandatory = $true)][object]$Inputs) + + $preflight = Get-CodexPreflight -Inputs $Inputs + $started = [DateTime]::UtcNow + $sessionId = [Guid]::NewGuid().ToString('D') + if ($preflight.status -ne 'compatible') { + $finished = [DateTime]::UtcNow + $failureText = [string]::Join('; ', @($preflight.reasons)) + return New-ExecutionResult -Descriptor $descriptor -Profile $Inputs.Profile -Run $Inputs.Run -Status incompatible -FinalResponseReason 'preflight_incompatible' -StartedUtc $started.ToString('o') -FinishedUtc $finished.ToString('o') -DurationSeconds ($finished - $started).TotalSeconds -Failure (New-ExecutionFailure -Code 'incompatible' -Message $failureText) -SessionId $sessionId -IsolationCapabilities ([ordered]@{ fresh_context = 'supported'; isolated_home_config = 'supported'; isolated_working_directory = 'supported'; filesystem_confinement = 'unsupported'; ambient_candidate_skill_exclusion = 'supported'; candidate_skill_exposure = 'supported' }) -IsolationMechanisms @('preflight-only') -Evidence ([ordered]@{ preflight = $preflight }) -AttemptCount 1 + } + + $commandInfo = Resolve-ExternalCommand -Name 'codex' + $auth = Get-CodexAuthSource -Provider ([string]$Inputs.Profile.Provider) + $environment = New-CodexEnvironment -Inputs $Inputs -Auth $auth + $lastResponsePath = 'evidence/codex-final.txt' + $directoryArgument = if ((Get-PlatformName) -eq 'linux') { '/run/repo' } else { $Inputs.Run.WorkingDirectoryPath } + $arguments = @( + 'exec', + '--ephemeral', + '--ignore-user-config', + '--ignore-rules', + '--skip-git-repo-check', + '--json', + '--color', 'never', + '--cd', $directoryArgument, + '--model', $Inputs.Profile.Model, + '--sandbox', 'workspace-write', + '--approve-for-me', + '--output-last-message', (Join-Path $Inputs.Run.RunRoot ($lastResponsePath -replace '/', [System.IO.Path]::DirectorySeparatorChar)) + ) + if (-not [string]::IsNullOrWhiteSpace([string]$Inputs.Profile.ReasoningEffort)) { + $arguments += @('-c', "model_reasoning_effort=$($Inputs.Profile.ReasoningEffort)") + } + $arguments += '-' + + if ((Get-PlatformName) -eq 'linux') { + $sandboxInfo = Resolve-SandboxCommand -Name 'bwrap' + $sandboxArguments = Get-LinuxCodexSandboxArguments -Inputs $Inputs -CommandInfo $commandInfo -Environment $environment + $process = Invoke-RunnerProcess -FileName $sandboxInfo.FileName -ArgumentList (@($sandboxArguments) + @($arguments)) -WorkingDirectory $Inputs.Run.WorkingDirectoryPath -Environment $environment -InputBytes $Inputs.Run.PromptBytes -TimeoutSeconds $Inputs.Profile.TimeoutSeconds + } else { + $sandboxInfo = Resolve-SandboxCommand -Name 'sandbox-exec' + $sandboxProfile = New-CodexMacosSandboxProfile -Inputs $Inputs -CommandInfo $commandInfo + $sandboxArguments = @('-f', $sandboxProfile, '--', $commandInfo.FileName) + @($commandInfo.Prefix) + @($arguments) + $process = Invoke-RunnerProcess -FileName $sandboxInfo.FileName -ArgumentList $sandboxArguments -WorkingDirectory $Inputs.Run.WorkingDirectoryPath -Environment $environment -InputBytes $Inputs.Run.PromptBytes -TimeoutSeconds $Inputs.Profile.TimeoutSeconds + } + $stdoutArtifact = Write-CodexCapture -RunData $Inputs -RelativePath 'evidence/codex-events.jsonl' -Text $process.Stdout + $stderrArtifact = Write-CodexCapture -RunData $Inputs -RelativePath 'evidence/codex-stderr.txt' -Text $process.Stderr + $artifacts = [System.Collections.Generic.List[object]]::new() + $artifacts.Add($stdoutArtifact) + $artifacts.Add($stderrArtifact) + + $parsed = ConvertFrom-JsonLines -Text $process.Stdout + $warnings = [System.Collections.Generic.List[string]]::new() + foreach ($parseError in @($parsed.Errors)) { $warnings.Add("Codex event parse error: $parseError") } + $finalText = $null + $threadId = $null + $turnFailure = $null + $usage = $null + $toolCalls = 0 + $commands = [System.Collections.Generic.List[object]]::new() + $files = [System.Collections.Generic.List[object]]::new() + $eventCounts = @{} + foreach ($event in @($parsed.Events)) { + $eventType = [string](Get-JsonProperty -Object $event -Name 'type' -Default '') + if ([string]::IsNullOrWhiteSpace($eventType)) { + $warnings.Add('Codex emitted an event without a type; it was ignored.') + continue + } + if ($eventCounts.ContainsKey($eventType)) { $eventCounts[$eventType]++ } else { $eventCounts[$eventType] = 1 } + switch ($eventType) { + 'thread.started' { $threadId = [string](Get-JsonProperty -Object $event -Name 'thread_id' -Default '') } + 'item.completed' { + $item = Get-JsonProperty -Object $event -Name 'item' -Default $null + $itemType = [string](Get-JsonProperty -Object $item -Name 'type' -Default '') + if ($itemType -eq 'agent_message') { + $candidate = [string](Get-JsonProperty -Object $item -Name 'text' -Default '') + if (-not [string]::IsNullOrWhiteSpace($candidate)) { $finalText = $candidate } + } elseif ($itemType -in @('command_execution', 'mcp_tool_call', 'file_change')) { + $toolCalls++ + if ($itemType -eq 'command_execution') { + $commands.Add([ordered]@{ type = $itemType; command = Get-JsonProperty -Object $item -Name 'command'; exit_code = Get-JsonProperty -Object $item -Name 'exit_code' }) + } else { + $files.Add([ordered]@{ type = $itemType; item = $itemType }) + } + } + } + 'turn.completed' { + $usage = Get-JsonProperty -Object $event -Name 'usage' -Default $null + } + 'turn.failed' { $turnFailure = Get-JsonProperty -Object $event -Name 'error' -Default 'Codex turn failed.' } + 'error' { $turnFailure = Get-JsonProperty -Object $event -Name 'message' -Default 'Codex emitted an error.' } + { $_ -in @('turn.started', 'item.started', 'item.updated') } { } + default { $warnings.Add("Unknown Codex event '$eventType' was preserved as a warning.") } + } + } + if (Test-Path -LiteralPath (Join-Path $Inputs.Run.RunRoot ($lastResponsePath -replace '/', [System.IO.Path]::DirectorySeparatorChar)) -PathType Leaf) { + $lastArtifact = New-ArtifactReference -Run $Inputs.Run -Path $lastResponsePath -Scope run -MediaType 'text/plain; charset=utf-8' + $artifacts.Add($lastArtifact) + if ([string]::IsNullOrWhiteSpace($finalText)) { + $finalText = [System.IO.File]::ReadAllText((Join-Path $Inputs.Run.RunRoot ($lastResponsePath -replace '/', [System.IO.Path]::DirectorySeparatorChar)), [System.Text.UTF8Encoding]::new($false)) + } + } + + $status = 'completed' + $reason = $null + $failure = $null + $exitStatus = if ($process.TimedOut) { $null } else { [Nullable[int]]$process.ExitCode } + if ($process.TimedOut) { + $status = 'timed_out' + $reason = 'codex_timeout' + $failure = New-ExecutionFailure -Code 'timed_out' -Message 'Codex did not finish before timeout_seconds.' + } elseif ($process.ExitCode -ne 0 -or $null -ne $turnFailure) { + $status = 'failed' + $reason = 'codex_failure' + $failure = New-ExecutionFailure -Code 'codex_failure' -Message ([string]$turnFailure) + } elseif ([string]::IsNullOrWhiteSpace($finalText)) { + $warnings.Add('Codex exited successfully without a final agent message.') + $reason = 'codex_did_not_return_final_response' + } + + $tokenMetric = if ($null -eq $usage) { + New-UnavailableMetric -Reason 'codex_did_not_expose_turn_usage' + } else { + $usageValue = [ordered]@{} + foreach ($name in @('input_tokens', 'cached_input_tokens', 'output_tokens', 'reasoning_output_tokens')) { + $value = Get-JsonProperty -Object $usage -Name $name -Default $null + if ($null -ne $value) { $usageValue[$name] = $value } + } + if ($usageValue.Count -eq 0) { New-UnavailableMetric -Reason 'codex_usage_event_had_no_supported_buckets' } else { New-AvailableMetric -Value $usageValue } + } + $telemetry = [ordered]@{ + transcript = New-AvailableMetric -Value ([ordered]@{ artifact = 'evidence/codex-events.jsonl'; complete = $true }) + tokens = $tokenMetric + tool_calls = New-AvailableMetric -Value $toolCalls + cost = New-UnavailableMetric -Reason 'codex_runner_does_not_estimate_cost' + } + $finished = [DateTime]::UtcNow + return New-ExecutionResult -Descriptor $descriptor -Profile $Inputs.Profile -Run $Inputs.Run -Status $status -FinalResponse $finalText -FinalResponseReason $reason -StartedUtc $process.StartedUtc.ToString('o') -FinishedUtc $finished.ToString('o') -DurationSeconds $process.DurationSeconds -ExitStatus $exitStatus -Failure $failure -SessionId (if ([string]::IsNullOrWhiteSpace($threadId)) { $sessionId } else { $threadId }) -IsolationCapabilities ([ordered]@{ fresh_context = 'supported'; isolated_home_config = 'supported'; isolated_working_directory = 'supported'; filesystem_confinement = 'supported'; ambient_candidate_skill_exclusion = 'supported'; candidate_skill_exposure = 'supported'; prompt_fidelity = 'supported'; model_configuration_lock = 'supported'; response_capture = 'supported' }) -IsolationMechanisms @('codex --ephemeral', '--ignore-user-config', '--ignore-rules', '--sandbox workspace-write', 'external bwrap/sandbox-exec filesystem sandbox', 'isolated CODEX_HOME', 'prompt on stdin') -Telemetry $telemetry -Artifacts @($artifacts) -Warnings @($warnings) -Evidence ([ordered]@{ thread_id = $threadId; event_counts = $eventCounts; commands = @($commands); files = @($files); prompt_first_input = $true; resume = $false; stdout_exit_code = $process.ExitCode; sandbox = if ((Get-PlatformName) -eq 'linux') { 'bwrap' } else { 'sandbox-exec' } }) -AttemptCount 1 +} + +try { + [void](Assert-RunnerDescriptor -Descriptor $descriptor) + switch ($Command) { + 'describe' { Write-RunnerJson -Value $descriptor -AsOutput } + 'preflight' { + $inputs = Resolve-CodexInputs + Write-RunnerJson -Value (Get-CodexPreflight -Inputs $inputs) -AsOutput + } + 'execute' { + $inputs = Resolve-CodexInputs + $result = Invoke-CodexExecute -Inputs $inputs + [void](Assert-ExecutionResult -Result $result) + Write-RunnerJson -Value $result -AsOutput + } + } +} catch { + Write-ProtocolError -Message $_.Exception.Message +} diff --git a/scripts/eval-runners/contracts/execution-profile.schema.json b/scripts/eval-runners/contracts/execution-profile.schema.json new file mode 100644 index 0000000..91aac6b --- /dev/null +++ b/scripts/eval-runners/contracts/execution-profile.schema.json @@ -0,0 +1,29 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "https://codebelt.net/schemas/agentic/eval-execution-profile/1", + "title": "Codebelt Agentic Eval Execution Profile", + "type": "object", + "additionalProperties": false, + "required": [ + "schema", + "runner", + "provider", + "model", + "reasoning_effort", + "configuration_profile", + "tool_profile", + "timeout_seconds", + "concurrency" + ], + "properties": { + "schema": { "const": "codebeltnet/agentic/eval-execution-profile/1" }, + "runner": { "type": ["string", "null"], "pattern": "^[a-z0-9][a-z0-9-]*$" }, + "provider": { "type": ["string", "null"], "minLength": 1 }, + "model": { "type": ["string", "null"], "minLength": 1 }, + "reasoning_effort": { "type": ["string", "null"], "minLength": 1 }, + "configuration_profile": { "type": "string", "minLength": 1 }, + "tool_profile": { "type": "string", "minLength": 1 }, + "timeout_seconds": { "type": "integer", "minimum": 1, "maximum": 86400 }, + "concurrency": { "type": "integer", "minimum": 1 } + } +} diff --git a/scripts/eval-runners/contracts/execution-result.schema.json b/scripts/eval-runners/contracts/execution-result.schema.json new file mode 100644 index 0000000..8cce4eb --- /dev/null +++ b/scripts/eval-runners/contracts/execution-result.schema.json @@ -0,0 +1,68 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "https://codebelt.net/schemas/agentic/eval-execution-result/1", + "title": "Codebelt Agentic Eval Execution Result", + "type": "object", + "additionalProperties": true, + "required": [ + "schema", + "protocol_version", + "run_id", + "session", + "status", + "run", + "final_response", + "runner", + "harness", + "requested", + "resolved", + "started_utc", + "finished_utc", + "duration_seconds", + "exit", + "input", + "isolation", + "telemetry", + "evidence", + "artifacts", + "warnings", + "compatibility_deviations", + "attempt_count" + ], + "properties": { + "schema": { "const": "codebeltnet/agentic/eval-execution-result/1" }, + "protocol_version": { "const": "codebeltnet/agentic/eval-runner-protocol/1" }, + "run_id": { "type": "string", "minLength": 1 }, + "session": { + "type": "object", + "required": ["id", "fresh", "resumed"], + "properties": { + "id": { "type": "string", "minLength": 1 }, + "fresh": { "const": true }, + "resumed": { "const": false } + } + }, + "status": { "enum": ["completed", "failed", "timed_out", "cancelled", "incompatible"] }, + "final_response": { + "type": "object", + "required": ["status"], + "properties": { + "status": { "enum": ["available", "unavailable"] }, + "text": { "type": "string" }, + "reason": { "type": "string" } + } + }, + "duration_seconds": { "type": "number", "minimum": 0 }, + "input": { + "type": "object", + "required": ["prompt_sha256", "run_json_sha256", "profile_sha256"] + }, + "isolation": { "type": "object" }, + "telemetry": { "type": "object" }, + "evidence": { "type": "object" }, + "artifacts": { "type": "array" }, + "warnings": { "type": "array", "items": { "type": "string" } }, + "compatibility_deviations": { "type": "array", "items": { "type": "string" } }, + "attempt_count": { "const": 1 } + } +} diff --git a/scripts/eval-runners/contracts/preflight-result.schema.json b/scripts/eval-runners/contracts/preflight-result.schema.json new file mode 100644 index 0000000..a3c3126 --- /dev/null +++ b/scripts/eval-runners/contracts/preflight-result.schema.json @@ -0,0 +1,31 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "https://codebelt.net/schemas/agentic/eval-runner-preflight/1", + "title": "Codebelt Agentic Eval Runner Preflight Result", + "type": "object", + "additionalProperties": true, + "required": [ + "schema", + "protocol_version", + "status", + "runner", + "harness", + "run", + "requested", + "checks", + "resolved_capabilities", + "mechanisms", + "warnings", + "reasons" + ], + "properties": { + "schema": { "const": "codebeltnet/agentic/eval-runner-preflight/1" }, + "protocol_version": { "const": "codebeltnet/agentic/eval-runner-protocol/1" }, + "status": { "enum": ["compatible", "incompatible"] }, + "checks": { "type": "array" }, + "resolved_capabilities": { "type": "object" }, + "mechanisms": { "type": "array", "items": { "type": "string" } }, + "warnings": { "type": "array", "items": { "type": "string" } }, + "reasons": { "type": "array", "items": { "type": "string" } } + } +} diff --git a/scripts/eval-runners/contracts/runner-descriptor.schema.json b/scripts/eval-runners/contracts/runner-descriptor.schema.json new file mode 100644 index 0000000..e4ae3b4 --- /dev/null +++ b/scripts/eval-runners/contracts/runner-descriptor.schema.json @@ -0,0 +1,41 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "https://codebelt.net/schemas/agentic/eval-runner-descriptor/1", + "title": "Codebelt Agentic Eval Runner Descriptor", + "type": "object", + "additionalProperties": true, + "required": [ + "schema", + "protocol_version", + "name", + "version", + "platforms", + "harness", + "capabilities", + "supported_telemetry", + "configuration_profiles", + "tool_profiles" + ], + "properties": { + "schema": { "const": "codebeltnet/agentic/eval-runner-descriptor/1" }, + "protocol_version": { "const": "codebeltnet/agentic/eval-runner-protocol/1" }, + "name": { "type": "string", "minLength": 1 }, + "version": { "type": "string", "minLength": 1 }, + "platforms": { "type": "array", "items": { "type": "string" } }, + "harness": { + "type": "object", + "required": ["name", "version"], + "properties": { + "name": { "type": "string", "minLength": 1 }, + "version": { "type": "string", "minLength": 1 } + } + }, + "capabilities": { + "type": "object", + "additionalProperties": { "enum": ["supported", "conditional", "unsupported"] } + }, + "supported_telemetry": { "type": "array", "items": { "type": "string" } }, + "configuration_profiles": { "type": "array", "items": { "type": "string" } }, + "tool_profiles": { "type": "array", "items": { "type": "string" } } + } +} diff --git a/scripts/eval-runners/fake/runner.ps1 b/scripts/eval-runners/fake/runner.ps1 new file mode 100644 index 0000000..7ced67a --- /dev/null +++ b/scripts/eval-runners/fake/runner.ps1 @@ -0,0 +1,277 @@ +<#! +.SYNOPSIS + Deterministic Eval Runner used for protocol conformance tests. + +.DESCRIPTION + This runner never invokes a model or a provider. It records the same + boundaries as a real runner and emits deterministic outcomes selected by + -Scenario or AGENTIC_FAKE_SCENARIO. +#> +[CmdletBinding()] +param( + [Parameter(Mandatory = $true, Position = 0)] + [ValidateSet('describe', 'preflight', 'execute')] + [string]$Command, + + [string]$Run, + [string]$Profile, + + [ValidateSet('normal', 'refusal', 'timeout', 'failure', 'incompatible', 'escape', 'unknown-event')] + [string]$Scenario +) + +$ErrorActionPreference = 'Stop' +Set-StrictMode -Version Latest +. (Join-Path $PSScriptRoot '..\runner-common.ps1') + +$descriptor = [ordered]@{ + schema = (Get-RunnerSchemaNames).Descriptor + protocol_version = (Get-RunnerSchemaNames).Protocol + name = 'fake' + version = '0.9.1-test' + platforms = @('windows', 'linux', 'macos') + harness = [ordered]@{ name = 'deterministic-fake'; version = '1' } + capabilities = [ordered]@{ + fresh_context = 'supported' + isolated_home_config = 'supported' + isolated_working_directory = 'supported' + filesystem_confinement = 'supported' + ambient_candidate_skill_exclusion = 'supported' + candidate_skill_exposure = 'supported' + prompt_fidelity = 'supported' + model_configuration_lock = 'supported' + response_capture = 'supported' + transcript_event_capture = 'supported' + token_telemetry = 'unsupported' + cache_token_telemetry = 'unsupported' + tool_call_telemetry = 'supported' + command_evidence = 'supported' + file_evidence = 'supported' + cost_telemetry = 'unsupported' + native_skill_activation_evidence = 'unsupported' + } + supported_telemetry = @('transcript_event_capture', 'tool_call_telemetry', 'command_evidence', 'file_evidence') + configuration_profiles = @('isolated-default') + tool_profiles = @('default') +} + +function Write-ProtocolError { + param([string]$Message) + + [Console]::Error.WriteLine($Message) + exit 2 +} + +function Get-FakeScenario { + if (-not [string]::IsNullOrWhiteSpace($Scenario)) { + return $Scenario + } + $fromEnvironment = [Environment]::GetEnvironmentVariable('AGENTIC_FAKE_SCENARIO') + if ([string]::IsNullOrWhiteSpace($fromEnvironment)) { + return 'normal' + } + return $fromEnvironment.ToLowerInvariant() +} + +function Resolve-FakeInputs { + if ([string]::IsNullOrWhiteSpace($Run) -or [string]::IsNullOrWhiteSpace($Profile)) { + throw 'preflight and execute require -Run and -Profile.' + } + $resolvedRun = Resolve-RunContract -RunPath $Run + $resolvedProfile = Resolve-ExecutionProfile -ProfilePath $Profile + return [pscustomobject]@{ Run = $resolvedRun; Profile = $resolvedProfile } +} + +function Get-FakePreflight { + param( + [Parameter(Mandatory = $true)][object]$Inputs, + [string]$ScenarioValue = 'normal' + ) + + $checks = [System.Collections.Generic.List[object]]::new() + $reasons = [System.Collections.Generic.List[string]]::new() + $warnings = [System.Collections.Generic.List[string]]::new() + $profile = $Inputs.Profile + $run = $Inputs.Run + + if ($profile.Runner -ne 'fake') { + $reasons.Add("execution-profile.json selects '$($profile.Runner)' rather than fake.") + } else { + $checks.Add((New-PreflightCheck -Name 'runner_selection' -Status passed -Detail 'The selected runner is fake.')) + } + + if ($profile.ConfigurationProfile -notin @($descriptor.configuration_profiles)) { + $reasons.Add("configuration_profile '$($profile.ConfigurationProfile)' is not supported by fake.") + } else { + $checks.Add((New-PreflightCheck -Name 'configuration_profile' -Status passed -Detail $profile.ConfigurationProfile)) + } + if ($profile.ToolProfile -notin @($descriptor.tool_profiles)) { + $reasons.Add("tool_profile '$($profile.ToolProfile)' is not supported by fake.") + } else { + $checks.Add((New-PreflightCheck -Name 'tool_profile' -Status passed -Detail $profile.ToolProfile)) + } + + $checks.Add((New-PreflightCheck -Name 'fresh_process' -Status passed -Detail 'Each fake execute command creates a new process and session id.')) + $checks.Add((New-PreflightCheck -Name 'prompt_fidelity' -Status passed -Detail 'The prompt bytes are read once and recorded without transformation.')) + $checks.Add((New-PreflightCheck -Name 'run_paths' -Status passed -Detail "repo=$($run.WorkingDirectoryPath); home=$($run.HomeDirectoryPath)")) + $checks.Add((New-PreflightCheck -Name 'filesystem_confinement' -Status passed -Detail 'Fake escape probes are rejected by the contained-path guard.')) + $checks.Add((New-PreflightCheck -Name 'candidate_skill_boundary' -Status passed -Detail "candidate_skill_exposed=$($run.CandidateSkillExposed)")) + if ($ScenarioValue -eq 'incompatible') { + $reasons.Add('The deterministic incompatible scenario was requested.') + } + + $capabilities = [ordered]@{} + foreach ($property in @($descriptor.capabilities.PSObject.Properties)) { + $capabilities[$property.Name] = $property.Value + } + if ($reasons.Count -gt 0) { + $warnings.Add('No execute process is started for an incompatible preflight.') + } + + return New-PreflightDocument -Descriptor $descriptor -Profile $profile -Run $run -Compatible ($reasons.Count -eq 0) -Checks @($checks) -Mechanisms @('pwsh-process', 'run-directory-contained-path-guard', 'isolated-home-directory') -ResolvedCapabilities $capabilities -Warnings @($warnings) -Reasons @($reasons) +} + +function Write-FakeEvidence { + param( + [Parameter(Mandatory = $true)][object]$Inputs, + [Parameter(Mandatory = $true)][string]$SessionId, + [Parameter(Mandatory = $true)][string]$ScenarioValue + ) + + $evidenceDirectory = Join-Path $Inputs.Run.RunRoot 'evidence' + New-Item -ItemType Directory -Path $evidenceDirectory -Force | Out-Null + $eventsPath = Join-Path $evidenceDirectory 'fake-events.jsonl' + $promptEvidencePath = Join-Path $evidenceDirectory 'prompt-delivery.json' + $boundaryEvidencePath = Join-Path $evidenceDirectory 'boundary-probes.json' + + $events = [System.Collections.Generic.List[string]]::new() + $events.Add((([ordered]@{ type = 'session.started'; session_id = $SessionId } | ConvertTo-Json -Compress))) + $events.Add((([ordered]@{ + type = 'task.input' + ordinal = 1 + prompt_sha256 = $Inputs.Run.PromptHash + byte_length = $Inputs.Run.PromptBytes.Length + first_task_input = $true + } | ConvertTo-Json -Compress))) + $events.Add((([ordered]@{ type = 'response.completed'; status = if ($ScenarioValue -eq 'refusal') { 'refusal' } else { 'completed' } } | ConvertTo-Json -Compress))) + if ($ScenarioValue -eq 'unknown-event') { + $events.Add((([ordered]@{ type = 'future.event.v99'; payload = 'ignored-by-conformance-adapter' } | ConvertTo-Json -Compress))) + } + [System.IO.File]::WriteAllText($eventsPath, ([string]::Join("`n", $events) + "`n"), [System.Text.UTF8Encoding]::new($false)) + + [ordered]@{ + prompt_sha256 = $Inputs.Run.PromptHash + first_task_input_sha256 = $Inputs.Run.PromptHash + first_task_input_bytes = $Inputs.Run.PromptBytes.Length + byte_exact = $true + candidate_skill_exposed = $Inputs.Run.CandidateSkillExposed + working_directory = $Inputs.Run.WorkingDirectoryPath + home_directory = $Inputs.Run.HomeDirectoryPath + global_rules_visible = $false + global_memory_visible = $false + global_plugins_visible = $false + global_same_name_skill_visible = $false + } | ConvertTo-Json -Depth 20 | Set-Content -LiteralPath $promptEvidencePath -Encoding utf8NoBOM + + $boundary = [ordered]@{ + read_outside_run = [ordered]@{ attempted = $false; blocked = $true; path = '../eval-metadata.json' } + write_outside_run = [ordered]@{ attempted = $false; blocked = $true; path = '../escape-write.txt' } + } + if ($ScenarioValue -eq 'escape') { + foreach ($probe in @('read_outside_run', 'write_outside_run')) { + $boundary[$probe].attempted = $true + try { + [void](Resolve-ContainedPath -BasePath $Inputs.Run.RunRoot -RelativePath ([string]$boundary[$probe].path) -FieldName $probe) + $boundary[$probe].blocked = $false + } catch { + $boundary[$probe].blocked = $true + $boundary[$probe].error = $_.Exception.Message + } + } + } + $boundary | ConvertTo-Json -Depth 20 | Set-Content -LiteralPath $boundaryEvidencePath -Encoding utf8NoBOM + + return @( + (New-ArtifactReference -Run $Inputs.Run -Path 'evidence/fake-events.jsonl' -Scope run -MediaType 'application/x-ndjson'), + (New-ArtifactReference -Run $Inputs.Run -Path 'evidence/prompt-delivery.json' -Scope run -MediaType 'application/json'), + (New-ArtifactReference -Run $Inputs.Run -Path 'evidence/boundary-probes.json' -Scope run -MediaType 'application/json') + ) +} + +function Invoke-FakeExecute { + param([Parameter(Mandatory = $true)][object]$Inputs) + + $scenarioValue = Get-FakeScenario + if ($scenarioValue -notin @('normal', 'refusal', 'timeout', 'failure', 'incompatible', 'escape', 'unknown-event')) { + throw "Unsupported fake scenario '$scenarioValue'." + } + + $preflight = Get-FakePreflight -Inputs $Inputs -ScenarioValue $scenarioValue + $started = [DateTime]::UtcNow + $sessionId = [Guid]::NewGuid().ToString('D') + $warnings = [System.Collections.Generic.List[string]]::new() + $deviations = [System.Collections.Generic.List[string]]::new() + + if ($preflight.status -eq 'incompatible') { + $finished = [DateTime]::UtcNow + return New-ExecutionResult -Descriptor $descriptor -Profile $Inputs.Profile -Run $Inputs.Run -Status incompatible -FinalResponseReason 'preflight_incompatible' -StartedUtc $started.ToString('o') -FinishedUtc $finished.ToString('o') -DurationSeconds ($finished - $started).TotalSeconds -Failure (New-ExecutionFailure -Code 'incompatible' -Message ([string]::Join('; ', @($preflight.reasons)))) -SessionId $sessionId -IsolationCapabilities ([ordered]@{ fresh_context = 'supported'; isolated_home_config = 'supported'; isolated_working_directory = 'supported'; filesystem_confinement = 'supported'; candidate_skill_exposure = if ($Inputs.Run.CandidateSkillExposed) { 'supported' } else { 'excluded' } }) -IsolationMechanisms @('pwsh-process', 'run-directory-contained-path-guard', 'isolated-home-directory') -CompatibilityDeviations @($deviations) -Evidence ([ordered]@{ preflight = $preflight }) -AttemptCount 1 + } + + $artifacts = @(Write-FakeEvidence -Inputs $Inputs -SessionId $sessionId -ScenarioValue $scenarioValue) + if ($scenarioValue -eq 'unknown-event') { + $warnings.Add('Unknown fake event future.event.v99 was preserved as an explicit warning.') + } + + $status = 'completed' + $finalResponse = 'Deterministic fake runner completed the blind eval arm.' + $finalReason = $null + $exitStatus = [Nullable[int]]0 + $failure = $null + if ($scenarioValue -eq 'refusal') { + $finalResponse = 'I cannot complete this request.' + $warnings.Add('The harness returned a refusal; it was normalized as a completed response, not retried.') + } elseif ($scenarioValue -eq 'timeout') { + $status = 'timed_out' + $finalResponse = $null + $finalReason = 'fake_timeout' + $exitStatus = $null + $failure = New-ExecutionFailure -Code 'timed_out' -Message 'The deterministic fake exceeded its configured execution window.' + } elseif ($scenarioValue -eq 'failure') { + $status = 'failed' + $finalResponse = $null + $finalReason = 'harness_failure' + $exitStatus = [Nullable[int]]17 + $failure = New-ExecutionFailure -Code 'fake_harness_failure' -Message 'The deterministic fake reported a harness failure.' + } + + $telemetry = [ordered]@{ + transcript = New-AvailableMetric -Value ([ordered]@{ artifact = 'evidence/fake-events.jsonl'; complete = $true }) + tokens = New-UnavailableMetric -Reason 'fake_harness_does_not_expose_usage' + tool_calls = New-AvailableMetric -Value 0 + cost = New-UnavailableMetric -Reason 'fake_harness_does_not_expose_cost' + } + $finished = [DateTime]::UtcNow + return New-ExecutionResult -Descriptor $descriptor -Profile $Inputs.Profile -Run $Inputs.Run -Status $status -FinalResponse $finalResponse -FinalResponseReason $finalReason -StartedUtc $started.ToString('o') -FinishedUtc $finished.ToString('o') -DurationSeconds ($finished - $started).TotalSeconds -ExitStatus $exitStatus -Failure $failure -SessionId $sessionId -IsolationCapabilities ([ordered]@{ fresh_context = 'supported'; isolated_home_config = 'supported'; isolated_working_directory = 'supported'; filesystem_confinement = 'supported'; ambient_candidate_skill_exclusion = 'supported'; candidate_skill_exposure = if ($Inputs.Run.CandidateSkillExposed) { 'supported' } else { 'excluded' }; prompt_fidelity = 'supported'; model_configuration_lock = 'supported'; response_capture = 'supported' }) -IsolationMechanisms @('pwsh-process', 'run-directory-contained-path-guard', 'isolated-home-directory') -Telemetry $telemetry -Artifacts $artifacts -Warnings @($warnings) -CompatibilityDeviations @($deviations) -Evidence ([ordered]@{ scenario = $scenarioValue; prompt_first_input = $true; resume = $false; preflight = $preflight }) -AttemptCount 1 +} + +try { + [void](Assert-RunnerDescriptor -Descriptor $descriptor) + switch ($Command) { + 'describe' { + Write-RunnerJson -Value $descriptor -AsOutput + } + 'preflight' { + $inputs = Resolve-FakeInputs + $preflight = Get-FakePreflight -Inputs $inputs -ScenarioValue (Get-FakeScenario) + Write-RunnerJson -Value $preflight -AsOutput + } + 'execute' { + $inputs = Resolve-FakeInputs + $result = Invoke-FakeExecute -Inputs $inputs + [void](Assert-ExecutionResult -Result $result) + Write-RunnerJson -Value $result -AsOutput + } + } +} catch { + Write-ProtocolError -Message $_.Exception.Message +} diff --git a/scripts/eval-runners/opencode/runner.ps1 b/scripts/eval-runners/opencode/runner.ps1 new file mode 100644 index 0000000..b1fe8d8 --- /dev/null +++ b/scripts/eval-runners/opencode/runner.ps1 @@ -0,0 +1,440 @@ +<#! +.SYNOPSIS + OpenCode Eval Runner adapter. + +.DESCRIPTION + This is the only place where OpenCode CLI flags, pure configuration, + sandbox process setup, and JSON event parsing are defined. +#> +[CmdletBinding()] +param( + [Parameter(Mandatory = $true, Position = 0)] + [ValidateSet('describe', 'preflight', 'execute')] + [string]$Command, + + [string]$Run, + [string]$Profile +) + +$ErrorActionPreference = 'Stop' +Set-StrictMode -Version Latest +. (Join-Path $PSScriptRoot '..\runner-common.ps1') + +$descriptor = [ordered]@{ + schema = (Get-RunnerSchemaNames).Descriptor + protocol_version = (Get-RunnerSchemaNames).Protocol + name = 'opencode' + version = '0.9.1' + platforms = @('linux', 'macos') + harness = [ordered]@{ name = 'OpenCode CLI'; version = 'current-supported' } + capabilities = [ordered]@{ + fresh_context = 'supported' + isolated_home_config = 'supported' + isolated_working_directory = 'supported' + filesystem_confinement = 'conditional' + ambient_candidate_skill_exclusion = 'supported' + candidate_skill_exposure = 'supported' + prompt_fidelity = 'supported' + model_configuration_lock = 'supported' + response_capture = 'supported' + transcript_event_capture = 'supported' + token_telemetry = 'conditional' + cache_token_telemetry = 'conditional' + tool_call_telemetry = 'supported' + command_evidence = 'conditional' + file_evidence = 'conditional' + cost_telemetry = 'conditional' + native_skill_activation_evidence = 'unsupported' + } + supported_telemetry = @('transcript_event_capture', 'token_telemetry', 'cache_token_telemetry', 'tool_call_telemetry', 'command_evidence', 'file_evidence', 'cost_telemetry') + configuration_profiles = @('isolated-default') + tool_profiles = @('default') +} + +function Write-ProtocolError { + param([string]$Message) + + [Console]::Error.WriteLine($Message) + exit 2 +} + +function Resolve-OpenCodeInputs { + if ([string]::IsNullOrWhiteSpace($Run) -or [string]::IsNullOrWhiteSpace($Profile)) { + throw 'preflight and execute require -Run and -Profile.' + } + return [pscustomobject]@{ + Run = Resolve-RunContract -RunPath $Run + Profile = Resolve-ExecutionProfile -ProfilePath $Profile + } +} + +function Invoke-OpenCodeCli { + param( + [Parameter(Mandatory = $true)][object]$CommandInfo, + [Parameter(Mandatory = $true)][string[]]$Arguments, + [Parameter(Mandatory = $true)][object]$Inputs, + [System.Collections.IDictionary]$Environment, + [byte[]]$InputBytes = @(), + [int]$TimeoutSeconds = 60 + ) + + $allArguments = @($CommandInfo.Prefix) + @($Arguments) + return Invoke-RunnerProcess -FileName $CommandInfo.FileName -ArgumentList $allArguments -WorkingDirectory $Inputs.Run.WorkingDirectoryPath -Environment $Environment -InputBytes $InputBytes -TimeoutSeconds $TimeoutSeconds +} + +function Get-OpenCodeHelpResult { + param( + [Parameter(Mandatory = $true)][object]$CommandInfo, + [Parameter(Mandatory = $true)][object]$Inputs + ) + + $environment = New-RunnerEnvironment -Run $Inputs.Run + return Invoke-OpenCodeCli -CommandInfo $CommandInfo -Arguments @('run', '--help') -Inputs $Inputs -Environment $environment -TimeoutSeconds 30 +} + +function Get-OpenCodeAuthVariable { + param([Parameter(Mandatory = $true)][string]$Provider) + + $variables = @(Get-ProviderAuthenticationVariables -Provider $Provider) + foreach ($name in $variables) { + if (-not [string]::IsNullOrWhiteSpace([Environment]::GetEnvironmentVariable($name))) { + return $name + } + } + return $null +} + +function Resolve-SandboxCommand { + param([Parameter(Mandatory = $true)][string]$Name) + + return Resolve-ExternalCommand -Name $Name +} + +function Get-OpenCodePreflight { + param([Parameter(Mandatory = $true)][object]$Inputs) + + $checks = [System.Collections.Generic.List[object]]::new() + $reasons = [System.Collections.Generic.List[string]]::new() + $warnings = [System.Collections.Generic.List[string]]::new() + $profile = $Inputs.Profile + $run = $Inputs.Run + $platform = Get-PlatformName + $commandInfo = Resolve-ExternalCommand -Name 'opencode' + $sandboxInfo = if ($platform -eq 'linux') { Resolve-SandboxCommand -Name 'bwrap' } elseif ($platform -eq 'macos') { Resolve-SandboxCommand -Name 'sandbox-exec' } else { $null } + + if ($profile.Runner -ne 'opencode') { + $reasons.Add("execution-profile.json selects '$($profile.Runner)' rather than opencode.") + } else { + $checks.Add((New-PreflightCheck -Name 'runner_selection' -Status passed -Detail 'The selected runner is opencode.')) + } + if ([string]::IsNullOrWhiteSpace($profile.Provider)) { + $reasons.Add('OpenCode requires a provider in execution-profile.json.') + } else { + $checks.Add((New-PreflightCheck -Name 'provider' -Status passed -Detail $profile.Provider)) + } + if ([string]::IsNullOrWhiteSpace($profile.Model)) { + $reasons.Add('OpenCode requires a model in execution-profile.json.') + } else { + $checks.Add((New-PreflightCheck -Name 'model' -Status passed -Detail ("{0}/{1}" -f $profile.Provider, $profile.Model))) + } + if ($profile.ConfigurationProfile -ne 'isolated-default') { + $reasons.Add("configuration_profile '$($profile.ConfigurationProfile)' is unsupported by opencode.") + } + if ($profile.ToolProfile -ne 'default') { + $reasons.Add("tool_profile '$($profile.ToolProfile)' is unsupported by opencode.") + } + if ($null -eq $commandInfo) { + $reasons.Add('The OpenCode CLI executable is not available on PATH.') + } else { + $checks.Add((New-PreflightCheck -Name 'harness_executable' -Status passed -Detail $commandInfo.Source)) + try { + $help = Get-OpenCodeHelpResult -CommandInfo $commandInfo -Inputs $Inputs + if ($help.TimedOut -or $help.ExitCode -ne 0) { + $reasons.Add("OpenCode run --help failed with exit status $($help.ExitCode).") + } else { + $helpText = [string]::Join("`n", @($help.Stdout, $help.Stderr)) + foreach ($flag in @('--pure', '--format', '--dir', '--model', '--auto')) { + if ($helpText -notmatch [regex]::Escape($flag)) { + $reasons.Add("The installed OpenCode CLI does not advertise required flag '$flag'.") + } + } + if ($reasons.Count -eq 0) { + $checks.Add((New-PreflightCheck -Name 'harness_contract' -Status passed -Detail 'OpenCode run advertises pure, noninteractive, model, directory, and structured-output controls.')) + } + } + } catch { + $reasons.Add("Could not inspect OpenCode CLI capabilities: $($_.Exception.Message)") + } + } + + $authVariable = if ([string]::IsNullOrWhiteSpace($profile.Provider)) { $null } else { Get-OpenCodeAuthVariable -Provider ([string]$profile.Provider) } + if ([string]::IsNullOrWhiteSpace($authVariable)) { + $reasons.Add("No narrow provider authentication environment variable is available for '$($profile.Provider)'. OpenCode global auth profiles are not copied into an eval run.") + } else { + $checks.Add((New-PreflightCheck -Name 'authentication' -Status passed -Detail "Provider credential will be passed only as $authVariable.")) + } + + if ($platform -notin @('linux', 'macos')) { + $reasons.Add("Platform '$platform' has no v0.9.1 OpenCode filesystem sandbox implementation.") + } elseif ($null -eq $sandboxInfo) { + $reasons.Add("Required $([string]$(if ($platform -eq 'linux') { 'bwrap' } else { 'sandbox-exec' })) isolation command is unavailable.") + } else { + $checks.Add((New-PreflightCheck -Name 'filesystem_confinement' -Status passed -Detail "External $($sandboxInfo.Source) sandbox confines the process to the staged run and required system runtime paths.")) + } + $checks.Add((New-PreflightCheck -Name 'fresh_session' -Status passed -Detail 'The adapter starts one new opencode run process and supplies no resume, continue, or session id.')) + $checks.Add((New-PreflightCheck -Name 'ambient_configuration' -Status passed -Detail 'The adapter uses --pure and isolated OpenCode configuration roots.')) + $checks.Add((New-PreflightCheck -Name 'prompt_fidelity' -Status passed -Detail 'The exact prompt bytes are sent on stdin as the first and only task input.')) + + $capabilities = [ordered]@{} + foreach ($capabilityName in @(Get-JsonPropertyNames -Object $descriptor.capabilities)) { + $value = [string](Get-JsonProperty -Object $descriptor.capabilities -Name $capabilityName) + if ($capabilityName -eq 'filesystem_confinement' -and $null -ne $sandboxInfo -and $platform -in @('linux', 'macos') -and $reasons.Count -eq 0) { + $value = 'supported' + } elseif ($capabilityName -eq 'filesystem_confinement') { + $value = 'unsupported' + } + $capabilities[$capabilityName] = $value + } + if ($platform -eq 'macos') { + $warnings.Add('macOS sandbox-exec is deprecated by Apple but is used only when present; a future runner revision may replace it with an equivalent supported mechanism.') + } + $harnessVersion = if ($null -eq $commandInfo) { 'unavailable' } else { 'available' } + $descriptorCopy = [ordered]@{} + foreach ($key in $descriptor.Keys) { $descriptorCopy[$key] = $descriptor[$key] } + $descriptorCopy.harness = [ordered]@{ name = 'OpenCode CLI'; version = $harnessVersion } + return New-PreflightDocument -Descriptor $descriptorCopy -Profile $profile -Run $run -Compatible ($reasons.Count -eq 0) -Checks @($checks) -Mechanisms @('--pure', 'isolated OPENCODE_CONFIG_DIR', 'external filesystem sandbox', 'prompt on stdin', 'no session continuation') -ResolvedCapabilities $capabilities -Warnings @($warnings) -Reasons @($reasons) +} + +function New-OpenCodeEnvironment { + param([Parameter(Mandatory = $true)][object]$Inputs) + + $configDirectory = Join-Path $Inputs.Run.HomeDirectoryPath 'opencode-config' + New-Item -ItemType Directory -Path $configDirectory -Force | Out-Null + $configPath = Join-Path $configDirectory 'opencode.json' + [System.IO.File]::WriteAllText($configPath, '{}', [System.Text.UTF8Encoding]::new($false)) + return New-RunnerEnvironment -Run $Inputs.Run -AuthenticationVariables @(Get-ProviderAuthenticationVariables -Provider ([string]$Inputs.Profile.Provider)) -Additional @{ + OPENCODE_CONFIG_DIR = $configDirectory + OPENCODE_CONFIG = $configPath + OPENCODE_DISABLE_AUTOUPDATE = '1' + OPENCODE_DISABLE_PROJECT_CONFIG = '1' + } +} + +function Get-LinuxSandboxArguments { + param( + [Parameter(Mandatory = $true)][object]$Inputs, + [Parameter(Mandatory = $true)][object]$CommandInfo, + [Parameter(Mandatory = $true)][System.Collections.IDictionary]$Environment + ) + + $args = [System.Collections.Generic.List[string]]::new() + foreach ($argument in @('--die-with-parent', '--new-session', '--unshare-pid')) { $args.Add($argument) } + foreach ($path in @('/usr', '/bin', '/lib', '/lib64', '/etc', '/opt')) { + if (Test-Path -LiteralPath $path) { + $args.Add('--ro-bind'); $args.Add($path); $args.Add($path) + } + } + $args.Add('--proc'); $args.Add('/proc') + $args.Add('--dev'); $args.Add('/dev') + $args.Add('--tmpfs'); $args.Add('/tmp') + $args.Add('--bind'); $args.Add($Inputs.Run.RunRoot); $args.Add('/run') + $commandSource = [string]$CommandInfo.Source + $commandDirectory = Split-Path -Parent $commandSource + if (-not ($commandSource.StartsWith('/usr/', [System.StringComparison]::Ordinal) -or $commandSource.StartsWith('/bin/', [System.StringComparison]::Ordinal) -or $commandSource.StartsWith('/opt/', [System.StringComparison]::Ordinal))) { + if (Test-Path -LiteralPath $commandDirectory -PathType Container) { + $args.Add('--ro-bind'); $args.Add($commandDirectory); $args.Add($commandDirectory) + } + } + $args.Add('--chdir'); $args.Add('/run/repo') + $insideEnvironment = [ordered]@{ + HOME = '/run/home' + XDG_CONFIG_HOME = '/run/home/.config' + XDG_DATA_HOME = '/run/home/.local/share' + XDG_CACHE_HOME = '/run/home/.cache' + TEMP = '/run/home/tmp' + TMP = '/run/home/tmp' + OPENCODE_CONFIG_DIR = '/run/home/opencode-config' + OPENCODE_CONFIG = '/run/home/opencode-config/opencode.json' + OPENCODE_DISABLE_AUTOUPDATE = [string]$Environment['OPENCODE_DISABLE_AUTOUPDATE'] + OPENCODE_DISABLE_PROJECT_CONFIG = [string]$Environment['OPENCODE_DISABLE_PROJECT_CONFIG'] + PATH = '/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin' + CI = '1' + NO_COLOR = '1' + } + foreach ($authName in @(Get-ProviderAuthenticationVariables -Provider ([string]$Inputs.Profile.Provider))) { + if ($Environment.Contains($authName) -and -not [string]::IsNullOrWhiteSpace([string]$Environment[$authName])) { + $insideEnvironment[$authName] = [string]$Environment[$authName] + } + } + foreach ($key in @($insideEnvironment.Keys)) { + $args.Add('--setenv'); $args.Add($key); $args.Add([string]$insideEnvironment[$key]) + } + $args.Add('--') + $args.Add($CommandInfo.FileName) + foreach ($prefix in @($CommandInfo.Prefix)) { $args.Add($prefix) } + return @($args) +} + +function New-MacosSandboxProfile { + param( + [Parameter(Mandatory = $true)][object]$Inputs, + [Parameter(Mandatory = $true)][object]$CommandInfo + ) + + $profilePath = Join-Path $Inputs.Run.HomeDirectoryPath 'opencode-sandbox.sb' + $runRoot = $Inputs.Run.RunRoot.Replace('\', '/') + $commandDirectory = (Split-Path -Parent ([string]$CommandInfo.Source)).Replace('\', '/') + $systemReadRoots = @('/usr', '/usr/local', '/bin', '/sbin', '/lib', '/libexec', '/System', '/Library', '/opt', '/private/var/db', $commandDirectory) + $lines = [System.Collections.Generic.List[string]]::new() + $lines.Add('(version 1)') + $lines.Add('(deny default)') + $lines.Add('(allow process*)') + $lines.Add('(allow network*)') + foreach ($root in $systemReadRoots | Sort-Object -Unique) { + if (-not [string]::IsNullOrWhiteSpace($root) -and (Test-Path -LiteralPath $root -PathType Container)) { + $escapedRoot = $root.Replace('"', '\"') + $lines.Add(('(allow file-read* (subpath "{0}"))' -f $escapedRoot)) + } + } + $escapedRunRoot = $runRoot.Replace('"', '\"') + $lines.Add(('(allow file-read* (subpath "{0}"))' -f $escapedRunRoot)) + $lines.Add(('(allow file-write* (subpath "{0}"))' -f $escapedRunRoot)) + $lines.Add('(allow file-read* (subpath "/dev"))') + $lines.Add('(allow file-write* (subpath "/dev/null"))') + [System.IO.File]::WriteAllText($profilePath, ([string]::Join("`n", $lines) + "`n"), [System.Text.UTF8Encoding]::new($false)) + return $profilePath +} + +function Write-OpenCodeCapture { + param( + [Parameter(Mandatory = $true)][object]$RunData, + [Parameter(Mandatory = $true)][string]$RelativePath, + [Parameter(Mandatory = $true)][string]$Text + ) + + $path = Join-Path $RunData.Run.RunRoot ($RelativePath -replace '/', [System.IO.Path]::DirectorySeparatorChar) + New-Item -ItemType Directory -Path (Split-Path -Parent $path) -Force | Out-Null + [System.IO.File]::WriteAllText($path, $Text, [System.Text.UTF8Encoding]::new($false)) + return New-ArtifactReference -Run $RunData.Run -Path $RelativePath -Scope run -MediaType (Get-MediaType -Path $RelativePath) +} + +function Invoke-OpenCodeExecute { + param([Parameter(Mandatory = $true)][object]$Inputs) + + $preflight = Get-OpenCodePreflight -Inputs $Inputs + $started = [DateTime]::UtcNow + $sessionId = [Guid]::NewGuid().ToString('D') + if ($preflight.status -ne 'compatible') { + $finished = [DateTime]::UtcNow + return New-ExecutionResult -Descriptor $descriptor -Profile $Inputs.Profile -Run $Inputs.Run -Status incompatible -FinalResponseReason 'preflight_incompatible' -StartedUtc $started.ToString('o') -FinishedUtc $finished.ToString('o') -DurationSeconds ($finished - $started).TotalSeconds -Failure (New-ExecutionFailure -Code 'incompatible' -Message ([string]::Join('; ', @($preflight.reasons)))) -SessionId $sessionId -IsolationCapabilities ([ordered]@{ fresh_context = 'supported'; isolated_home_config = 'supported'; isolated_working_directory = 'supported'; filesystem_confinement = 'unsupported'; ambient_candidate_skill_exclusion = 'supported'; candidate_skill_exposure = 'supported' }) -IsolationMechanisms @('preflight-only') -Evidence ([ordered]@{ preflight = $preflight }) -AttemptCount 1 + } + + $commandInfo = Resolve-ExternalCommand -Name 'opencode' + $environment = New-OpenCodeEnvironment -Inputs $Inputs + $model = "{0}/{1}" -f $Inputs.Profile.Provider, $Inputs.Profile.Model + $directoryArgument = if ((Get-PlatformName) -eq 'linux') { '/run/repo' } else { $Inputs.Run.WorkingDirectoryPath } + $arguments = @('run', '--format', 'json', '--pure', '--dir', $directoryArgument, '--model', $model, '--auto') + if (-not [string]::IsNullOrWhiteSpace([string]$Inputs.Profile.ReasoningEffort)) { + $arguments += @('--variant', $Inputs.Profile.ReasoningEffort) + } + + if ((Get-PlatformName) -eq 'linux') { + $sandboxInfo = Resolve-SandboxCommand -Name 'bwrap' + $sandboxArguments = Get-LinuxSandboxArguments -Inputs $Inputs -CommandInfo $commandInfo -Environment $environment + $process = Invoke-RunnerProcess -FileName $sandboxInfo.FileName -ArgumentList (@($sandboxArguments) + @($arguments)) -WorkingDirectory $Inputs.Run.WorkingDirectoryPath -Environment $environment -InputBytes $Inputs.Run.PromptBytes -TimeoutSeconds $Inputs.Profile.TimeoutSeconds + } else { + $sandboxInfo = Resolve-SandboxCommand -Name 'sandbox-exec' + $sandboxProfile = New-MacosSandboxProfile -Inputs $Inputs -CommandInfo $commandInfo + $sandboxArguments = @('-f', $sandboxProfile, '--', $commandInfo.FileName) + @($commandInfo.Prefix) + $arguments + $process = Invoke-RunnerProcess -FileName $sandboxInfo.FileName -ArgumentList $sandboxArguments -WorkingDirectory $Inputs.Run.WorkingDirectoryPath -Environment $environment -InputBytes $Inputs.Run.PromptBytes -TimeoutSeconds $Inputs.Profile.TimeoutSeconds + } + + $stdoutArtifact = Write-OpenCodeCapture -RunData $Inputs -RelativePath 'evidence/opencode-events.jsonl' -Text $process.Stdout + $stderrArtifact = Write-OpenCodeCapture -RunData $Inputs -RelativePath 'evidence/opencode-stderr.txt' -Text $process.Stderr + $artifacts = [System.Collections.Generic.List[object]]::new() + $artifacts.Add($stdoutArtifact); $artifacts.Add($stderrArtifact) + $parsed = ConvertFrom-JsonLines -Text $process.Stdout + $warnings = [System.Collections.Generic.List[string]]::new() + foreach ($parseError in @($parsed.Errors)) { $warnings.Add("OpenCode event parse error: $parseError") } + $finalTextParts = [System.Collections.Generic.List[string]]::new() + $eventCounts = @{} + $toolCalls = 0 + $commands = [System.Collections.Generic.List[object]]::new() + $usageBuckets = [ordered]@{} + $failureMessage = $null + foreach ($event in @($parsed.Events)) { + $eventType = [string](Get-JsonProperty -Object $event -Name 'type' -Default '') + if ([string]::IsNullOrWhiteSpace($eventType)) { + $warnings.Add('OpenCode emitted an event without a type; it was ignored.') + continue + } + if ($eventCounts.ContainsKey($eventType)) { $eventCounts[$eventType]++ } else { $eventCounts[$eventType] = 1 } + $part = Get-JsonProperty -Object $event -Name 'part' -Default $null + switch ($eventType) { + 'text' { + $text = Get-JsonProperty -Object $event -Name 'text' -Default (Get-JsonProperty -Object $part -Name 'text' -Default '') + if (-not [string]::IsNullOrWhiteSpace([string]$text)) { $finalTextParts.Add([string]$text) } + } + 'step_finish' { + $tokens = Get-JsonProperty -Object $part -Name 'tokens' -Default (Get-JsonProperty -Object $event -Name 'tokens' -Default $null) + if ($null -ne $tokens) { + foreach ($name in @('input', 'output', 'reasoning', 'cache_read', 'cache_write')) { + $value = Get-JsonProperty -Object $tokens -Name $name -Default $null + if ($null -ne $value) { $usageBuckets[$name] = $value } + } + } + $costValue = Get-JsonProperty -Object $part -Name 'cost' -Default (Get-JsonProperty -Object $event -Name 'cost' -Default $null) + if ($null -ne $costValue) { $usageBuckets['cost'] = $costValue } + } + 'tool_use' { + $toolCalls++ + $toolName = Get-JsonProperty -Object $part -Name 'tool' -Default (Get-JsonProperty -Object $event -Name 'tool' -Default '') + $commands.Add([ordered]@{ tool = [string]$toolName }) + } + 'error' { + $failureMessage = [string](Get-JsonProperty -Object $event -Name 'message' -Default (Get-JsonProperty -Object $part -Name 'message' -Default 'OpenCode emitted an error.')) + } + 'step_start' { } + 'reasoning' { } + default { $warnings.Add("Unknown OpenCode event '$eventType' was preserved as a warning.") } + } + } + $finalText = if ($finalTextParts.Count -gt 0) { [string]::Join('', $finalTextParts) } else { $null } + $status = 'completed' + $reason = $null + $failure = $null + $exitStatus = if ($process.TimedOut) { $null } else { [Nullable[int]]$process.ExitCode } + if ($process.TimedOut) { + $status = 'timed_out'; $reason = 'opencode_timeout'; $failure = New-ExecutionFailure -Code 'timed_out' -Message 'OpenCode did not finish before timeout_seconds.' + } elseif ($process.ExitCode -ne 0 -or $null -ne $failureMessage) { + $status = 'failed'; $reason = 'opencode_failure'; $failure = New-ExecutionFailure -Code 'opencode_failure' -Message ([string]$failureMessage) + } elseif ([string]::IsNullOrWhiteSpace($finalText)) { + $reason = 'opencode_did_not_return_final_response'; $warnings.Add('OpenCode exited successfully without a text response.') + } + $telemetry = [ordered]@{ + transcript = New-AvailableMetric -Value ([ordered]@{ artifact = 'evidence/opencode-events.jsonl'; complete = $true }) + tokens = if ($usageBuckets.Count -eq 0) { New-UnavailableMetric -Reason 'opencode_did_not_expose_usage' } else { New-AvailableMetric -Value $usageBuckets } + tool_calls = New-AvailableMetric -Value $toolCalls + cost = if ($usageBuckets.Contains('cost')) { New-AvailableMetric -Value $usageBuckets['cost'] } else { New-UnavailableMetric -Reason 'opencode_did_not_expose_cost' } + } + return New-ExecutionResult -Descriptor $descriptor -Profile $Inputs.Profile -Run $Inputs.Run -Status $status -FinalResponse $finalText -FinalResponseReason $reason -StartedUtc $process.StartedUtc.ToString('o') -FinishedUtc $process.FinishedUtc.ToString('o') -DurationSeconds $process.DurationSeconds -ExitStatus $exitStatus -Failure $failure -SessionId $sessionId -IsolationCapabilities ([ordered]@{ fresh_context = 'supported'; isolated_home_config = 'supported'; isolated_working_directory = 'supported'; filesystem_confinement = 'supported'; ambient_candidate_skill_exclusion = 'supported'; candidate_skill_exposure = 'supported'; prompt_fidelity = 'supported'; model_configuration_lock = 'supported'; response_capture = 'supported' }) -IsolationMechanisms @('--pure', 'isolated OPENCODE_CONFIG_DIR', 'bwrap/sandbox-exec', 'prompt on stdin', 'no session continuation') -Telemetry $telemetry -Artifacts @($artifacts) -Warnings @($warnings) -Evidence ([ordered]@{ event_counts = $eventCounts; commands = @($commands); prompt_first_input = $true; resume = $false; model_argument = $model; sandbox = if ((Get-PlatformName) -eq 'linux') { 'bwrap' } else { 'sandbox-exec' } }) -AttemptCount 1 +} + +try { + [void](Assert-RunnerDescriptor -Descriptor $descriptor) + switch ($Command) { + 'describe' { Write-RunnerJson -Value $descriptor -AsOutput } + 'preflight' { + $inputs = Resolve-OpenCodeInputs + Write-RunnerJson -Value (Get-OpenCodePreflight -Inputs $inputs) -AsOutput + } + 'execute' { + $inputs = Resolve-OpenCodeInputs + $result = Invoke-OpenCodeExecute -Inputs $inputs + [void](Assert-ExecutionResult -Result $result) + Write-RunnerJson -Value $result -AsOutput + } + } +} catch { + Write-ProtocolError -Message $_.Exception.Message +} diff --git a/scripts/eval-runners/resolve-runner.ps1 b/scripts/eval-runners/resolve-runner.ps1 new file mode 100644 index 0000000..9517dac --- /dev/null +++ b/scripts/eval-runners/resolve-runner.ps1 @@ -0,0 +1,33 @@ +<#! +.SYNOPSIS + Resolves one package-local Eval Runner without guessing or falling back. +#> +[CmdletBinding()] +param( + [Parameter(Mandatory = $true, Position = 0)] + [string]$Runner +) + +$ErrorActionPreference = 'Stop' +Set-StrictMode -Version Latest + +$protocol = 'codebeltnet/agentic/eval-runner-protocol/1' +if ($Runner -notmatch '^[a-z0-9][a-z0-9-]*$') { + throw "Runner name '$Runner' is not a safe package-local runner name." +} + +$runnerPath = Join-Path (Join-Path $PSScriptRoot $Runner) 'runner.ps1' +if (-not (Test-Path -LiteralPath $runnerPath -PathType Leaf)) { + throw "Selected Eval Runner '$Runner' is unavailable in this package." +} + +$resolved = (Resolve-Path -LiteralPath $runnerPath).Path +$root = (Resolve-Path -LiteralPath $PSScriptRoot).Path +$relative = [System.IO.Path]::GetRelativePath($root, $resolved).Replace('\', '/') + +[ordered]@{ + schema = 'codebeltnet/agentic/eval-runner-resolution/1' + protocol_version = $protocol + runner = $Runner + path = $relative +} | ConvertTo-Json -Depth 10 -Compress diff --git a/scripts/eval-runners/runner-common.ps1 b/scripts/eval-runners/runner-common.ps1 new file mode 100644 index 0000000..4efcb8d --- /dev/null +++ b/scripts/eval-runners/runner-common.ps1 @@ -0,0 +1,808 @@ +Set-StrictMode -Version Latest +$ErrorActionPreference = 'Stop' + +function Get-RunnerSchemaNames { + return [ordered]@{ + Protocol = 'codebeltnet/agentic/eval-runner-protocol/1' + Descriptor = 'codebeltnet/agentic/eval-runner-descriptor/1' + Preflight = 'codebeltnet/agentic/eval-runner-preflight/1' + Profile = 'codebeltnet/agentic/eval-execution-profile/1' + Result = 'codebeltnet/agentic/eval-execution-result/1' + PortableResult = 'codebeltnet/agentic/eval-result/2' + Run = 'codebeltnet/agentic/eval-run/1' + } +} + +function Get-JsonProperty { + param( + [object]$Object, + [string]$Name, + [object]$Default = $null + ) + + if ($null -ne $Object -and $Object -is [System.Collections.IDictionary] -and $Object.Contains($Name)) { + if ($null -ne $Object[$Name]) { + return $Object[$Name] + } + return $Default + } + + if ($null -ne $Object -and $Object.PSObject.Properties.Name -contains $Name -and $null -ne $Object.$Name) { + return $Object.$Name + } + + return $Default +} + +function Get-JsonPropertyNames { + param([object]$Object) + + if ($null -eq $Object) { + return @() + } + if ($Object -is [System.Collections.IDictionary]) { + return @($Object.Keys | ForEach-Object { [string]$_ }) + } + return @($Object.PSObject.Properties.Name) +} + +function Test-JsonProperty { + param( + [object]$Object, + [Parameter(Mandatory = $true)][string]$Name + ) + + return (Get-JsonPropertyNames -Object $Object) -contains $Name +} + +function Read-RunnerJson { + param([Parameter(Mandatory = $true)][string]$Path) + + if (-not (Test-Path -LiteralPath $Path -PathType Leaf)) { + throw "JSON file '$Path' does not exist." + } + + return [System.IO.File]::ReadAllText((Resolve-Path -LiteralPath $Path).Path, [System.Text.UTF8Encoding]::new($false)) | ConvertFrom-Json +} + +function Write-RunnerJson { + param( + [Parameter(Mandatory = $true)][object]$Value, + [switch]$AsOutput + ) + + $json = ((ConvertTo-Json -InputObject $Value -Depth 100) + [Environment]::NewLine) + if ($AsOutput) { + [Console]::Out.Write($json) + return + } + + return $json +} + +function Get-Sha256HexFromBytes { + param([Parameter(Mandatory = $true)][byte[]]$Bytes) + + $sha = [System.Security.Cryptography.SHA256]::Create() + try { + return ([Convert]::ToHexString($sha.ComputeHash($Bytes))).ToLowerInvariant() + } finally { + $sha.Dispose() + } +} + +function Get-Sha256HexFromFile { + param([Parameter(Mandatory = $true)][string]$Path) + + $resolved = (Resolve-Path -LiteralPath $Path -ErrorAction Stop).Path + return Get-Sha256HexFromBytes -Bytes ([System.IO.File]::ReadAllBytes($resolved)) +} + +function Test-Sha256 { + param([string]$Value) + + return -not [string]::IsNullOrWhiteSpace($Value) -and $Value -match '^[0-9a-fA-F]{64}$' +} + +function Test-PathInside { + param( + [Parameter(Mandatory = $true)][string]$BasePath, + [Parameter(Mandatory = $true)][string]$CandidatePath + ) + + $base = ([System.IO.Path]::GetFullPath($BasePath)).TrimEnd([char[]]@('\', '/')) + $candidate = ([System.IO.Path]::GetFullPath($CandidatePath)).TrimEnd([char[]]@('\', '/')) + return $candidate -eq $base -or $candidate.StartsWith($base + [System.IO.Path]::DirectorySeparatorChar, [System.StringComparison]::OrdinalIgnoreCase) +} + +function Assert-SafeRelativePath { + param( + [Parameter(Mandatory = $true)][string]$RelativePath, + [Parameter(Mandatory = $true)][string]$FieldName + ) + + if ([string]::IsNullOrWhiteSpace($RelativePath) -or [System.IO.Path]::IsPathRooted($RelativePath) -or $RelativePath -match '^[A-Za-z]:') { + throw "$FieldName must be a non-empty relative path." + } + + $normalized = $RelativePath.Replace('\', '/') + if (($normalized -split '/') -contains '..') { + throw "$FieldName must not contain a parent-directory segment." + } +} + +function Resolve-ContainedPath { + param( + [Parameter(Mandatory = $true)][string]$BasePath, + [Parameter(Mandatory = $true)][string]$RelativePath, + [Parameter(Mandatory = $true)][string]$FieldName, + [ValidateSet('Any', 'File', 'Directory')][string]$Kind = 'Any' + ) + + Assert-SafeRelativePath -RelativePath $RelativePath -FieldName $FieldName + $resolvedBase = (Resolve-Path -LiteralPath $BasePath -ErrorAction Stop).Path + $candidate = [System.IO.Path]::GetFullPath((Join-Path $resolvedBase ($RelativePath -replace '/', [System.IO.Path]::DirectorySeparatorChar))) + if (-not (Test-PathInside -BasePath $resolvedBase -CandidatePath $candidate)) { + throw "$FieldName resolves outside the run directory." + } + + $exists = switch ($Kind) { + 'File' { Test-Path -LiteralPath $candidate -PathType Leaf } + 'Directory' { Test-Path -LiteralPath $candidate -PathType Container } + default { Test-Path -LiteralPath $candidate } + } + if (-not $exists) { + throw "$FieldName '$RelativePath' does not exist under '$resolvedBase'." + } + + $resolvedCandidate = (Resolve-Path -LiteralPath $candidate -ErrorAction Stop).Path + if (-not (Test-PathInside -BasePath $resolvedBase -CandidatePath $resolvedCandidate)) { + throw "$FieldName resolves through a link outside the run directory." + } + return $resolvedCandidate +} + +function Get-PlatformName { + if ($IsWindows) { return 'windows' } + if ($IsMacOS) { return 'macos' } + if ($IsLinux) { return 'linux' } + return 'unknown' +} + +function Resolve-RunContract { + param([Parameter(Mandatory = $true)][string]$RunPath) + + $resolvedRunPath = (Resolve-Path -LiteralPath $RunPath -ErrorAction Stop).Path + $runRoot = Split-Path -Parent $resolvedRunPath + $run = Read-RunnerJson -Path $resolvedRunPath + $schemas = Get-RunnerSchemaNames + + if ([string]$run.schema -ne $schemas.Run) { + throw "run.json must declare '$($schemas.Run)'." + } + if (-not [bool]$run.freshContextRequired -or -not [bool]$run.filesystemIsolationRequired -or -not [bool]$run.isolatedHomeRequired) { + throw 'run.json must require fresh context, filesystem isolation, and isolated home.' + } + + $mode = [string]$run.mode + if ($mode -notin @('with_skill', 'without_skill')) { + throw "run.json mode '$mode' is not with_skill or without_skill." + } + + $promptPath = Resolve-ContainedPath -BasePath $runRoot -RelativePath ([string]$run.promptFile) -FieldName 'promptFile' -Kind File + $workingPath = Resolve-ContainedPath -BasePath $runRoot -RelativePath ([string]$run.workingDirectory) -FieldName 'workingDirectory' -Kind Directory + $homePath = Resolve-ContainedPath -BasePath $runRoot -RelativePath ([string]$run.homeDirectory) -FieldName 'homeDirectory' -Kind Directory + + $skillPath = $null + if ($mode -eq 'with_skill') { + if ([string]::IsNullOrWhiteSpace([string]$run.skillDirectory)) { + throw 'with_skill run.json must declare skillDirectory.' + } + $skillPath = Resolve-ContainedPath -BasePath $runRoot -RelativePath ([string]$run.skillDirectory) -FieldName 'skillDirectory' -Kind Directory + if (-not (Test-Path -LiteralPath (Join-Path $skillPath 'SKILL.md') -PathType Leaf)) { + throw 'with_skill skillDirectory must contain SKILL.md.' + } + } else { + if ($null -ne $run.skillDirectory -and -not [string]::IsNullOrWhiteSpace([string]$run.skillDirectory)) { + throw 'without_skill run.json must not declare skillDirectory.' + } + $skillRoot = Join-Path $runRoot 'skill' + if (Test-Path -LiteralPath $skillRoot) { + throw 'without_skill run must not contain a skill directory.' + } + } + + $promptBytes = [System.IO.File]::ReadAllBytes($promptPath) + $fixtureHash = [string](Get-JsonProperty -Object $run -Name 'fixtureHash' -Default '') + if (-not (Test-Sha256 -Value $fixtureHash)) { + throw 'run.json fixtureHash must be a SHA-256 value.' + } + if ($mode -eq 'with_skill' -and -not (Test-Sha256 -Value ([string]$run.skillHash))) { + throw 'with_skill run.json skillHash must be a SHA-256 value.' + } + + return [pscustomobject]@{ + RunPath = $resolvedRunPath + RunRoot = $runRoot + Contract = $run + EvalId = [int]$run.evalId + EvalName = [string]$run.evalName + Mode = $mode + PromptPath = $promptPath + PromptBytes = $promptBytes + PromptHash = Get-Sha256HexFromBytes -Bytes $promptBytes + WorkingDirectoryPath = $workingPath + HomeDirectoryPath = $homePath + SkillDirectoryPath = $skillPath + CandidateSkillExposed = $mode -eq 'with_skill' + FixtureHash = $fixtureHash + SkillHash = if ($mode -eq 'with_skill') { [string]$run.skillHash } else { $null } + } +} + +function Assert-ProfileHasNoSecrets { + param([Parameter(Mandatory = $true)][object]$Profile) + + foreach ($property in @($Profile.PSObject.Properties)) { + if ([string]$property.Name -match '(?i)(secret|token|password|credential|api[_-]?key|private[_-]?key)') { + throw "execution-profile.json must not contain secret-bearing field '$($property.Name)'." + } + } +} + +function Resolve-ExecutionProfile { + param([Parameter(Mandatory = $true)][string]$ProfilePath) + + $resolvedProfilePath = (Resolve-Path -LiteralPath $ProfilePath -ErrorAction Stop).Path + $profile = Read-RunnerJson -Path $resolvedProfilePath + $schemas = Get-RunnerSchemaNames + if ([string]$profile.schema -ne $schemas.Profile) { + throw "execution-profile.json must declare '$($schemas.Profile)'." + } + Assert-ProfileHasNoSecrets -Profile $profile + + $allowedProperties = @('schema', 'runner', 'provider', 'model', 'reasoning_effort', 'configuration_profile', 'tool_profile', 'timeout_seconds', 'concurrency') + foreach ($propertyName in @(Get-JsonPropertyNames -Object $profile)) { + if ($allowedProperties -notcontains $propertyName) { + throw "execution-profile.json contains unsupported field '$propertyName'." + } + } + + $timeout = [int](Get-JsonProperty -Object $profile -Name 'timeout_seconds' -Default 0) + $concurrency = [int](Get-JsonProperty -Object $profile -Name 'concurrency' -Default 0) + if ($timeout -lt 1 -or $timeout -gt 86400) { + throw 'execution-profile.json timeout_seconds must be between 1 and 86400.' + } + if ($concurrency -lt 1) { + throw 'execution-profile.json concurrency must be at least 1.' + } + if ([string]::IsNullOrWhiteSpace([string]$profile.configuration_profile) -or [string]::IsNullOrWhiteSpace([string]$profile.tool_profile)) { + throw 'execution-profile.json must declare configuration_profile and tool_profile.' + } + + $runnerValue = [string](Get-JsonProperty -Object $profile -Name 'runner' -Default '') + $providerValue = [string](Get-JsonProperty -Object $profile -Name 'provider' -Default '') + $modelValue = [string](Get-JsonProperty -Object $profile -Name 'model' -Default '') + if (-not [string]::IsNullOrWhiteSpace($runnerValue) -and $runnerValue -notmatch '^[a-z0-9][a-z0-9-]*$') { + throw 'execution-profile.json runner must be a safe lowercase runner name.' + } + return [pscustomobject]@{ + Path = $resolvedProfilePath + Profile = $profile + Hash = Get-Sha256HexFromFile -Path $resolvedProfilePath + Runner = if ([string]::IsNullOrWhiteSpace($runnerValue)) { $null } else { $runnerValue } + Provider = if ([string]::IsNullOrWhiteSpace($providerValue)) { $null } else { $providerValue } + Model = if ([string]::IsNullOrWhiteSpace($modelValue)) { $null } else { $modelValue } + ReasoningEffort = if ([string]::IsNullOrWhiteSpace([string]$profile.reasoning_effort)) { $null } else { [string]$profile.reasoning_effort } + ConfigurationProfile = [string]$profile.configuration_profile + ToolProfile = [string]$profile.tool_profile + TimeoutSeconds = $timeout + Concurrency = $concurrency + } +} + +function Assert-RunnerDescriptor { + param([Parameter(Mandatory = $true)][object]$Descriptor) + + $schemas = Get-RunnerSchemaNames + if ([string]$Descriptor.schema -ne $schemas.Descriptor) { + throw "Runner descriptor must declare '$($schemas.Descriptor)'." + } + if ([string]$Descriptor.protocol_version -ne $schemas.Protocol) { + throw "Runner descriptor protocol_version must be '$($schemas.Protocol)'." + } + foreach ($field in @('name', 'version', 'platforms', 'harness', 'capabilities', 'configuration_profiles', 'tool_profiles')) { + if (-not (Test-JsonProperty -Object $Descriptor -Name $field)) { + throw "Runner descriptor is missing '$field'." + } + } + if ([string]::IsNullOrWhiteSpace([string]$Descriptor.name) -or [string]::IsNullOrWhiteSpace([string]$Descriptor.version)) { + throw 'Runner descriptor name and version must be non-empty.' + } + if (-not (Test-JsonProperty -Object $Descriptor.harness -Name 'name') -or -not (Test-JsonProperty -Object $Descriptor.harness -Name 'version')) { + throw 'Runner descriptor harness must declare name and version.' + } + + foreach ($capabilityName in @(Get-JsonPropertyNames -Object $Descriptor.capabilities)) { + $capabilityValue = Get-JsonProperty -Object $Descriptor.capabilities -Name $capabilityName + if ([string]$capabilityValue -notin @('supported', 'conditional', 'unsupported')) { + throw "Runner capability '$capabilityName' must be supported, conditional, or unsupported." + } + } + + $required = @( + 'fresh_context', + 'isolated_home_config', + 'isolated_working_directory', + 'filesystem_confinement', + 'ambient_candidate_skill_exclusion', + 'candidate_skill_exposure', + 'prompt_fidelity', + 'model_configuration_lock', + 'response_capture' + ) + foreach ($name in $required) { + if (-not (Test-JsonProperty -Object $Descriptor.capabilities -Name $name)) { + throw "Runner descriptor is missing required capability '$name'." + } + } + + return $true +} + +function New-PreflightCheck { + param( + [Parameter(Mandatory = $true)][string]$Name, + [Parameter(Mandatory = $true)][ValidateSet('passed', 'failed', 'unavailable', 'not_applicable')][string]$Status, + [Parameter(Mandatory = $true)][string]$Detail + ) + + return [ordered]@{ name = $Name; status = $Status; detail = $Detail } +} + +function New-PreflightDocument { + param( + [Parameter(Mandatory = $true)][object]$Descriptor, + [Parameter(Mandatory = $true)][object]$Profile, + [Parameter(Mandatory = $true)][object]$Run, + [Parameter(Mandatory = $true)][bool]$Compatible, + [object[]]$Checks = @(), + [string[]]$Mechanisms = @(), + [object]$ResolvedCapabilities = $null, + [string[]]$Warnings = @(), + [string[]]$Reasons = @() + ) + + $schemas = Get-RunnerSchemaNames + return [ordered]@{ + schema = $schemas.Preflight + protocol_version = $schemas.Protocol + status = if ($Compatible) { 'compatible' } else { 'incompatible' } + runner = [ordered]@{ name = [string]$Descriptor.name; version = [string]$Descriptor.version } + harness = $Descriptor.harness + run = [ordered]@{ eval_id = $Run.EvalId; eval_name = $Run.EvalName; configuration = $Run.Mode } + requested = [ordered]@{ + provider = $Profile.Provider + model = $Profile.Model + reasoning_effort = $Profile.ReasoningEffort + configuration_profile = $Profile.ConfigurationProfile + tool_profile = $Profile.ToolProfile + timeout_seconds = $Profile.TimeoutSeconds + } + checks = @($Checks) + resolved_capabilities = if ($null -eq $ResolvedCapabilities) { [ordered]@{} } else { $ResolvedCapabilities } + mechanisms = @($Mechanisms) + warnings = @($Warnings) + reasons = @($Reasons) + } +} + +function New-UnavailableMetric { + param([Parameter(Mandatory = $true)][string]$Reason) + + return [ordered]@{ status = 'unavailable'; reason = $Reason } +} + +function New-AvailableMetric { + param([Parameter(Mandatory = $true)][object]$Value) + + return [ordered]@{ status = 'available'; value = $Value } +} + +function New-ExecutionFailure { + param( + [Parameter(Mandatory = $true)][string]$Code, + [Parameter(Mandatory = $true)][string]$Message + ) + + return [ordered]@{ code = $Code; message = $Message } +} + +function New-ExecutionResult { + param( + [Parameter(Mandatory = $true)][object]$Descriptor, + [Parameter(Mandatory = $true)][object]$Profile, + [Parameter(Mandatory = $true)][object]$Run, + [Parameter(Mandatory = $true)][ValidateSet('completed', 'failed', 'timed_out', 'cancelled', 'incompatible')][string]$Status, + [string]$FinalResponse, + [string]$FinalResponseReason, + [string]$StartedUtc, + [string]$FinishedUtc, + [double]$DurationSeconds = 0, + [Nullable[int]]$ExitStatus, + [object]$Failure, + [string]$SessionId, + [hashtable]$IsolationCapabilities, + [string[]]$IsolationMechanisms = @(), + [object]$Telemetry = $null, + [object[]]$Artifacts = @(), + [string[]]$Warnings = @(), + [string[]]$CompatibilityDeviations = @(), + [object]$Evidence = $null, + [int]$AttemptCount = 1 + ) + + $schemas = Get-RunnerSchemaNames + $hasResponse = -not [string]::IsNullOrWhiteSpace($FinalResponse) + $started = if ([string]::IsNullOrWhiteSpace($StartedUtc)) { [DateTime]::UtcNow } else { [DateTime]::Parse($StartedUtc).ToUniversalTime() } + $finished = if ([string]::IsNullOrWhiteSpace($FinishedUtc)) { [DateTime]::UtcNow } else { [DateTime]::Parse($FinishedUtc).ToUniversalTime() } + $isolation = [ordered]@{ + status = if ($Status -eq 'completed' -or $Status -eq 'failed' -or $Status -eq 'timed_out' -or $Status -eq 'cancelled') { 'verified' } else { 'unverified' } + capabilities = if ($null -eq $IsolationCapabilities) { [ordered]@{} } else { $IsolationCapabilities } + mechanisms = @($IsolationMechanisms) + } + + return [ordered]@{ + schema = $schemas.Result + protocol_version = $schemas.Protocol + run_id = [Guid]::NewGuid().ToString('D') + session = [ordered]@{ + id = if ([string]::IsNullOrWhiteSpace($SessionId)) { [Guid]::NewGuid().ToString('D') } else { $SessionId } + fresh = $true + resumed = $false + } + status = $Status + run = [ordered]@{ + eval_id = $Run.EvalId + eval_name = $Run.EvalName + configuration = $Run.Mode + } + final_response = if ($hasResponse) { + [ordered]@{ status = 'available'; text = $FinalResponse } + } else { + [ordered]@{ status = 'unavailable'; reason = if ([string]::IsNullOrWhiteSpace($FinalResponseReason)) { 'harness_did_not_return_a_final_response' } else { $FinalResponseReason } } + } + runner = [ordered]@{ name = [string]$Descriptor.name; version = [string]$Descriptor.version } + harness = $Descriptor.harness + requested = [ordered]@{ + provider = $Profile.Provider + model = $Profile.Model + reasoning_effort = $Profile.ReasoningEffort + configuration_profile = $Profile.ConfigurationProfile + tool_profile = $Profile.ToolProfile + } + resolved = [ordered]@{ + provider = $Profile.Provider + model = $Profile.Model + reasoning_effort = $Profile.ReasoningEffort + configuration_profile = $Profile.ConfigurationProfile + tool_profile = $Profile.ToolProfile + } + started_utc = $started.ToString('yyyy-MM-ddTHH:mm:ss.fffZ') + finished_utc = $finished.ToString('yyyy-MM-ddTHH:mm:ss.fffZ') + duration_seconds = [Math]::Max(0, [Math]::Round($DurationSeconds, 3)) + exit = [ordered]@{ status = $ExitStatus; failure = $Failure } + input = [ordered]@{ + prompt_sha256 = $Run.PromptHash + run_json_sha256 = Get-Sha256HexFromFile -Path $Run.RunPath + profile_sha256 = $Profile.Hash + } + isolation = $isolation + telemetry = if ($null -eq $Telemetry) { + [ordered]@{ + transcript = New-UnavailableMetric -Reason 'harness_did_not_expose_transcript' + tokens = New-UnavailableMetric -Reason 'harness_did_not_expose_usage' + tool_calls = New-UnavailableMetric -Reason 'harness_did_not_expose_tool_calls' + cost = New-UnavailableMetric -Reason 'harness_did_not_expose_cost' + } + } else { $Telemetry } + evidence = if ($null -eq $Evidence) { [ordered]@{} } else { $Evidence } + artifacts = @($Artifacts) + warnings = @($Warnings) + compatibility_deviations = @($CompatibilityDeviations) + attempt_count = $AttemptCount + } +} + +function Assert-ExecutionResult { + param([Parameter(Mandatory = $true)][object]$Result) + + $schemas = Get-RunnerSchemaNames + if ([string]$Result.schema -ne $schemas.Result) { + throw "execution-result.json must declare '$($schemas.Result)'." + } + if ([string]$Result.protocol_version -ne $schemas.Protocol) { + throw "execution-result.json protocol_version must be '$($schemas.Protocol)'." + } + if ([string]$Result.status -notin @('completed', 'failed', 'timed_out', 'cancelled', 'incompatible')) { + throw "execution-result.json status '$($Result.status)' is unsupported." + } + foreach ($field in @('run_id', 'runner', 'harness', 'requested', 'resolved', 'started_utc', 'finished_utc', 'duration_seconds', 'exit', 'final_response', 'input', 'isolation', 'telemetry', 'evidence', 'artifacts', 'warnings')) { + if (-not (Test-JsonProperty -Object $Result -Name $field)) { + throw "execution-result.json is missing '$field'." + } + } + if ([string]::IsNullOrWhiteSpace([string]$Result.run_id)) { + throw 'execution-result.json run_id must be non-empty.' + } + if (-not (Test-JsonProperty -Object $Result.session -Name 'id') -or + -not [bool](Get-JsonProperty -Object $Result.session -Name 'fresh' -Default $false) -or + [bool](Get-JsonProperty -Object $Result.session -Name 'resumed' -Default $true)) { + throw 'execution-result.json must identify a fresh, non-resumed session.' + } + if ([int](Get-JsonProperty -Object $Result -Name 'attempt_count' -Default 0) -ne 1) { + throw 'execution-result.json attempt_count must be exactly 1; quality retries are not allowed.' + } + foreach ($hashField in @('prompt_sha256', 'run_json_sha256', 'profile_sha256')) { + if (-not (Test-Sha256 -Value ([string]$Result.input.$hashField))) { + throw "execution-result.json input.$hashField must be a SHA-256 value." + } + } + if ([double]$Result.duration_seconds -lt 0) { + throw 'execution-result.json duration_seconds must not be negative.' + } + $responseStatus = [string]$Result.final_response.status + if ($responseStatus -eq 'available') { + if (-not (Test-JsonProperty -Object $Result.final_response -Name 'text')) { + throw 'Available final_response must contain text.' + } + } elseif ($responseStatus -eq 'unavailable') { + if ([string]::IsNullOrWhiteSpace([string]$Result.final_response.reason)) { + throw 'Unavailable final_response must contain a reason.' + } + } else { + throw "final_response status '$responseStatus' is unsupported." + } + + foreach ($metricName in @(Get-JsonPropertyNames -Object $Result.telemetry)) { + $metric = Get-JsonProperty -Object $Result.telemetry -Name $metricName + $status = [string](Get-JsonProperty -Object $metric -Name 'status' -Default '') + if ($status -notin @('available', 'unavailable')) { + throw "Telemetry '$metricName' must declare available or unavailable status." + } + if ($status -eq 'unavailable' -and [string]::IsNullOrWhiteSpace([string](Get-JsonProperty -Object $metric -Name 'reason' -Default ''))) { + throw "Unavailable telemetry '$metricName' must declare a reason." + } + } + + foreach ($artifact in @($Result.artifacts)) { + $path = [string](Get-JsonProperty -Object $artifact -Name 'path' -Default '') + $scope = [string](Get-JsonProperty -Object $artifact -Name 'scope' -Default '') + Assert-SafeRelativePath -RelativePath $path -FieldName 'artifact.path' + if ($scope -notin @('run', 'package')) { + throw "artifact.scope '$scope' must be run or package." + } + if (-not (Test-Sha256 -Value ([string]$artifact.sha256))) { + throw 'artifact.sha256 must be a SHA-256 value.' + } + if ([int64]$artifact.size -lt 0 -or [string]::IsNullOrWhiteSpace([string]$artifact.media_type)) { + throw 'artifact must declare non-negative size and media_type.' + } + } + + return $true +} + +function New-RunnerEnvironment { + param( + [Parameter(Mandatory = $true)][object]$Run, + [string[]]$AuthenticationVariables = @(), + [hashtable]$Additional = @{} + ) + + $environment = [ordered]@{} + foreach ($name in @('PATH', 'SystemRoot', 'WINDIR', 'ComSpec', 'PATHEXT', 'LANG', 'LC_ALL', 'TZ', 'SSL_CERT_FILE', 'NODE_PATH')) { + $value = [Environment]::GetEnvironmentVariable($name) + if (-not [string]::IsNullOrWhiteSpace($value)) { + $environment[$name] = $value + } + } + + $tempPath = Join-Path $Run.HomeDirectoryPath 'tmp' + foreach ($directory in @($Run.HomeDirectoryPath, $tempPath, (Join-Path $Run.HomeDirectoryPath '.config'), (Join-Path $Run.HomeDirectoryPath '.local/share'), (Join-Path $Run.HomeDirectoryPath '.cache'))) { + New-Item -ItemType Directory -Path $directory -Force | Out-Null + } + + $environment['HOME'] = $Run.HomeDirectoryPath + $environment['USERPROFILE'] = $Run.HomeDirectoryPath + $environment['XDG_CONFIG_HOME'] = Join-Path $Run.HomeDirectoryPath '.config' + $environment['XDG_DATA_HOME'] = Join-Path $Run.HomeDirectoryPath '.local/share' + $environment['XDG_CACHE_HOME'] = Join-Path $Run.HomeDirectoryPath '.cache' + $environment['TEMP'] = $tempPath + $environment['TMP'] = $tempPath + $environment['CI'] = '1' + $environment['NO_COLOR'] = '1' + + foreach ($name in $AuthenticationVariables | Sort-Object -Unique) { + $value = [Environment]::GetEnvironmentVariable($name) + if (-not [string]::IsNullOrWhiteSpace($value)) { + $environment[$name] = $value + } + } + foreach ($key in $Additional.Keys) { + $environment[$key] = [string]$Additional[$key] + } + + return $environment +} + +function Resolve-ExternalCommand { + param([Parameter(Mandatory = $true)][string]$Name) + + $command = Get-Command $Name -ErrorAction SilentlyContinue + if ($null -eq $command) { + return $null + } + + $source = [string]$command.Source + $extension = [System.IO.Path]::GetExtension($source).ToLowerInvariant() + if ($extension -eq '.ps1') { + $pwsh = Get-Command pwsh -ErrorAction SilentlyContinue + if ($null -eq $pwsh) { + return $null + } + return [pscustomobject]@{ FileName = [string]$pwsh.Source; Prefix = @('-NoProfile', '-File', $source); Source = $source } + } + + return [pscustomobject]@{ FileName = $source; Prefix = @(); Source = $source } +} + +function Invoke-RunnerProcess { + param( + [Parameter(Mandatory = $true)][string]$FileName, + [string[]]$ArgumentList = @(), + [Parameter(Mandatory = $true)][string]$WorkingDirectory, + [System.Collections.IDictionary]$Environment = @{}, + [byte[]]$InputBytes = @(), + [int]$TimeoutSeconds = 900 + ) + + $start = [DateTime]::UtcNow + $startInfo = [System.Diagnostics.ProcessStartInfo]::new() + $startInfo.FileName = $FileName + $startInfo.WorkingDirectory = $WorkingDirectory + $startInfo.UseShellExecute = $false + $startInfo.CreateNoWindow = $true + $startInfo.RedirectStandardInput = $true + $startInfo.RedirectStandardOutput = $true + $startInfo.RedirectStandardError = $true + foreach ($argument in @($ArgumentList)) { + [void]$startInfo.ArgumentList.Add([string]$argument) + } + $startInfo.Environment.Clear() + foreach ($key in $Environment.Keys) { + $startInfo.Environment[$key] = [string]$Environment[$key] + } + + $process = [System.Diagnostics.Process]::new() + $process.StartInfo = $startInfo + try { + if (-not $process.Start()) { + throw "Could not start '$FileName'." + } + + if ($null -ne $InputBytes -and $InputBytes.Length -gt 0) { + $process.StandardInput.BaseStream.Write($InputBytes, 0, $InputBytes.Length) + } + $process.StandardInput.Close() + $stdoutTask = $process.StandardOutput.ReadToEndAsync() + $stderrTask = $process.StandardError.ReadToEndAsync() + $timeoutMilliseconds = [Math]::Min([int64]::MaxValue, [int64]$TimeoutSeconds * 1000) + $exited = $process.WaitForExit([int]([Math]::Min($timeoutMilliseconds, [int]::MaxValue))) + $timedOut = -not $exited + if ($timedOut) { + try { $process.Kill($true) } catch { } + $process.WaitForExit() + } + $stdout = $stdoutTask.GetAwaiter().GetResult() + $stderr = $stderrTask.GetAwaiter().GetResult() + $finish = [DateTime]::UtcNow + + return [pscustomobject]@{ + ExitCode = if ($timedOut) { $null } else { $process.ExitCode } + TimedOut = $timedOut + Stdout = $stdout + Stderr = $stderr + StartedUtc = $start + FinishedUtc = $finish + DurationSeconds = [Math]::Round(($finish - $start).TotalSeconds, 3) + } + } finally { + $process.Dispose() + } +} + +function Get-ProviderAuthenticationVariables { + param([string]$Provider) + + $normalized = ([string]$Provider).ToLowerInvariant() + switch -Regex ($normalized) { + '^openai$|^chatgpt$' { return @('OPENAI_API_KEY') } + '^anthropic$' { return @('ANTHROPIC_API_KEY') } + '^google$|^google-vertex$|^gemini$' { return @('GOOGLE_API_KEY', 'GEMINI_API_KEY') } + '^openrouter$' { return @('OPENROUTER_API_KEY') } + '^xai$|^x-ai$' { return @('XAI_API_KEY') } + '^mistral$' { return @('MISTRAL_API_KEY') } + default { return @() } + } +} + +function Test-EnvironmentVariablePresent { + param([string[]]$Names) + + foreach ($name in @($Names)) { + if (-not [string]::IsNullOrWhiteSpace([Environment]::GetEnvironmentVariable($name))) { + return $true + } + } + return $false +} + +function New-ArtifactReference { + param( + [Parameter(Mandatory = $true)][object]$Run, + [Parameter(Mandatory = $true)][string]$Path, + [ValidateSet('run', 'package')][string]$Scope = 'run', + [string]$MediaType = 'application/octet-stream' + ) + + Assert-SafeRelativePath -RelativePath $Path -FieldName 'artifact.path' + $base = if ($Scope -eq 'run') { $Run.RunRoot } else { Split-Path -Parent (Split-Path -Parent $Run.RunRoot) } + $full = Resolve-ContainedPath -BasePath $base -RelativePath $Path -FieldName 'artifact.path' -Kind File + return [ordered]@{ + path = $Path.Replace('\', '/') + scope = $Scope + sha256 = Get-Sha256HexFromFile -Path $full + size = (Get-Item -LiteralPath $full).Length + media_type = $MediaType + } +} + +function Get-MediaType { + param([string]$Path) + + switch ([System.IO.Path]::GetExtension($Path).ToLowerInvariant()) { + '.jsonl' { return 'application/x-ndjson' } + '.json' { return 'application/json' } + '.txt' { return 'text/plain; charset=utf-8' } + '.md' { return 'text/markdown; charset=utf-8' } + default { return 'application/octet-stream' } + } +} + +function ConvertFrom-JsonLines { + param([Parameter(Mandatory = $true)][string]$Text) + + $events = [System.Collections.Generic.List[object]]::new() + $errors = [System.Collections.Generic.List[string]]::new() + $lineNumber = 0 + foreach ($line in ($Text -split "`r?`n")) { + $lineNumber++ + if ([string]::IsNullOrWhiteSpace($line)) { continue } + try { + $events.Add(($line | ConvertFrom-Json)) + } catch { + $errors.Add("line ${lineNumber}: $($_.Exception.Message)") + } + } + return [pscustomobject]@{ Events = @($events); Errors = @($errors) } +} + +function Get-OutputTextFromFinalResponse { + param([object]$Result) + + if ([string](Get-JsonProperty -Object $Result.final_response -Name 'status' -Default '') -eq 'available') { + return [string]$Result.final_response.text + } + return $null +} diff --git a/scripts/eval-runners/tests/fixtures/codex-events.jsonl b/scripts/eval-runners/tests/fixtures/codex-events.jsonl new file mode 100644 index 0000000..3a3399f --- /dev/null +++ b/scripts/eval-runners/tests/fixtures/codex-events.jsonl @@ -0,0 +1,5 @@ +{"type":"thread.started","thread_id":"fixture-thread"} +{"type":"turn.started"} +{"type":"item.completed","item":{"type":"agent_message","text":"fixture response"}} +{"type":"turn.completed","usage":{"input_tokens":12,"cached_input_tokens":3,"output_tokens":4}} +{"type":"future.event.v99","payload":"unknown"} diff --git a/scripts/eval-runners/tests/fixtures/opencode-events.jsonl b/scripts/eval-runners/tests/fixtures/opencode-events.jsonl new file mode 100644 index 0000000..d0a705a --- /dev/null +++ b/scripts/eval-runners/tests/fixtures/opencode-events.jsonl @@ -0,0 +1,5 @@ +{"type":"step_start","part":{"id":"fixture-step"}} +{"type":"text","part":{"text":"fixture response"}} +{"type":"tool_use","part":{"tool":"read"}} +{"type":"step_finish","part":{"tokens":{"input":12,"output":4},"cost":0.01}} +{"type":"future.event.v99","payload":"unknown"} diff --git a/scripts/eval-runners/tests/test-runner-conformance.ps1 b/scripts/eval-runners/tests/test-runner-conformance.ps1 new file mode 100644 index 0000000..77ca026 --- /dev/null +++ b/scripts/eval-runners/tests/test-runner-conformance.ps1 @@ -0,0 +1,275 @@ +<#! +.SYNOPSIS + Deterministic conformance suite for the common Eval Runner protocol. + +.DESCRIPTION + Creates an ephemeral package under the system temp directory, invokes only + the fake runner, and checks the contracts and recorded event fixtures. It + never invokes Codex, OpenCode, or a live model. +#> +[CmdletBinding()] +param() + +$ErrorActionPreference = 'Stop' +Set-StrictMode -Version Latest + +$runnerRoot = (Resolve-Path (Join-Path $PSScriptRoot '..')).Path +$repoRoot = (Resolve-Path (Join-Path $PSScriptRoot '..\..\..')).Path +. (Join-Path $runnerRoot 'runner-common.ps1') + +function Assert-True { + param([bool]$Condition, [string]$Message) + if (-not $Condition) { throw "ASSERT: $Message" } +} + +function Assert-Equal { + param([object]$Expected, [object]$Actual, [string]$Message) + if ([string]$Expected -ne [string]$Actual) { throw "ASSERT: $Message (expected '$Expected', got '$Actual')" } +} + +function Assert-Throws { + param([scriptblock]$Action, [string]$Message) + $thrown = $false + try { & $Action } catch { $thrown = $true } + if (-not $thrown) { throw "ASSERT: $Message" } +} + +function Write-TestJson { + param([string]$Path, [object]$Value) + New-Item -ItemType Directory -Path (Split-Path -Parent $Path) -Force | Out-Null + [System.IO.File]::WriteAllText($Path, ((ConvertTo-Json -InputObject $Value -Depth 100) + [Environment]::NewLine), [System.Text.UTF8Encoding]::new($false)) +} + +function Invoke-Fake { + param( + [string]$FakePath, + [string]$Command, + [string]$RunPath, + [string]$ProfilePath, + [string]$Scenario = '' + ) + + $arguments = @('-NoProfile', '-File', $FakePath, $Command, '-Run', $RunPath, '-Profile', $ProfilePath) + if (-not [string]::IsNullOrWhiteSpace($Scenario)) { $arguments += @('-Scenario', $Scenario) } + $output = & pwsh @arguments + if ($LASTEXITCODE -ne 0) { throw "Fake runner '$Command' failed: $([string]::Join(' ', @($output)))" } + $json = [string]::Join([Environment]::NewLine, @($output)) + if ([string]::IsNullOrWhiteSpace($json)) { throw "Fake runner '$Command' returned no JSON." } + return $json | ConvertFrom-Json +} + +function New-TestRun { + param( + [string]$IterationDirectory, + [ValidateSet('with_skill', 'without_skill')][string]$Configuration + ) + + $evalDirectory = Join-Path $IterationDirectory 'conformance' + $runRoot = Join-Path $evalDirectory $Configuration + $repo = Join-Path $runRoot 'repo' + $homeDirectory = Join-Path $runRoot 'home' + New-Item -ItemType Directory -Path $repo,$homeDirectory -Force | Out-Null + [System.IO.File]::WriteAllText((Join-Path $homeDirectory 'README.txt'), 'isolated home', [System.Text.UTF8Encoding]::new($false)) + $prompt = "# task`r`n`r`nByte fidelity: Δ and emoji 🚀.`r`n" + [System.IO.File]::WriteAllBytes((Join-Path $runRoot 'prompt.md'), [System.Text.UTF8Encoding]::new($false).GetBytes($prompt)) + if ($Configuration -eq 'with_skill') { + $skill = Join-Path $runRoot 'skill\candidate' + New-Item -ItemType Directory -Path $skill -Force | Out-Null + [System.IO.File]::WriteAllText((Join-Path $skill 'SKILL.md'), '# candidate', [System.Text.UTF8Encoding]::new($false)) + $skillDirectory = 'skill/candidate' + $skillHash = ('b' * 64) + } else { + $skillDirectory = $null + $skillHash = $null + } + $run = [ordered]@{ + schema = (Get-RunnerSchemaNames).Run + evalId = 1 + evalName = 'conformance' + skillName = if ($Configuration -eq 'with_skill') { 'candidate' } else { $null } + iteration = 1 + mode = $Configuration + promptFile = 'prompt.md' + workingDirectory = 'repo' + homeDirectory = 'home' + skillDirectory = $skillDirectory + freshContextRequired = $true + filesystemIsolationRequired = $true + isolatedHomeRequired = $true + gitWorkspace = $false + inputFiles = @() + fixtureHash = ('a' * 64) + skillHash = $skillHash + contract = [ordered]@{ + sandboxRoot = '.' + workingDirectory = 'repo' + homeDirectory = 'home' + mustNotReadOutsideSandbox = $true + mustNotExposeGlobalSkillsOrConfig = $true + } + } + $path = Join-Path $runRoot 'run.json' + Write-TestJson -Path $path -Value $run + return [pscustomobject]@{ Root = $runRoot; Path = $path; Contract = $run } +} + +$testRoot = Join-Path ([System.IO.Path]::GetTempPath()) ('agentic-runner-conformance-' + [Guid]::NewGuid().ToString('N')) +try { + $iteration = Join-Path $testRoot 'iteration-1' + New-Item -ItemType Directory -Path $iteration -Force | Out-Null + $canaryPath = Join-Path $testRoot 'eval-metadata.json' + [System.IO.File]::WriteAllText($canaryPath, 'grading-key-canary', [System.Text.UTF8Encoding]::new($false)) + $globalRoot = Join-Path $testRoot 'seeded-global-profile' + New-Item -ItemType Directory -Path $globalRoot -Force | Out-Null + [System.IO.File]::WriteAllText((Join-Path $globalRoot 'same-name-SKILL.md'), 'must remain invisible', [System.Text.UTF8Encoding]::new($false)) + [Environment]::SetEnvironmentVariable('AGENTIC_FAKE_GLOBAL_RULES', $globalRoot, 'Process') + [Environment]::SetEnvironmentVariable('AGENTIC_FAKE_MEMORY', 'seeded-memory', 'Process') + [Environment]::SetEnvironmentVariable('AGENTIC_FAKE_PLUGINS', 'seeded-plugins', 'Process') + + $profilePath = Join-Path $iteration 'execution-profile.json' + Write-TestJson -Path $profilePath -Value ([ordered]@{ + schema = (Get-RunnerSchemaNames).Profile + runner = 'fake' + provider = 'fixture-provider' + model = 'fixture-model' + reasoning_effort = 'high' + configuration_profile = 'isolated-default' + tool_profile = 'default' + timeout_seconds = 30 + concurrency = 1 + }) + $unsupportedProfilePath = Join-Path $iteration 'unsupported-profile.json' + Write-TestJson -Path $unsupportedProfilePath -Value ([ordered]@{ + schema = (Get-RunnerSchemaNames).Profile + runner = 'fake' + provider = 'fixture-provider' + model = 'fixture-model' + reasoning_effort = $null + configuration_profile = 'unsupported' + tool_profile = 'default' + timeout_seconds = 30 + concurrency = 1 + }) + + $with = New-TestRun -IterationDirectory $iteration -Configuration with_skill + $without = New-TestRun -IterationDirectory $iteration -Configuration without_skill + $fakePath = Join-Path $runnerRoot 'fake\runner.ps1' + + $descriptor = Invoke-Fake -FakePath $fakePath -Command describe -Run $with.Path -Profile $profilePath + [void](Assert-RunnerDescriptor -Descriptor $descriptor) + Assert-Equal 'fake' $descriptor.name 'descriptor identity' + Assert-Equal (Get-RunnerSchemaNames).Protocol $descriptor.protocol_version 'descriptor protocol' + Assert-Throws { Assert-RunnerDescriptor -Descriptor ([pscustomobject]@{ schema = $descriptor.schema; protocol_version = 'changed'; name = 'fake' }) } 'changed protocol must fail descriptor validation' + + $unsupported = Invoke-Fake -FakePath $fakePath -Command preflight -Run $with.Path -Profile $unsupportedProfilePath + Assert-Equal 'incompatible' $unsupported.status 'unsupported capability/profile must fail during preflight' + Assert-True (@($unsupported.reasons).Count -gt 0) 'incompatible preflight must explain its reason' + + $withResult = Invoke-Fake -FakePath $fakePath -Command execute -Run $with.Path -Profile $profilePath + $withoutResult = Invoke-Fake -FakePath $fakePath -Command execute -Run $without.Path -Profile $profilePath + foreach ($result in @($withResult, $withoutResult)) { + [void](Assert-ExecutionResult -Result $result) + Assert-Equal 'completed' $result.status 'normal completion status' + Assert-True $result.session.fresh 'fresh session flag' + Assert-True (-not $result.session.resumed) 'resume must be false' + Assert-Equal 1 $result.attempt_count 'answer-quality retry is forbidden' + Assert-Equal 'fixture-provider' $result.requested.provider 'provider must pass unchanged' + Assert-Equal 'fixture-model' $result.requested.model 'model must pass unchanged' + Assert-Equal 'high' $result.requested.reasoning_effort 'reasoning effort must pass unchanged' + Assert-Equal 'isolated-default' $result.requested.configuration_profile 'configuration profile must pass unchanged' + Assert-Equal 'default' $result.requested.tool_profile 'tool profile must pass unchanged' + Assert-Equal 'unavailable' $result.telemetry.tokens.status 'missing token telemetry must be explicit' + Assert-True ($result.telemetry.tokens.PSObject.Properties.Name -notcontains 'value') 'missing token telemetry must not contain a zero placeholder' + foreach ($artifact in @($result.artifacts)) { + Assert-True ($artifact.path -notmatch '(^|/|\\)\.\.(/|\\|$)') 'artifact path must not escape the run' + $artifactPath = Join-Path $($with.Root) ($artifact.path -replace '/', [System.IO.Path]::DirectorySeparatorChar) + if ($result.run.configuration -eq 'without_skill') { $artifactPath = Join-Path $($without.Root) ($artifact.path -replace '/', [System.IO.Path]::DirectorySeparatorChar) } + Assert-True (Test-Path -LiteralPath $artifactPath -PathType Leaf) 'artifact must exist inside its run' + Assert-Equal $artifact.sha256 ((Get-FileHash -Algorithm SHA256 -LiteralPath $artifactPath).Hash.ToLowerInvariant()) 'artifact hash' + Assert-Equal $artifact.size (Get-Item -LiteralPath $artifactPath).Length 'artifact size' + Assert-True (-not [string]::IsNullOrWhiteSpace($artifact.media_type)) 'artifact media type' + } + } + Assert-True ($withResult.session.id -ne $withoutResult.session.id) 'paired arms must have distinct session ids' + Assert-True ($withResult.run.configuration -ne $withoutResult.run.configuration) 'paired arms must retain distinct configurations' + + $withPromptEvidence = Get-Content (Join-Path $with.Root 'evidence\prompt-delivery.json') -Raw | ConvertFrom-Json + $withoutPromptEvidence = Get-Content (Join-Path $without.Root 'evidence\prompt-delivery.json') -Raw | ConvertFrom-Json + foreach ($pair in @( + [pscustomobject]@{ Run = $with; Evidence = $withPromptEvidence; Result = $withResult } + [pscustomobject]@{ Run = $without; Evidence = $withoutPromptEvidence; Result = $withoutResult } + )) { + $promptBytes = [System.IO.File]::ReadAllBytes((Join-Path $pair.Run.Root 'prompt.md')) + Assert-Equal (Get-Sha256HexFromBytes -Bytes $promptBytes) $pair.Evidence.first_task_input_sha256 'prompt must be the first task input byte-for-byte' + Assert-Equal $promptBytes.Length $pair.Evidence.first_task_input_bytes 'prompt byte length' + Assert-True $pair.Evidence.byte_exact 'prompt fidelity evidence' + Assert-Equal (Join-Path $pair.Run.Root 'repo') $pair.Evidence.working_directory 'working directory' + Assert-Equal (Join-Path $pair.Run.Root 'home') $pair.Evidence.home_directory 'isolated home' + Assert-True (-not $pair.Evidence.global_rules_visible -and -not $pair.Evidence.global_memory_visible -and -not $pair.Evidence.global_plugins_visible -and -not $pair.Evidence.global_same_name_skill_visible) 'seeded ambient rules/memory/plugins/skill must remain invisible' + } + Assert-True $withPromptEvidence.candidate_skill_exposed 'candidate skill is exposed only for with_skill' + Assert-True (-not $withoutPromptEvidence.candidate_skill_exposed) 'candidate skill is excluded for without_skill' + Assert-True (Test-Path -LiteralPath (Join-Path $with.Root 'skill\candidate\SKILL.md')) 'with_skill has staged skill' + Assert-True (-not (Test-Path -LiteralPath (Join-Path $without.Root 'skill'))) 'without_skill has no staged skill' + + $escape = Invoke-Fake -FakePath $fakePath -Command execute -Run $with.Path -Profile $profilePath -Scenario escape + $escapeEvidence = Get-Content (Join-Path $with.Root 'evidence\boundary-probes.json') -Raw | ConvertFrom-Json + Assert-True $escapeEvidence.read_outside_run.attempted 'read escape probe was exercised' + Assert-True $escapeEvidence.read_outside_run.blocked 'read escape probe was blocked' + Assert-True $escapeEvidence.write_outside_run.attempted 'write escape probe was exercised' + Assert-True $escapeEvidence.write_outside_run.blocked 'write escape probe was blocked' + Assert-True (-not (Test-Path -LiteralPath (Join-Path $testRoot 'escape-write.txt'))) 'escape write did not create a file' + + $refusal = Invoke-Fake -FakePath $fakePath -Command execute -Run $without.Path -Profile $profilePath -Scenario refusal + Assert-Equal 'completed' $refusal.status 'refusal is a completed captured response' + Assert-True $refusal.final_response.text.Contains('cannot') 'refusal response is retained' + $timeout = Invoke-Fake -FakePath $fakePath -Command execute -Run $without.Path -Profile $profilePath -Scenario timeout + Assert-Equal 'timed_out' $timeout.status 'timeout normalization' + Assert-Equal 'unavailable' $timeout.final_response.status 'timeout has no final response' + Assert-True ($null -eq $timeout.exit.status) 'timeout exit status is unavailable' + $failure = Invoke-Fake -FakePath $fakePath -Command execute -Run $without.Path -Profile $profilePath -Scenario failure + Assert-Equal 'failed' $failure.status 'harness failure normalization' + Assert-Equal 17 $failure.exit.status 'harness failure exit status' + $incompatible = Invoke-Fake -FakePath $fakePath -Command execute -Run $without.Path -Profile $profilePath -Scenario incompatible + Assert-Equal 'incompatible' $incompatible.status 'incompatible normalization' + $unknown = Invoke-Fake -FakePath $fakePath -Command execute -Run $without.Path -Profile $profilePath -Scenario unknown-event + Assert-True (@($unknown.warnings | Where-Object { $_ -match 'future\.event\.v99' }).Count -gt 0) 'unknown events produce explicit warnings' + + foreach ($fixture in @('codex-events.jsonl', 'opencode-events.jsonl')) { + $fixturePath = Join-Path $PSScriptRoot "fixtures\$fixture" + $parsed = ConvertFrom-JsonLines -Text ([System.IO.File]::ReadAllText($fixturePath, [System.Text.UTF8Encoding]::new($false))) + Assert-Equal 0 $parsed.Errors.Count "recorded $fixture has valid JSONL" + Assert-True ($parsed.Events.Count -ge 4) "recorded $fixture has events" + Assert-True (@($parsed.Events | Where-Object { $_.type -eq 'future.event.v99' }).Count -eq 1) "recorded $fixture includes an unknown event" + } + + $prepareText = [System.IO.File]::ReadAllText((Join-Path $repoRoot 'scripts\prepare-skill-evals.ps1'), [System.Text.UTF8Encoding]::new($false)) + $reportText = [System.IO.File]::ReadAllText((Join-Path $repoRoot 'scripts\generate-eval-report.ps1'), [System.Text.UTF8Encoding]::new($false)) + Assert-True ($prepareText -notmatch '(?i)codex\s+exec|opencode\s+run') 'portable preparation must not contain harness-specific CLI invocations' + Assert-True ($reportText -notmatch '(?i)codex\s+exec|opencode\s+run') 'reporting must not contain harness-specific branches' + + $rawPath = Join-Path $iteration 'conformance\results\with-skill.execution-result.json' + $resultPath = Join-Path $iteration 'conformance\results\with-skill.result.json' + New-Item -ItemType Directory -Path (Split-Path -Parent $rawPath) -Force | Out-Null + $bridgeResult = Invoke-Fake -FakePath $fakePath -Command execute -Run $with.Path -Profile $profilePath + Write-TestJson -Path $rawPath -Value $bridgeResult + $bridgePath = Join-Path $runnerRoot 'bridge-execution-result.ps1' + $bridgeOutput = & pwsh -NoProfile -File $bridgePath -Run $with.Path -ExecutionResult $rawPath -Result $resultPath + if ($LASTEXITCODE -ne 0) { throw "execution-result bridge failed: $([string]::Join(' ', @($bridgeOutput)))" } + $portable = Get-Content -LiteralPath $resultPath -Raw | ConvertFrom-Json + Assert-Equal 'codebeltnet/agentic/eval-result/2' $portable.schema 'bridge preserves existing result schema' + Assert-Equal 'completed' $portable.execution_status 'bridge carries execution status' + Assert-Equal 'fixture-model' $portable.model 'bridge carries resolved model' + Assert-Equal 'fixture-provider' $portable.provider 'bridge carries resolved provider' + Assert-True ($null -eq $portable.total_tokens) 'bridge keeps unavailable total tokens unavailable' + Assert-Equal 0 $portable.tool_calls 'bridge carries available tool-call count' + Assert-True $portable.isolation.transcript_captured 'bridge carries transcript availability' + Assert-True (@($portable.output_files).Count -gt 0) 'bridge carries confined evidence paths' + + Write-Output 'Eval Runner conformance: PASS' +} finally { + [Environment]::SetEnvironmentVariable('AGENTIC_FAKE_GLOBAL_RULES', $null, 'Process') + [Environment]::SetEnvironmentVariable('AGENTIC_FAKE_MEMORY', $null, 'Process') + [Environment]::SetEnvironmentVariable('AGENTIC_FAKE_PLUGINS', $null, 'Process') + if (Test-Path -LiteralPath $testRoot) { Remove-Item -LiteralPath $testRoot -Recurse -Force } +} diff --git a/scripts/generate-eval-report.ps1 b/scripts/generate-eval-report.ps1 index 63be59f..5481eb4 100644 --- a/scripts/generate-eval-report.ps1 +++ b/scripts/generate-eval-report.ps1 @@ -412,6 +412,9 @@ function Get-ReportRun { stdout = [string](Get-Property -Object $Result -Name 'stdout' -Default '') stderr = [string](Get-Property -Object $Result -Name 'stderr' -Default '') exit_status = Get-Property -Object $Result -Name 'exit_status' -Default $null + execution_status = Get-Property -Object $Result -Name 'execution_status' -Default $null + execution_run_id = Get-Property -Object $Result -Name 'execution_run_id' -Default $null + execution_result_file = Get-Property -Object $Result -Name 'execution_result_file' -Default $null metrics = $metrics isolation = Get-Property -Object $Result -Name 'isolation' -Default $null grades = @($grades) @@ -469,7 +472,8 @@ function Write-FirstPartyReport { if ($null -ne $result) { $run = Get-ReportRun -Result $result -Configuration $configuration -EvalName ([string]$entry.eval_name) -EvalId ([int]$metadata.eval_id) -Assertions $assertions -RunPackageDirectory (Join-Path $evalDirectory $configuration) -EvalDirectory $evalDirectory -IterationPath $IterationPath $runMap[$configuration] = $run - if (-not [string]::IsNullOrWhiteSpace([string]$run.output) -or @($run.output_files).Count -gt 0) { $completedRuns++ } + $executionStatus = [string](Get-Property -Object $result -Name 'execution_status' -Default '') + if (-not [string]::IsNullOrWhiteSpace([string]$run.output) -or @($run.output_files).Count -gt 0 -or ($executionStatus -and $executionStatus -ne 'unrun')) { $completedRuns++ } if (-not [string]::IsNullOrWhiteSpace([string]$run.model) -and -not $allModels.Contains([string]$run.model)) { $allModels.Add([string]$run.model) } @@ -544,14 +548,19 @@ function Write-UpstreamGrading { $duration = Get-Property -Object $Result -Name 'duration_seconds' -Default $null $tokens = Get-Property -Object $Result -Name 'total_tokens' -Default $null $toolCalls = Get-Property -Object $Result -Name 'tool_calls' -Default $null + $exitStatus = Get-Property -Object $Result -Name 'exit_status' -Default $null + $errorsEncountered = if ($null -eq $exitStatus -or [string]::IsNullOrWhiteSpace([string]$exitStatus)) { $null } elseif ([int]$exitStatus -eq 0) { 0 } else { 1 } - $durationSeconds = if ($null -eq $duration) { 0.0 } else { [double]$duration } - $totalTokens = if ($null -eq $tokens) { 0 } else { [int64]$tokens } - Write-JsonFile -Path (Join-Path $RunDirectory 'timing.json') -Value ([ordered]@{ - total_tokens = $totalTokens - duration_ms = [math]::Round($durationSeconds * 1000, 0) - total_duration_seconds = $durationSeconds - }) + $timing = [ordered]@{} + if ($null -ne $duration -and -not [string]::IsNullOrWhiteSpace([string]$duration)) { + $durationSeconds = [double]$duration + $timing.duration_ms = [math]::Round($durationSeconds * 1000, 0) + $timing.total_duration_seconds = $durationSeconds + } + if ($null -ne $tokens -and -not [string]::IsNullOrWhiteSpace([string]$tokens)) { + $timing.total_tokens = [int64]$tokens + } + Write-JsonFile -Path (Join-Path $RunDirectory 'timing.json') -Value $timing $gradingDocument = [ordered]@{ expectations = @($expectations) @@ -563,7 +572,7 @@ function Write-UpstreamGrading { } execution_metrics = [ordered]@{ total_tool_calls = $toolCalls - errors_encountered = if ([int](Get-Property -Object $Result -Name 'exit_status' -Default 0) -eq 0) { 0 } else { 1 } + errors_encountered = $errorsEncountered } # The upstream aggregator reads timing.json when grading.json does not claim a duration. Keep the # portable run's timing in that sibling file so both elapsed time and token usage survive aggregation. diff --git a/scripts/prepare-skill-evals.ps1 b/scripts/prepare-skill-evals.ps1 index c73787a..13b2951 100644 --- a/scripts/prepare-skill-evals.ps1 +++ b/scripts/prepare-skill-evals.ps1 @@ -42,6 +42,31 @@ .PARAMETER Force Overwrite an existing iteration directory. +.PARAMETER Runner + Optional runner name written to execution-profile.json. If omitted, the profile remains unselected and an external + orchestrator must fail clearly rather than guess a runner. + +.PARAMETER Provider + Optional provider name written to execution-profile.json. + +.PARAMETER Model + Optional model identifier written to execution-profile.json. + +.PARAMETER ReasoningEffort + Optional runner-supported reasoning/effort setting written to execution-profile.json. + +.PARAMETER ConfigurationProfile + Runner configuration profile. Defaults to isolated-default. + +.PARAMETER ToolProfile + Runner tool profile. Defaults to default. + +.PARAMETER TimeoutSeconds + Per-arm runner timeout. Defaults to 900 seconds. + +.PARAMETER Concurrency + Requested external-orchestrator concurrency. Defaults to 1. It does not change paired-arm semantics. + .PARAMETER Changed Prepares a package for every repo-managed skill this branch changed, including uncommitted work. This is the form the eval completion gate uses after adding or modifying a skill. @@ -93,6 +118,40 @@ param( [Parameter(ParameterSetName = 'Changed')] [switch]$Force, + [Parameter(ParameterSetName = 'Prepare')] + [Parameter(ParameterSetName = 'Changed')] + [string]$Runner, + + [Parameter(ParameterSetName = 'Prepare')] + [Parameter(ParameterSetName = 'Changed')] + [string]$Provider, + + [Parameter(ParameterSetName = 'Prepare')] + [Parameter(ParameterSetName = 'Changed')] + [string]$Model, + + [Parameter(ParameterSetName = 'Prepare')] + [Parameter(ParameterSetName = 'Changed')] + [string]$ReasoningEffort, + + [Parameter(ParameterSetName = 'Prepare')] + [Parameter(ParameterSetName = 'Changed')] + [string]$ConfigurationProfile = 'isolated-default', + + [Parameter(ParameterSetName = 'Prepare')] + [Parameter(ParameterSetName = 'Changed')] + [string]$ToolProfile = 'default', + + [Parameter(ParameterSetName = 'Prepare')] + [Parameter(ParameterSetName = 'Changed')] + [ValidateRange(1, 86400)] + [int]$TimeoutSeconds = 900, + + [Parameter(ParameterSetName = 'Prepare')] + [Parameter(ParameterSetName = 'Changed')] + [ValidateRange(1, 128)] + [int]$Concurrency = 1, + [Parameter(ParameterSetName = 'Collect', Mandatory = $true)] [string]$CollectResults ) @@ -112,6 +171,9 @@ $packageSchema = 'codebeltnet/agentic/eval-package/2' $metadataSchema = 'codebeltnet/agentic/eval-metadata/2' $resultSchema = 'codebeltnet/agentic/eval-result/2' $runSchema = 'codebeltnet/agentic/eval-run/1' +$executionProfileSchema = 'codebeltnet/agentic/eval-execution-profile/1' +$executionResultSchema = 'codebeltnet/agentic/eval-execution-result/1' +$runnerProtocolSchema = 'codebeltnet/agentic/eval-runner-protocol/1' $maxFixtureInlineBytes = 32768 # A materialized run is hermetic: the harness treats the run directory as the worker's sandbox root, mounts repo/ as @@ -127,6 +189,7 @@ $runDirectoryNames = [ordered]@{ } $reportToolRelativePath = 'tools/generate-eval-report.ps1' $skillCreatorToolRelativePath = 'tools/skill-creator' +$evalRunnerToolRelativePath = 'tools/eval-runners' $skillCreatorEvalFiles = @( 'LICENSE.txt', 'agents/grader.md', @@ -518,6 +581,9 @@ function New-ResultStub { stdout = '' stderr = '' exit_status = $null + execution_status = 'unrun' + execution_run_id = '' + execution_result_file = '' duration_seconds = $null total_tokens = $null tool_calls = $null @@ -542,6 +608,20 @@ function New-ResultStub { } } +function New-ExecutionProfile { + return [ordered]@{ + schema = $executionProfileSchema + runner = if ([string]::IsNullOrWhiteSpace($Runner)) { $null } else { $Runner } + provider = if ([string]::IsNullOrWhiteSpace($Provider)) { $null } else { $Provider } + model = if ([string]::IsNullOrWhiteSpace($Model)) { $null } else { $Model } + reasoning_effort = if ([string]::IsNullOrWhiteSpace($ReasoningEffort)) { $null } else { $ReasoningEffort } + configuration_profile = $ConfigurationProfile + tool_profile = $ToolProfile + timeout_seconds = $TimeoutSeconds + concurrency = $Concurrency + } +} + function Get-JsonProperty { param( [object]$Object, @@ -879,6 +959,32 @@ function Copy-SkillCreatorEvalTools { return $destinationRoot } +function Copy-EvalRunnerTools { + param( + [string]$RepoRoot, + [string]$IterationDirectory + ) + + $sourceRoot = Join-Path (Join-Path $RepoRoot 'scripts') 'eval-runners' + if (-not (Test-Path -LiteralPath $sourceRoot -PathType Container)) { + throw "Missing Eval Runner protocol source '$sourceRoot'." + } + + $destinationRoot = Join-Path $IterationDirectory $evalRunnerToolRelativePath + $files = Get-ChildItem -LiteralPath $sourceRoot -Recurse -File -Force | + Where-Object { $_.FullName -notmatch '[\\/]tests[\\/]' } | + ForEach-Object { Get-RelativePath -BasePath $sourceRoot -FullPath $_.FullName } | + Sort-Object + foreach ($relative in $files) { + $source = Join-Path $sourceRoot ($relative -replace '/', [System.IO.Path]::DirectorySeparatorChar) + $destination = Join-Path $destinationRoot ($relative -replace '/', [System.IO.Path]::DirectorySeparatorChar) + New-Item -ItemType Directory -Path (Split-Path -Parent $destination) -Force | Out-Null + Copy-Item -LiteralPath $source -Destination $destination -Force + } + + return $destinationRoot +} + # The candidate skill's fingerprint, computed from the source over exactly the files Copy-SkillTree stages. The staged # copy in each with_skill run must reproduce this value, which is how preparation proves the worker received the # revision under development rather than a globally installed one. @@ -1108,6 +1214,8 @@ function Invoke-PrepareMode { [void](Copy-ReportTool -RepoRoot $repoRoot -IterationDirectory $iterationDirectory) $skillCreatorSourcePath = Resolve-SkillCreatorSourcePath -RequestedPath $null [void](Copy-SkillCreatorEvalTools -IterationDirectory $iterationDirectory -SourcePath $skillCreatorSourcePath) + [void](Copy-EvalRunnerTools -RepoRoot $repoRoot -IterationDirectory $iterationDirectory) + ConvertTo-JsonFile -Path (Join-Path $iterationDirectory 'execution-profile.json') -Value (New-ExecutionProfile) $skillText = [System.IO.File]::ReadAllText($skillMarkdownPath, $utf8NoBom) $skillBody = if ($skillText -match '(?ms)\A---\r?\n.*?\r?\n---\r?\n(?.*)\z') { $Matches['body'] } else { $skillText } @@ -1239,12 +1347,14 @@ function Invoke-PrepareMode { prompt_file = "with_skill/$($runDirectoryNames.Prompt)" run_manifest = "with_skill/$($runDirectoryNames.Run)" result_file = 'results/with-skill.result.json' + execution_result_file = 'results/with-skill.execution-result.json' } without_skill = [ordered]@{ run_directory = 'without_skill' prompt_file = "without_skill/$($runDirectoryNames.Prompt)" run_manifest = "without_skill/$($runDirectoryNames.Run)" result_file = 'results/without-skill.result.json' + execution_result_file = 'results/without-skill.execution-result.json' } } assumptions = @($assumptions) @@ -1275,6 +1385,7 @@ function Invoke-PrepareMode { home_directory = "$evalName/with_skill/$($runDirectoryNames.Home)" skill_directory = "$evalName/with_skill/$($runDirectoryNames.Skill)/$Skill" result = "$evalName/results/with-skill.result.json" + execution_result = "$evalName/results/with-skill.execution-result.json" } without_skill = [ordered]@{ mode = 'without_skill' @@ -1285,6 +1396,7 @@ function Invoke-PrepareMode { home_directory = "$evalName/without_skill/$($runDirectoryNames.Home)" skill_directory = $null result = "$evalName/results/without-skill.result.json" + execution_result = "$evalName/results/without-skill.execution-result.json" } } }) @@ -1297,8 +1409,12 @@ function Invoke-PrepareMode { iteration = $iterationNumber generated_utc = $generatedUtc configurations = @('with_skill', 'without_skill') - execution = 'external_handoff' + execution = 'runner_handoff' runner_prompt = 'RUN-THIS.prompt.md' + execution_profile = 'execution-profile.json' + runner_protocol = $runnerProtocolSchema + runner_tools = $evalRunnerToolRelativePath + execution_result_schema = $executionResultSchema report = [ordered]@{ tool = $reportToolRelativePath template = 'tools/eval-report-template.html' @@ -1358,8 +1474,9 @@ function Invoke-PrepareMode { Write-Host 'Point the harness at that path. Do not reproduce its contents in chat: a pasted copy' Write-Host 'loses the absolute paths it depends on, and the harness then cannot find the package.' Write-Host '' - Write-Host 'The runner makes the selected agent the evaluator, grader, and report producer. It must create' - Write-Host 'one isolated fresh worker per run, then grade the collected results and generate both report artifacts.' + Write-Host 'The runner-aware handoff uses execution-profile.json and the package-local Eval Runner protocol.' + Write-Host 'The selected external orchestrator must preflight and execute one runner process per blind arm,' + Write-Host 'then bridge, grade, and report the collected results.' Write-Host '' Write-Host 'This script prepared prompts only. It did not run them, and nothing here will.' Write-Host 'The selected evaluator should finish the package in one run. If it cannot write back to this package,' @@ -1375,122 +1492,53 @@ function New-RunnerPrompt { ) $builder = [System.Text.StringBuilder]::new() - [void]$builder.AppendLine('# Run, grade, and report this evaluation package') - [void]$builder.AppendLine() - [void]$builder.AppendLine('START NOW. You are the evaluator, grader, and report producer for this package. Do not ask me which role to perform, whether to run the workers, or whether to continue to grading. Execute the complete workflow in this one handoff and return the finished report.') - [void]$builder.AppendLine() - [void]$builder.AppendLine('Do not execute evaluation prompts in the current agent context. Create an isolated fresh-context worker for every run, then continue in this coordinator context with grading and report generation after all workers finish.') - [void]$builder.AppendLine() - [void]$builder.AppendLine('Everything you need is in this directory:') - [void]$builder.AppendLine() - [void]$builder.AppendLine(" $IterationDirectory") - [void]$builder.AppendLine() - [void]$builder.AppendLine("This file is `"$(Join-Path $IterationDirectory 'RUN-THIS.prompt.md')`". Use that package path as authoritative. If the package is inaccessible, report the concrete access failure; do not ask the user to choose another workflow.") - [void]$builder.AppendLine() - [void]$builder.AppendLine("It holds $(@($ManifestEvals).Count) eval case(s), each with a `with_skill` and `without_skill` run. Every run is a self-contained directory that holds a `prompt.md`, a `run.json` contract, a `repo/` working tree, an isolated `home/`, and - for `with_skill` only - a `skill/` directory with the candidate. The runs differ only in whether that candidate skill is present. You run the experiment, grade the completed results, and generate the report.") + $profilePath = Join-Path $IterationDirectory 'execution-profile.json' + $resolverPath = Join-Path $IterationDirectory "$evalRunnerToolRelativePath/resolve-runner.ps1" + $bridgePath = Join-Path $IterationDirectory "$evalRunnerToolRelativePath/bridge-execution-result.ps1" + $reportPath = Join-Path $IterationDirectory $reportToolRelativePath + [void]$builder.AppendLine('# Execute, grade, and report this evaluation package') [void]$builder.AppendLine() - [void]$builder.AppendLine('A person selected this harness and handed you this package. That authorizes these runs, grading, aggregation, and report generation. The repository prohibition on automated model evaluation binds repository scripts and the agent that prepared the package; it does not bind the user-directed external executor completing this specific handoff.') + [void]$builder.AppendLine('START NOW. You are the external Eval Orchestrator for this user-directed handoff. Complete execution, deterministic grading, optional judgement, aggregation, and reporting in this run. Do not execute evaluation prompts in the current agent context.') [void]$builder.AppendLine() - [void]$builder.AppendLine('## Execute now') - [void]$builder.AppendLine() - [void]$builder.AppendLine('This package assumes the harness can give each run a fresh context, an isolated HOME/config root, an isolated working directory, and a filesystem sandbox scoped to the run directory. A harness that cannot provide fresh, independent sessions is incompatible with these evals. Continue without asking for role clarification, and report per run which guarantees you satisfied - fresh context, isolated HOME/config, isolated CWD, filesystem sandbox, candidate skill exposure, and transcript capture.') + [void]$builder.AppendLine("Package: $IterationDirectory") + [void]$builder.AppendLine("Profile: $profilePath") + [void]$builder.AppendLine("Runner resolver: $resolverPath") [void]$builder.AppendLine() - [void]$builder.AppendLine('## Orchestration contract') + [void]$builder.AppendLine('This is a runner-aware package. `run.json` is the existing portable one-arm contract: it defines the prompt, staged files, working directory, isolated home, candidate-skill exposure, and required isolation. `execution-profile.json` selects the runner/provider/model/configuration and carries the execution limits. The selected runner defines how its harness satisfies the contract.') [void]$builder.AppendLine() - [void]$builder.AppendLine('1. For every eval case, create one isolated fresh-context worker for `with_skill` and a second isolated fresh-context worker for `without_skill`. Never reuse a worker or session between runs, between cases, or between iterations.') - [void]$builder.AppendLine('2. Launch each worker from its own run directory, which is the worker''s sandbox root. Set the working directory to that run''s `repo/`, set HOME and the platform-equivalent profile and config roots to its `home/`, and confine filesystem access to the run directory. Read the run''s `run.json` for the exact contract: `workingDirectory`, `homeDirectory`, `skillDirectory`, and the fresh-context, filesystem, and home isolation flags.') - [void]$builder.AppendLine('3. Give each worker only its `prompt.md` and the files already staged in its run directory. Do not expose this runner, `manifest.json`, any `eval-metadata.json`, `comparison.md`, result files, grading criteria, expectations, the paired run, another case''s output, or any note that an experiment is underway. All of those live outside the run directory, so keeping the worker inside it keeps them hidden.') - [void]$builder.AppendLine('4. The candidate skill is already inlined in the with_skill run''s `prompt.md` and staged under its `skill/` directory. Do not load, summarize, or add it yourself. The without_skill run carries no skill instructions and no `skill/` directory; do not expose the candidate skill to that worker by any route, including a globally installed copy.') - [void]$builder.AppendLine('5. Send each `prompt.md` unchanged as the worker''s first message. The input files are already real files in the worker''s `repo/`; the worker reads and edits them there rather than from attachments.') - [void]$builder.AppendLine('6. Use the same model, version, configuration, tools, and limits for every worker. Disable persistent memory or cross-session recall. Independent runs may execute concurrently when the selected harness and token budget allow it.') - [void]$builder.AppendLine('7. Record the worker''s complete response, transcript when available, token usage, elapsed time, and tool-call count. When the harness exposes them, also record the shell commands, files read and written, stdout and stderr, and exit status, and which isolation guarantees you satisfied. Record refusals, questions, and failures as results. Do not retry to improve an answer.') - [void]$builder.AppendLine('8. Work only inside this package. Do not read or modify the source repository around it. Do not begin grading until every available worker has completed or failed and its result is recorded.') + [void]$builder.AppendLine('A human selected the external orchestrator and authorized this handoff. Repository preparation, validation, CI, hooks, and automatic completion gates remain model-free. Do not substitute a generic worker, another runner, or an improvised isolation scheme if the selected runner is unavailable or incompatible.') [void]$builder.AppendLine() - [void]$builder.AppendLine('For each case in `manifest.json`, the `runs.with_skill` and `runs.without_skill` entries give each run''s directory, its `prompt`, its `run_manifest` (`run.json`), and the `result` file to write. Run the two prompts in separate workers, then overwrite the matching result file without reading its existing contents. A partial package is valid: record every completed run, continue to grading/reporting, and mark missing arms honestly instead of asking what to do next.') + [void]$builder.AppendLine('## Phase 1: execute blind arms') [void]$builder.AppendLine() - [void]$builder.AppendLine('## Result shape') + [void]$builder.AppendLine('1. Read `manifest.json` and `execution-profile.json`. If `runner` is null, unavailable, or unsupported, fail clearly; do not guess a default. The profile contains no credentials.') + [void]$builder.AppendLine('2. Resolve the selected package-local runner with the resolver. Ask it for `describe` and validate its protocol, descriptor, and capability declarations before running an arm. Do not invent harness-specific CLI commands.') + [void]$builder.AppendLine('3. For every eval case, use the exact `run_manifest` path from `manifest.json` and the same profile path. Preflight each arm, then invoke the runner exactly once with `execute`. The runner receives only `run.json` and `execution-profile.json`; it must never receive or inspect expected output, assertions, grading, paired output, benchmark data, or human feedback.') + [void]$builder.AppendLine('4. Keep `with_skill` and `without_skill` in fresh independent processes/sessions. Use the same model, provider, configuration, tools, and limits. The runner must send each `prompt.md` unchanged as the first task input and must enforce the run contract, including the baseline skill exclusion and filesystem boundary.') + [void]$builder.AppendLine('5. Save the runner''s single normalized JSON response unchanged as the matching `execution_result` path. Preserve the complete final response, status, telemetry, evidence references, hashes, isolation mechanisms, warnings, and compatibility deviations. Do not retry for answer quality. A refusal is a result; timeout, harness failure, and incompatibility are results.') + [void]$builder.AppendLine('6. If the runner cannot satisfy a required guarantee, keep the normalized status `incompatible` and stop that arm. Never fall back to the old generic isolated-worker behavior and never substitute a different runner.') [void]$builder.AppendLine() - [void]$builder.AppendLine('```json') - [void]$builder.AppendLine('{') - [void]$builder.AppendLine(' "schema": "codebeltnet/agentic/eval-result/2",') - [void]$builder.AppendLine(" `"iteration`": $IterationNumber,") - [void]$builder.AppendLine(' "eval_id": 1,') - [void]$builder.AppendLine(' "eval_name": "the directory name",') - [void]$builder.AppendLine(' "configuration": "with_skill",') - [void]$builder.AppendLine(' "model": "the exact model id you used",') - [void]$builder.AppendLine(' "provider": "who served it",') - [void]$builder.AppendLine(' "harness": "what you are",') - [void]$builder.AppendLine(' "executed_utc": "2026-01-01T00:00:00Z",') - [void]$builder.AppendLine(' "output": "the complete response the run produced",') - [void]$builder.AppendLine(' "output_files": ["paths of any files the run wrote"],') - [void]$builder.AppendLine(' "transcript": "the complete worker transcript when the harness exposes it",') - [void]$builder.AppendLine(' "shell_commands": ["commands the run executed, when exposed"],') - [void]$builder.AppendLine(' "files_read": ["paths the run read, when exposed"],') - [void]$builder.AppendLine(' "files_written": ["paths the run wrote, when exposed"],') - [void]$builder.AppendLine(' "exit_status": 0,') - [void]$builder.AppendLine(' "duration_seconds": 12.5,') - [void]$builder.AppendLine(' "total_tokens": 1234,') - [void]$builder.AppendLine(' "tool_calls": 6,') - [void]$builder.AppendLine(' "turns": 12,') - [void]$builder.AppendLine(' "base_input_tokens": 27,') - [void]$builder.AppendLine(' "output_tokens": 3800,') - [void]$builder.AppendLine(' "cache_read_tokens": 515605,') - [void]$builder.AppendLine(' "cache_write_1h_tokens": 129582,') - [void]$builder.AppendLine(' "estimated_cost_usd": 2.27,') - [void]$builder.AppendLine(' "model_effort": "high",') - [void]$builder.AppendLine(' "isolation": {') - [void]$builder.AppendLine(' "fresh_context": true,') - [void]$builder.AppendLine(' "isolated_home": true,') - [void]$builder.AppendLine(' "isolated_cwd": true,') - [void]$builder.AppendLine(' "filesystem_sandbox": true,') - [void]$builder.AppendLine(' "candidate_skill_exposed": true,') - [void]$builder.AppendLine(' "transcript_captured": true') - [void]$builder.AppendLine(' },') - [void]$builder.AppendLine(' "grading": [],') - [void]$builder.AppendLine(' "notes": "anything that would change how this result reads"') - [void]$builder.AppendLine('}') + [void]$builder.AppendLine('The package-local process surface is:') + [void]$builder.AppendLine('```text') + [void]$builder.AppendLine("pwsh -NoProfile -File `"$resolverPath`" ") + [void]$builder.AppendLine('runner.ps1 describe') + [void]$builder.AppendLine("runner.ps1 preflight -Run `"`" -Profile `"$profilePath`"") + [void]$builder.AppendLine("runner.ps1 execute -Run `"`" -Profile `"$profilePath`"") [void]$builder.AppendLine('```') + [void]$builder.AppendLine('Use the resolver output to locate `runner.ps1`; `` is data from the profile, not a branch in this orchestration contract. The runner command must be invoked once per arm and its stdout must remain one JSON execution result.') [void]$builder.AppendLine() - [void]$builder.AppendLine('`transcript`, `shell_commands`, `files_read`, `files_written`, `exit_status`, `duration_seconds`, `total_tokens`, `tool_calls`, the optional efficiency telemetry fields, and every `isolation` flag are optional. Include each when the harness exposes it and omit it otherwise. Never estimate a missing value. For `with_skill`, set `isolation.candidate_skill_exposed` to how the skill actually reached the worker.') - [void]$builder.AppendLine() - [void]$builder.AppendLine('`configuration` is `with_skill` or `without_skill` and must match the prompt you ran. Read `eval_id` and `eval_name` from `manifest.json`; do not send them to the worker. Put the full model response in `output`. If it is very long, write it beside the result file and list that path in `output_files` with a summary in `output`.') - [void]$builder.AppendLine() - [void]$builder.AppendLine('`output` is the model''s message in full, including questions, caveats, explanations, or a refusal. Tool output is evidence from the run, not a replacement for the model response. Put the full worker event history in `transcript` when the harness exposes it.') + [void]$builder.AppendLine('Do not read any `eval-metadata.json`, expected output, assertions, result grading, or paired output during Phase 1. Those files remain outside every run directory and are the grading key.') [void]$builder.AppendLine() - [void]$builder.AppendLine('## Grade and report immediately') + [void]$builder.AppendLine('## Phase 2: bridge, grade, and report') [void]$builder.AppendLine() - [void]$builder.AppendLine('After all available workers finish, read each eval''s `eval-metadata.json`. Only now may you read `expected_output` and `assertions`; they are the grading key and were intentionally hidden from the workers.') - [void]$builder.AppendLine('1. Grade every completed result against every assertion. Use deterministic checks for mechanical assertions and concrete output, transcript, and file evidence for process assertions. Use judgement only where the assertion is genuinely qualitative, and say so in the evidence. Never infer a tool or file action from the model''s self-report when process evidence is absent.') - [void]$builder.AppendLine('2. Write grading back into the matching result file using exactly `grading[].text`, `grading[].passed`, and `grading[].evidence`. Use `passed: null` when an assertion cannot be judged from captured evidence. Do not grade a missing run as passed.') - [void]$builder.AppendLine('The package carries Anthropic skill-creator under `tools/skill-creator`. Use `tools/skill-creator/agents/grader.md` for grading guidance, and use `tools/skill-creator/scripts/aggregate_benchmark.py` plus `tools/skill-creator/eval-viewer/generate_review.py` as the source-of-truth aggregation and review tools.') - $reportCommand = 'pwsh -NoProfile -File "' + (Join-Path $IterationDirectory $reportToolRelativePath) + '" -IterationDirectory "' + $IterationDirectory + '"' - [void]$builder.AppendLine(('3. Run the package report adapter now; do not ask the user to run a second command: ' + $reportCommand + '. It stages the recorded results into the upstream skill-creator workspace contract, invokes the exact upstream aggregator and static viewer, then writes the first-party side-by-side `report.html`, the exact upstream `skill-creator-report.html`, `benchmark.json`, and `benchmark.md` at the package root.')) - [void]$builder.AppendLine('4. If the harness can open local files, open `report.html` after it is written. Otherwise return its absolute path as the primary artifact. Do not wait for browser feedback before finishing the handoff.') + [void]$builder.AppendLine('1. After every available arm has completed or failed, validate each `execution-result.json` and run the package bridge. The bridge checks prompt/run/profile hashes and artifact confinement, then writes the existing `eval-result/2` file while preserving unavailable telemetry as null/unavailable.') + [void]$builder.AppendLine((' `pwsh -NoProfile -File "' + $bridgePath + '" -Run "" -ExecutionResult "" -Result ""`')) + [void]$builder.AppendLine('2. Only now read each eval''s `eval-metadata.json` and reveal `expected_output` and `assertions` to the Grader. Follow `tools/skill-creator/agents/grader.md`; grade deterministically first, then use optional model judgement only where deterministic evidence cannot decide. Never infer tool or file behavior from model self-report without process evidence.') + [void]$builder.AppendLine('3. Write only `grading[].text`, `grading[].passed`, and `grading[].evidence` for grading. Do not alter raw execution results. Use null for genuinely unavailable judgement and leave missing arms visibly missing.') + [void]$builder.AppendLine(('4. Run the existing package report adapter now: `pwsh -NoProfile -File "' + $reportPath + '" -IterationDirectory "' + $IterationDirectory + '"`. It remains the bridge to Anthropic skill-creator''s grader-compatible aggregator and viewer; do not replace it with harness-specific reporting. The packaged compatibility tools remain `scripts/aggregate_benchmark.py` and `eval-viewer/generate_review.py`.')) [void]$builder.AppendLine() - [void]$builder.AppendLine('The report is the completion artifact. Do not stop after worker execution, do not return a prose-only recap, and do not ask whether grading or HTML generation is wanted.') + [void]$builder.AppendLine('The completion artifacts are `report.html`, `skill-creator-report.html`, `benchmark.json`, and `benchmark.md` at the package root. Return their absolute paths, completed and missing arm counts, runner/provider/model identity, and a concise evidence-backed summary. If the package cannot be written from the external environment, return one paste-ready block containing the completed result objects and report artifacts.') [void]$builder.AppendLine() - [void]$builder.AppendLine('## Final handoff') - [void]$builder.AppendLine() - [void]$builder.AppendLine('The finished artifacts are the first-party paired review and the exact upstream skill-creator viewer report, not a request for another command. If you can write to the package machine, leave every result, grading field, `benchmark.json`, `benchmark.md`, `report.html`, and `skill-creator-report.html` in place. Return the absolute first-party report path, the completed/expected run count, any missing arms, the model/provider, and a concise quality summary.') - [void]$builder.AppendLine() - [void]$builder.AppendLine('If you cannot write to the package machine, return one fenced JSON block containing every completed result object, including its `grading` array, plus the generated report as an artifact when the harness supports file handoff. Do not return separate blocks or a human summary in place of the result objects. State any missing arms and the concrete artifact-transfer limitation.') - [void]$builder.AppendLine() - [void]$builder.AppendLine('If you cannot write to that machine - a different product, a browser, a sandbox that shares no disk with it - the results have to travel as text. End with one fenced block, and say plainly that it is meant to be pasted into the repository session as-is:') - [void]$builder.AppendLine() - [void]$builder.AppendLine('```') - [void]$builder.AppendLine('Eval results, grading, and report artifact.') - [void]$builder.AppendLine("Package: $IterationDirectory") - [void]$builder.AppendLine('Model: via , harness ') - [void]$builder.AppendLine() - [void]$builder.AppendLine('') - [void]$builder.AppendLine() - [void]$builder.AppendLine('Still unfilled: ') - [void]$builder.AppendLine('```') - [void]$builder.AppendLine() - [void]$builder.AppendLine('One block covering everything you ran, not one per case, and the outputs go in it verbatim - a summary written for a human to skim cannot be graded against assertions.') - [void]$builder.AppendLine() - [void]$builder.AppendLine('The repository collector is only a fallback when result files were transferred without the report artifacts: `pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -CollectResults `. It validates the returned files and invokes the same packaged skill-creator aggregator and viewer; it is not the normal next step after this prompt.') - + [void]$builder.AppendLine("This package contains $(@($ManifestEvals).Count) eval case(s), each with paired `with_skill` and `without_skill` runs. The human reviewer remains the final evaluator.") return $builder.ToString() } @@ -1505,7 +1553,7 @@ function New-PackageReadme { $builder = [System.Text.StringBuilder]::new() [void]$builder.AppendLine("# Eval package: $SkillName (iteration $IterationNumber)") [void]$builder.AppendLine() - [void]$builder.AppendLine('Prepared by `scripts/prepare-skill-evals.ps1` in `codebeltnet/agentic`. Nothing in this package was executed. You choose the harness, provider, and model; the selected external evaluator runs both configurations, grades them, and generates the report.') + [void]$builder.AppendLine('Prepared by `scripts/prepare-skill-evals.ps1` in `codebeltnet/agentic`. Nothing in this package was executed. `execution-profile.json` selects the user-chosen Eval Runner, provider, model, and limits; the external Eval Orchestrator runs both configurations, grades them, and generates the report.') [void]$builder.AppendLine() [void]$builder.AppendLine('## What is here') [void]$builder.AppendLine() @@ -1514,6 +1562,7 @@ function New-PackageReadme { } [void]$builder.AppendLine() [void]$builder.AppendLine('Each eval directory holds the grading key (`eval-metadata.json`), result stubs under `results/`, and two hermetic run directories: `with_skill/` and `without_skill/`. A run directory holds `prompt.md`, a `run.json` contract, a `repo/` working tree materialized from the fixtures, an isolated `home/`, and - for `with_skill` only - a `skill/` directory with the candidate skill. The grading key and results sit outside both run directories, so a worker confined to its run directory never sees them.') + [void]$builder.AppendLine('The package root also holds `execution-profile.json`, the package-local Eval Runner protocol under `tools/eval-runners/`, and raw `execution-result.json` paths beside the existing result stubs. `run.json` defines what one blind arm must execute; the profile defines with what runner/model/configuration; the selected runner defines how.') [void]$builder.AppendLine('The package also carries the exact Anthropic skill-creator assets used after execution under `tools/skill-creator`: `tools/skill-creator/agents/grader.md`, `tools/skill-creator/agents/comparator.md`, `tools/skill-creator/agents/analyzer.md`, `tools/skill-creator/references/schemas.md`, `tools/skill-creator/scripts/aggregate_benchmark.py`, and `tools/skill-creator/eval-viewer/generate_review.py` plus `tools/skill-creator/eval-viewer/viewer.html`.') [void]$builder.AppendLine() [void]$builder.AppendLine('## Isolation model') @@ -1524,12 +1573,11 @@ function New-PackageReadme { [void]$builder.AppendLine() [void]$builder.AppendLine('## How to run') [void]$builder.AppendLine() - [void]$builder.AppendLine('1. Pick one model and configuration. Use the same one for every run in this iteration.') - [void]$builder.AppendLine('2. For each eval, launch a fresh worker for `with_skill/` with its run directory as the sandbox root, `repo/` as the working directory, and `home/` as HOME. Send `prompt.md` as the first message. Read `run.json` for the contract.') - [void]$builder.AppendLine('3. Launch a second fresh worker for `without_skill/` the same way. Never reuse a worker between runs.') - [void]$builder.AppendLine('4. Save each response into the matching file under the eval''s `results/` directory, grade every completed result using the packaged `agents/grader.md` guidance, and run `tools/generate-eval-report.ps1`. The resulting `report.html` is the first-party side-by-side review; `skill-creator-report.html` is the exact upstream viewer.') + [void]$builder.AppendLine('1. Read `execution-profile.json`. If no runner is selected, fail clearly instead of guessing. Resolve the selected package-local runner and run `describe`, then `preflight`, for every arm.') + [void]$builder.AppendLine('2. Invoke `execute` exactly once for each `run.json`, preserving its one JSON `execution-result.json` unchanged. The runner must provide a fresh process/session, isolated home/config, isolated CWD, filesystem confinement, baseline skill exclusion, prompt fidelity, model/configuration lock, and complete response capture or return `incompatible`.') + [void]$builder.AppendLine('3. After all arms complete or fail, run `tools/eval-runners/bridge-execution-result.ps1` for each raw result. It writes the existing `eval-result/2` file and preserves explicit unavailable telemetry. Only then read the grading key, grade with `tools/skill-creator/agents/grader.md`, and run `tools/generate-eval-report.ps1`.') [void]$builder.AppendLine() - [void]$builder.AppendLine('`RUN-THIS.prompt.md` turns a harness that can create isolated workers or sessions into the evaluator, grader, and report producer. It reads the package, creates one new worker per run from its run directory, keeps runner instructions and grading data out of every worker, records results, grades after collection, and invokes Anthropic skill-creator''s aggregator and static viewer through the package adapter. It never executes an eval prompt in its own context.') + [void]$builder.AppendLine('`RUN-THIS.prompt.md` is the external Eval Orchestrator handoff. It selects the package-local runner from the profile, invokes the common protocol once per blind arm, bridges raw evidence into the existing result shape, reveals grading material only after execution, and invokes Anthropic skill-creator''s compatible aggregator and static viewer through the package adapter. It never executes an eval prompt in its own context.') [void]$builder.AppendLine() [void]$builder.AppendLine('A harness that cannot provide fresh, independent sessions with isolated working and config roots is incompatible with these evals. `-CollectResults` accepts a partial iteration and reports unfilled runs as missing.') [void]$builder.AppendLine() @@ -1686,10 +1734,23 @@ function Invoke-CollectMode { $evalDirectory = Join-Path $iterationDirectory $entry.directory $metadata = [System.IO.File]::ReadAllText((Join-Path $evalDirectory 'eval-metadata.json'), $utf8NoBom) | ConvertFrom-Json $observed = @{} + $runnerAware = $manifest.PSObject.Properties.Name -contains 'execution_profile' + $bridgePath = if ($runnerAware) { Join-Path $iterationDirectory ($manifest.runner_tools + '/bridge-execution-result.ps1') } else { $null } foreach ($configuration in @('with_skill', 'without_skill')) { $fileName = if ($configuration -eq 'with_skill') { 'with-skill.result.json' } else { 'without-skill.result.json' } $resultPath = Join-Path (Join-Path $evalDirectory 'results') $fileName + $runEntry = Get-JsonProperty -Object $entry.runs -Name $configuration -Default $null + $rawRelative = Get-JsonProperty -Object $runEntry -Name 'execution_result' -Default $null + $rawPath = if ([string]::IsNullOrWhiteSpace([string]$rawRelative)) { $null } else { Join-Path $iterationDirectory $rawRelative } + $runPath = Join-Path $iterationDirectory (Get-JsonProperty -Object $runEntry -Name 'run_manifest' -Default '') + if ($runnerAware -and $null -ne $rawPath -and (Test-Path -LiteralPath $rawPath -PathType Leaf)) { + $bridgeOutput = & pwsh -NoProfile -File $bridgePath -Run $runPath -ExecutionResult $rawPath -Result $resultPath 2>&1 + if ($LASTEXITCODE -ne 0) { + $errors.Add("$($entry.eval_name)/$configuration - execution-result bridge failed: $([string]::Join(' ', @($bridgeOutput)))") + continue + } + } if (-not (Test-Path -LiteralPath $resultPath)) { $warnings.Add("$($entry.eval_name)/$configuration - no result file at results/$fileName.") continue @@ -1713,7 +1774,9 @@ function Invoke-CollectMode { $outputText = [string](Get-JsonProperty -Object $result -Name 'output' -Default '') $outputFiles = @(Get-JsonProperty -Object $result -Name 'output_files' -Default @()) - $hasOutput = -not [string]::IsNullOrWhiteSpace($outputText) -or $outputFiles.Count -gt 0 + $executionStatus = [string](Get-JsonProperty -Object $result -Name 'execution_status' -Default '') + $hasExecution = -not [string]::IsNullOrWhiteSpace($executionStatus) -and $executionStatus -ne 'unrun' + $hasOutput = -not [string]::IsNullOrWhiteSpace($outputText) -or $outputFiles.Count -gt 0 -or $hasExecution if (-not $hasOutput) { $warnings.Add("$($entry.eval_name)/$configuration - not run yet (empty output and no output_files).") continue @@ -1755,6 +1818,7 @@ function Invoke-CollectMode { TranscriptRecorded = -not [string]::IsNullOrWhiteSpace($transcriptText) ProcessEvidence = $hasProcessEvidence IsolationReport = $isolationReport + ExecutionStatus = $executionStatus DurationSeconds = Get-JsonProperty -Object $result -Name 'duration_seconds' TotalTokens = Get-JsonProperty -Object $result -Name 'total_tokens' ToolCalls = Get-JsonProperty -Object $result -Name 'tool_calls' diff --git a/scripts/validate-skill-templates.ps1 b/scripts/validate-skill-templates.ps1 index f66755c..e7f176f 100644 --- a/scripts/validate-skill-templates.ps1 +++ b/scripts/validate-skill-templates.ps1 @@ -1191,6 +1191,7 @@ Add-ValidationResult -Results $results -Name 'Repository automation cannot launc $automationPaths = @($automationPaths | Where-Object { $normalized = $_ -replace '\\', '/' ($normalized.StartsWith('scripts/') -or $normalized.StartsWith('.github/')) -and + -not $normalized.StartsWith('scripts/eval-runners/') -and $normalized -ne 'scripts/validate-skill-templates.ps1' -and $automationExtensions -contains [System.IO.Path]::GetExtension($normalized).ToLowerInvariant() } | Sort-Object -Unique) @@ -1231,10 +1232,29 @@ Add-ValidationResult -Results $results -Name 'Repository automation cannot launc Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle 'this repository does not provide an opt-in path around that rule.' Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle 'Model-backed comparisons are not a repository completion gate.' Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle 'This rule is Priority 1.' + Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle 'A human-selected external Eval Orchestrator may invoke an explicitly selected package-local Eval Runner' Assert-Contains -Name 'README.md' -Content $readme -Needle 'There is no repository opt-in switch.' + Assert-Contains -Name 'README.md' -Content $readme -Needle 'Eval Runner' Assert-Contains -Name 'README.md' -Content $readme -Needle 'validate-skill-templates.ps1 -MetadataOnly' } +Add-ValidationResult -Results $results -Name 'Eval Runner protocol conformance remains deterministic' -Action { + if (-not [string]::IsNullOrWhiteSpace($Ref)) { + return + } + $conformancePath = Join-Path $repoRoot 'scripts/eval-runners/tests/test-runner-conformance.ps1' + if (-not (Test-Path -LiteralPath $conformancePath -PathType Leaf)) { + throw 'The Eval Runner conformance suite is missing.' + } + $conformanceOutput = & pwsh -NoProfile -File $conformancePath 2>&1 + if ($LASTEXITCODE -ne 0) { + throw "Eval Runner conformance failed: $($conformanceOutput -join [Environment]::NewLine)" + } + if (@($conformanceOutput -join [Environment]::NewLine) -notmatch 'Eval Runner conformance:\s+PASS') { + throw 'Eval Runner conformance did not report PASS.' + } +} + Add-ValidationResult -Results $results -Name 'Skill evaluation prepares portable prompts instead of executing them' -Action { $agents = Get-FileText -RepoRoot $repoRoot -RelativePath 'AGENTS.md' -GitRef $Ref $readme = Get-FileText -RepoRoot $repoRoot -RelativePath 'README.md' -GitRef $Ref @@ -1333,17 +1353,15 @@ Add-ValidationResult -Results $results -Name 'Skill evaluation prepares portable } $runner = [System.IO.File]::ReadAllText($runnerPath, $utf8NoBom) foreach ($needle in @( - 'START NOW. You are the evaluator, grader, and report producer', + 'START NOW. You are the external Eval Orchestrator', 'Do not execute evaluation prompts in the current agent context.', - 'create one isolated fresh-context worker for `with_skill` and a second isolated fresh-context worker for `without_skill`', - 'Never reuse a worker or session between runs', - 'Do not expose this runner', - 'The candidate skill is already inlined in the with_skill run', - 'Launch each worker from its own run directory', - 'Use the same model, version, configuration, tools, and limits for every worker.', - 'Record the worker''s complete response, transcript when available, token usage, elapsed time, and tool-call count.', - 'Do not begin grading until every available worker has completed or failed', - '## Grade and report immediately', + 'execution-profile.json` selects the runner/provider/model/configuration', + 'invoke the runner exactly once with `execute`', + 'never receive or inspect expected output, assertions, grading, paired output, benchmark data, or human feedback', + 'Never fall back to the old generic isolated-worker behavior', + 'execution-result.json', + 'The bridge checks prompt/run/profile hashes and artifact confinement', + 'Only now read each eval''s `eval-metadata.json`', 'grading[].text', 'tools/skill-creator/agents/grader.md', 'scripts/aggregate_benchmark.py', @@ -1381,6 +1399,9 @@ Add-ValidationResult -Results $results -Name 'Skill evaluation prepares portable throw "$($entry.eval_name)/$configuration manifest entry must declare '$pathProperty'." } } + if ($run.PSObject.Properties.Name -notcontains 'execution_result') { + throw "$($entry.eval_name)/$configuration manifest entry must declare 'execution_result'." + } foreach ($mustExist in @($run.prompt, $run.run_manifest, $run.working_directory, $run.home_directory)) { if (-not (Test-Path -LiteralPath (Join-Path $iterationDirectory $mustExist))) { throw "$($entry.eval_name)/$configuration manifest path '$mustExist' does not exist." @@ -1509,6 +1530,30 @@ Add-ValidationResult -Results $results -Name 'Skill evaluation prepares portable } } + if ([string]$manifest.execution -ne 'runner_handoff' -or + [string]$manifest.execution_profile -ne 'execution-profile.json' -or + [string]$manifest.runner_protocol -ne 'codebeltnet/agentic/eval-runner-protocol/1' -or + [string]$manifest.runner_tools -ne 'tools/eval-runners' -or + [string]$manifest.execution_result_schema -ne 'codebeltnet/agentic/eval-execution-result/1') { + throw 'Runner-aware packages must declare the execution profile, runner protocol, runner tools, and execution-result schema.' + } + $profilePath = Join-Path $iterationDirectory ([string]$manifest.execution_profile) + $profile = [System.IO.File]::ReadAllText($profilePath, $utf8NoBom) | ConvertFrom-Json + foreach ($profileField in @('schema', 'runner', 'provider', 'model', 'reasoning_effort', 'configuration_profile', 'tool_profile', 'timeout_seconds', 'concurrency')) { + if ($profile.PSObject.Properties.Name -notcontains $profileField) { + throw "execution-profile.json must declare '$profileField'." + } + } + if ([string]$profile.schema -ne 'codebeltnet/agentic/eval-execution-profile/1' -or + [int]$profile.timeout_seconds -lt 1 -or [int]$profile.concurrency -lt 1) { + throw 'execution-profile.json has an invalid schema or execution limit.' + } + foreach ($runnerTool in @('runner-common.ps1', 'resolve-runner.ps1', 'bridge-execution-result.ps1', 'fake/runner.ps1', 'codex/runner.ps1', 'opencode/runner.ps1', 'contracts/execution-profile.schema.json', 'contracts/execution-result.schema.json')) { + if (-not (Test-Path -LiteralPath (Join-Path $iterationDirectory "tools/eval-runners/$runnerTool") -PathType Leaf)) { + throw "Prepared package is missing runner tool '$runnerTool'." + } + } + $collectOutput = & pwsh -NoProfile -File $scriptPath -CollectResults $iterationDirectory 2>&1 if ($LASTEXITCODE -ne 0) { throw "prepare-skill-evals.ps1 -CollectResults failed on an unrun package: $($collectOutput -join [Environment]::NewLine)" From 9e416d29f19972c5c9a255a05bb0b7a260093638 Mon Sep 17 00:00:00 2001 From: "aicia[bot]" Date: Fri, 21 Aug 2026 19:45:35 +0200 Subject: [PATCH 03/39] =?UTF-8?q?=F0=9F=92=AC=20update=20installation=20an?= =?UTF-8?q?d=20runner=20guidance?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Update CONTRIBUTING.md and README.md to document runner infrastructure, installation procedures, and eval orchestrator workflow. --- CONTRIBUTING.md | 4 ++-- README.md | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index fbfd760..e075545 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -98,7 +98,7 @@ Run it after the last skill edit and before `scripts/sync-skill-install.ps1`, wh pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -Skill ``` -The script writes `.bot/-workspace/iteration-/` with one directory per eval. Each holds the grading key `eval-metadata.json` and result stubs under `results/` at the eval-case level, plus two hermetic run directories, `with_skill/` and `without_skill/`. A run directory is the worker's sandbox root: `prompt.md`, a `run.json` contract, a `repo/` working tree materialized from the fixtures, an isolated `home/`, and - for `with_skill` only - a `skill//` copy of the candidate. The grading key and results sit outside both run directories. At the root it writes `manifest.json`, `execution-profile.json`, the package-local Eval Runner protocol, the package report adapter, the exact Anthropic skill-creator grader/aggregator/viewer assets, and `RUN-THIS.prompt.md`, the one prompt you hand to the external Eval Orchestrator. That orchestrator resolves and preflights the selected runner, invokes its common `describe`/`preflight`/`execute` surface once per fresh blind arm, stores raw `execution-result.json`, bridges the results, grades only after execution, and runs the adapter, which invokes `aggregate_benchmark.py` and `eval-viewer/generate_review.py --static`. It never runs an eval prompt in the coordinator context and never reuses a worker. Both worker prompts carry the same task, materialized repository, and response contract; only the operating instructions and the presence of `skill/` differ, and neither prompt identifies itself as an eval. `.gitignore` covers `.bot/*`, so nothing there reaches git. The script refuses an `-OutputRoot` inside the repository but outside `.bot/`; pass an explicit temp path when the harness does not need repository-local storage. +The script writes `.bot/-workspace/iteration-/` with one directory per eval. Each holds the grading key `eval-metadata.json` and result stubs under `results/` at the eval-case level, plus two paired run directories, `with_skill/` and `without_skill/`. A run directory is the worker's run root: `prompt.md`, a `run.json` contract, a `repo/` working tree materialized from the fixtures, an isolated `home/`, and - for `with_skill` only - a `skill//` copy of the candidate. The grading key and results sit outside both run directories. At the root it writes `manifest.json`, `execution-profile.json`, the package-local Eval Runner protocol, the package report adapter, the exact Anthropic skill-creator grader/aggregator/viewer assets, and `RUN-THIS.prompt.md`, the one prompt you hand to the external Eval Orchestrator. That orchestrator resolves and preflights the selected runner, invokes its common `describe`/`preflight`/`execute` surface once per fresh blind arm, stores raw `execution-result.json`, bridges the results, grades only after execution, and runs the adapter, which invokes `aggregate_benchmark.py` and `eval-viewer/generate_review.py --static`. It never runs an eval prompt in the coordinator context and never reuses a worker. Both worker prompts carry the same task, materialized repository, and response contract; only the operating instructions and the presence of `skill/` differ, and neither prompt identifies itself as an eval. `.gitignore` covers `.bot/*`, so nothing there reaches git. The script refuses an `-OutputRoot` inside the repository but outside `.bot/`; pass an explicit temp path when the harness does not need repository-local storage. Repository preparation, validation, CI, hooks, deterministic tests, and automatic completion gates never run those prompts or invoke a model. That boundary is the Priority 1 rule in `AGENTS.md`, and preparing a prompt is not permission to execute one. A human-selected external Eval Orchestrator handed a specific package may invoke the selected package-local Eval Runner; this explicit handoff boundary does not weaken the repository prohibition or authorize CI/live evals. @@ -110,7 +110,7 @@ Record each external runner result in the matching `execution-result.json` first pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -CollectResults ``` -That writes `comparison.md`, the first-party side-by-side `report.html`, the exact upstream `skill-creator-report.html`, and the upstream `benchmark.json`/`benchmark.md`, while flagging missing arms, unrun configurations, and mixed models. The normal external Eval Orchestrator grades in the same handoff using deterministic checks for mechanical assertions and evidence-backed judgement where an assertion is genuinely qualitative. Repository automation remains deterministic and never invokes a model. Initial real runners are Codex and OpenCode; the deterministic fake runner is the CI conformance harness. Native skill activation, portability scoring, and additional runners are not part of v0.9.1. +That writes `comparison.md`, the first-party side-by-side `report.html`, the exact upstream `skill-creator-report.html`, and the upstream `benchmark.json`/`benchmark.md`, while flagging missing arms, unrun configurations, and mixed models. The normal external Eval Orchestrator grades in the same handoff using deterministic checks for mechanical assertions and evidence-backed judgement where an assertion is genuinely qualitative. Repository automation remains deterministic and never invokes a model. Codex, OpenCode, and Cline are the conforming real runners; the deterministic fake runner is the CI conformance harness. Hard filesystem confinement is reported as strict versus pragmatic confidence and is not a universal Windows prerequisite. Freebuff remains planned/blocked until its official CLI provides a supported noninteractive machine-readable transport. Native skill activation, portability scoring, and additional runners are not part of v0.9.1. The eval package is a temp artifact. Do not commit it, its prompts, or its results unless the change explicitly calls for checked-in examples. diff --git a/README.md b/README.md index 1f17d69..2254e0f 100644 --- a/README.md +++ b/README.md @@ -14,7 +14,7 @@ Another repo rule is intentionally strict: every repo-managed skill ships with i Skill validation is local and deterministic. The Priority 1 **AI/LLM Evaluation Automation Prohibition** in `AGENTS.md` forbids repository preparation, validation, CI, hooks, deterministic tests, automatic fan-out, graders, and completion gates from using an authenticated Copilot, Claude, Codex, Gemini, or other model account. There is no repository opt-in switch. A human-selected external Eval Orchestrator may invoke an explicitly selected package-local Eval Runner for a package it was handed; that boundary never authorizes live model execution in CI or automatic repository workflows. -Evaluation keeps Anthropic's `skill-creator` methodology and portable paired-run conventions while replacing only the execution transport. `run.json` remains the runner-neutral one-arm contract; `execution-profile.json` selects the runner/provider/model/configuration; and `execution-result.json` normalizes one blind execution before the existing `eval-result/2` bridge and reports. The initial real runners are Codex and OpenCode, with a deterministic fake runner used for conformance. The repository prepares the paired candidate and baseline inputs as a portable package and stops; the external Eval Orchestrator resolves, preflights, and invokes the selected Eval Runner once per fresh arm, then grades and reports. No provider has a privileged runtime role, and native skill activation is not evaluated in v0.9.1. Adding or modifying a skill triggers package preparation automatically, as a completion gate an agent cannot skip: +Evaluation keeps Anthropic's `skill-creator` methodology and portable paired-run conventions while replacing only the execution transport. `run.json` remains the runner-neutral one-arm contract; `execution-profile.json` selects the runner/provider/model/configuration; and `execution-result.json` normalizes one blind execution before the existing `eval-result/2` bridge and reports. The conforming real runners are Codex, OpenCode, and Cline, with a deterministic fake runner used for conformance. The repository prepares the paired candidate and baseline inputs as a portable package and stops; the external Eval Orchestrator resolves, preflights, and invokes the selected Eval Runner once per fresh arm, then grades and reports. Hard filesystem confinement raises reported isolation confidence from pragmatic to strict but is not a universal platform prerequisite; mandatory experimental controls remain fail-closed. Freebuff remains planned/blocked until it exposes a supported noninteractive machine-readable transport. No provider has a privileged runtime role, and native skill activation is not evaluated in v0.9.1. Adding or modifying a skill triggers package preparation automatically, as a completion gate an agent cannot skip: ```powershell pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -Changed @@ -22,7 +22,7 @@ pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -Changed That resolves every skill the branch changed and prepares a package for each. `-Skill ` prepares one on demand. Packages land in the gitignored `.bot/-workspace/`, so a harness that refuses to work outside the repository folder can still reach them without anything entering the working tree. -Each eval becomes a directory holding the grading key (`eval-metadata.json` with the expected output, assertions, and fixture and skill hashes) and prefilled result stubs, plus two hermetic run directories. `with_skill/` is a self-contained sandbox root: a `prompt.md` with the effective skill instructions inlined, a `run.json` contract naming only paths inside the run, a `repo/` working tree materialized from the fixtures as real files, an isolated empty `home/`, and a `skill//` copy of the exact candidate revision. `without_skill/` is the same run with a byte-identical `repo/`, no `skill/` directory, and no mention of the skill. The grading key and results sit outside both run directories, so a worker confined to its run directory never sees them. Neither prompt identifies itself as an eval or names its configuration. `RUN-THIS.prompt.md` makes the user-selected agent the Eval Orchestrator: it reads the profile, resolves the selected runner, validates `describe`, preflights each `run.json`, invokes one fresh runner process per arm, stores raw `execution-result.json`, bridges it into the existing result shape, reveals the grading key only after execution, grades, and invokes the package adapter. The adapter stages the results into Anthropic's upstream benchmark workspace, runs `aggregate_benchmark.py`, writes the exact upstream `skill-creator-report.html`, and writes a first-party `report.html` with paired outputs, expected outcomes, assertion evidence, telemetry, transcripts, and downloadable feedback, plus `benchmark.json` and `benchmark.md`. Missing telemetry is displayed as unavailable rather than estimated. A runner that cannot prove the required isolation returns `incompatible`; there is no generic fallback. `-CollectResults ` remains a deterministic fallback for transferred artifacts without reports. Packages land in gitignored `.bot/` storage by default and are not committed. +Each eval becomes a directory holding the grading key (`eval-metadata.json` with the expected output, assertions, and fixture and skill hashes) and prefilled result stubs, plus two paired run directories. `with_skill/` is a self-contained run root: a `prompt.md` with the effective skill instructions inlined, a `run.json` contract naming only paths inside the run, a `repo/` working tree materialized from the fixtures as real files, an isolated empty `home/`, and a `skill//` copy of the exact candidate revision. `without_skill/` is the same run with a byte-identical `repo/`, no `skill/` directory, and no mention of the skill. The grading key and results sit outside both run directories, so workers are not intentionally given them. Neither prompt identifies itself as an eval or names its configuration. `RUN-THIS.prompt.md` makes the user-selected agent the Eval Orchestrator: it reads the profile, resolves the selected runner, validates `describe`, preflights each `run.json`, invokes one fresh runner process per arm, stores raw `execution-result.json`, bridges it into the existing result shape, reveals the grading key only after execution, grades, and invokes the package adapter. The adapter stages the results into Anthropic's upstream benchmark workspace, runs `aggregate_benchmark.py`, writes the exact upstream `skill-creator-report.html`, and writes a first-party `report.html` with paired outputs, expected outcomes, assertion evidence, telemetry, transcripts, and downloadable feedback, plus `benchmark.json` and `benchmark.md`. Missing telemetry is displayed as unavailable rather than estimated. A runner that cannot satisfy the mandatory experimental controls returns `incompatible`; lack of hard filesystem confinement downgrades the result to pragmatic isolation. `-CollectResults ` remains a deterministic fallback for transferred artifacts without reports. Packages land in gitignored `.bot/` storage by default and are not committed. The four roles are intentionally separate: the Eval Runner is the harness-specific executor for one blind arm; the Eval Orchestrator coordinates the external handoff; the Grader assesses results only after execution; and the Human Reviewer remains the final evaluator. Behavioral evaluation is portable across supported runners. Native activation is harness-specific and out of scope for v0.9.1. Anthropic is the methodology and compatibility reference, not a required Claude runtime. From a1353992eda10f3ba8e560c3fb04b67f241e2162 Mon Sep 17 00:00:00 2001 From: "aicia[bot]" Date: Fri, 21 Aug 2026 19:45:43 +0200 Subject: [PATCH 04/39] =?UTF-8?q?=F0=9F=94=A8=20enhance=20runner=20isolati?= =?UTF-8?q?on=20assessment=20and=20add=20cline=20support?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add isolation capability assessment and preflight validation to detect unsupported runner configurations. Implement Cline runner adapter for eval orchestration. Enhance common runner utilities with sandbox path mapping, external command versioning, and environment management. Expand conformance tests with additional event fixtures. --- scripts/eval-runners/README.md | 49 +- .../eval-runners/bridge-execution-result.ps1 | 23 +- scripts/eval-runners/cline/runner.ps1 | 529 ++++++++++++++++++ scripts/eval-runners/codex/runner.ps1 | 198 ++++--- .../contracts/execution-result.schema.json | 27 +- .../contracts/preflight-result.schema.json | 11 + scripts/eval-runners/fake/runner.ps1 | 5 +- scripts/eval-runners/freebuff-readiness.md | 34 ++ scripts/eval-runners/opencode/runner.ps1 | 147 +++-- scripts/eval-runners/runner-common.ps1 | 314 ++++++++++- .../tests/fixtures/cline-events.jsonl | 11 + .../tests/test-runner-conformance.ps1 | 352 +++++++++++- scripts/generate-eval-report.ps1 | 6 + scripts/prepare-skill-evals.ps1 | 7 +- scripts/validate-skill-templates.ps1 | 14 +- 15 files changed, 1576 insertions(+), 151 deletions(-) create mode 100644 scripts/eval-runners/cline/runner.ps1 create mode 100644 scripts/eval-runners/freebuff-readiness.md create mode 100644 scripts/eval-runners/tests/fixtures/cline-events.jsonl diff --git a/scripts/eval-runners/README.md b/scripts/eval-runners/README.md index c22d9da..cda4723 100644 --- a/scripts/eval-runners/README.md +++ b/scripts/eval-runners/README.md @@ -13,9 +13,13 @@ run.json + execution-profile.json -> runner -> execution-result.json `run.json` is the existing portable one-arm contract. It owns the prompt, working directory, isolated home, staged candidate skill, and required -isolation semantics. `execution-profile.json` selects the runner and execution -configuration. It contains no credentials or secrets. `execution-result.json` -normalizes one blind execution and keeps grading separate from raw evidence. +experimental controls. Its `filesystemIsolationRequired` and +`mustNotReadOutsideSandbox` fields describe the staged worker-facing package +boundary; they do not claim that the host has a hard OS filesystem sandbox. +`execution-profile.json` selects the runner and +execution configuration. It contains no credentials or secrets. +`execution-result.json` normalizes one blind execution and keeps grading +separate from raw evidence. Every runner exposes the same process surface: @@ -33,19 +37,30 @@ refusals, timeouts, failures, and incompatibility. The package resolver selects a named child directory under this directory. It does not guess a runner and does not fall back to an improvised worker. A selected runner that cannot satisfy the required contract returns -`incompatible`. +`incompatible`. Hard OS-level filesystem confinement is a confidence signal, +not a universal prerequisite: a run with all mandatory experimental controls +proven reports `strict` isolation when hard confinement is proven and +`pragmatic` isolation when it is not. A missing fresh context, controlled skill +boundary, prompt fidelity, result capture, or other mandatory control remains +incompatible. -The fake runner is deterministic and is the conformance reference. The Codex -and OpenCode runners are thin harness-specific adapters. Their native CLI +The fake runner is deterministic and is the conformance reference. Codex, +OpenCode, and Cline are thin harness-specific adapters. Their native CLI flags, environment setup, event parsing, authentication injection, and -isolation checks stay inside their own directories. - -The Codex adapter requires the current `codex exec` contract and an external -`bwrap` (Linux) or `sandbox-exec` (macOS) boundary in addition to Codex's -`workspace-write` sandbox. It fails closed on Windows because this slice does -not claim package-level filesystem read confinement there. The OpenCode -adapter uses the same platform split, `--pure`, isolated configuration roots, -and a narrowly selected provider credential; it also fails closed when its -external sandbox or credential requirement is unavailable. Neither adapter -copies a global skill directory, memory store, plugin set, or normal agent -profile into a run. +isolation checks stay inside their own directories. Windows is supported in +pragmatic mode when the native CLI satisfies the mandatory controls. + +Codex uses `--ask-for-approval never` with `exec --sandbox workspace-write`; +it does not combine explicit sandbox selection with `--approve-for-me`. +OpenCode uses `run --format json --auto` with isolated global/config roots and +preserves repository-owned project configuration; it does not depend on +`OPENCODE_DISABLE_PROJECT_CONFIG` or use `--pure`. Cline uses `--json`, +`--auto-approve true`, `--retries 0`, `--config `, +`--data-dir /.cline/data`, and run-local hooks; it passes no session +id. All three capture an exact observable CLI version and pass only a narrow +provider environment credential when available. None copies a global skill +directory, memory store, plugin set, or normal agent profile into a run. + +Freebuff is currently documented as planned/blocked. Its supported CLI remains +TUI-oriented and does not provide the required one-prompt, noninteractive, +machine-readable fresh-session transport, so no Freebuff runner is advertised. diff --git a/scripts/eval-runners/bridge-execution-result.ps1 b/scripts/eval-runners/bridge-execution-result.ps1 index 284fea5..b7aa352 100644 --- a/scripts/eval-runners/bridge-execution-result.ps1 +++ b/scripts/eval-runners/bridge-execution-result.ps1 @@ -180,6 +180,15 @@ try { $existingGrading = @(Get-ExistingGrading -ResultPath $resultPath) $caps = Get-JsonProperty -Object $raw.isolation -Name 'capabilities' -Default ([ordered]@{}) + $requestedModel = [string](Get-JsonProperty -Object $raw.requested -Name 'model' -Default '') + $requestedProvider = [string](Get-JsonProperty -Object $raw.requested -Name 'provider' -Default '') + $resolvedModelValue = Get-JsonProperty -Object $raw.resolved -Name 'model' -Default $null + $resolvedProviderValue = Get-JsonProperty -Object $raw.resolved -Name 'provider' -Default $null + $resolvedModel = if ($null -eq $resolvedModelValue) { '' } else { [string]$resolvedModelValue } + $resolvedProvider = if ($null -eq $resolvedProviderValue) { '' } else { [string]$resolvedProviderValue } + $resolutionStatus = [string](Get-JsonProperty -Object $raw.resolved -Name 'status' -Default 'unavailable') + $resolutionReason = [string](Get-JsonProperty -Object $raw.resolved -Name 'reason' -Default '') + $notes.Add("configuration_resolution=$resolutionStatus") $portableResult = [ordered]@{ schema = (Get-RunnerSchemaNames).PortableResult skill_name = if ($runData.Mode -eq 'with_skill') { [string](Get-JsonProperty -Object $runData.Contract -Name 'skillName' -Default '') } else { '' } @@ -187,8 +196,14 @@ try { eval_id = $runData.EvalId eval_name = $runData.EvalName configuration = $runData.Mode - model = [string](Get-JsonProperty -Object $raw.resolved -Name 'model' -Default (Get-JsonProperty -Object $raw.requested -Name 'model' -Default '')) - provider = [string](Get-JsonProperty -Object $raw.resolved -Name 'provider' -Default (Get-JsonProperty -Object $raw.requested -Name 'provider' -Default '')) + model = if ([string]::IsNullOrWhiteSpace($resolvedModel)) { $requestedModel } else { $resolvedModel } + provider = if ([string]::IsNullOrWhiteSpace($resolvedProvider)) { $requestedProvider } else { $resolvedProvider } + requested_model = $requestedModel + requested_provider = $requestedProvider + resolved_model = $resolvedModel + resolved_provider = $resolvedProvider + configuration_resolution_status = $resolutionStatus + configuration_resolution_reason = $resolutionReason harness = "$(Get-JsonProperty -Object $raw.harness -Name 'name' -Default 'unknown') $(Get-JsonProperty -Object $raw.harness -Name 'version' -Default '')".Trim() executed_utc = [string]$raw.finished_utc output = $output @@ -212,6 +227,10 @@ try { estimated_cost_usd = $costValue model_effort = [string](Get-JsonProperty -Object $raw.resolved -Name 'reasoning_effort' -Default '') isolation = [ordered]@{ + level = Get-JsonProperty -Object $raw.isolation -Name 'level' -Default 'unsupported' + status = Get-JsonProperty -Object $raw.isolation -Name 'status' -Default 'unverified' + hard_filesystem_confinement = Get-JsonProperty -Object $raw.isolation -Name 'hard_filesystem_confinement' -Default $false + mechanisms = @(Get-JsonProperty -Object $raw.isolation -Name 'mechanisms' -Default @()) fresh_context = Get-CapabilityBoolean (Get-JsonProperty -Object $caps -Name 'fresh_context' -Default $null) isolated_home = Get-CapabilityBoolean (Get-JsonProperty -Object $caps -Name 'isolated_home_config' -Default $null) isolated_cwd = Get-CapabilityBoolean (Get-JsonProperty -Object $caps -Name 'isolated_working_directory' -Default $null) diff --git a/scripts/eval-runners/cline/runner.ps1 b/scripts/eval-runners/cline/runner.ps1 new file mode 100644 index 0000000..22109aa --- /dev/null +++ b/scripts/eval-runners/cline/runner.ps1 @@ -0,0 +1,529 @@ +<#! +.SYNOPSIS + Cline Eval Runner adapter. + +.DESCRIPTION + The adapter uses Cline's supported headless JSON/NDJSON surface. It starts + one fresh process per arm, supplies the prompt on stdin, uses run-local + data/config and hooks directories, and never supplies a session id. +#> +[CmdletBinding()] +param( + [Parameter(Mandatory = $true, Position = 0)] + [ValidateSet('describe', 'preflight', 'execute')] + [string]$Command, + + [string]$Run, + [string]$Profile +) + +$ErrorActionPreference = 'Stop' +Set-StrictMode -Version Latest +. (Join-Path $PSScriptRoot '..\runner-common.ps1') + +$descriptor = [ordered]@{ + schema = (Get-RunnerSchemaNames).Descriptor + protocol_version = (Get-RunnerSchemaNames).Protocol + name = 'cline' + version = '0.9.1' + platforms = @('windows', 'linux', 'macos') + harness = [ordered]@{ name = 'Cline CLI'; version = 'unavailable' } + capabilities = [ordered]@{ + fresh_context = 'supported' + isolated_home_config = 'supported' + isolated_working_directory = 'supported' + filesystem_confinement = 'conditional' + ambient_candidate_skill_exclusion = 'supported' + candidate_skill_exposure = 'supported' + prompt_fidelity = 'supported' + model_configuration_lock = 'supported' + response_capture = 'supported' + transcript_event_capture = 'supported' + token_telemetry = 'conditional' + cache_token_telemetry = 'conditional' + tool_call_telemetry = 'conditional' + command_evidence = 'conditional' + file_evidence = 'conditional' + cost_telemetry = 'conditional' + credential_child_filtering = 'conditional' + native_skill_activation_evidence = 'unsupported' + } + supported_telemetry = @('transcript_event_capture', 'token_telemetry', 'cache_token_telemetry', 'tool_call_telemetry', 'command_evidence', 'file_evidence', 'cost_telemetry') + configuration_profiles = @('isolated-default') + tool_profiles = @('default') +} + +function Write-ProtocolError { + param([string]$Message) + + [Console]::Error.WriteLine($Message) + exit 2 +} + +function Resolve-ClineInputs { + if ([string]::IsNullOrWhiteSpace($Run) -or [string]::IsNullOrWhiteSpace($Profile)) { + throw 'preflight and execute require -Run and -Profile.' + } + return [pscustomobject]@{ + Run = Resolve-RunContract -RunPath $Run + Profile = Resolve-ExecutionProfile -ProfilePath $Profile + } +} + +function Invoke-ClineCli { + param( + [Parameter(Mandatory = $true)][object]$CommandInfo, + [Parameter(Mandatory = $true)][string[]]$Arguments, + [Parameter(Mandatory = $true)][object]$Inputs, + [System.Collections.IDictionary]$Environment, + [byte[]]$InputBytes = @(), + [int]$TimeoutSeconds = 60 + ) + + return Invoke-RunnerProcess -FileName $CommandInfo.FileName -ArgumentList (@($CommandInfo.Prefix) + @($Arguments)) -WorkingDirectory $Inputs.Run.WorkingDirectoryPath -Environment $Environment -InputBytes $InputBytes -TimeoutSeconds $TimeoutSeconds +} + +function Get-ClineDescriptor { + $copy = [ordered]@{} + foreach ($key in $descriptor.Keys) { $copy[$key] = $descriptor[$key] } + $commandInfo = Resolve-ExternalCommand -Name 'cline' + $version = 'unavailable' + if ($null -ne $commandInfo) { + $observation = Get-ExternalCommandVersion -CommandInfo $commandInfo + $version = [string]$observation.Version + } + $copy.harness = [ordered]@{ name = 'Cline CLI'; version = $version } + return $copy +} + +function New-ClineEnvironment { + param([Parameter(Mandatory = $true)][object]$Inputs) + + $clineRoot = Join-Path $Inputs.Run.HomeDirectoryPath '.cline' + $dataDirectory = Join-Path $clineRoot 'data' + $settingsDirectory = Join-Path $dataDirectory 'settings' + $sandboxDataDirectory = Join-Path $clineRoot 'sandbox-data' + $teamDataDirectory = Join-Path $dataDirectory 'teams' + $hooksDirectory = Join-Path $clineRoot 'hooks' + foreach ($directory in @($clineRoot, $dataDirectory, $settingsDirectory, $sandboxDataDirectory, $teamDataDirectory, $hooksDirectory)) { + New-Item -ItemType Directory -Path $directory -Force | Out-Null + } + return [pscustomobject]@{ + Environment = New-RunnerEnvironment -Run $Inputs.Run -AuthenticationVariables @(Get-ProviderAuthenticationVariables -Provider ([string]$Inputs.Profile.Provider)) -Additional @{ + CLINE_DATA_DIR = $dataDirectory + CLINE_SANDBOX_DATA_DIR = $sandboxDataDirectory + CLINE_HOOKS_DIR = $hooksDirectory + CLINE_SESSION_BACKEND_MODE = 'local' + } + Root = $clineRoot + DataDirectory = $dataDirectory + SettingsDirectory = $settingsDirectory + SandboxDataDirectory = $sandboxDataDirectory + TeamDataDirectory = $teamDataDirectory + HooksDirectory = $hooksDirectory + ConfigPath = $clineRoot + } +} + +function Get-ClineInsideEnvironment { + param( + [Parameter(Mandatory = $true)][object]$Inputs, + [Parameter(Mandatory = $true)][object]$EnvironmentData + ) + + $inside = [ordered]@{ + HOME = '/run/home' + USERPROFILE = '/run/home' + XDG_CONFIG_HOME = '/run/home/.config' + XDG_DATA_HOME = '/run/home/.local/share' + XDG_CACHE_HOME = '/run/home/.cache' + TEMP = '/run/home/tmp' + TMP = '/run/home/tmp' + CLINE_DATA_DIR = '/run/home/.cline/data' + CLINE_SANDBOX_DATA_DIR = '/run/home/.cline/sandbox-data' + CLINE_HOOKS_DIR = '/run/home/.cline/hooks' + CLINE_SESSION_BACKEND_MODE = 'local' + PATH = '/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin' + CI = '1' + NO_COLOR = '1' + } + foreach ($name in @(Get-ProviderAuthenticationVariables -Provider ([string]$Inputs.Profile.Provider))) { + if ($EnvironmentData.Environment.Contains($name) -and -not [string]::IsNullOrWhiteSpace([string]$EnvironmentData.Environment[$name])) { + $inside[$name] = [string]$EnvironmentData.Environment[$name] + } + } + return $inside +} + +function New-ClineCliArguments { + param( + [Parameter(Mandatory = $true)][object]$Inputs, + [Parameter(Mandatory = $true)][object]$EnvironmentData, + [ValidateSet('windows', 'linux', 'macos', 'unknown')][string]$VisiblePlatform = (Get-PlatformName) + ) + + $workingDirectory = Get-SandboxVisiblePath -HostPath $Inputs.Run.WorkingDirectoryPath -RunRoot $Inputs.Run.RunRoot -Platform $VisiblePlatform + $configPath = Get-SandboxVisiblePath -HostPath $EnvironmentData.ConfigPath -RunRoot $Inputs.Run.RunRoot -Platform $VisiblePlatform + $dataRoot = Get-SandboxVisiblePath -HostPath $EnvironmentData.DataDirectory -RunRoot $Inputs.Run.RunRoot -Platform $VisiblePlatform + $hooksDirectory = Get-SandboxVisiblePath -HostPath $EnvironmentData.HooksDirectory -RunRoot $Inputs.Run.RunRoot -Platform $VisiblePlatform + $arguments = [System.Collections.Generic.List[string]]::new() + foreach ($argument in @('--json', '--auto-approve', 'true', '--cwd', $workingDirectory, '--config', $configPath, '--data-dir', $dataRoot, '--hooks-dir', $hooksDirectory, '--provider', $Inputs.Profile.Provider, '--model', $Inputs.Profile.Model, '--retries', '0', '--timeout', [string]$Inputs.Profile.TimeoutSeconds)) { + $arguments.Add([string]$argument) + } + if (-not [string]::IsNullOrWhiteSpace([string]$Inputs.Profile.ReasoningEffort)) { + $arguments.Add('--thinking') + $arguments.Add([string]$Inputs.Profile.ReasoningEffort) + } + return @($arguments) +} + +function Get-ClineCapabilityMap { + param( + [Parameter(Mandatory = $true)][object]$Inputs, + [bool]$HardFilesystemConfinement = $false + ) + + $capabilities = [ordered]@{} + foreach ($capabilityName in @(Get-JsonPropertyNames -Object $descriptor.capabilities)) { + $capabilities[$capabilityName] = [string](Get-JsonProperty -Object $descriptor.capabilities -Name $capabilityName) + } + $capabilities['filesystem_confinement'] = if ($HardFilesystemConfinement) { 'supported' } else { 'unsupported' } + $capabilities['candidate_skill_exposure'] = if ($Inputs.Run.CandidateSkillExposed) { 'supported' } else { 'excluded' } + return $capabilities +} + +function Get-ClinePreflight { + param([Parameter(Mandatory = $true)][object]$Inputs) + + $checks = [System.Collections.Generic.List[object]]::new() + $reasons = [System.Collections.Generic.List[string]]::new() + $warnings = [System.Collections.Generic.List[string]]::new() + $profile = $Inputs.Profile + $run = $Inputs.Run + $platform = Get-PlatformName + $commandInfo = Resolve-ExternalCommand -Name 'cline' + $sandboxInfo = if ($platform -eq 'linux') { Resolve-ExternalCommand -Name 'bwrap' } elseif ($platform -eq 'macos') { Resolve-ExternalCommand -Name 'sandbox-exec' } else { $null } + $versionObservation = $null + + if ($profile.Runner -ne 'cline') { + $reasons.Add("execution-profile.json selects '$($profile.Runner)' rather than cline.") + } else { + $checks.Add((New-PreflightCheck -Name 'runner_selection' -Status passed -Detail 'The selected runner is cline.')) + } + if ([string]::IsNullOrWhiteSpace($profile.Provider)) { + $reasons.Add('Cline requires a provider in execution-profile.json.') + } else { + $checks.Add((New-PreflightCheck -Name 'provider' -Status passed -Detail $profile.Provider)) + } + if ([string]::IsNullOrWhiteSpace($profile.Model)) { + $reasons.Add('Cline requires a model in execution-profile.json.') + } else { + $checks.Add((New-PreflightCheck -Name 'model' -Status passed -Detail $profile.Model)) + } + if ($profile.ConfigurationProfile -ne 'isolated-default') { $reasons.Add("configuration_profile '$($profile.ConfigurationProfile)' is unsupported by cline.") } + if ($profile.ToolProfile -ne 'default') { $reasons.Add("tool_profile '$($profile.ToolProfile)' is unsupported by cline.") } + + $environmentData = New-ClineEnvironment -Inputs $Inputs + if ($null -eq $commandInfo) { + $reasons.Add('The Cline CLI executable is not available on PATH.') + } else { + $checks.Add((New-PreflightCheck -Name 'harness_executable' -Status passed -Detail $commandInfo.Source)) + try { + $versionObservation = Get-ExternalCommandVersion -CommandInfo $commandInfo -WorkingDirectory $run.WorkingDirectoryPath -Environment $environmentData.Environment -TimeoutSeconds 30 + if (-not $versionObservation.Available) { + $reasons.Add('The Cline CLI did not expose an exact observable version through --version.') + $checks.Add((New-PreflightCheck -Name 'harness_version' -Status unavailable -Detail 'cline --version did not return a usable version string.')) + } else { + $checks.Add((New-PreflightCheck -Name 'harness_version' -Status passed -Detail ([string]$versionObservation.Version))) + } + $help = Invoke-ClineCli -CommandInfo $commandInfo -Arguments @('--retries', '0', '--help') -Inputs $Inputs -Environment $environmentData.Environment -TimeoutSeconds 30 + if ($help.TimedOut -or $help.ExitCode -ne 0) { + $reasons.Add("Cline --retries 0 --help failed with exit status $($help.ExitCode).") + } else { + $helpText = [string]::Join("`n", @($help.Stdout, $help.Stderr)) + foreach ($flag in @('--json', '--auto-approve', '--cwd', '--config', '--data-dir', '--hooks-dir', '--provider', '--model', '--thinking', '--timeout', '--retries')) { + if ($helpText -notmatch [regex]::Escape($flag)) { $reasons.Add("The installed Cline CLI does not advertise required flag '$flag'.") } + } + $visiblePlatform = if ($platform -eq 'linux' -and $null -ne $sandboxInfo) { 'linux' } else { $platform } + $constructed = New-ClineCliArguments -Inputs $Inputs -EnvironmentData $environmentData -VisiblePlatform $visiblePlatform + foreach ($forbidden in @('--id', '--continue', '--session', '--yolo', '--zen', '--tui')) { + if (@($constructed) -contains $forbidden) { $reasons.Add("The constructed Cline invocation must not use resume or interactive option '$forbidden'.") } + } + $retryIndex = [Array]::IndexOf([string[]]$constructed, '--retries') + if ($retryIndex -lt 0 -or $constructed[$retryIndex + 1] -ne '0') { $reasons.Add('The constructed Cline invocation must set --retries 0.') } + if ($reasons.Count -eq 0) { + $checks.Add((New-PreflightCheck -Name 'harness_contract' -Status passed -Detail 'Cline advertises JSON/NDJSON output, isolated directories, provider/model selection, timeout, thinking, auto-approval, and zero internal retries.')) + } + } + } catch { + $reasons.Add("Could not inspect Cline CLI capabilities: $($_.Exception.Message)") + } + } + + $authVariables = @(Get-ProviderAuthenticationVariables -Provider ([string]$profile.Provider)) + $authVariable = $null + foreach ($name in $authVariables) { + if (-not [string]::IsNullOrWhiteSpace([Environment]::GetEnvironmentVariable($name))) { $authVariable = $name; break } + } + if ([string]::IsNullOrWhiteSpace($authVariable)) { + $reasons.Add("No narrow provider authentication environment variable is available for '$($profile.Provider)'; ambient Cline auth profiles are not copied into an eval run.") + } else { + $checks.Add((New-PreflightCheck -Name 'authentication' -Status passed -Detail "Provider credential will be passed only as $authVariable.")) + } + + if ($platform -notin @('linux', 'macos')) { + $checks.Add((New-PreflightCheck -Name 'filesystem_confinement' -Status not_applicable -Detail "Platform '$platform' has no configured external hard-confinement mechanism; pragmatic isolation remains available.")) + $warnings.Add("Platform '$platform' has no external hard filesystem confinement in this adapter; execution will report pragmatic isolation.") + } elseif ($null -eq $sandboxInfo) { + $sandboxName = if ($platform -eq 'linux') { 'bwrap' } else { 'sandbox-exec' } + $checks.Add((New-PreflightCheck -Name 'filesystem_confinement' -Status unavailable -Detail "External '$sandboxName' is unavailable; pragmatic isolation remains available.")) + $warnings.Add("External '$sandboxName' was unavailable; execution will report pragmatic isolation.") + } else { + $checks.Add((New-PreflightCheck -Name 'filesystem_confinement' -Status passed -Detail "External $($sandboxInfo.Source) confines Cline to the staged run and run-local data/config roots.")) + } + $checks.Add((New-PreflightCheck -Name 'fresh_session' -Status passed -Detail 'The adapter starts one new Cline process, supplies no --id, and never reuses a session.')) + $checks.Add((New-PreflightCheck -Name 'retry_semantics' -Status passed -Detail '--retries 0 disables Cline consecutive operational retries; the runner still starts exactly one semantic process with attempt_count=1.')) + $checks.Add((New-PreflightCheck -Name 'ambient_configuration' -Status passed -Detail 'HOME, Cline data/config, hooks, sessions, and plugin roots are run-local and empty; no ambient user profile is copied.')) + $checks.Add((New-PreflightCheck -Name 'prompt_fidelity' -Status passed -Detail 'The exact prompt bytes are sent on stdin as the first and only task input.')) + $warnings.Add('Cline does not expose a supported child-tool environment filter in this CLI contract; the runner removes unrelated inherited variables but cannot independently prove that the selected provider credential is hidden from every Cline-launched tool.') + + $hardConfinement = $null -ne $sandboxInfo -and $platform -in @('linux', 'macos') + $capabilities = Get-ClineCapabilityMap -Inputs $Inputs -HardFilesystemConfinement $hardConfinement + $harnessVersion = if ($null -eq $versionObservation) { 'unavailable' } else { [string]$versionObservation.Version } + $descriptorCopy = [ordered]@{} + foreach ($key in $descriptor.Keys) { $descriptorCopy[$key] = $descriptor[$key] } + $descriptorCopy.harness = [ordered]@{ name = 'Cline CLI'; version = $harnessVersion } + $mechanisms = [System.Collections.Generic.List[string]]::new() + foreach ($mechanism in @('cline --json', '--auto-approve true', '--retries 0', 'no --id session resume', 'run-local HOME', 'run-local Cline data/config/hooks directories', 'prompt on stdin')) { $mechanisms.Add($mechanism) } + if ($hardConfinement) { $mechanisms.Add("external $($sandboxInfo.Source) filesystem sandbox") } else { $mechanisms.Add('pragmatic process/environment isolation without hard filesystem confinement') } + return New-PreflightDocument -Descriptor $descriptorCopy -Profile $profile -Run $run -Compatible ($reasons.Count -eq 0) -Checks @($checks) -Mechanisms @($mechanisms) -ResolvedCapabilities $capabilities -Warnings @($warnings) -Reasons @($reasons) +} + +function Write-ClineCapture { + param( + [Parameter(Mandatory = $true)][object]$RunData, + [Parameter(Mandatory = $true)][string]$RelativePath, + [Parameter(Mandatory = $true)][AllowEmptyString()][string]$Text + ) + + $path = Join-Path $RunData.Run.RunRoot ($RelativePath -replace '/', [System.IO.Path]::DirectorySeparatorChar) + New-Item -ItemType Directory -Path (Split-Path -Parent $path) -Force | Out-Null + [System.IO.File]::WriteAllText($path, $Text, [System.Text.UTF8Encoding]::new($false)) + return New-ArtifactReference -Run $RunData.Run -Path $RelativePath -Scope run -MediaType (Get-MediaType -Path $RelativePath) +} + +function Invoke-ClineExecute { + param([Parameter(Mandatory = $true)][object]$Inputs) + + $preflight = Get-ClinePreflight -Inputs $Inputs + $started = [DateTime]::UtcNow + $sessionId = [Guid]::NewGuid().ToString('D') + $executionDescriptor = [ordered]@{} + foreach ($key in $descriptor.Keys) { $executionDescriptor[$key] = $descriptor[$key] } + $executionDescriptor.harness = $preflight.harness + if ($preflight.status -ne 'compatible') { + $finished = [DateTime]::UtcNow + return New-ExecutionResult -Descriptor $executionDescriptor -Profile $Inputs.Profile -Run $Inputs.Run -Status incompatible -FinalResponseReason 'preflight_incompatible' -StartedUtc $started.ToString('o') -FinishedUtc $finished.ToString('o') -DurationSeconds ($finished - $started).TotalSeconds -Failure (New-ExecutionFailure -Code 'incompatible' -Message ([string]::Join('; ', @($preflight.reasons)))) -SessionId $sessionId -IsolationCapabilities ([ordered]@{}) -IsolationMechanisms @('preflight-only') -Evidence ([ordered]@{ preflight = $preflight; resume = $false; session_id_supplied = $false }) -AttemptCount 1 + } + + $commandInfo = Resolve-ExternalCommand -Name 'cline' + $environmentData = New-ClineEnvironment -Inputs $Inputs + $platform = Get-PlatformName + $sandboxInfo = if ($platform -eq 'linux') { Resolve-ExternalCommand -Name 'bwrap' } elseif ($platform -eq 'macos') { Resolve-ExternalCommand -Name 'sandbox-exec' } else { $null } + $hardFilesystem = $null -ne $sandboxInfo -and $platform -in @('linux', 'macos') + $visiblePlatform = if ($hardFilesystem) { $platform } elseif ($platform -eq 'linux') { 'unknown' } else { $platform } + $arguments = New-ClineCliArguments -Inputs $Inputs -EnvironmentData $environmentData -VisiblePlatform $visiblePlatform + if ($platform -eq 'linux' -and $hardFilesystem) { + $insideEnvironment = Get-ClineInsideEnvironment -Inputs $Inputs -EnvironmentData $environmentData + $sandboxArguments = Get-LinuxEvalSandboxArguments -Inputs $Inputs -CommandInfo $commandInfo -InsideEnvironment $insideEnvironment -ReadOnlyRoots @('/usr', '/usr/local', '/bin', '/sbin', '/lib', '/lib64', '/libexec', '/etc', '/opt') + $process = Invoke-RunnerProcess -FileName $sandboxInfo.FileName -ArgumentList (@($sandboxArguments) + @($arguments)) -WorkingDirectory $Inputs.Run.WorkingDirectoryPath -Environment $environmentData.Environment -InputBytes $Inputs.Run.PromptBytes -TimeoutSeconds $Inputs.Profile.TimeoutSeconds + } elseif ($platform -eq 'macos' -and $hardFilesystem) { + $sandboxProfile = New-MacosEvalSandboxProfile -Inputs $Inputs -CommandInfo $commandInfo -ReadOnlyRoots @('/usr', '/usr/local', '/bin', '/sbin', '/lib', '/libexec', '/System', '/Library', '/opt', '/private/var/db') + $sandboxArguments = @('-f', $sandboxProfile, '--', $commandInfo.FileName) + @($commandInfo.Prefix) + @($arguments) + $process = Invoke-RunnerProcess -FileName $sandboxInfo.FileName -ArgumentList $sandboxArguments -WorkingDirectory $Inputs.Run.WorkingDirectoryPath -Environment $environmentData.Environment -InputBytes $Inputs.Run.PromptBytes -TimeoutSeconds $Inputs.Profile.TimeoutSeconds + } else { + $process = Invoke-ClineCli -CommandInfo $commandInfo -Arguments $arguments -Inputs $Inputs -Environment $environmentData.Environment -InputBytes $Inputs.Run.PromptBytes -TimeoutSeconds $Inputs.Profile.TimeoutSeconds + } + + $stdoutArtifact = Write-ClineCapture -RunData $Inputs -RelativePath 'evidence/cline-events.jsonl' -Text $process.Stdout + $stderrArtifact = Write-ClineCapture -RunData $Inputs -RelativePath 'evidence/cline-stderr.txt' -Text $process.Stderr + $artifacts = [System.Collections.Generic.List[object]]::new() + $artifacts.Add($stdoutArtifact); $artifacts.Add($stderrArtifact) + $parsed = ConvertFrom-JsonLines -Text $process.Stdout + $warnings = [System.Collections.Generic.List[string]]::new() + foreach ($parseError in @($parsed.Errors)) { $warnings.Add("Cline event parse error: $parseError") } + $eventCounts = @{} + $contentEnd = [System.Collections.Generic.List[string]]::new() + $contentStart = [System.Collections.Generic.List[string]]::new() + $jsonText = [System.Collections.Generic.List[string]]::new() + $jsonPartialText = [System.Collections.Generic.List[string]]::new() + $completionResultText = [System.Collections.Generic.List[string]]::new() + $finalText = $null + $returnedSessionId = $null + $failureMessage = $null + $usage = [ordered]@{} + $toolCalls = [System.Collections.Generic.List[object]]::new() + $commands = [System.Collections.Generic.List[object]]::new() + $files = [System.Collections.Generic.List[object]]::new() + foreach ($event in @($parsed.Events)) { + $topType = [string](Get-JsonProperty -Object $event -Name 'type' -Default '') + if ([string]::IsNullOrWhiteSpace($topType)) { + $warnings.Add('Cline emitted an event without a type; it was ignored.') + continue + } + if ($eventCounts.ContainsKey($topType)) { $eventCounts[$topType]++ } else { $eventCounts[$topType] = 1 } + $inner = Get-JsonProperty -Object $event -Name 'event' -Default $null + $eventType = if ($topType -eq 'agent_event' -and $null -ne $inner) { [string](Get-JsonProperty -Object $inner -Name 'type' -Default '') } else { $topType } + $payload = if ($topType -eq 'agent_event' -and $null -ne $inner) { $inner } else { $event } + + if ($topType -in @('say', 'ask')) { + $subtypeName = if ($topType -eq 'say') { 'say' } else { 'ask' } + $subtype = [string](Get-JsonProperty -Object $event -Name $subtypeName -Default '') + $text = [string](Get-JsonProperty -Object $event -Name 'text' -Default '') + $isPartial = [bool](Get-JsonProperty -Object $event -Name 'partial' -Default $false) + if (-not [string]::IsNullOrWhiteSpace($text)) { + if ($subtype -eq 'completion_result') { $completionResultText.Add($text) } + elseif ($subtype -eq 'text') { + if ($isPartial) { $jsonPartialText.Add($text) } else { $jsonText.Add($text) } + } + } + if ($topType -eq 'ask') { + if ($subtype -eq 'api_req_failed') { + $failureMessage = if ([string]::IsNullOrWhiteSpace($text)) { 'Cline reported an API request failure.' } else { $text } + } elseif ($subtype -in @('followup', 'plan_mode_respond', 'act_mode_respond')) { + $failureMessage = 'Cline requested interactive input during a noninteractive eval run.' + } elseif ($subtype -in @('use_mcp_server', 'command', 'tool')) { + $toolCalls.Add([ordered]@{ type = $subtype; name = $subtype }) + if ($subtype -eq 'command' -and -not [string]::IsNullOrWhiteSpace($text)) { + $commands.Add([ordered]@{ command = $text }) + } + } + } elseif ($subtype -in @('tool', 'command', 'command_output', 'mcp_server_request_started', 'mcp_server_response')) { + $toolName = [string](Get-JsonProperty -Object $event -Name 'name' -Default (Get-JsonProperty -Object $event -Name 'tool' -Default '')) + $toolCalls.Add([ordered]@{ type = $subtype; name = $toolName }) + if ($subtype -eq 'command' -and -not [string]::IsNullOrWhiteSpace($text)) { + $commands.Add([ordered]@{ command = $text }) + } + } + if ($subtype -eq 'completion_result' -and -not [string]::IsNullOrWhiteSpace($text)) { + $finalText = $text + } + if ($subtype -eq 'api_req_finished' -and -not [string]::IsNullOrWhiteSpace($text)) { + try { + $finishedUsage = $text | ConvertFrom-Json + foreach ($name in @('inputTokens', 'outputTokens', 'totalTokens', 'cacheReadTokens', 'cacheWriteTokens', 'cost')) { + $value = Get-JsonProperty -Object $finishedUsage -Name $name -Default $null + if ($null -ne $value) { $usage[$name] = $value } + } + } catch { + $warnings.Add('Cline api_req_finished text was not a usage JSON object; it was retained in the transcript.') + } + } + if ($subtype -notin @('task', 'error', 'api_req_started', 'api_req_finished', 'api_req_retried', 'api_req_retry_delayed', 'api_req_deleted', 'text', 'reasoning', 'completion_result', 'user_feedback', 'user_feedback_diff', 'command_output', 'tool', 'shell_integration_warning', 'browser_action', 'browser_action_result', 'command', 'mcp_server_request_started', 'mcp_server_response', 'new_task_started', 'new_task', 'subtask_result', 'checkpoint_saved', 'rooignore_error', 'diff_error', 'followup', 'plan_mode_respond', 'act_mode_respond', 'api_req_failed', 'use_mcp_server', 'resume_task', 'resume_completed_task', 'mistake_limit_reached', 'finishTask')) { + $warnings.Add("Unknown Cline $topType subtype '$subtype' was preserved as a warning.") + } + continue + } + switch ($eventType) { + 'done' { + $candidate = Get-JsonProperty -Object $payload -Name 'text' -Default '' + if (-not [string]::IsNullOrWhiteSpace([string]$candidate)) { $finalText = [string]$candidate } + $returnedSessionId = [string](Get-JsonProperty -Object $payload -Name 'sessionId' -Default (Get-JsonProperty -Object $payload -Name 'session_id' -Default $returnedSessionId)) + $doneUsage = Get-JsonProperty -Object $payload -Name 'usage' -Default $null + if ($null -ne $doneUsage) { + foreach ($name in @('inputTokens', 'outputTokens', 'totalTokens', 'cacheReadTokens', 'cacheWriteTokens', 'cost')) { + $value = Get-JsonProperty -Object $doneUsage -Name $name -Default $null + if ($null -ne $value) { $usage[$name] = $value } + } + } + } + 'content_end' { $text = [string](Get-JsonProperty -Object $payload -Name 'text' -Default ''); if (-not [string]::IsNullOrWhiteSpace($text)) { $contentEnd.Add($text) } } + 'content_start' { $text = [string](Get-JsonProperty -Object $payload -Name 'text' -Default ''); if (-not [string]::IsNullOrWhiteSpace($text)) { $contentStart.Add($text) } } + 'usage' { + foreach ($name in @('inputTokens', 'outputTokens', 'totalTokens', 'cacheReadTokens', 'cacheWriteTokens', 'cost')) { + $value = Get-JsonProperty -Object $payload -Name $name -Default $null + if ($null -ne $value) { $usage[$name] = $value } + } + } + 'tool_call' { $toolCalls.Add([ordered]@{ type = $eventType; name = [string](Get-JsonProperty -Object $payload -Name 'name' -Default (Get-JsonProperty -Object $payload -Name 'tool' -Default '')) }) } + 'tool_use' { $toolCalls.Add([ordered]@{ type = $eventType; name = [string](Get-JsonProperty -Object $payload -Name 'name' -Default (Get-JsonProperty -Object $payload -Name 'tool' -Default '')) }) } + 'command' { $commands.Add([ordered]@{ command = Get-JsonProperty -Object $payload -Name 'command' -Default (Get-JsonProperty -Object $payload -Name 'text' -Default '') }) } + 'file' { $files.Add([ordered]@{ path = Get-JsonProperty -Object $payload -Name 'path' -Default '' }) } + 'error' { $failureMessage = [string](Get-JsonProperty -Object $payload -Name 'message' -Default 'Cline emitted an error.') } + 'hook_event' { } + 'agent_start' { } + 'agent_end' { } + 'iteration_start' { } + 'iteration_end' { } + default { $warnings.Add("Unknown Cline event '$topType/$eventType' was preserved as a warning.") } + } + if ($eventType -eq 'usage') { + $eventUsage = Get-JsonProperty -Object $payload -Name 'usage' -Default $null + if ($null -ne $eventUsage) { + foreach ($name in @('inputTokens', 'outputTokens', 'totalTokens', 'cacheReadTokens', 'cacheWriteTokens', 'cost')) { + $value = Get-JsonProperty -Object $eventUsage -Name $name -Default $null + if ($null -ne $value) { $usage[$name] = $value } + } + } + } + } + if ([string]::IsNullOrWhiteSpace($finalText)) { + if ($contentEnd.Count -gt 0) { $finalText = [string]::Join('', @($contentEnd)) } + elseif ($contentStart.Count -gt 0) { $finalText = [string]::Join('', @($contentStart)) } + elseif ($completionResultText.Count -gt 0) { $finalText = [string]$completionResultText[$completionResultText.Count - 1] } + elseif ($jsonText.Count -gt 0) { $finalText = [string]$jsonText[$jsonText.Count - 1] } + elseif ($jsonPartialText.Count -gt 0) { $finalText = [string]::Join('', @($jsonPartialText)) } + } + + $status = 'completed' + $reason = $null + $failure = $null + $exitStatus = if ($process.TimedOut) { $null } else { [Nullable[int]]$process.ExitCode } + if ($process.TimedOut) { + $status = 'timed_out'; $reason = 'cline_timeout'; $failure = New-ExecutionFailure -Code 'timed_out' -Message 'Cline did not finish before timeout_seconds.' + } elseif ($process.ExitCode -ne 0 -or $null -ne $failureMessage) { + $failureDetail = if ([string]::IsNullOrWhiteSpace($failureMessage)) { 'Cline exited unsuccessfully.' } else { $failureMessage } + $status = 'failed'; $reason = 'cline_failure'; $failure = New-ExecutionFailure -Code 'cline_failure' -Message $failureDetail + } elseif ([string]::IsNullOrWhiteSpace($finalText)) { + $reason = 'cline_did_not_return_final_response'; $warnings.Add('Cline exited successfully without a final response event.') + } + $telemetry = [ordered]@{ + transcript = New-AvailableMetric -Value ([ordered]@{ artifact = 'evidence/cline-events.jsonl'; complete = $true }) + tokens = if ($usage.Count -eq 0) { New-UnavailableMetric -Reason 'cline_did_not_expose_usage' } else { New-AvailableMetric -Value $usage } + tool_calls = New-AvailableMetric -Value $toolCalls.Count + cost = if ($usage.Contains('cost')) { New-AvailableMetric -Value $usage['cost'] } else { New-UnavailableMetric -Reason 'cline_did_not_expose_cost' } + } + $finished = [DateTime]::UtcNow + $capabilities = Get-ClineCapabilityMap -Inputs $Inputs -HardFilesystemConfinement $hardFilesystem + $mechanisms = [System.Collections.Generic.List[string]]::new() + foreach ($mechanism in @('cline --json', '--auto-approve true', '--retries 0', 'no --id session resume', 'run-local HOME', 'run-local Cline data/config/hooks directories', 'prompt on stdin')) { $mechanisms.Add($mechanism) } + if ($hardFilesystem) { $mechanisms.Add("external $($sandboxInfo.Source) filesystem sandbox") } else { $mechanisms.Add('pragmatic process/environment isolation without hard filesystem confinement'); $warnings.Add('Hard filesystem confinement was unavailable; the completed arm is reported as pragmatic isolation.') } + $credentialEvidence = [ordered]@{ + provider_environment_variables = @(Get-ProviderAuthenticationVariables -Provider ([string]$Inputs.Profile.Provider)) + unrelated_environment_excluded = $true + child_tool_visibility = 'not_exposed_by_runner_environment; Cline child filtering is not independently observable' + value_observed = $false + } + $sessionResultId = if ([string]::IsNullOrWhiteSpace($returnedSessionId)) { $sessionId } else { $returnedSessionId } + $sandboxEvidence = if (-not $hardFilesystem) { 'unavailable' } elseif ($platform -eq 'linux') { 'bwrap' } else { 'sandbox-exec' } + return New-ExecutionResult -Descriptor $executionDescriptor -Profile $Inputs.Profile -Run $Inputs.Run -Status $status -FinalResponse $finalText -FinalResponseReason $reason -StartedUtc $process.StartedUtc.ToString('o') -FinishedUtc $finished.ToString('o') -DurationSeconds $process.DurationSeconds -ExitStatus $exitStatus -Failure $failure -SessionId $sessionResultId -IsolationCapabilities $capabilities -IsolationMechanisms @($mechanisms) -ResolvedConfiguration ([ordered]@{ status = 'accepted_request'; reason = 'Cline accepted the requested provider, model, thinking, and configuration but did not expose concrete backend resolution.'; observations = [ordered]@{ provider = $Inputs.Profile.Provider; model = $Inputs.Profile.Model; reasoning_effort = $Inputs.Profile.ReasoningEffort; retries = 0 } }) -Telemetry $telemetry -Artifacts @($artifacts) -Warnings @($warnings) -Evidence ([ordered]@{ event_counts = $eventCounts; commands = @($commands); files = @($files); prompt_first_input = $true; resume = $false; session_id_supplied = $false; retry_argument = 0; sandbox = $sandboxEvidence; credential = $credentialEvidence }) -AttemptCount 1 +} + +try { + [void](Assert-RunnerDescriptor -Descriptor $descriptor) + switch ($Command) { + 'describe' { Write-RunnerJson -Value (Get-ClineDescriptor) -AsOutput } + 'preflight' { + $inputs = Resolve-ClineInputs + Write-RunnerJson -Value (Get-ClinePreflight -Inputs $inputs) -AsOutput + } + 'execute' { + $inputs = Resolve-ClineInputs + $result = Invoke-ClineExecute -Inputs $inputs + [void](Assert-ExecutionResult -Result $result) + Write-RunnerJson -Value $result -AsOutput + } + } +} catch { + Write-ProtocolError -Message $_.Exception.Message +} diff --git a/scripts/eval-runners/codex/runner.ps1 b/scripts/eval-runners/codex/runner.ps1 index f63f423..e83bb4a 100644 --- a/scripts/eval-runners/codex/runner.ps1 +++ b/scripts/eval-runners/codex/runner.ps1 @@ -4,7 +4,7 @@ .DESCRIPTION This is the only place where Codex CLI flags, CODEX_HOME handling, JSONL - event parsing, and Codex sandbox limitations are defined. + event parsing, and Codex isolation limitations are defined. #> [CmdletBinding()] param( @@ -25,8 +25,8 @@ $descriptor = [ordered]@{ protocol_version = (Get-RunnerSchemaNames).Protocol name = 'codex' version = '0.9.1' - platforms = @('linux', 'macos') - harness = [ordered]@{ name = 'OpenAI Codex CLI'; version = 'current-supported' } + platforms = @('windows', 'linux', 'macos') + harness = [ordered]@{ name = 'OpenAI Codex CLI'; version = 'unavailable' } capabilities = [ordered]@{ fresh_context = 'supported' isolated_home_config = 'supported' @@ -44,6 +44,7 @@ $descriptor = [ordered]@{ command_evidence = 'conditional' file_evidence = 'conditional' cost_telemetry = 'conditional' + credential_child_filtering = 'supported' native_skill_activation_evidence = 'unsupported' } supported_telemetry = @('transcript_event_capture', 'token_telemetry', 'cache_token_telemetry', 'tool_call_telemetry', 'command_evidence', 'file_evidence', 'cost_telemetry') @@ -86,7 +87,7 @@ function Get-CodexAuthSource { } $authPath = Join-Path $codexHome 'auth.json' if (Test-Path -LiteralPath $authPath -PathType Leaf) { - return [pscustomobject]@{ Kind = 'file'; Name = 'auth.json'; Path = (Resolve-Path -LiteralPath $authPath).Path } + return [pscustomobject]@{ Kind = 'file_unsupported'; Name = 'auth.json'; Path = (Resolve-Path -LiteralPath $authPath).Path } } return [pscustomobject]@{ Kind = 'missing'; Name = $null; Path = $null } @@ -109,11 +110,12 @@ function Invoke-CodexCli { function Get-CodexHelpResult { param( [Parameter(Mandatory = $true)][object]$CommandInfo, - [Parameter(Mandatory = $true)][object]$Inputs + [Parameter(Mandatory = $true)][object]$Inputs, + [string[]]$Arguments = @('--ask-for-approval', 'never', 'exec', '--help') ) $environment = New-RunnerEnvironment -Run $Inputs.Run - return Invoke-CodexCli -CommandInfo $CommandInfo -Arguments @('exec', '--help') -Inputs $Inputs -Environment $environment -TimeoutSeconds 30 + return Invoke-CodexCli -CommandInfo $CommandInfo -Arguments $Arguments -Inputs $Inputs -Environment $environment -TimeoutSeconds 30 } function Resolve-SandboxCommand { @@ -122,6 +124,55 @@ function Resolve-SandboxCommand { return Resolve-ExternalCommand -Name $Name } +function Get-CodexDescriptor { + $copy = [ordered]@{} + foreach ($key in $descriptor.Keys) { $copy[$key] = $descriptor[$key] } + $commandInfo = Resolve-ExternalCommand -Name 'codex' + $version = 'unavailable' + if ($null -ne $commandInfo) { + $observation = Get-ExternalCommandVersion -CommandInfo $commandInfo + $version = [string]$observation.Version + } + $copy.harness = [ordered]@{ name = 'OpenAI Codex CLI'; version = $version } + return $copy +} + +function New-CodexCliArguments { + param( + [Parameter(Mandatory = $true)][object]$Inputs, + [Parameter(Mandatory = $true)][string]$LastResponsePath, + [ValidateSet('windows', 'linux', 'macos', 'unknown')][string]$VisiblePlatform = (Get-PlatformName) + ) + + $directoryArgument = Get-SandboxVisiblePath -HostPath $Inputs.Run.WorkingDirectoryPath -RunRoot $Inputs.Run.RunRoot -Platform $VisiblePlatform + $outputArgument = Get-SandboxVisiblePath -HostPath $LastResponsePath -RunRoot $Inputs.Run.RunRoot -Platform $VisiblePlatform + $arguments = [System.Collections.Generic.List[string]]::new() + foreach ($argument in @('--ask-for-approval', 'never', 'exec', '--ephemeral', '--ignore-user-config', '--ignore-rules', '--skip-git-repo-check', '--json', '--color', 'never', '--cd', $directoryArgument, '--model', $Inputs.Profile.Model, '--sandbox', 'workspace-write', '--config', 'shell_environment_policy.inherit=none', '--output-last-message', $outputArgument)) { + $arguments.Add([string]$argument) + } + if (-not [string]::IsNullOrWhiteSpace([string]$Inputs.Profile.ReasoningEffort)) { + $arguments.Add('-c') + $arguments.Add("model_reasoning_effort=$($Inputs.Profile.ReasoningEffort)") + } + $arguments.Add('-') + return @($arguments) +} + +function Get-CodexCapabilityMap { + param( + [Parameter(Mandatory = $true)][object]$Inputs, + [bool]$HardFilesystemConfinement = $false + ) + + $capabilities = [ordered]@{} + foreach ($capabilityName in @(Get-JsonPropertyNames -Object $descriptor.capabilities)) { + $capabilities[$capabilityName] = [string](Get-JsonProperty -Object $descriptor.capabilities -Name $capabilityName) + } + $capabilities['filesystem_confinement'] = if ($HardFilesystemConfinement) { 'supported' } else { 'unsupported' } + $capabilities['candidate_skill_exposure'] = if ($Inputs.Run.CandidateSkillExposed) { 'supported' } else { 'excluded' } + return $capabilities +} + function Get-CodexPreflight { param([Parameter(Mandatory = $true)][object]$Inputs) @@ -138,6 +189,7 @@ function Get-CodexPreflight { default { $null } } $sandboxInfo = if ([string]::IsNullOrWhiteSpace([string]$sandboxName)) { $null } else { Resolve-SandboxCommand -Name $sandboxName } + $versionObservation = $null if ($profile.Runner -ne 'codex') { $reasons.Add("execution-profile.json selects '$($profile.Runner)' rather than codex.") @@ -166,18 +218,41 @@ function Get-CodexPreflight { } else { $checks.Add((New-PreflightCheck -Name 'harness_executable' -Status passed -Detail $commandInfo.Source)) try { + $versionObservation = Get-ExternalCommandVersion -CommandInfo $commandInfo -WorkingDirectory $run.WorkingDirectoryPath -Environment (New-RunnerEnvironment -Run $run) -TimeoutSeconds 30 + if (-not $versionObservation.Available) { + $reasons.Add('The Codex CLI did not expose an exact observable version through --version.') + $checks.Add((New-PreflightCheck -Name 'harness_version' -Status unavailable -Detail 'codex --version did not return a usable version string.')) + } else { + $checks.Add((New-PreflightCheck -Name 'harness_version' -Status passed -Detail ([string]$versionObservation.Version))) + } + + $globalHelp = Get-CodexHelpResult -CommandInfo $commandInfo -Inputs $Inputs -Arguments @('--help') $help = Get-CodexHelpResult -CommandInfo $commandInfo -Inputs $Inputs + if ($globalHelp.TimedOut -or $globalHelp.ExitCode -ne 0) { + $reasons.Add("Codex --help failed with exit status $($globalHelp.ExitCode).") + } if ($help.TimedOut -or $help.ExitCode -ne 0) { - $reasons.Add("Codex exec --help failed with exit status $($help.ExitCode).") + $reasons.Add("Codex --ask-for-approval never exec --help failed with exit status $($help.ExitCode).") } else { - $helpText = [string]::Join("`n", @($help.Stdout, $help.Stderr)) - foreach ($flag in @('--ephemeral', '--ignore-user-config', '--ignore-rules', '--json', '--output-last-message', '--sandbox', '--approve-for-me')) { + $helpText = [string]::Join("`n", @($globalHelp.Stdout, $globalHelp.Stderr, $help.Stdout, $help.Stderr)) + foreach ($flag in @('--ask-for-approval', '--ephemeral', '--ignore-user-config', '--ignore-rules', '--json', '--output-last-message', '--sandbox', '--cd', '--model', '--config')) { if ($helpText -notmatch [regex]::Escape($flag)) { $reasons.Add("The installed Codex CLI does not advertise required flag '$flag'.") } } + $visiblePlatform = if ($platform -eq 'linux' -and $null -ne $sandboxInfo) { 'linux' } else { $platform } + $constructed = New-CodexCliArguments -Inputs $Inputs -LastResponsePath (Join-Path $run.RunRoot 'evidence/codex-final.txt') -VisiblePlatform $visiblePlatform + if (@($constructed) -contains '--approve-for-me') { + $reasons.Add('The constructed Codex invocation must not combine --approve-for-me with explicit --sandbox selection.') + } + $sandboxIndex = [Array]::IndexOf([string[]]$constructed, '--sandbox') + $approvalIndex = [Array]::IndexOf([string[]]$constructed, '--ask-for-approval') + $execIndex = [Array]::IndexOf([string[]]$constructed, 'exec') + if ($approvalIndex -lt 0 -or $execIndex -lt 0 -or $approvalIndex -gt $execIndex -or $sandboxIndex -lt 0) { + $reasons.Add('The constructed Codex invocation must set --ask-for-approval never before exec and retain --sandbox workspace-write.') + } if ($reasons.Count -eq 0) { - $checks.Add((New-PreflightCheck -Name 'harness_contract' -Status passed -Detail 'Codex exec advertises the required noninteractive, ephemeral, isolated, and structured-output flags.')) + $checks.Add((New-PreflightCheck -Name 'harness_contract' -Status passed -Detail 'Codex accepts the constructed noninteractive invocation: --ask-for-approval never, exec, --sandbox workspace-write, ephemeral JSON output, and isolated configuration controls.')) } } } catch { @@ -187,44 +262,38 @@ function Get-CodexPreflight { $auth = Get-CodexAuthSource -Provider ([string]$profile.Provider) if ($auth.Kind -eq 'missing') { - $reasons.Add('No narrow Codex authentication source is available (provider environment variable or CODEX_HOME/auth.json).') + $reasons.Add('No narrow Codex provider API-key environment variable is available.') + } elseif ($auth.Kind -eq 'file_unsupported') { + $reasons.Add('Codex auth.json cannot be copied into the worker HOME: the evaluated agent could read that credential file. Set the provider API-key environment variable instead.') } else { - $checks.Add((New-PreflightCheck -Name 'authentication' -Status passed -Detail "Authentication is available through $($auth.Kind) and will be injected into the isolated run only.")) + $checks.Add((New-PreflightCheck -Name 'authentication' -Status passed -Detail "Authentication is available through the narrow $($auth.Name) environment variable; the child shell policy is set to inherit=none.")) } - if ($platform -notin @('linux', 'macos')) { - $reasons.Add("Platform '$platform' is not supported by the Codex runner.") + if ($null -eq $sandboxName) { + $checks.Add((New-PreflightCheck -Name 'filesystem_confinement' -Status not_applicable -Detail "Platform '$platform' has no configured external hard-confinement mechanism; pragmatic isolation remains available.")) + $warnings.Add("Platform '$platform' has no external hard filesystem confinement in this adapter; execution will report pragmatic isolation.") } elseif ($null -eq $sandboxInfo) { - $reasons.Add("Required external filesystem sandbox '$sandboxName' is unavailable; Codex cannot prove package-boundary read confinement.") + $checks.Add((New-PreflightCheck -Name 'filesystem_confinement' -Status unavailable -Detail "External '$sandboxName' is unavailable; pragmatic isolation remains available.")) + $warnings.Add("External '$sandboxName' was unavailable; execution will report pragmatic isolation.") } else { $checks.Add((New-PreflightCheck -Name 'filesystem_confinement' -Status passed -Detail "External $sandboxName confines Codex to the staged run and required system runtime paths; Codex sandbox=workspace-write remains enabled inside it.")) } $checks.Add((New-PreflightCheck -Name 'fresh_session' -Status passed -Detail 'The adapter uses --ephemeral and never supplies a resume, continue, or session identifier.')) - $checks.Add((New-PreflightCheck -Name 'ambient_configuration' -Status passed -Detail 'The adapter uses an isolated CODEX_HOME plus --ignore-user-config and --ignore-rules.')) + $checks.Add((New-PreflightCheck -Name 'ambient_configuration' -Status passed -Detail 'The adapter uses an isolated CODEX_HOME plus --ignore-user-config and --ignore-rules; unrelated inherited environment variables are removed.')) $checks.Add((New-PreflightCheck -Name 'run_paths' -Status passed -Detail "--cd $($run.WorkingDirectoryPath); CODEX_HOME under $($run.HomeDirectoryPath)")) + $checks.Add((New-PreflightCheck -Name 'credential_boundary' -Status passed -Detail 'Only the selected provider API-key variable is passed to Codex; auth files are never copied into the worker HOME.')) - $capabilities = [ordered]@{} - foreach ($capabilityName in @(Get-JsonPropertyNames -Object $descriptor.capabilities)) { - $value = [string](Get-JsonProperty -Object $descriptor.capabilities -Name $capabilityName) - if ($capabilityName -eq 'filesystem_confinement' -and $platform -in @('linux', 'macos') -and $commandInfo -ne $null -and $sandboxInfo -ne $null) { - $value = 'supported' - } elseif ($capabilityName -eq 'filesystem_confinement') { - $value = 'unsupported' - } - $capabilities[$capabilityName] = $value - } - if ($platform -eq 'windows') { - $warnings.Add('The installed Codex CLI may run on Windows, but this runner does not claim the required package-level filesystem read boundary there.') - } elseif ($null -eq $sandboxInfo) { - $warnings.Add("The Codex CLI was inspected, but external sandbox '$sandboxName' was unavailable.") - } - - $harnessVersion = if ($null -eq $commandInfo) { 'unavailable' } else { 'available' } + $hardConfinement = $null -ne $sandboxInfo -and $platform -in @('linux', 'macos') + $capabilities = Get-CodexCapabilityMap -Inputs $Inputs -HardFilesystemConfinement $hardConfinement + $harnessVersion = if ($null -eq $versionObservation) { 'unavailable' } else { [string]$versionObservation.Version } $descriptorCopy = [ordered]@{} foreach ($key in $descriptor.Keys) { $descriptorCopy[$key] = $descriptor[$key] } $descriptorCopy.harness = [ordered]@{ name = 'OpenAI Codex CLI'; version = $harnessVersion } - return New-PreflightDocument -Descriptor $descriptorCopy -Profile $profile -Run $run -Compatible ($reasons.Count -eq 0) -Checks @($checks) -Mechanisms @('codex --ephemeral', '--ignore-user-config', '--ignore-rules', '--sandbox workspace-write', "external $sandboxName filesystem sandbox", 'isolated CODEX_HOME', 'prompt on stdin') -ResolvedCapabilities $capabilities -Warnings @($warnings) -Reasons @($reasons) + $mechanisms = [System.Collections.Generic.List[string]]::new() + foreach ($mechanism in @('--ask-for-approval never', 'codex exec --ephemeral', '--ignore-user-config', '--ignore-rules', '--sandbox workspace-write', 'shell_environment_policy.inherit=none', 'isolated CODEX_HOME', 'prompt on stdin', 'no session continuation')) { $mechanisms.Add($mechanism) } + if ($hardConfinement) { $mechanisms.Add("external $sandboxName filesystem sandbox") } else { $mechanisms.Add('pragmatic process/environment isolation without hard filesystem confinement') } + return New-PreflightDocument -Descriptor $descriptorCopy -Profile $profile -Run $run -Compatible ($reasons.Count -eq 0) -Checks @($checks) -Mechanisms @($mechanisms) -ResolvedCapabilities $capabilities -Warnings @($warnings) -Reasons @($reasons) } function New-CodexEnvironment { @@ -236,8 +305,8 @@ function New-CodexEnvironment { $codexHome = Join-Path $Inputs.Run.HomeDirectoryPath '.codex' New-Item -ItemType Directory -Path $codexHome -Force | Out-Null $environment = New-RunnerEnvironment -Run $Inputs.Run -AuthenticationVariables @(Get-ProviderAuthenticationVariables -Provider ([string]$Inputs.Profile.Provider)) -Additional @{ CODEX_HOME = $codexHome } - if ($Auth.Kind -eq 'file') { - Copy-Item -LiteralPath $Auth.Path -Destination (Join-Path $codexHome 'auth.json') -Force | Out-Null + if ($Auth.Kind -ne 'environment') { + throw 'Codex execution requires a provider environment credential; file credentials are not safe to expose in the worker HOME.' } return $environment } @@ -329,7 +398,7 @@ function Write-CodexCapture { param( [Parameter(Mandatory = $true)][object]$RunData, [Parameter(Mandatory = $true)][string]$RelativePath, - [Parameter(Mandatory = $true)][string]$Text + [Parameter(Mandatory = $true)][AllowEmptyString()][string]$Text ) $path = Join-Path $RunData.Run.RunRoot ($RelativePath -replace '/', [System.IO.Path]::DirectorySeparatorChar) @@ -345,45 +414,34 @@ function Invoke-CodexExecute { $preflight = Get-CodexPreflight -Inputs $Inputs $started = [DateTime]::UtcNow $sessionId = [Guid]::NewGuid().ToString('D') + $executionDescriptor = [ordered]@{} + foreach ($key in $descriptor.Keys) { $executionDescriptor[$key] = $descriptor[$key] } + $executionDescriptor.harness = $preflight.harness if ($preflight.status -ne 'compatible') { $finished = [DateTime]::UtcNow $failureText = [string]::Join('; ', @($preflight.reasons)) - return New-ExecutionResult -Descriptor $descriptor -Profile $Inputs.Profile -Run $Inputs.Run -Status incompatible -FinalResponseReason 'preflight_incompatible' -StartedUtc $started.ToString('o') -FinishedUtc $finished.ToString('o') -DurationSeconds ($finished - $started).TotalSeconds -Failure (New-ExecutionFailure -Code 'incompatible' -Message $failureText) -SessionId $sessionId -IsolationCapabilities ([ordered]@{ fresh_context = 'supported'; isolated_home_config = 'supported'; isolated_working_directory = 'supported'; filesystem_confinement = 'unsupported'; ambient_candidate_skill_exclusion = 'supported'; candidate_skill_exposure = 'supported' }) -IsolationMechanisms @('preflight-only') -Evidence ([ordered]@{ preflight = $preflight }) -AttemptCount 1 + return New-ExecutionResult -Descriptor $executionDescriptor -Profile $Inputs.Profile -Run $Inputs.Run -Status incompatible -FinalResponseReason 'preflight_incompatible' -StartedUtc $started.ToString('o') -FinishedUtc $finished.ToString('o') -DurationSeconds ($finished - $started).TotalSeconds -Failure (New-ExecutionFailure -Code 'incompatible' -Message $failureText) -SessionId $sessionId -IsolationCapabilities ([ordered]@{}) -IsolationMechanisms @('preflight-only') -Evidence ([ordered]@{ preflight = $preflight; resume = $false }) -AttemptCount 1 } $commandInfo = Resolve-ExternalCommand -Name 'codex' $auth = Get-CodexAuthSource -Provider ([string]$Inputs.Profile.Provider) $environment = New-CodexEnvironment -Inputs $Inputs -Auth $auth $lastResponsePath = 'evidence/codex-final.txt' - $directoryArgument = if ((Get-PlatformName) -eq 'linux') { '/run/repo' } else { $Inputs.Run.WorkingDirectoryPath } - $arguments = @( - 'exec', - '--ephemeral', - '--ignore-user-config', - '--ignore-rules', - '--skip-git-repo-check', - '--json', - '--color', 'never', - '--cd', $directoryArgument, - '--model', $Inputs.Profile.Model, - '--sandbox', 'workspace-write', - '--approve-for-me', - '--output-last-message', (Join-Path $Inputs.Run.RunRoot ($lastResponsePath -replace '/', [System.IO.Path]::DirectorySeparatorChar)) - ) - if (-not [string]::IsNullOrWhiteSpace([string]$Inputs.Profile.ReasoningEffort)) { - $arguments += @('-c', "model_reasoning_effort=$($Inputs.Profile.ReasoningEffort)") - } - $arguments += '-' + $platform = Get-PlatformName + $sandboxInfo = if ($platform -eq 'linux') { Resolve-SandboxCommand -Name 'bwrap' } elseif ($platform -eq 'macos') { Resolve-SandboxCommand -Name 'sandbox-exec' } else { $null } + $hardFilesystem = $null -ne $sandboxInfo -and $platform -in @('linux', 'macos') + $visiblePlatform = if ($hardFilesystem) { $platform } elseif ($platform -eq 'linux') { 'unknown' } else { $platform } + $arguments = New-CodexCliArguments -Inputs $Inputs -LastResponsePath (Join-Path $Inputs.Run.RunRoot ($lastResponsePath -replace '/', [System.IO.Path]::DirectorySeparatorChar)) -VisiblePlatform $visiblePlatform - if ((Get-PlatformName) -eq 'linux') { - $sandboxInfo = Resolve-SandboxCommand -Name 'bwrap' + if ($platform -eq 'linux' -and $hardFilesystem) { $sandboxArguments = Get-LinuxCodexSandboxArguments -Inputs $Inputs -CommandInfo $commandInfo -Environment $environment $process = Invoke-RunnerProcess -FileName $sandboxInfo.FileName -ArgumentList (@($sandboxArguments) + @($arguments)) -WorkingDirectory $Inputs.Run.WorkingDirectoryPath -Environment $environment -InputBytes $Inputs.Run.PromptBytes -TimeoutSeconds $Inputs.Profile.TimeoutSeconds - } else { - $sandboxInfo = Resolve-SandboxCommand -Name 'sandbox-exec' + } elseif ($platform -eq 'macos' -and $hardFilesystem) { $sandboxProfile = New-CodexMacosSandboxProfile -Inputs $Inputs -CommandInfo $commandInfo $sandboxArguments = @('-f', $sandboxProfile, '--', $commandInfo.FileName) + @($commandInfo.Prefix) + @($arguments) $process = Invoke-RunnerProcess -FileName $sandboxInfo.FileName -ArgumentList $sandboxArguments -WorkingDirectory $Inputs.Run.WorkingDirectoryPath -Environment $environment -InputBytes $Inputs.Run.PromptBytes -TimeoutSeconds $Inputs.Profile.TimeoutSeconds + } else { + $process = Invoke-CodexCli -CommandInfo $commandInfo -Arguments $arguments -Inputs $Inputs -Environment $environment -InputBytes $Inputs.Run.PromptBytes -TimeoutSeconds $Inputs.Profile.TimeoutSeconds } $stdoutArtifact = Write-CodexCapture -RunData $Inputs -RelativePath 'evidence/codex-events.jsonl' -Text $process.Stdout $stderrArtifact = Write-CodexCapture -RunData $Inputs -RelativePath 'evidence/codex-stderr.txt' -Text $process.Stderr @@ -477,13 +535,27 @@ function Invoke-CodexExecute { cost = New-UnavailableMetric -Reason 'codex_runner_does_not_estimate_cost' } $finished = [DateTime]::UtcNow - return New-ExecutionResult -Descriptor $descriptor -Profile $Inputs.Profile -Run $Inputs.Run -Status $status -FinalResponse $finalText -FinalResponseReason $reason -StartedUtc $process.StartedUtc.ToString('o') -FinishedUtc $finished.ToString('o') -DurationSeconds $process.DurationSeconds -ExitStatus $exitStatus -Failure $failure -SessionId (if ([string]::IsNullOrWhiteSpace($threadId)) { $sessionId } else { $threadId }) -IsolationCapabilities ([ordered]@{ fresh_context = 'supported'; isolated_home_config = 'supported'; isolated_working_directory = 'supported'; filesystem_confinement = 'supported'; ambient_candidate_skill_exclusion = 'supported'; candidate_skill_exposure = 'supported'; prompt_fidelity = 'supported'; model_configuration_lock = 'supported'; response_capture = 'supported' }) -IsolationMechanisms @('codex --ephemeral', '--ignore-user-config', '--ignore-rules', '--sandbox workspace-write', 'external bwrap/sandbox-exec filesystem sandbox', 'isolated CODEX_HOME', 'prompt on stdin') -Telemetry $telemetry -Artifacts @($artifacts) -Warnings @($warnings) -Evidence ([ordered]@{ thread_id = $threadId; event_counts = $eventCounts; commands = @($commands); files = @($files); prompt_first_input = $true; resume = $false; stdout_exit_code = $process.ExitCode; sandbox = if ((Get-PlatformName) -eq 'linux') { 'bwrap' } else { 'sandbox-exec' } }) -AttemptCount 1 + $sessionResultId = if ([string]::IsNullOrWhiteSpace($threadId)) { $sessionId } else { $threadId } + $capabilities = Get-CodexCapabilityMap -Inputs $Inputs -HardFilesystemConfinement $hardFilesystem + $mechanisms = [System.Collections.Generic.List[string]]::new() + foreach ($mechanism in @('--ask-for-approval never', 'codex exec --ephemeral', '--ignore-user-config', '--ignore-rules', '--sandbox workspace-write', 'shell_environment_policy.inherit=none', 'isolated CODEX_HOME', 'prompt on stdin', 'no session continuation')) { $mechanisms.Add($mechanism) } + if ($hardFilesystem) { $mechanisms.Add("external $($sandboxInfo.Source) filesystem sandbox") } else { $mechanisms.Add('pragmatic process/environment isolation without hard filesystem confinement') } + if (-not $hardFilesystem) { $warnings.Add('Hard filesystem confinement was unavailable; the completed arm is reported as pragmatic isolation.') } + $sandboxEvidence = if (-not $hardFilesystem) { 'unavailable' } elseif ($platform -eq 'linux') { 'bwrap' } else { 'sandbox-exec' } + $credentialEvidence = [ordered]@{ + source = $auth.Kind + provider_environment_variable = $auth.Name + unrelated_environment_excluded = $true + child_tool_visibility = 'codex_shell_environment_policy_inherit_none' + value_observed = $false + } + return New-ExecutionResult -Descriptor $executionDescriptor -Profile $Inputs.Profile -Run $Inputs.Run -Status $status -FinalResponse $finalText -FinalResponseReason $reason -StartedUtc $process.StartedUtc.ToString('o') -FinishedUtc $finished.ToString('o') -DurationSeconds $process.DurationSeconds -ExitStatus $exitStatus -Failure $failure -SessionId $sessionResultId -IsolationCapabilities $capabilities -IsolationMechanisms @($mechanisms) -ResolvedConfiguration ([ordered]@{ status = 'accepted_request'; reason = 'Codex accepted the requested provider, model, and configuration but did not expose concrete backend resolution.'; observations = [ordered]@{ provider = $Inputs.Profile.Provider; model = $Inputs.Profile.Model; reasoning_effort = $Inputs.Profile.ReasoningEffort } }) -Telemetry $telemetry -Artifacts @($artifacts) -Warnings @($warnings) -Evidence ([ordered]@{ thread_id = $threadId; event_counts = $eventCounts; commands = @($commands); files = @($files); prompt_first_input = $true; resume = $false; stdout_exit_code = $process.ExitCode; sandbox = $sandboxEvidence; output_last_message_argument = (Get-SandboxVisiblePath -HostPath (Join-Path $Inputs.Run.RunRoot ($lastResponsePath -replace '/', [System.IO.Path]::DirectorySeparatorChar)) -RunRoot $Inputs.Run.RunRoot -Platform $visiblePlatform); credential = $credentialEvidence }) -AttemptCount 1 } try { [void](Assert-RunnerDescriptor -Descriptor $descriptor) switch ($Command) { - 'describe' { Write-RunnerJson -Value $descriptor -AsOutput } + 'describe' { Write-RunnerJson -Value (Get-CodexDescriptor) -AsOutput } 'preflight' { $inputs = Resolve-CodexInputs Write-RunnerJson -Value (Get-CodexPreflight -Inputs $inputs) -AsOutput diff --git a/scripts/eval-runners/contracts/execution-result.schema.json b/scripts/eval-runners/contracts/execution-result.schema.json index 8cce4eb..258133c 100644 --- a/scripts/eval-runners/contracts/execution-result.schema.json +++ b/scripts/eval-runners/contracts/execution-result.schema.json @@ -52,12 +52,37 @@ "reason": { "type": "string" } } }, + "requested": { "type": "object" }, + "resolved": { + "type": "object", + "properties": { + "provider": { "type": ["string", "null"] }, + "model": { "type": ["string", "null"] }, + "reasoning_effort": { "type": ["string", "null"] }, + "configuration_profile": { "type": ["string", "null"] }, + "tool_profile": { "type": ["string", "null"] }, + "status": { "enum": ["unavailable", "accepted_request", "resolved"] }, + "accepted": { "type": "object" } + } + }, "duration_seconds": { "type": "number", "minimum": 0 }, "input": { "type": "object", "required": ["prompt_sha256", "run_json_sha256", "profile_sha256"] }, - "isolation": { "type": "object" }, + "isolation": { + "type": "object", + "required": ["status", "level", "hard_filesystem_confinement", "capabilities", "mechanisms"], + "properties": { + "status": { "enum": ["verified", "unverified"] }, + "level": { "enum": ["strict", "pragmatic", "unsupported"] }, + "hard_filesystem_confinement": { "type": "boolean" }, + "capabilities": { "type": "object" }, + "mechanisms": { "type": "array", "items": { "type": "string" } }, + "required_controls": { "type": "array", "items": { "type": "string" } }, + "unproven_controls": { "type": "array", "items": { "type": "string" } } + } + }, "telemetry": { "type": "object" }, "evidence": { "type": "object" }, "artifacts": { "type": "array" }, diff --git a/scripts/eval-runners/contracts/preflight-result.schema.json b/scripts/eval-runners/contracts/preflight-result.schema.json index a3c3126..c3351ca 100644 --- a/scripts/eval-runners/contracts/preflight-result.schema.json +++ b/scripts/eval-runners/contracts/preflight-result.schema.json @@ -14,6 +14,7 @@ "requested", "checks", "resolved_capabilities", + "isolation", "mechanisms", "warnings", "reasons" @@ -24,6 +25,16 @@ "status": { "enum": ["compatible", "incompatible"] }, "checks": { "type": "array" }, "resolved_capabilities": { "type": "object" }, + "isolation": { + "type": "object", + "required": ["level", "status", "hard_filesystem_confinement", "unproven_controls"], + "properties": { + "level": { "enum": ["strict", "pragmatic", "unsupported"] }, + "status": { "enum": ["verified", "unverified"] }, + "hard_filesystem_confinement": { "type": "boolean" }, + "unproven_controls": { "type": "array", "items": { "type": "string" } } + } + }, "mechanisms": { "type": "array", "items": { "type": "string" } }, "warnings": { "type": "array", "items": { "type": "string" } }, "reasons": { "type": "array", "items": { "type": "string" } } diff --git a/scripts/eval-runners/fake/runner.ps1 b/scripts/eval-runners/fake/runner.ps1 index 7ced67a..9453c56 100644 --- a/scripts/eval-runners/fake/runner.ps1 +++ b/scripts/eval-runners/fake/runner.ps1 @@ -121,9 +121,10 @@ function Get-FakePreflight { } $capabilities = [ordered]@{} - foreach ($property in @($descriptor.capabilities.PSObject.Properties)) { - $capabilities[$property.Name] = $property.Value + foreach ($capabilityName in @(Get-JsonPropertyNames -Object $descriptor.capabilities)) { + $capabilities[$capabilityName] = Get-JsonProperty -Object $descriptor.capabilities -Name $capabilityName } + $capabilities['candidate_skill_exposure'] = if ($run.CandidateSkillExposed) { 'supported' } else { 'excluded' } if ($reasons.Count -gt 0) { $warnings.Add('No execute process is started for an incompatible preflight.') } diff --git a/scripts/eval-runners/freebuff-readiness.md b/scripts/eval-runners/freebuff-readiness.md new file mode 100644 index 0000000..e836068 --- /dev/null +++ b/scripts/eval-runners/freebuff-readiness.md @@ -0,0 +1,34 @@ +# Freebuff runner readiness + +Status: planned and blocked. No Freebuff Eval Runner is shipped or advertised +until the official Freebuff CLI provides a supported noninteractive transport. + +The current upstream [Freebuff README](https://github.com/CodebuffAI/freebuff/blob/main/README.md) +documents the `freebuff` TUI. The current upstream +[headless CLI request](https://github.com/CodebuffAI/freebuff/issues/947) asks +for a print/headless mode with machine-readable output; it is evidence that +the required transport is not currently part of the supported CLI contract. +The locally installed CLI was also inspected without a model request: +`freebuff --version` reported `0.0.150`, and its help exposed `login`, +`--continue`, `--cwd`, and `--version`, but no prompt argument, print mode, +JSON/NDJSON output mode, or supported noninteractive session protocol. + +The missing capability is therefore the complete one-prompt-in, +machine-readable-result-out, fresh-session transport. TUI keystroke +automation, screen scraping, PTY emulation, private transport use, and the +paid Codebuff SDK are not substitutes for that capability. + +Freebuff becomes ready only when the supported official CLI can demonstrate all +of the following without a human TTY: + +1. Accept exactly one task/prompt deterministically. +2. Start a fresh independent session without resume or reuse. +3. Select or identify the requested model and configuration. +4. Return the complete final result and any available structured events. +5. Expose a timeout and an enforceable run-local HOME/config/workspace setup. +6. Permit the common Eval Runner contract to keep grading material and paired + arm data out of the worker. + +At that point a runner may be added under `freebuff/runner.ps1` using the +unchanged `describe`, `preflight`, and `execute` protocol. Until then, the +absence of a runner is intentional. diff --git a/scripts/eval-runners/opencode/runner.ps1 b/scripts/eval-runners/opencode/runner.ps1 index b1fe8d8..ab2971e 100644 --- a/scripts/eval-runners/opencode/runner.ps1 +++ b/scripts/eval-runners/opencode/runner.ps1 @@ -3,8 +3,9 @@ OpenCode Eval Runner adapter. .DESCRIPTION - This is the only place where OpenCode CLI flags, pure configuration, - sandbox process setup, and JSON event parsing are defined. + This is the only place where OpenCode CLI flags, project/global + configuration handling, sandbox process setup, and JSON event parsing are + defined. #> [CmdletBinding()] param( @@ -25,8 +26,8 @@ $descriptor = [ordered]@{ protocol_version = (Get-RunnerSchemaNames).Protocol name = 'opencode' version = '0.9.1' - platforms = @('linux', 'macos') - harness = [ordered]@{ name = 'OpenCode CLI'; version = 'current-supported' } + platforms = @('windows', 'linux', 'macos') + harness = [ordered]@{ name = 'OpenCode CLI'; version = 'unavailable' } capabilities = [ordered]@{ fresh_context = 'supported' isolated_home_config = 'supported' @@ -44,6 +45,7 @@ $descriptor = [ordered]@{ command_evidence = 'conditional' file_evidence = 'conditional' cost_telemetry = 'conditional' + credential_child_filtering = 'conditional' native_skill_activation_evidence = 'unsupported' } supported_telemetry = @('transcript_event_capture', 'token_telemetry', 'cache_token_telemetry', 'tool_call_telemetry', 'command_evidence', 'file_evidence', 'cost_telemetry') @@ -110,6 +112,53 @@ function Resolve-SandboxCommand { return Resolve-ExternalCommand -Name $Name } +function Get-OpenCodeDescriptor { + $copy = [ordered]@{} + foreach ($key in $descriptor.Keys) { $copy[$key] = $descriptor[$key] } + $commandInfo = Resolve-ExternalCommand -Name 'opencode' + $version = 'unavailable' + if ($null -ne $commandInfo) { + $observation = Get-ExternalCommandVersion -CommandInfo $commandInfo + $version = [string]$observation.Version + } + $copy.harness = [ordered]@{ name = 'OpenCode CLI'; version = $version } + return $copy +} + +function New-OpenCodeCliArguments { + param( + [Parameter(Mandatory = $true)][object]$Inputs, + [ValidateSet('windows', 'linux', 'macos', 'unknown')][string]$VisiblePlatform = (Get-PlatformName) + ) + + $directoryArgument = Get-SandboxVisiblePath -HostPath $Inputs.Run.WorkingDirectoryPath -RunRoot $Inputs.Run.RunRoot -Platform $VisiblePlatform + $model = "{0}/{1}" -f $Inputs.Profile.Provider, $Inputs.Profile.Model + $arguments = [System.Collections.Generic.List[string]]::new() + foreach ($argument in @('run', '--format', 'json', '--dir', $directoryArgument, '--model', $model, '--auto')) { + $arguments.Add([string]$argument) + } + if (-not [string]::IsNullOrWhiteSpace([string]$Inputs.Profile.ReasoningEffort)) { + $arguments.Add('--variant') + $arguments.Add([string]$Inputs.Profile.ReasoningEffort) + } + return @($arguments) +} + +function Get-OpenCodeCapabilityMap { + param( + [Parameter(Mandatory = $true)][object]$Inputs, + [bool]$HardFilesystemConfinement = $false + ) + + $capabilities = [ordered]@{} + foreach ($capabilityName in @(Get-JsonPropertyNames -Object $descriptor.capabilities)) { + $capabilities[$capabilityName] = [string](Get-JsonProperty -Object $descriptor.capabilities -Name $capabilityName) + } + $capabilities['filesystem_confinement'] = if ($HardFilesystemConfinement) { 'supported' } else { 'unsupported' } + $capabilities['candidate_skill_exposure'] = if ($Inputs.Run.CandidateSkillExposed) { 'supported' } else { 'excluded' } + return $capabilities +} + function Get-OpenCodePreflight { param([Parameter(Mandatory = $true)][object]$Inputs) @@ -121,6 +170,7 @@ function Get-OpenCodePreflight { $platform = Get-PlatformName $commandInfo = Resolve-ExternalCommand -Name 'opencode' $sandboxInfo = if ($platform -eq 'linux') { Resolve-SandboxCommand -Name 'bwrap' } elseif ($platform -eq 'macos') { Resolve-SandboxCommand -Name 'sandbox-exec' } else { $null } + $versionObservation = $null if ($profile.Runner -ne 'opencode') { $reasons.Add("execution-profile.json selects '$($profile.Runner)' rather than opencode.") @@ -148,18 +198,30 @@ function Get-OpenCodePreflight { } else { $checks.Add((New-PreflightCheck -Name 'harness_executable' -Status passed -Detail $commandInfo.Source)) try { + $versionObservation = Get-ExternalCommandVersion -CommandInfo $commandInfo -WorkingDirectory $run.WorkingDirectoryPath -Environment (New-RunnerEnvironment -Run $run) -TimeoutSeconds 30 + if (-not $versionObservation.Available) { + $reasons.Add('The OpenCode CLI did not expose an exact observable version through --version.') + $checks.Add((New-PreflightCheck -Name 'harness_version' -Status unavailable -Detail 'opencode --version did not return a usable version string.')) + } else { + $checks.Add((New-PreflightCheck -Name 'harness_version' -Status passed -Detail ([string]$versionObservation.Version))) + } $help = Get-OpenCodeHelpResult -CommandInfo $commandInfo -Inputs $Inputs if ($help.TimedOut -or $help.ExitCode -ne 0) { $reasons.Add("OpenCode run --help failed with exit status $($help.ExitCode).") } else { $helpText = [string]::Join("`n", @($help.Stdout, $help.Stderr)) - foreach ($flag in @('--pure', '--format', '--dir', '--model', '--auto')) { + foreach ($flag in @('--format', '--dir', '--model', '--auto')) { if ($helpText -notmatch [regex]::Escape($flag)) { $reasons.Add("The installed OpenCode CLI does not advertise required flag '$flag'.") } } + $visiblePlatform = if ($platform -eq 'linux' -and $null -ne $sandboxInfo) { 'linux' } else { $platform } + $constructed = New-OpenCodeCliArguments -Inputs $Inputs -VisiblePlatform $visiblePlatform + foreach ($forbidden in @('--pure', '--continue', '--session')) { + if (@($constructed) -contains $forbidden) { $reasons.Add("The constructed OpenCode invocation must not use session or project-suppression option '$forbidden'.") } + } if ($reasons.Count -eq 0) { - $checks.Add((New-PreflightCheck -Name 'harness_contract' -Status passed -Detail 'OpenCode run advertises pure, noninteractive, model, directory, and structured-output controls.')) + $checks.Add((New-PreflightCheck -Name 'harness_contract' -Status passed -Detail 'OpenCode run advertises noninteractive, model, directory, and structured-output controls; the adapter intentionally does not use --pure.')) } } } catch { @@ -175,34 +237,33 @@ function Get-OpenCodePreflight { } if ($platform -notin @('linux', 'macos')) { - $reasons.Add("Platform '$platform' has no v0.9.1 OpenCode filesystem sandbox implementation.") + $checks.Add((New-PreflightCheck -Name 'filesystem_confinement' -Status not_applicable -Detail "Platform '$platform' has no configured external hard-confinement mechanism; pragmatic isolation remains available.")) + $warnings.Add("Platform '$platform' has no external hard filesystem confinement in this adapter; execution will report pragmatic isolation.") } elseif ($null -eq $sandboxInfo) { - $reasons.Add("Required $([string]$(if ($platform -eq 'linux') { 'bwrap' } else { 'sandbox-exec' })) isolation command is unavailable.") + $sandboxName = if ($platform -eq 'linux') { 'bwrap' } else { 'sandbox-exec' } + $checks.Add((New-PreflightCheck -Name 'filesystem_confinement' -Status unavailable -Detail "External '$sandboxName' is unavailable; pragmatic isolation remains available.")) + $warnings.Add("External '$sandboxName' was unavailable; execution will report pragmatic isolation.") } else { $checks.Add((New-PreflightCheck -Name 'filesystem_confinement' -Status passed -Detail "External $($sandboxInfo.Source) sandbox confines the process to the staged run and required system runtime paths.")) } $checks.Add((New-PreflightCheck -Name 'fresh_session' -Status passed -Detail 'The adapter starts one new opencode run process and supplies no resume, continue, or session id.')) - $checks.Add((New-PreflightCheck -Name 'ambient_configuration' -Status passed -Detail 'The adapter uses --pure and isolated OpenCode configuration roots.')) + $checks.Add((New-PreflightCheck -Name 'ambient_configuration' -Status passed -Detail 'The adapter isolates global/user configuration roots and deliberately preserves repository-owned project configuration; OPENCODE_DISABLE_PROJECT_CONFIG is not used.')) $checks.Add((New-PreflightCheck -Name 'prompt_fidelity' -Status passed -Detail 'The exact prompt bytes are sent on stdin as the first and only task input.')) + $warnings.Add('OpenCode does not expose a supported child-tool environment filter in this CLI contract; the runner removes unrelated inherited variables but cannot independently prove that the selected provider credential is hidden from every OpenCode-launched tool.') - $capabilities = [ordered]@{} - foreach ($capabilityName in @(Get-JsonPropertyNames -Object $descriptor.capabilities)) { - $value = [string](Get-JsonProperty -Object $descriptor.capabilities -Name $capabilityName) - if ($capabilityName -eq 'filesystem_confinement' -and $null -ne $sandboxInfo -and $platform -in @('linux', 'macos') -and $reasons.Count -eq 0) { - $value = 'supported' - } elseif ($capabilityName -eq 'filesystem_confinement') { - $value = 'unsupported' - } - $capabilities[$capabilityName] = $value - } + $hardConfinement = $null -ne $sandboxInfo -and $platform -in @('linux', 'macos') + $capabilities = Get-OpenCodeCapabilityMap -Inputs $Inputs -HardFilesystemConfinement $hardConfinement if ($platform -eq 'macos') { $warnings.Add('macOS sandbox-exec is deprecated by Apple but is used only when present; a future runner revision may replace it with an equivalent supported mechanism.') } - $harnessVersion = if ($null -eq $commandInfo) { 'unavailable' } else { 'available' } + $harnessVersion = if ($null -eq $versionObservation) { 'unavailable' } else { [string]$versionObservation.Version } $descriptorCopy = [ordered]@{} foreach ($key in $descriptor.Keys) { $descriptorCopy[$key] = $descriptor[$key] } $descriptorCopy.harness = [ordered]@{ name = 'OpenCode CLI'; version = $harnessVersion } - return New-PreflightDocument -Descriptor $descriptorCopy -Profile $profile -Run $run -Compatible ($reasons.Count -eq 0) -Checks @($checks) -Mechanisms @('--pure', 'isolated OPENCODE_CONFIG_DIR', 'external filesystem sandbox', 'prompt on stdin', 'no session continuation') -ResolvedCapabilities $capabilities -Warnings @($warnings) -Reasons @($reasons) + $mechanisms = [System.Collections.Generic.List[string]]::new() + foreach ($mechanism in @('opencode run --format json', '--auto', 'isolated OPENCODE_CONFIG_DIR', 'isolated OPENCODE_CONFIG', 'isolated HOME/XDG roots', 'repository-owned project configuration preserved', 'prompt on stdin', 'no session continuation')) { $mechanisms.Add($mechanism) } + if ($hardConfinement) { $mechanisms.Add("external $($sandboxInfo.Source) filesystem sandbox") } else { $mechanisms.Add('pragmatic process/environment isolation without hard filesystem confinement') } + return New-PreflightDocument -Descriptor $descriptorCopy -Profile $profile -Run $run -Compatible ($reasons.Count -eq 0) -Checks @($checks) -Mechanisms @($mechanisms) -ResolvedCapabilities $capabilities -Warnings @($warnings) -Reasons @($reasons) } function New-OpenCodeEnvironment { @@ -216,7 +277,6 @@ function New-OpenCodeEnvironment { OPENCODE_CONFIG_DIR = $configDirectory OPENCODE_CONFIG = $configPath OPENCODE_DISABLE_AUTOUPDATE = '1' - OPENCODE_DISABLE_PROJECT_CONFIG = '1' } } @@ -248,6 +308,7 @@ function Get-LinuxSandboxArguments { $args.Add('--chdir'); $args.Add('/run/repo') $insideEnvironment = [ordered]@{ HOME = '/run/home' + USERPROFILE = '/run/home' XDG_CONFIG_HOME = '/run/home/.config' XDG_DATA_HOME = '/run/home/.local/share' XDG_CACHE_HOME = '/run/home/.cache' @@ -256,7 +317,6 @@ function Get-LinuxSandboxArguments { OPENCODE_CONFIG_DIR = '/run/home/opencode-config' OPENCODE_CONFIG = '/run/home/opencode-config/opencode.json' OPENCODE_DISABLE_AUTOUPDATE = [string]$Environment['OPENCODE_DISABLE_AUTOUPDATE'] - OPENCODE_DISABLE_PROJECT_CONFIG = [string]$Environment['OPENCODE_DISABLE_PROJECT_CONFIG'] PATH = '/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin' CI = '1' NO_COLOR = '1' @@ -309,7 +369,7 @@ function Write-OpenCodeCapture { param( [Parameter(Mandatory = $true)][object]$RunData, [Parameter(Mandatory = $true)][string]$RelativePath, - [Parameter(Mandatory = $true)][string]$Text + [Parameter(Mandatory = $true)][AllowEmptyString()][string]$Text ) $path = Join-Path $RunData.Run.RunRoot ($RelativePath -replace '/', [System.IO.Path]::DirectorySeparatorChar) @@ -324,29 +384,32 @@ function Invoke-OpenCodeExecute { $preflight = Get-OpenCodePreflight -Inputs $Inputs $started = [DateTime]::UtcNow $sessionId = [Guid]::NewGuid().ToString('D') + $executionDescriptor = [ordered]@{} + foreach ($key in $descriptor.Keys) { $executionDescriptor[$key] = $descriptor[$key] } + $executionDescriptor.harness = $preflight.harness if ($preflight.status -ne 'compatible') { $finished = [DateTime]::UtcNow - return New-ExecutionResult -Descriptor $descriptor -Profile $Inputs.Profile -Run $Inputs.Run -Status incompatible -FinalResponseReason 'preflight_incompatible' -StartedUtc $started.ToString('o') -FinishedUtc $finished.ToString('o') -DurationSeconds ($finished - $started).TotalSeconds -Failure (New-ExecutionFailure -Code 'incompatible' -Message ([string]::Join('; ', @($preflight.reasons)))) -SessionId $sessionId -IsolationCapabilities ([ordered]@{ fresh_context = 'supported'; isolated_home_config = 'supported'; isolated_working_directory = 'supported'; filesystem_confinement = 'unsupported'; ambient_candidate_skill_exclusion = 'supported'; candidate_skill_exposure = 'supported' }) -IsolationMechanisms @('preflight-only') -Evidence ([ordered]@{ preflight = $preflight }) -AttemptCount 1 + return New-ExecutionResult -Descriptor $executionDescriptor -Profile $Inputs.Profile -Run $Inputs.Run -Status incompatible -FinalResponseReason 'preflight_incompatible' -StartedUtc $started.ToString('o') -FinishedUtc $finished.ToString('o') -DurationSeconds ($finished - $started).TotalSeconds -Failure (New-ExecutionFailure -Code 'incompatible' -Message ([string]::Join('; ', @($preflight.reasons)))) -SessionId $sessionId -IsolationCapabilities ([ordered]@{}) -IsolationMechanisms @('preflight-only') -Evidence ([ordered]@{ preflight = $preflight; resume = $false }) -AttemptCount 1 } $commandInfo = Resolve-ExternalCommand -Name 'opencode' $environment = New-OpenCodeEnvironment -Inputs $Inputs + $platform = Get-PlatformName + $sandboxInfo = if ($platform -eq 'linux') { Resolve-SandboxCommand -Name 'bwrap' } elseif ($platform -eq 'macos') { Resolve-SandboxCommand -Name 'sandbox-exec' } else { $null } + $hardFilesystem = $null -ne $sandboxInfo -and $platform -in @('linux', 'macos') + $visiblePlatform = if ($hardFilesystem) { $platform } elseif ($platform -eq 'linux') { 'unknown' } else { $platform } $model = "{0}/{1}" -f $Inputs.Profile.Provider, $Inputs.Profile.Model - $directoryArgument = if ((Get-PlatformName) -eq 'linux') { '/run/repo' } else { $Inputs.Run.WorkingDirectoryPath } - $arguments = @('run', '--format', 'json', '--pure', '--dir', $directoryArgument, '--model', $model, '--auto') - if (-not [string]::IsNullOrWhiteSpace([string]$Inputs.Profile.ReasoningEffort)) { - $arguments += @('--variant', $Inputs.Profile.ReasoningEffort) - } + $arguments = New-OpenCodeCliArguments -Inputs $Inputs -VisiblePlatform $visiblePlatform - if ((Get-PlatformName) -eq 'linux') { - $sandboxInfo = Resolve-SandboxCommand -Name 'bwrap' + if ($platform -eq 'linux' -and $hardFilesystem) { $sandboxArguments = Get-LinuxSandboxArguments -Inputs $Inputs -CommandInfo $commandInfo -Environment $environment $process = Invoke-RunnerProcess -FileName $sandboxInfo.FileName -ArgumentList (@($sandboxArguments) + @($arguments)) -WorkingDirectory $Inputs.Run.WorkingDirectoryPath -Environment $environment -InputBytes $Inputs.Run.PromptBytes -TimeoutSeconds $Inputs.Profile.TimeoutSeconds - } else { - $sandboxInfo = Resolve-SandboxCommand -Name 'sandbox-exec' + } elseif ($platform -eq 'macos' -and $hardFilesystem) { $sandboxProfile = New-MacosSandboxProfile -Inputs $Inputs -CommandInfo $commandInfo $sandboxArguments = @('-f', $sandboxProfile, '--', $commandInfo.FileName) + @($commandInfo.Prefix) + $arguments $process = Invoke-RunnerProcess -FileName $sandboxInfo.FileName -ArgumentList $sandboxArguments -WorkingDirectory $Inputs.Run.WorkingDirectoryPath -Environment $environment -InputBytes $Inputs.Run.PromptBytes -TimeoutSeconds $Inputs.Profile.TimeoutSeconds + } else { + $process = Invoke-OpenCodeCli -CommandInfo $commandInfo -Arguments $arguments -Inputs $Inputs -Environment $environment -InputBytes $Inputs.Run.PromptBytes -TimeoutSeconds $Inputs.Profile.TimeoutSeconds } $stdoutArtifact = Write-OpenCodeCapture -RunData $Inputs -RelativePath 'evidence/opencode-events.jsonl' -Text $process.Stdout @@ -417,13 +480,25 @@ function Invoke-OpenCodeExecute { tool_calls = New-AvailableMetric -Value $toolCalls cost = if ($usageBuckets.Contains('cost')) { New-AvailableMetric -Value $usageBuckets['cost'] } else { New-UnavailableMetric -Reason 'opencode_did_not_expose_cost' } } - return New-ExecutionResult -Descriptor $descriptor -Profile $Inputs.Profile -Run $Inputs.Run -Status $status -FinalResponse $finalText -FinalResponseReason $reason -StartedUtc $process.StartedUtc.ToString('o') -FinishedUtc $process.FinishedUtc.ToString('o') -DurationSeconds $process.DurationSeconds -ExitStatus $exitStatus -Failure $failure -SessionId $sessionId -IsolationCapabilities ([ordered]@{ fresh_context = 'supported'; isolated_home_config = 'supported'; isolated_working_directory = 'supported'; filesystem_confinement = 'supported'; ambient_candidate_skill_exclusion = 'supported'; candidate_skill_exposure = 'supported'; prompt_fidelity = 'supported'; model_configuration_lock = 'supported'; response_capture = 'supported' }) -IsolationMechanisms @('--pure', 'isolated OPENCODE_CONFIG_DIR', 'bwrap/sandbox-exec', 'prompt on stdin', 'no session continuation') -Telemetry $telemetry -Artifacts @($artifacts) -Warnings @($warnings) -Evidence ([ordered]@{ event_counts = $eventCounts; commands = @($commands); prompt_first_input = $true; resume = $false; model_argument = $model; sandbox = if ((Get-PlatformName) -eq 'linux') { 'bwrap' } else { 'sandbox-exec' } }) -AttemptCount 1 + $capabilities = Get-OpenCodeCapabilityMap -Inputs $Inputs -HardFilesystemConfinement $hardFilesystem + $mechanisms = [System.Collections.Generic.List[string]]::new() + foreach ($mechanism in @('opencode run --format json', '--auto', 'isolated OPENCODE_CONFIG_DIR', 'isolated OPENCODE_CONFIG', 'isolated HOME/XDG roots', 'repository-owned project configuration preserved', 'prompt on stdin', 'no session continuation')) { $mechanisms.Add($mechanism) } + if ($hardFilesystem) { $mechanisms.Add("external $($sandboxInfo.Source) filesystem sandbox") } else { $mechanisms.Add('pragmatic process/environment isolation without hard filesystem confinement'); $warnings.Add('Hard filesystem confinement was unavailable; the completed arm is reported as pragmatic isolation.') } + $sandboxEvidence = if (-not $hardFilesystem) { 'unavailable' } elseif ($platform -eq 'linux') { 'bwrap' } else { 'sandbox-exec' } + $credentialNames = @(Get-ProviderAuthenticationVariables -Provider ([string]$Inputs.Profile.Provider)) + $credentialEvidence = [ordered]@{ + provider_environment_variables = $credentialNames + unrelated_environment_excluded = $true + child_tool_visibility = 'provider_credential_may_be_visible_to_native_child_tools; no supported child filter is exposed' + value_observed = $false + } + return New-ExecutionResult -Descriptor $executionDescriptor -Profile $Inputs.Profile -Run $Inputs.Run -Status $status -FinalResponse $finalText -FinalResponseReason $reason -StartedUtc $process.StartedUtc.ToString('o') -FinishedUtc $process.FinishedUtc.ToString('o') -DurationSeconds $process.DurationSeconds -ExitStatus $exitStatus -Failure $failure -SessionId $sessionId -IsolationCapabilities $capabilities -IsolationMechanisms @($mechanisms) -ResolvedConfiguration ([ordered]@{ status = 'accepted_request'; reason = 'OpenCode accepted the requested provider, model, and configuration but did not expose concrete backend resolution.'; observations = [ordered]@{ provider = $Inputs.Profile.Provider; model = $Inputs.Profile.Model; reasoning_effort = $Inputs.Profile.ReasoningEffort } }) -Telemetry $telemetry -Artifacts @($artifacts) -Warnings @($warnings) -Evidence ([ordered]@{ event_counts = $eventCounts; commands = @($commands); prompt_first_input = $true; resume = $false; model_argument = $model; sandbox = $sandboxEvidence; project_configuration = 'repository_owned_project_config_preserved'; disable_project_config_environment = $false; credential = $credentialEvidence }) -AttemptCount 1 } try { [void](Assert-RunnerDescriptor -Descriptor $descriptor) switch ($Command) { - 'describe' { Write-RunnerJson -Value $descriptor -AsOutput } + 'describe' { Write-RunnerJson -Value (Get-OpenCodeDescriptor) -AsOutput } 'preflight' { $inputs = Resolve-OpenCodeInputs Write-RunnerJson -Value (Get-OpenCodePreflight -Inputs $inputs) -AsOutput diff --git a/scripts/eval-runners/runner-common.ps1 b/scripts/eval-runners/runner-common.ps1 index 4efcb8d..253f0da 100644 --- a/scripts/eval-runners/runner-common.ps1 +++ b/scripts/eval-runners/runner-common.ps1 @@ -81,7 +81,7 @@ function Write-RunnerJson { } function Get-Sha256HexFromBytes { - param([Parameter(Mandatory = $true)][byte[]]$Bytes) + param([Parameter(Mandatory = $true)][AllowEmptyCollection()][byte[]]$Bytes) $sha = [System.Security.Cryptography.SHA256]::Create() try { @@ -169,6 +169,116 @@ function Get-PlatformName { return 'unknown' } +function Get-SandboxVisiblePath { + param( + [Parameter(Mandatory = $true)][string]$HostPath, + [Parameter(Mandatory = $true)][string]$RunRoot, + [ValidateSet('windows', 'linux', 'macos', 'unknown')][string]$Platform = (Get-PlatformName), + [string]$MountRoot = '/run' + ) + + $fullHostPath = [System.IO.Path]::GetFullPath($HostPath) + if ($Platform -ne 'linux' -or -not (Test-PathInside -BasePath $RunRoot -CandidatePath $fullHostPath)) { + return $fullHostPath + } + + $relative = [System.IO.Path]::GetRelativePath(([System.IO.Path]::GetFullPath($RunRoot)), $fullHostPath).Replace('\', '/') + if ($relative -eq '.') { + return $MountRoot.TrimEnd('/') + } + return $MountRoot.TrimEnd('/') + '/' + $relative.TrimStart('/') +} + +function Get-ObservableVersionFromText { + param([string]$Text) + + foreach ($line in ($Text -split "`r?`n")) { + $trimmed = $line.Trim() + if (-not [string]::IsNullOrWhiteSpace($trimmed)) { + return $trimmed + } + } + return $null +} + +function New-RunnerProbeEnvironment { + $environment = [ordered]@{} + foreach ($name in @('PATH', 'SystemRoot', 'WINDIR', 'ComSpec', 'PATHEXT', 'LANG', 'LC_ALL', 'TZ', 'SSL_CERT_FILE', 'NODE_PATH')) { + $value = [Environment]::GetEnvironmentVariable($name) + if (-not [string]::IsNullOrWhiteSpace($value)) { + $environment[$name] = $value + } + } + $environment['CI'] = '1' + $environment['NO_COLOR'] = '1' + return $environment +} + +function Get-ExternalCommandVersion { + param( + [Parameter(Mandatory = $true)][object]$CommandInfo, + [string]$WorkingDirectory = '', + [System.Collections.IDictionary]$Environment = (New-RunnerProbeEnvironment), + [int]$TimeoutSeconds = 30 + ) + + $probeDirectory = $WorkingDirectory + $ownsProbeDirectory = $false + if ([string]::IsNullOrWhiteSpace($probeDirectory)) { + $probeDirectory = Join-Path ([System.IO.Path]::GetTempPath()) ('agentic-version-probe-' + [Guid]::NewGuid().ToString('N')) + New-Item -ItemType Directory -Path $probeDirectory -Force | Out-Null + $ownsProbeDirectory = $true + } + try { + $process = Invoke-RunnerProcess -FileName $CommandInfo.FileName -ArgumentList (@($CommandInfo.Prefix) + @('--version')) -WorkingDirectory $probeDirectory -Environment $Environment -TimeoutSeconds $TimeoutSeconds + $text = [string]::Join("`n", @($process.Stdout, $process.Stderr)) + $version = Get-ObservableVersionFromText -Text $text + return [pscustomobject]@{ + Version = if ($process.TimedOut -or $process.ExitCode -ne 0 -or [string]::IsNullOrWhiteSpace($version)) { 'unavailable' } else { $version } + Available = (-not $process.TimedOut -and $process.ExitCode -eq 0 -and -not [string]::IsNullOrWhiteSpace($version)) + Process = $process + } + } catch { + return [pscustomobject]@{ Version = 'unavailable'; Available = $false; Process = $null; Error = $_.Exception.Message } + } finally { + if ($ownsProbeDirectory -and (Test-Path -LiteralPath $probeDirectory)) { + Remove-Item -LiteralPath $probeDirectory -Recurse -Force -ErrorAction SilentlyContinue + } + } +} + +function Get-IsolationCapabilityAssessment { + param([System.Collections.IDictionary]$Capabilities) + + $required = @( + 'fresh_context', + 'isolated_home_config', + 'isolated_working_directory', + 'ambient_candidate_skill_exclusion', + 'candidate_skill_exposure', + 'prompt_fidelity', + 'model_configuration_lock', + 'response_capture' + ) + $unproven = [System.Collections.Generic.List[string]]::new() + foreach ($name in $required) { + $value = if ($null -ne $Capabilities -and $Capabilities.Contains($name)) { [string]$Capabilities[$name] } else { 'unavailable' } + $valid = if ($name -eq 'candidate_skill_exposure') { $value -in @('supported', 'excluded') } else { $value -eq 'supported' } + if (-not $valid) { $unproven.Add($name) } + } + + $filesystemValue = if ($null -ne $Capabilities -and $Capabilities.Contains('filesystem_confinement')) { [string]$Capabilities['filesystem_confinement'] } else { 'unavailable' } + $hardFilesystem = $filesystemValue -eq 'supported' + $mandatoryProven = $unproven.Count -eq 0 + return [pscustomobject]@{ + MandatoryProven = $mandatoryProven + HardFilesystemConfinement = $hardFilesystem + Level = if (-not $mandatoryProven) { 'unsupported' } elseif ($hardFilesystem) { 'strict' } else { 'pragmatic' } + Unproven = $unproven.ToArray() + Required = @($required) + } +} + function Resolve-RunContract { param([Parameter(Mandatory = $true)][string]$RunPath) @@ -181,7 +291,7 @@ function Resolve-RunContract { throw "run.json must declare '$($schemas.Run)'." } if (-not [bool]$run.freshContextRequired -or -not [bool]$run.filesystemIsolationRequired -or -not [bool]$run.isolatedHomeRequired) { - throw 'run.json must require fresh context, filesystem isolation, and isolated home.' + throw 'run.json must require a fresh context, a staged filesystem/workspace boundary, and an isolated home.' } $mode = [string]$run.mode @@ -374,10 +484,15 @@ function New-PreflightDocument { ) $schemas = Get-RunnerSchemaNames + $capabilitiesForAssessment = if ($null -eq $ResolvedCapabilities) { [ordered]@{} } else { $ResolvedCapabilities } + $assessment = Get-IsolationCapabilityAssessment -Capabilities $capabilitiesForAssessment + $effectiveCompatible = $Compatible -and $assessment.MandatoryProven + $unprovenControls = [string[]]$assessment.Unproven + if (-not $effectiveCompatible) { $unprovenControls = [string[]](@($assessment.Unproven) + @('preflight')) } return [ordered]@{ schema = $schemas.Preflight protocol_version = $schemas.Protocol - status = if ($Compatible) { 'compatible' } else { 'incompatible' } + status = if ($effectiveCompatible) { 'compatible' } else { 'incompatible' } runner = [ordered]@{ name = [string]$Descriptor.name; version = [string]$Descriptor.version } harness = $Descriptor.harness run = [ordered]@{ eval_id = $Run.EvalId; eval_name = $Run.EvalName; configuration = $Run.Mode } @@ -391,6 +506,12 @@ function New-PreflightDocument { } checks = @($Checks) resolved_capabilities = if ($null -eq $ResolvedCapabilities) { [ordered]@{} } else { $ResolvedCapabilities } + isolation = [ordered]@{ + level = if ($effectiveCompatible) { $assessment.Level } else { 'unsupported' } + status = if ($effectiveCompatible) { 'verified' } else { 'unverified' } + hard_filesystem_confinement = if ($effectiveCompatible) { $assessment.HardFilesystemConfinement } else { $false } + unproven_controls = $unprovenControls + } mechanisms = @($Mechanisms) warnings = @($Warnings) reasons = @($Reasons) @@ -432,8 +553,9 @@ function New-ExecutionResult { [Nullable[int]]$ExitStatus, [object]$Failure, [string]$SessionId, - [hashtable]$IsolationCapabilities, + [System.Collections.IDictionary]$IsolationCapabilities, [string[]]$IsolationMechanisms = @(), + [object]$ResolvedConfiguration = $null, [object]$Telemetry = $null, [object[]]$Artifacts = @(), [string[]]$Warnings = @(), @@ -443,13 +565,60 @@ function New-ExecutionResult { ) $schemas = Get-RunnerSchemaNames - $hasResponse = -not [string]::IsNullOrWhiteSpace($FinalResponse) + $assessment = Get-IsolationCapabilityAssessment -Capabilities $IsolationCapabilities + $effectiveStatus = $Status + $effectiveFinalResponse = $FinalResponse + $effectiveFinalResponseReason = $FinalResponseReason + $effectiveExitStatus = $ExitStatus + $effectiveFailure = $Failure + $effectiveDeviations = [System.Collections.Generic.List[string]]::new() + foreach ($deviation in @($CompatibilityDeviations)) { $effectiveDeviations.Add([string]$deviation) } + if ($Status -ne 'incompatible' -and -not $assessment.MandatoryProven) { + $effectiveStatus = 'incompatible' + $effectiveFinalResponse = $null + $effectiveFinalResponseReason = 'isolation_controls_unproven' + $effectiveExitStatus = $null + $effectiveFailure = New-ExecutionFailure -Code 'isolation_unproven' -Message ("Mandatory isolation controls were not proven: {0}." -f ([string]::Join(', ', @($assessment.Unproven)))) + $effectiveDeviations.Add('execution_rejected_because_mandatory_isolation_controls_were_unproven') + } + $hasResponse = -not [string]::IsNullOrWhiteSpace($effectiveFinalResponse) $started = if ([string]::IsNullOrWhiteSpace($StartedUtc)) { [DateTime]::UtcNow } else { [DateTime]::Parse($StartedUtc).ToUniversalTime() } $finished = if ([string]::IsNullOrWhiteSpace($FinishedUtc)) { [DateTime]::UtcNow } else { [DateTime]::Parse($FinishedUtc).ToUniversalTime() } $isolation = [ordered]@{ - status = if ($Status -eq 'completed' -or $Status -eq 'failed' -or $Status -eq 'timed_out' -or $Status -eq 'cancelled') { 'verified' } else { 'unverified' } + status = if ($effectiveStatus -ne 'incompatible' -and $assessment.MandatoryProven) { 'verified' } else { 'unverified' } + level = if ($effectiveStatus -eq 'incompatible') { 'unsupported' } else { $assessment.Level } + hard_filesystem_confinement = if ($effectiveStatus -eq 'incompatible') { $false } else { $assessment.HardFilesystemConfinement } capabilities = if ($null -eq $IsolationCapabilities) { [ordered]@{} } else { $IsolationCapabilities } mechanisms = @($IsolationMechanisms) + required_controls = @($assessment.Required) + unproven_controls = [string[]]$assessment.Unproven + } + + $resolved = [ordered]@{ + provider = $null + model = $null + reasoning_effort = $null + configuration_profile = $null + tool_profile = $null + status = 'unavailable' + reason = 'harness_only_confirmed_the_requested_configuration' + accepted = [ordered]@{ + provider = $Profile.Provider + model = $Profile.Model + reasoning_effort = $Profile.ReasoningEffort + configuration_profile = $Profile.ConfigurationProfile + tool_profile = $Profile.ToolProfile + } + } + if ($null -ne $ResolvedConfiguration) { + $resolved.status = [string](Get-JsonProperty -Object $ResolvedConfiguration -Name 'status' -Default 'resolved') + $resolved.reason = Get-JsonProperty -Object $ResolvedConfiguration -Name 'reason' -Default $null + foreach ($name in @('provider', 'model', 'reasoning_effort', 'configuration_profile', 'tool_profile')) { + $value = Get-JsonProperty -Object $ResolvedConfiguration -Name $name -Default $null + if ($null -ne $value) { $resolved[$name] = $value } + } + $observations = Get-JsonProperty -Object $ResolvedConfiguration -Name 'observations' -Default $null + if ($null -ne $observations) { $resolved.observations = $observations } } return [ordered]@{ @@ -461,16 +630,16 @@ function New-ExecutionResult { fresh = $true resumed = $false } - status = $Status + status = $effectiveStatus run = [ordered]@{ eval_id = $Run.EvalId eval_name = $Run.EvalName configuration = $Run.Mode } final_response = if ($hasResponse) { - [ordered]@{ status = 'available'; text = $FinalResponse } + [ordered]@{ status = 'available'; text = $effectiveFinalResponse } } else { - [ordered]@{ status = 'unavailable'; reason = if ([string]::IsNullOrWhiteSpace($FinalResponseReason)) { 'harness_did_not_return_a_final_response' } else { $FinalResponseReason } } + [ordered]@{ status = 'unavailable'; reason = if ([string]::IsNullOrWhiteSpace($effectiveFinalResponseReason)) { 'harness_did_not_return_a_final_response' } else { $effectiveFinalResponseReason } } } runner = [ordered]@{ name = [string]$Descriptor.name; version = [string]$Descriptor.version } harness = $Descriptor.harness @@ -481,17 +650,11 @@ function New-ExecutionResult { configuration_profile = $Profile.ConfigurationProfile tool_profile = $Profile.ToolProfile } - resolved = [ordered]@{ - provider = $Profile.Provider - model = $Profile.Model - reasoning_effort = $Profile.ReasoningEffort - configuration_profile = $Profile.ConfigurationProfile - tool_profile = $Profile.ToolProfile - } + resolved = $resolved started_utc = $started.ToString('yyyy-MM-ddTHH:mm:ss.fffZ') finished_utc = $finished.ToString('yyyy-MM-ddTHH:mm:ss.fffZ') duration_seconds = [Math]::Max(0, [Math]::Round($DurationSeconds, 3)) - exit = [ordered]@{ status = $ExitStatus; failure = $Failure } + exit = [ordered]@{ status = $effectiveExitStatus; failure = $effectiveFailure } input = [ordered]@{ prompt_sha256 = $Run.PromptHash run_json_sha256 = Get-Sha256HexFromFile -Path $Run.RunPath @@ -509,7 +672,7 @@ function New-ExecutionResult { evidence = if ($null -eq $Evidence) { [ordered]@{} } else { $Evidence } artifacts = @($Artifacts) warnings = @($Warnings) - compatibility_deviations = @($CompatibilityDeviations) + compatibility_deviations = @($effectiveDeviations) attempt_count = $AttemptCount } } @@ -543,6 +706,46 @@ function Assert-ExecutionResult { if ([int](Get-JsonProperty -Object $Result -Name 'attempt_count' -Default 0) -ne 1) { throw 'execution-result.json attempt_count must be exactly 1; quality retries are not allowed.' } + $isolationStatus = [string](Get-JsonProperty -Object $Result.isolation -Name 'status' -Default '') + $isolationLevel = [string](Get-JsonProperty -Object $Result.isolation -Name 'level' -Default '') + $hardFilesystem = [bool](Get-JsonProperty -Object $Result.isolation -Name 'hard_filesystem_confinement' -Default $false) + if ($isolationStatus -notin @('verified', 'unverified')) { + throw "execution-result.json isolation.status '$isolationStatus' is unsupported." + } + if ($isolationLevel -notin @('strict', 'pragmatic', 'unsupported')) { + throw "execution-result.json isolation.level '$isolationLevel' is unsupported." + } + if ($Result.status -eq 'incompatible') { + if ($isolationStatus -ne 'unverified' -or $isolationLevel -ne 'unsupported') { + throw 'An incompatible execution must report unverified, unsupported isolation.' + } + } else { + if ($isolationStatus -ne 'verified' -or $isolationLevel -eq 'unsupported') { + throw 'A non-incompatible execution must prove the mandatory experimental controls.' + } + if ($isolationLevel -eq 'strict' -and -not $hardFilesystem) { + throw 'Strict isolation must report hard filesystem confinement.' + } + if ($isolationLevel -eq 'pragmatic' -and $hardFilesystem) { + throw 'Pragmatic isolation must not claim hard filesystem confinement.' + } + $requiredControls = @('fresh_context', 'isolated_home_config', 'isolated_working_directory', 'ambient_candidate_skill_exclusion', 'candidate_skill_exposure', 'prompt_fidelity', 'model_configuration_lock', 'response_capture') + foreach ($control in $requiredControls) { + $value = [string](Get-JsonProperty -Object $Result.isolation.capabilities -Name $control -Default 'unavailable') + if ($control -eq 'candidate_skill_exposure') { + if ($value -notin @('supported', 'excluded')) { throw "Mandatory isolation capability '$control' is not proven." } + } elseif ($value -ne 'supported') { + throw "Mandatory isolation capability '$control' is not proven." + } + } + } + $resolvedStatus = [string](Get-JsonProperty -Object $Result.resolved -Name 'status' -Default '') + if ($resolvedStatus -notin @('unavailable', 'accepted_request', 'resolved')) { + throw "execution-result.json resolved.status '$resolvedStatus' is unsupported." + } + if (-not (Test-JsonProperty -Object $Result.resolved -Name 'accepted')) { + throw 'execution-result.json resolved must preserve the requested configuration as accepted evidence.' + } foreach ($hashField in @('prompt_sha256', 'run_json_sha256', 'profile_sha256')) { if (-not (Test-Sha256 -Value ([string]$Result.input.$hashField))) { throw "execution-result.json input.$hashField must be a SHA-256 value." @@ -636,10 +839,82 @@ function New-RunnerEnvironment { return $environment } +function Get-LinuxEvalSandboxArguments { + param( + [Parameter(Mandatory = $true)][object]$Inputs, + [Parameter(Mandatory = $true)][object]$CommandInfo, + [Parameter(Mandatory = $true)][System.Collections.IDictionary]$InsideEnvironment, + [string[]]$ReadOnlyRoots = @('/usr', '/usr/local', '/bin', '/sbin', '/lib', '/lib64', '/libexec', '/etc', '/opt') + ) + + $arguments = [System.Collections.Generic.List[string]]::new() + foreach ($argument in @('--die-with-parent', '--new-session', '--unshare-pid')) { $arguments.Add($argument) } + foreach ($path in $ReadOnlyRoots) { + if (Test-Path -LiteralPath $path -PathType Container) { + $arguments.Add('--ro-bind'); $arguments.Add($path); $arguments.Add($path) + } + } + $arguments.Add('--proc'); $arguments.Add('/proc') + $arguments.Add('--dev'); $arguments.Add('/dev') + $arguments.Add('--tmpfs'); $arguments.Add('/tmp') + $arguments.Add('--bind'); $arguments.Add($Inputs.Run.RunRoot); $arguments.Add('/run') + $commandSource = [string]$CommandInfo.Source + $commandDirectory = Split-Path -Parent $commandSource + if (-not ($commandSource.StartsWith('/usr/', [System.StringComparison]::Ordinal) -or $commandSource.StartsWith('/bin/', [System.StringComparison]::Ordinal) -or $commandSource.StartsWith('/opt/', [System.StringComparison]::Ordinal))) { + if (Test-Path -LiteralPath $commandDirectory -PathType Container) { + $arguments.Add('--ro-bind'); $arguments.Add($commandDirectory); $arguments.Add($commandDirectory) + } + } + $arguments.Add('--chdir'); $arguments.Add('/run/repo') + foreach ($key in @($InsideEnvironment.Keys)) { + $arguments.Add('--setenv'); $arguments.Add([string]$key); $arguments.Add([string]$InsideEnvironment[$key]) + } + $arguments.Add('--') + $arguments.Add($CommandInfo.FileName) + foreach ($prefix in @($CommandInfo.Prefix)) { $arguments.Add($prefix) } + return @($arguments) +} + +function New-MacosEvalSandboxProfile { + param( + [Parameter(Mandatory = $true)][object]$Inputs, + [Parameter(Mandatory = $true)][object]$CommandInfo, + [string[]]$ReadOnlyRoots = @('/usr', '/usr/local', '/bin', '/sbin', '/lib', '/libexec', '/System', '/Library', '/opt', '/private/var/db') + ) + + $profilePath = Join-Path $Inputs.Run.HomeDirectoryPath 'eval-sandbox.sb' + $runRoot = $Inputs.Run.RunRoot.Replace('\', '/') + $commandDirectory = (Split-Path -Parent ([string]$CommandInfo.Source)).Replace('\', '/') + $lines = [System.Collections.Generic.List[string]]::new() + $lines.Add('(version 1)') + $lines.Add('(deny default)') + $lines.Add('(allow process*)') + $lines.Add('(allow network*)') + foreach ($root in @($ReadOnlyRoots + @($commandDirectory)) | Sort-Object -Unique) { + if (-not [string]::IsNullOrWhiteSpace($root) -and (Test-Path -LiteralPath $root -PathType Container)) { + $escapedRoot = $root.Replace('\', '/').Replace('"', '\"') + $lines.Add(('(allow file-read* (subpath "{0}"))' -f $escapedRoot)) + } + } + $escapedRunRoot = $runRoot.Replace('"', '\"') + $lines.Add(('(allow file-read* (subpath "{0}"))' -f $escapedRunRoot)) + $lines.Add(('(allow file-write* (subpath "{0}"))' -f $escapedRunRoot)) + $lines.Add('(allow file-read* (subpath "/dev"))') + $lines.Add('(allow file-write* (subpath "/dev/null"))') + [System.IO.File]::WriteAllText($profilePath, ([string]::Join("`n", $lines) + "`n"), [System.Text.UTF8Encoding]::new($false)) + return $profilePath +} + function Resolve-ExternalCommand { param([Parameter(Mandatory = $true)][string]$Name) $command = Get-Command $Name -ErrorAction SilentlyContinue + if ($null -eq $command) { + foreach ($candidateName in @("$Name.ps1", "$Name.cmd", "$Name.exe")) { + $command = Get-Command $candidateName -ErrorAction SilentlyContinue + if ($null -ne $command) { break } + } + } if ($null -eq $command) { return $null } @@ -663,7 +938,7 @@ function Invoke-RunnerProcess { [string[]]$ArgumentList = @(), [Parameter(Mandatory = $true)][string]$WorkingDirectory, [System.Collections.IDictionary]$Environment = @{}, - [byte[]]$InputBytes = @(), + [AllowEmptyCollection()][byte[]]$InputBytes = @(), [int]$TimeoutSeconds = 900 ) @@ -733,6 +1008,7 @@ function Get-ProviderAuthenticationVariables { '^openrouter$' { return @('OPENROUTER_API_KEY') } '^xai$|^x-ai$' { return @('XAI_API_KEY') } '^mistral$' { return @('MISTRAL_API_KEY') } + '^cline$' { return @('CLINE_API_KEY') } default { return @() } } } diff --git a/scripts/eval-runners/tests/fixtures/cline-events.jsonl b/scripts/eval-runners/tests/fixtures/cline-events.jsonl new file mode 100644 index 0000000..21d2a65 --- /dev/null +++ b/scripts/eval-runners/tests/fixtures/cline-events.jsonl @@ -0,0 +1,11 @@ +{"type":"hook_event","hookEventName":"agent_start","sessionId":"fixture-session"} +{"type":"say","say":"text","text":"2 + 2 = 4.","partial":false} +{"type":"say","say":"tool","name":"read_file","text":"fixture.md","partial":false} +{"type":"say","say":"completion_result","text":"2 + 2 = 4.","partial":false} +{"type":"agent_event","event":{"type":"iteration_start","iteration":1}} +{"type":"agent_event","event":{"type":"content_start","contentType":"text","text":"2 + 2 = "}} +{"type":"agent_event","event":{"type":"content_end","contentType":"text","text":"2 + 2 = 4."}} +{"type":"agent_event","event":{"type":"usage","inputTokens":5022,"outputTokens":28,"totalTokens":5050,"cost":0.01}} +{"type":"agent_event","event":{"type":"done","reason":"completed","text":"2 + 2 = 4.","sessionId":"fixture-session","iterations":1,"usage":{"inputTokens":5022,"outputTokens":28,"totalTokens":5050,"cost":0.01}}} +{"type":"future.event.v99","payload":"preserve-as-warning"} +{"type":"hook_event","hookEventName":"agent_end","sessionId":"fixture-session"} diff --git a/scripts/eval-runners/tests/test-runner-conformance.ps1 b/scripts/eval-runners/tests/test-runner-conformance.ps1 index 77ca026..8a61bef 100644 --- a/scripts/eval-runners/tests/test-runner-conformance.ps1 +++ b/scripts/eval-runners/tests/test-runner-conformance.ps1 @@ -3,9 +3,10 @@ Deterministic conformance suite for the common Eval Runner protocol. .DESCRIPTION - Creates an ephemeral package under the system temp directory, invokes only - the fake runner, and checks the contracts and recorded event fixtures. It - never invokes Codex, OpenCode, or a live model. + Creates ephemeral packages under the system temp directory, invokes the + deterministic fake runner and recorded fake CLI processes, and checks the + contracts and recorded event fixtures. It never invokes a real harness or + a live model. #> [CmdletBinding()] param() @@ -22,6 +23,268 @@ function Assert-True { if (-not $Condition) { throw "ASSERT: $Message" } } +function Invoke-AdapterJson { + param( + [Parameter(Mandatory = $true)][string]$RunnerPath, + [Parameter(Mandatory = $true)][string]$Command, + [Parameter(Mandatory = $true)][string]$RunPath, + [Parameter(Mandatory = $true)][string]$ProfilePath + ) + + $output = & pwsh -NoProfile -File $RunnerPath $Command -Run $RunPath -Profile $ProfilePath + if ($LASTEXITCODE -ne 0) { throw "Recorded runner '$Command' failed for '$RunnerPath': $([string]::Join(' ', @($output)))" } + $json = [string]::Join([Environment]::NewLine, @($output)) + if ([string]::IsNullOrWhiteSpace($json)) { throw "Recorded runner '$Command' returned no JSON for '$RunnerPath'." } + return $json | ConvertFrom-Json +} + +function Invoke-RecordedRunnerTests { + $recordedRoot = Join-Path ([System.IO.Path]::GetTempPath()) ('agentic-recorded-runner-' + [Guid]::NewGuid().ToString('N')) +$recordedOldPath = $env:PATH +$recordedOldOpenAi = $env:OPENAI_API_KEY +$recordedOldCodexHome = $env:CODEX_HOME +$recordedOldGlobalSecret = $env:AGENTIC_GLOBAL_SECRET +$recordedOldProjectDisable = $env:OPENCODE_DISABLE_PROJECT_CONFIG +try { + $fakeBin = Join-Path $recordedRoot 'bin' + New-Item -ItemType Directory -Path $fakeBin -Force | Out-Null + $recordedIteration = Join-Path $recordedRoot 'iteration-1' + New-Item -ItemType Directory -Path $recordedIteration -Force | Out-Null + $with = New-TestRun -IterationDirectory $recordedIteration -Configuration with_skill + $without = New-TestRun -IterationDirectory $recordedIteration -Configuration without_skill + [System.IO.File]::WriteAllText((Join-Path $with.Root 'repo\opencode.json'), '{"fixture_project_config":true}', [System.Text.UTF8Encoding]::new($false)) + [System.IO.File]::WriteAllText((Join-Path $without.Root 'repo\opencode.json'), '{"fixture_project_config":true}', [System.Text.UTF8Encoding]::new($false)) + $fakeCli = @' +[CmdletBinding()] +param([Parameter(ValueFromRemainingArguments = $true)][string[]]$RemainingArguments) +$harness = [System.IO.Path]::GetFileNameWithoutExtension($MyInvocation.MyCommand.Path) +$logPath = Join-Path (Get-Location).Path ("{0}-fake-cli-log.jsonl" -f $harness) +$arguments = @($RemainingArguments | ForEach-Object { [string]$_ }) +$authNames = @('OPENAI_API_KEY', 'ANTHROPIC_API_KEY', 'GOOGLE_API_KEY', 'GEMINI_API_KEY', 'OPENROUTER_API_KEY', 'XAI_API_KEY', 'MISTRAL_API_KEY', 'CLINE_API_KEY') +$authPresent = @($authNames | Where-Object { -not [string]::IsNullOrWhiteSpace([Environment]::GetEnvironmentVariable($_)) }) +$record = [ordered]@{ + args = $arguments + working_directory = (Get-Location).Path + home = [Environment]::GetEnvironmentVariable('HOME') + userprofile = [Environment]::GetEnvironmentVariable('USERPROFILE') + auth_names_present = $authPresent + unrelated_present = -not [string]::IsNullOrWhiteSpace([Environment]::GetEnvironmentVariable('AGENTIC_GLOBAL_SECRET')) + disable_project_config_present = -not [string]::IsNullOrWhiteSpace([Environment]::GetEnvironmentVariable('OPENCODE_DISABLE_PROJECT_CONFIG')) + project_config_visible = Test-Path -LiteralPath (Join-Path (Get-Location).Path 'opencode.json') -PathType Leaf + stdin_received = $false +} +if ($arguments -contains '--version') { + $version = switch ($harness) { 'codex' { 'recorded-codex 9.1' } 'opencode' { 'recorded-opencode 9.2' } default { 'recorded-cline 9.3' } } + [IO.File]::AppendAllText($logPath, (($record | ConvertTo-Json -Compress) + [Environment]::NewLine), [Text.UTF8Encoding]::new($false)) + Write-Output $version + exit 0 +} +if ($arguments -contains '--help') { + $help = switch ($harness) { + 'codex' { '--ask-for-approval never --ephemeral --ignore-user-config --ignore-rules --json --output-last-message --sandbox --cd --model --config --approve-for-me' } + 'opencode' { '--format --dir --model --auto --pure --continue --session' } + default { '--json --auto-approve --cwd --config --data-dir --hooks-dir --provider --model --thinking --timeout --retries --id' } + } + [IO.File]::AppendAllText($logPath, (($record | ConvertTo-Json -Compress) + [Environment]::NewLine), [Text.UTF8Encoding]::new($false)) + Write-Output $help + exit 0 +} +$stdinText = [Console]::In.ReadToEnd() +$record.stdin_received = -not [string]::IsNullOrEmpty($stdinText) +$probeCommand = '$result = [ordered]@{ provider_visible = -not [string]::IsNullOrWhiteSpace([Environment]::GetEnvironmentVariable(''OPENAI_API_KEY'')); auth_file_visible = Test-Path -LiteralPath (Join-Path ([Environment]::GetEnvironmentVariable(''HOME'')) ''.codex/auth.json''); global_secret_visible = -not [string]::IsNullOrWhiteSpace([Environment]::GetEnvironmentVariable(''AGENTIC_GLOBAL_SECRET'')); project_disable_visible = -not [string]::IsNullOrWhiteSpace([Environment]::GetEnvironmentVariable(''OPENCODE_DISABLE_PROJECT_CONFIG'')) }; $result | ConvertTo-Json -Compress' +$probeInfo = [Diagnostics.ProcessStartInfo]::new() +$probeInfo.FileName = (Get-Command pwsh).Source +$probeInfo.UseShellExecute = $false +$probeInfo.CreateNoWindow = $true +$probeInfo.RedirectStandardOutput = $true +$probeInfo.RedirectStandardError = $true +$probeInfo.WorkingDirectory = (Get-Location).Path +$probeInfo.ArgumentList.Add('-NoProfile') +$probeInfo.ArgumentList.Add('-Command') +$probeInfo.ArgumentList.Add($probeCommand) +$probeInfo.Environment.Clear() +$probeInfo.Environment['PATH'] = [Environment]::GetEnvironmentVariable('PATH') +$probeInfo.Environment['HOME'] = [Environment]::GetEnvironmentVariable('HOME') +$probeInfo.Environment['USERPROFILE'] = [Environment]::GetEnvironmentVariable('USERPROFILE') +if ($harness -ne 'codex') { $probeInfo.Environment['OPENAI_API_KEY'] = [Environment]::GetEnvironmentVariable('OPENAI_API_KEY') } +$probe = [Diagnostics.Process]::new() +$probe.StartInfo = $probeInfo +try { + [void]$probe.Start() + $probeOutput = $probe.StandardOutput.ReadToEnd() + $probeError = $probe.StandardError.ReadToEnd() + $probe.WaitForExit() + if ($probe.ExitCode -ne 0) { throw "worker credential probe failed: $probeError" } + $probeResult = $probeOutput | ConvertFrom-Json + $record.worker_provider_visible = [bool]$probeResult.provider_visible + $record.worker_auth_file_visible = [bool]$probeResult.auth_file_visible + $record.worker_global_secret_visible = [bool]$probeResult.global_secret_visible + $record.worker_project_disable_visible = [bool]$probeResult.project_disable_visible +} finally { + $probe.Dispose() +} +[IO.File]::AppendAllText($logPath, (($record | ConvertTo-Json -Compress) + [Environment]::NewLine), [Text.UTF8Encoding]::new($false)) +if ($harness -eq 'codex') { + $outputIndex = [Array]::IndexOf([string[]]$arguments, '--output-last-message') + if ($outputIndex -ge 0 -and $outputIndex + 1 -lt $arguments.Count) { + $outputPath = $arguments[$outputIndex + 1] + New-Item -ItemType Directory -Path (Split-Path -Parent $outputPath) -Force | Out-Null + [IO.File]::WriteAllText($outputPath, 'recorded Codex final response', [Text.UTF8Encoding]::new($false)) + } + Write-Output '{"type":"thread.started","thread_id":"recorded-thread"}' + Write-Output '{"type":"item.completed","item":{"type":"agent_message","text":"recorded Codex final response"}}' + Write-Output '{"type":"turn.completed","usage":{"input_tokens":2,"output_tokens":3}}' + Write-Output '{"type":"future.event.v99","payload":"fixture"}' +} elseif ($harness -eq 'opencode') { + Write-Output '{"type":"text","text":"recorded OpenCode final response"}' + Write-Output '{"type":"step_finish","part":{"tokens":{"input":2,"output":3},"cost":0.01}}' + Write-Output '{"type":"future.event.v99","payload":"fixture"}' +} else { + Write-Output '{"type":"say","say":"text","text":"recorded Cline progress","partial":false}' + Write-Output '{"type":"say","say":"tool","name":"read_file","text":"fixture.md","partial":false}' + Write-Output '{"type":"say","say":"completion_result","text":"recorded Cline final response","partial":false}' + Write-Output '{"type":"say","say":"api_req_finished","text":"{\"inputTokens\":2,\"outputTokens\":3,\"totalTokens\":5}"}' + Write-Output '{"type":"future.event.v99","payload":"fixture"}' +} +'@ + foreach ($harness in @('codex', 'opencode', 'cline')) { + [System.IO.File]::WriteAllText((Join-Path $fakeBin "$harness.ps1"), $fakeCli, [System.Text.UTF8Encoding]::new($false)) + } + $env:PATH = "$fakeBin$([System.IO.Path]::PathSeparator)$recordedOldPath" + $env:OPENAI_API_KEY = 'recorded-canary-not-logged' + $env:AGENTIC_GLOBAL_SECRET = 'recorded-unrelated-canary-not-logged' + $env:OPENCODE_DISABLE_PROJECT_CONFIG = '1' + $recordedProfiles = [ordered]@{} + foreach ($runnerName in @('codex', 'opencode', 'cline')) { + $profilePath = Join-Path $recordedRoot "$runnerName-profile.json" + Write-TestJson -Path $profilePath -Value ([ordered]@{ + schema = (Get-RunnerSchemaNames).Profile + runner = $runnerName + provider = 'openai' + model = 'fixture-model' + reasoning_effort = 'medium' + configuration_profile = 'isolated-default' + tool_profile = 'default' + timeout_seconds = 30 + concurrency = 1 + }) + $recordedProfiles[$runnerName] = $profilePath + } + $resolvedRecordedCodex = Resolve-ExternalCommand -Name 'codex' + Assert-Equal (Join-Path $fakeBin 'codex.ps1') $resolvedRecordedCodex.Source 'recorded Codex command is selected before the installed CLI' + $recordedVersion = Get-ExternalCommandVersion -CommandInfo $resolvedRecordedCodex -WorkingDirectory (Join-Path $with.Root 'repo') + if (-not $recordedVersion.Available) { throw "recorded Codex --version is not observable (exit=$($recordedVersion.Process.ExitCode), timed_out=$($recordedVersion.Process.TimedOut), stdout='$($recordedVersion.Process.Stdout)', stderr='$($recordedVersion.Process.Stderr)')" } + Assert-Equal 'recorded-codex 9.1' $recordedVersion.Version 'recorded Codex exact version helper' + foreach ($runnerName in @('codex', 'opencode', 'cline')) { + $runnerPath = Join-Path $runnerRoot "$runnerName\runner.ps1" + $description = Invoke-AdapterJson -RunnerPath $runnerPath -Command describe -RunPath $with.Path -ProfilePath $recordedProfiles[$runnerName] + [void](Assert-RunnerDescriptor -Descriptor $description) + $expectedVersion = switch ($runnerName) { 'codex' { 'recorded-codex 9.1' } 'opencode' { 'recorded-opencode 9.2' } default { 'recorded-cline 9.3' } } + Assert-Equal $expectedVersion $description.harness.version "$runnerName exact describe version" + $preflightWith = Invoke-AdapterJson -RunnerPath $runnerPath -Command preflight -RunPath $with.Path -ProfilePath $recordedProfiles[$runnerName] + $preflightWithout = Invoke-AdapterJson -RunnerPath $runnerPath -Command preflight -RunPath $without.Path -ProfilePath $recordedProfiles[$runnerName] + Assert-Equal 'compatible' $preflightWith.status "$runnerName with_skill pragmatic preflight" + Assert-Equal 'compatible' $preflightWithout.status "$runnerName without_skill pragmatic preflight" + Assert-Equal $expectedVersion $preflightWith.harness.version "$runnerName exact preflight version" + Assert-Equal 'pragmatic' $preflightWith.isolation.level "$runnerName pragmatic preflight level" + if ($runnerName -ne 'codex') { + Assert-True (@($preflightWith.warnings | Where-Object { $_ -match 'child-tool environment filter' }).Count -gt 0) "$runnerName reports the child credential-filter limitation" + } + $resultWith = Invoke-AdapterJson -RunnerPath $runnerPath -Command execute -RunPath $with.Path -ProfilePath $recordedProfiles[$runnerName] + $resultWithout = Invoke-AdapterJson -RunnerPath $runnerPath -Command execute -RunPath $without.Path -ProfilePath $recordedProfiles[$runnerName] + foreach ($result in @($resultWith, $resultWithout)) { + [void](Assert-ExecutionResult -Result $result) + Assert-Equal 'completed' $result.status "$runnerName recorded completion" + Assert-Equal $expectedVersion $result.harness.version "$runnerName exact execution version" + Assert-Equal 'accepted_request' $result.resolved.status "$runnerName accepted configuration provenance" + Assert-True ($null -eq $result.resolved.model) "$runnerName does not claim concrete model resolution" + Assert-Equal 'pragmatic' $result.isolation.level "$runnerName pragmatic execution level" + Assert-True (-not $result.isolation.hard_filesystem_confinement) "$runnerName pragmatic execution has no hard confinement" + Assert-Equal 1 $result.attempt_count "$runnerName one semantic attempt" + Assert-Equal 'available' $result.final_response.status "$runnerName captures final response" + $resultRoot = if ($result.run.configuration -eq 'with_skill') { $with.Root } else { $without.Root } + foreach ($artifact in @($result.artifacts)) { + Assert-True ($artifact.path -notmatch '(^|/|\\)\.\.(/|\\|$)') "$runnerName artifact path remains relative" + Assert-True (Test-Path -LiteralPath (Join-Path $resultRoot ($artifact.path -replace '/', [System.IO.Path]::DirectorySeparatorChar)) -PathType Leaf) "$runnerName artifact exists inside its run" + } + } + $logPath = Join-Path $with.Root "repo\$runnerName-fake-cli-log.jsonl" + Assert-True (Test-Path -LiteralPath $logPath -PathType Leaf) "$runnerName recorded process log exists" + $records = @(Get-Content -LiteralPath $logPath | ForEach-Object { $_ | ConvertFrom-Json }) + $executionRecords = @($records | Where-Object { $_.stdin_received -eq $true }) + Assert-Equal 1 $executionRecords.Count "$runnerName one execution process per checked arm" + $execution = $executionRecords[0] + Assert-True (-not $execution.unrelated_present) "$runnerName does not pass unrelated credential canary" + Assert-True (-not $execution.disable_project_config_present) "$runnerName does not pass ambient project-disable override" + Assert-True (-not $execution.worker_auth_file_visible) "$runnerName worker probe cannot read a copied Codex auth file" + Assert-True (-not $execution.worker_global_secret_visible) "$runnerName worker probe cannot read the parent/global canary" + Assert-True (-not $execution.worker_project_disable_visible) "$runnerName worker probe cannot read the parent project-disable variable" + if ($runnerName -eq 'codex') { + Assert-True (-not $execution.worker_provider_visible) 'Codex shell policy hides the provider API-key variable from the worker probe' + } else { + Assert-True $execution.worker_provider_visible "$runnerName credential visibility limitation is recorded by the worker probe" + } + $args = @($execution.args) + foreach ($forbidden in @('--continue', '--session', '--resume')) { Assert-True ($args -notcontains $forbidden) "$runnerName does not pass '$forbidden'" } + if ($runnerName -eq 'codex') { + Assert-True ($args -contains '--ask-for-approval') 'Codex uses explicit approval policy' + Assert-True ($args -contains 'never') 'Codex approval policy is never' + Assert-True ($args -contains '--sandbox' -and $args -contains 'workspace-write') 'Codex retains workspace-write sandbox' + Assert-True ($args -notcontains '--approve-for-me') 'Codex avoids the conflicting approve-for-me flag' + $outputIndex = [Array]::IndexOf([string[]]$args, '--output-last-message') + Assert-Equal (Join-Path $with.Root 'evidence\codex-final.txt') $args[$outputIndex + 1] 'Codex output path is host-visible on Windows' + } elseif ($runnerName -eq 'opencode') { + Assert-True ($args -notcontains '--pure') 'OpenCode preserves repository-owned project configuration' + Assert-True ($args -contains '--auto') 'OpenCode is noninteractive' + Assert-True $execution.project_config_visible 'OpenCode paired arm retains repository-owned project configuration' + } else { + $retryIndex = [Array]::IndexOf([string[]]$args, '--retries') + Assert-Equal '0' $args[$retryIndex + 1] 'Cline disables internal retries' + Assert-True ($args -notcontains '--id') 'Cline does not resume a session' + Assert-True ($args -contains '--json') 'Cline uses structured output' + $configIndex = [Array]::IndexOf([string[]]$args, '--config') + Assert-True ($args[$configIndex + 1] -match '(?i)[\\/]\.cline$') 'Cline uses the documented isolated config root' + $dataIndex = [Array]::IndexOf([string[]]$args, '--data-dir') + Assert-True ($args[$dataIndex + 1] -match '(?i)[\\/]\.cline[\\/]data$') 'Cline data-dir is the isolated data root' + Assert-Equal 'available' $resultWith.telemetry.tool_calls.status 'Cline reports available tool-call telemetry' + Assert-True ([int]$resultWith.telemetry.tool_calls.value -ge 1) 'Cline parses documented tool events' + } + $logText = [System.IO.File]::ReadAllText($logPath, [System.Text.UTF8Encoding]::new($false)) + Assert-True ($logText -notmatch 'recorded-canary|recorded-unrelated-canary') "$runnerName logs do not contain credential values" + $withoutLogPath = Join-Path $without.Root "repo\$runnerName-fake-cli-log.jsonl" + Assert-True (Test-Path -LiteralPath $withoutLogPath -PathType Leaf) "$runnerName baseline process log exists" + $withoutRecords = @(Get-Content -LiteralPath $withoutLogPath | ForEach-Object { $_ | ConvertFrom-Json }) + Assert-Equal 1 @($withoutRecords | Where-Object { $_.stdin_received -eq $true }).Count "$runnerName baseline has one execution process" + $withoutLogText = [System.IO.File]::ReadAllText($withoutLogPath, [System.Text.UTF8Encoding]::new($false)) + Assert-True ($withoutLogText -notmatch 'recorded-canary|recorded-unrelated-canary') "$runnerName baseline log does not contain credential values" + } + $staleCli = $fakeCli.Replace("'opencode' { '--format --dir --model --auto --pure --continue --session' }", "'opencode' { '--format --dir --model --pure --continue --session' }") + [System.IO.File]::WriteAllText((Join-Path $fakeBin 'opencode.ps1'), $staleCli, [System.Text.UTF8Encoding]::new($false)) + $stalePreflight = Invoke-AdapterJson -RunnerPath (Join-Path $runnerRoot 'opencode\runner.ps1') -Command preflight -RunPath $with.Path -ProfilePath $recordedProfiles['opencode'] + Assert-Equal 'incompatible' $stalePreflight.status 'stale OpenCode help contract is rejected during preflight' + Assert-True (@($stalePreflight.reasons | Where-Object { $_ -match '--auto' }).Count -gt 0) 'stale OpenCode option failure identifies the missing flag' + [System.IO.File]::WriteAllText((Join-Path $fakeBin 'opencode.ps1'), $fakeCli, [System.Text.UTF8Encoding]::new($false)) + $fileAuthHome = Join-Path $recordedRoot 'codex-file-auth' + New-Item -ItemType Directory -Path $fileAuthHome -Force | Out-Null + [System.IO.File]::WriteAllText((Join-Path $fileAuthHome 'auth.json'), '{"canary":"not-logged"}', [System.Text.UTF8Encoding]::new($false)) + $env:OPENAI_API_KEY = $null + $env:CODEX_HOME = $fileAuthHome + $fileAuthPreflight = Invoke-AdapterJson -RunnerPath (Join-Path $runnerRoot 'codex\runner.ps1') -Command preflight -RunPath $with.Path -ProfilePath $recordedProfiles['codex'] + Assert-Equal 'incompatible' $fileAuthPreflight.status 'Codex file-only authentication is fail-closed' + Assert-True (@($fileAuthPreflight.reasons | Where-Object { $_ -match 'auth\.json' }).Count -gt 0) 'Codex file-auth limitation is explicit' + $env:OPENAI_API_KEY = 'recorded-canary-not-logged' + $env:CODEX_HOME = $recordedOldCodexHome + Write-Output 'Real runner deterministic adapter conformance: PASS' +} finally { + $env:PATH = $recordedOldPath + $env:OPENAI_API_KEY = $recordedOldOpenAi + $env:CODEX_HOME = $recordedOldCodexHome + $env:AGENTIC_GLOBAL_SECRET = $recordedOldGlobalSecret + $env:OPENCODE_DISABLE_PROJECT_CONFIG = $recordedOldProjectDisable + if (Test-Path -LiteralPath $recordedRoot) { Remove-Item -LiteralPath $recordedRoot -Recurse -Force } +} +} + function Assert-Equal { param([object]$Expected, [object]$Actual, [string]$Message) if ([string]$Expected -ne [string]$Actual) { throw "ASSERT: $Message (expected '$Expected', got '$Actual')" } @@ -235,6 +498,59 @@ try { $unknown = Invoke-Fake -FakePath $fakePath -Command execute -Run $without.Path -Profile $profilePath -Scenario unknown-event Assert-True (@($unknown.warnings | Where-Object { $_ -match 'future\.event\.v99' }).Count -gt 0) 'unknown events produce explicit warnings' + $resolvedWith = Resolve-RunContract -RunPath $with.Path + $resolvedProfile = Resolve-ExecutionProfile -ProfilePath $profilePath + $mandatoryCapabilities = [ordered]@{ + fresh_context = 'supported' + isolated_home_config = 'supported' + isolated_working_directory = 'supported' + ambient_candidate_skill_exclusion = 'supported' + candidate_skill_exposure = 'supported' + prompt_fidelity = 'supported' + model_configuration_lock = 'supported' + response_capture = 'supported' + } + $pragmaticCapabilities = [ordered]@{} + foreach ($name in $mandatoryCapabilities.Keys) { $pragmaticCapabilities[$name] = $mandatoryCapabilities[$name] } + $pragmaticCapabilities['filesystem_confinement'] = 'unsupported' + $strictCapabilities = [ordered]@{} + foreach ($name in $mandatoryCapabilities.Keys) { $strictCapabilities[$name] = $mandatoryCapabilities[$name] } + $strictCapabilities['filesystem_confinement'] = 'supported' + $pragmaticResult = New-ExecutionResult -Descriptor $descriptor -Profile $resolvedProfile -Run $resolvedWith -Status completed -FinalResponse 'pragmatic response' -ExitStatus ([Nullable[int]]0) -IsolationCapabilities $pragmaticCapabilities -AttemptCount 1 + [void](Assert-ExecutionResult -Result $pragmaticResult) + Assert-Equal 'completed' $pragmaticResult.status 'pragmatic completed result remains usable' + Assert-Equal 'pragmatic' $pragmaticResult.isolation.level 'missing hard confinement downgrades confidence' + Assert-True (-not $pragmaticResult.isolation.hard_filesystem_confinement) 'pragmatic result does not claim hard confinement' + $strictResult = New-ExecutionResult -Descriptor $descriptor -Profile $resolvedProfile -Run $resolvedWith -Status completed -FinalResponse 'strict response' -ExitStatus ([Nullable[int]]0) -IsolationCapabilities $strictCapabilities -AttemptCount 1 + [void](Assert-ExecutionResult -Result $strictResult) + Assert-Equal 'strict' $strictResult.isolation.level 'proven hard confinement reports strict isolation' + Assert-True $strictResult.isolation.hard_filesystem_confinement 'strict result claims hard confinement' + $failedResult = New-ExecutionResult -Descriptor $descriptor -Profile $resolvedProfile -Run $resolvedWith -Status failed -ExitStatus ([Nullable[int]]17) -Failure (New-ExecutionFailure -Code 'fixture_failure' -Message 'fixture failure') -IsolationCapabilities $pragmaticCapabilities -AttemptCount 1 + [void](Assert-ExecutionResult -Result $failedResult) + Assert-Equal 'failed' $failedResult.status 'failed execution keeps proven pragmatic isolation' + Assert-Equal 'verified' $failedResult.isolation.status 'failed execution retains control verification' + $timedOutResult = New-ExecutionResult -Descriptor $descriptor -Profile $resolvedProfile -Run $resolvedWith -Status timed_out -IsolationCapabilities $strictCapabilities -AttemptCount 1 + [void](Assert-ExecutionResult -Result $timedOutResult) + Assert-Equal 'timed_out' $timedOutResult.status 'timed out execution keeps proven strict isolation' + $missingCapability = [ordered]@{} + foreach ($name in $mandatoryCapabilities.Keys) { $missingCapability[$name] = $mandatoryCapabilities[$name] } + $missingCapability.Remove('response_capture') + $rejectedResult = New-ExecutionResult -Descriptor $descriptor -Profile $resolvedProfile -Run $resolvedWith -Status completed -FinalResponse 'must be rejected' -IsolationCapabilities $missingCapability -AttemptCount 1 + [void](Assert-ExecutionResult -Result $rejectedResult) + Assert-Equal 'incompatible' $rejectedResult.status 'unproven mandatory control rejects completion' + Assert-Equal 'unverified' $rejectedResult.isolation.status 'rejected completion is unverified' + Assert-Equal 'unsupported' $rejectedResult.isolation.level 'rejected completion has unsupported isolation' + $preflightRejected = New-ExecutionResult -Descriptor $descriptor -Profile $resolvedProfile -Run $resolvedWith -Status incompatible -FinalResponseReason 'preflight_incompatible' -IsolationCapabilities ([ordered]@{}) -AttemptCount 1 + [void](Assert-ExecutionResult -Result $preflightRejected) + Assert-Equal 'unverified' $preflightRejected.isolation.status 'preflight incompatibility is unverified' + Assert-Equal 'unsupported' $preflightRejected.isolation.level 'preflight incompatibility is unsupported' + $translated = Get-SandboxVisiblePath -HostPath (Join-Path $with.Root 'repo\file.txt') -RunRoot $with.Root -Platform 'linux' + Assert-Equal '/run/repo/file.txt' $translated 'Linux hard sandbox paths use the child namespace' + $macPath = Get-SandboxVisiblePath -HostPath (Join-Path $with.Root 'repo\file.txt') -RunRoot $with.Root -Platform 'macos' + Assert-Equal ([System.IO.Path]::GetFullPath((Join-Path $with.Root 'repo\file.txt'))) $macPath 'macOS sandbox paths remain host-visible' + Assert-Equal 'recorded-cli 1.2.3' (Get-ObservableVersionFromText "`nrecorded-cli 1.2.3`n") 'observable version capture keeps the exact line' + Assert-True ($null -eq (Get-ObservableVersionFromText "`n `n")) 'empty version output has no observable value' + foreach ($fixture in @('codex-events.jsonl', 'opencode-events.jsonl')) { $fixturePath = Join-Path $PSScriptRoot "fixtures\$fixture" $parsed = ConvertFrom-JsonLines -Text ([System.IO.File]::ReadAllText($fixturePath, [System.Text.UTF8Encoding]::new($false))) @@ -242,11 +558,17 @@ try { Assert-True ($parsed.Events.Count -ge 4) "recorded $fixture has events" Assert-True (@($parsed.Events | Where-Object { $_.type -eq 'future.event.v99' }).Count -eq 1) "recorded $fixture includes an unknown event" } + $clineFixture = ConvertFrom-JsonLines -Text ([System.IO.File]::ReadAllText((Join-Path $PSScriptRoot 'fixtures\cline-events.jsonl'), [System.Text.UTF8Encoding]::new($false))) + Assert-Equal 0 $clineFixture.Errors.Count 'recorded cline fixture has valid JSONL' + Assert-True ($clineFixture.Events.Count -ge 9) 'recorded cline fixture has events' + Assert-True (@($clineFixture.Events | Where-Object { $_.type -eq 'say' -and $_.say -eq 'tool' }).Count -eq 1) 'recorded cline fixture includes documented tool output' + Assert-True (@($clineFixture.Events | Where-Object { $_.type -eq 'say' -and $_.say -eq 'completion_result' }).Count -eq 1) 'recorded cline fixture includes documented completion_result output' + Assert-True (@($clineFixture.Events | Where-Object { $_.type -eq 'future.event.v99' }).Count -eq 1) 'recorded cline fixture includes an unknown event' $prepareText = [System.IO.File]::ReadAllText((Join-Path $repoRoot 'scripts\prepare-skill-evals.ps1'), [System.Text.UTF8Encoding]::new($false)) $reportText = [System.IO.File]::ReadAllText((Join-Path $repoRoot 'scripts\generate-eval-report.ps1'), [System.Text.UTF8Encoding]::new($false)) - Assert-True ($prepareText -notmatch '(?i)codex\s+exec|opencode\s+run') 'portable preparation must not contain harness-specific CLI invocations' - Assert-True ($reportText -notmatch '(?i)codex\s+exec|opencode\s+run') 'reporting must not contain harness-specific branches' + Assert-True ($prepareText -notmatch '(?i)codex\s+exec|opencode\s+run|cline\s+--') 'portable preparation must not contain harness-specific CLI invocations' + Assert-True ($reportText -notmatch '(?i)codex\s+exec|opencode\s+run|cline\s+--') 'reporting must not contain harness-specific branches' $rawPath = Join-Path $iteration 'conformance\results\with-skill.execution-result.json' $resultPath = Join-Path $iteration 'conformance\results\with-skill.result.json' @@ -264,8 +586,26 @@ try { Assert-True ($null -eq $portable.total_tokens) 'bridge keeps unavailable total tokens unavailable' Assert-Equal 0 $portable.tool_calls 'bridge carries available tool-call count' Assert-True $portable.isolation.transcript_captured 'bridge carries transcript availability' + Assert-Equal 'strict' $portable.isolation.level 'bridge carries isolation confidence level' + Assert-Equal 'verified' $portable.isolation.status 'bridge carries isolation verification status' + Assert-True (@($portable.isolation.mechanisms).Count -gt 0) 'bridge carries isolation mechanisms' Assert-True (@($portable.output_files).Count -gt 0) 'bridge carries confined evidence paths' + $acceptedBridgeResult = $bridgeResult | ConvertTo-Json -Depth 100 | ConvertFrom-Json + $acceptedBridgeResult.resolved.status = 'accepted_request' + $acceptedBridgeResult.resolved.provider = $null + $acceptedBridgeResult.resolved.model = $null + $acceptedBridgeResult.resolved.reason = 'fixture accepted the requested alias without exposing backend resolution.' + Write-TestJson -Path $rawPath -Value $acceptedBridgeResult + $acceptedBridgeOutput = & pwsh -NoProfile -File $bridgePath -Run $with.Path -ExecutionResult $rawPath -Result $resultPath + if ($LASTEXITCODE -ne 0) { throw "accepted-configuration bridge failed: $([string]::Join(' ', @($acceptedBridgeOutput)))" } + $acceptedPortable = Get-Content -LiteralPath $resultPath -Raw | ConvertFrom-Json + Assert-Equal 'fixture-model' $acceptedPortable.model 'bridge keeps requested model compatibility label' + Assert-Equal 'fixture-model' $acceptedPortable.requested_model 'bridge records requested model separately' + Assert-True ([string]::IsNullOrWhiteSpace([string]$acceptedPortable.resolved_model)) 'bridge does not invent a resolved model' + Assert-Equal 'accepted_request' $acceptedPortable.configuration_resolution_status 'bridge carries configuration provenance' + Assert-True ([string]$acceptedPortable.notes -match 'configuration_resolution=accepted_request') 'bridge notes configuration provenance' + Write-Output 'Eval Runner conformance: PASS' } finally { [Environment]::SetEnvironmentVariable('AGENTIC_FAKE_GLOBAL_RULES', $null, 'Process') @@ -273,3 +613,5 @@ try { [Environment]::SetEnvironmentVariable('AGENTIC_FAKE_PLUGINS', $null, 'Process') if (Test-Path -LiteralPath $testRoot) { Remove-Item -LiteralPath $testRoot -Recurse -Force } } + +Invoke-RecordedRunnerTests diff --git a/scripts/generate-eval-report.ps1 b/scripts/generate-eval-report.ps1 index 5481eb4..a0123be 100644 --- a/scripts/generate-eval-report.ps1 +++ b/scripts/generate-eval-report.ps1 @@ -401,6 +401,12 @@ function Get-ReportRun { feedback_key = "eval-$EvalId-$Configuration" model = [string](Get-Property -Object $Result -Name 'model' -Default '') provider = [string](Get-Property -Object $Result -Name 'provider' -Default '') + requested_model = [string](Get-Property -Object $Result -Name 'requested_model' -Default '') + requested_provider = [string](Get-Property -Object $Result -Name 'requested_provider' -Default '') + resolved_model = [string](Get-Property -Object $Result -Name 'resolved_model' -Default '') + resolved_provider = [string](Get-Property -Object $Result -Name 'resolved_provider' -Default '') + configuration_resolution_status = [string](Get-Property -Object $Result -Name 'configuration_resolution_status' -Default '') + configuration_resolution_reason = [string](Get-Property -Object $Result -Name 'configuration_resolution_reason' -Default '') harness = [string](Get-Property -Object $Result -Name 'harness' -Default '') executed_utc = [string](Get-Property -Object $Result -Name 'executed_utc' -Default '') output = $output diff --git a/scripts/prepare-skill-evals.ps1 b/scripts/prepare-skill-evals.ps1 index 13b2951..eede3f3 100644 --- a/scripts/prepare-skill-evals.ps1 +++ b/scripts/prepare-skill-evals.ps1 @@ -1052,8 +1052,9 @@ function New-RunManifest { } } -# Fail package generation the moment a run violates an isolation invariant, so a contaminated package never reaches a -# harness. These checks operate on the materialized run directories, not on prose. +# Fail package generation the moment a run violates an experimental isolation invariant, so a contaminated package never +# reaches a harness. The filesystemIsolationRequired field declares the staged workspace boundary; hard OS confinement is +# evaluated separately by the selected runner and reported as strict or pragmatic confidence. function Assert-RunIsolation { param( [string]$EvalName, @@ -1106,7 +1107,7 @@ function Assert-RunIsolation { throw "$EvalName/$configuration run.json must require fresh context." } if (-not [bool]$runManifest.filesystemIsolationRequired -or -not [bool]$runManifest.isolatedHomeRequired) { - throw "$EvalName/$configuration run.json must require filesystem and home isolation." + throw "$EvalName/$configuration run.json must require the staged workspace boundary and isolated home." } # 6. No run manifest references the source repository, and 7. none references a global skill install. diff --git a/scripts/validate-skill-templates.ps1 b/scripts/validate-skill-templates.ps1 index e7f176f..0831431 100644 --- a/scripts/validate-skill-templates.ps1 +++ b/scripts/validate-skill-templates.ps1 @@ -1432,14 +1432,14 @@ Add-ValidationResult -Results $results -Name 'Skill evaluation prepares portable } } - # 10. run.json requires fresh context and isolation; 6/7. it references nothing outside the run package. + # 10. run.json requires fresh context, the staged workspace boundary, and isolated home; 6/7. it references nothing outside the run package. $withRunJson = [System.IO.File]::ReadAllText((Join-Path $withRunDir 'run.json'), $utf8NoBom) $withoutRunJson = [System.IO.File]::ReadAllText((Join-Path $withoutRunDir 'run.json'), $utf8NoBom) $withRun = $withRunJson | ConvertFrom-Json $withoutRun = $withoutRunJson | ConvertFrom-Json foreach ($run in @($withRun, $withoutRun)) { if (-not [bool]$run.freshContextRequired -or -not [bool]$run.filesystemIsolationRequired -or -not [bool]$run.isolatedHomeRequired) { - throw "$($entry.eval_name) run.json must require fresh context, filesystem, and home isolation." + throw "$($entry.eval_name) run.json must require fresh context, the staged workspace boundary, and isolated home." } if ([string]$run.workingDirectory -ne 'repo' -or [string]$run.homeDirectory -ne 'home') { throw "$($entry.eval_name) run.json must set workingDirectory=repo and homeDirectory=home." @@ -1548,7 +1548,15 @@ Add-ValidationResult -Results $results -Name 'Skill evaluation prepares portable [int]$profile.timeout_seconds -lt 1 -or [int]$profile.concurrency -lt 1) { throw 'execution-profile.json has an invalid schema or execution limit.' } - foreach ($runnerTool in @('runner-common.ps1', 'resolve-runner.ps1', 'bridge-execution-result.ps1', 'fake/runner.ps1', 'codex/runner.ps1', 'opencode/runner.ps1', 'contracts/execution-profile.schema.json', 'contracts/execution-result.schema.json')) { + $runnerTools = [System.Collections.Generic.List[string]]::new() + foreach ($runnerTool in @('runner-common.ps1', 'resolve-runner.ps1', 'bridge-execution-result.ps1', 'contracts/execution-profile.schema.json', 'contracts/execution-result.schema.json')) { $runnerTools.Add($runnerTool) } + $runnerSourceRoot = Join-Path $repoRoot 'scripts/eval-runners' + foreach ($runnerDirectory in Get-ChildItem -LiteralPath $runnerSourceRoot -Directory -Force | Sort-Object Name) { + if (Test-Path -LiteralPath (Join-Path $runnerDirectory.FullName 'runner.ps1') -PathType Leaf) { + $runnerTools.Add("$($runnerDirectory.Name)/runner.ps1") + } + } + foreach ($runnerTool in $runnerTools) { if (-not (Test-Path -LiteralPath (Join-Path $iterationDirectory "tools/eval-runners/$runnerTool") -PathType Leaf)) { throw "Prepared package is missing runner tool '$runnerTool'." } From dc29f6815be4feca4b38f58ce2b91361ff060182 Mon Sep 17 00:00:00 2001 From: "aicia[bot]" Date: Fri, 21 Aug 2026 21:46:27 +0200 Subject: [PATCH 05/39] =?UTF-8?q?=E2=9C=A8=20introduce=20github=20copilot?= =?UTF-8?q?=20eval=20runner?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add GitHub Copilot CLI as a supported Eval Runner alongside Cline, Codex, and OpenCode. The runner handles Copilot-specific authentication (GitHub tokens), isolation constraints, and JSONL-based event output parsing. Includes conformance tests and reference documentation for the runner protocol compliance. --- scripts/eval-runners/README.md | 45 +- .../eval-runners/github-copilot/runner.ps1 | 606 ++++++++++++++++++ .../tests/fixtures/copilot-events.jsonl | 9 + .../tests/test-runner-conformance.ps1 | 115 +++- 4 files changed, 747 insertions(+), 28 deletions(-) create mode 100644 scripts/eval-runners/github-copilot/runner.ps1 create mode 100644 scripts/eval-runners/tests/fixtures/copilot-events.jsonl diff --git a/scripts/eval-runners/README.md b/scripts/eval-runners/README.md index cda4723..9378fc6 100644 --- a/scripts/eval-runners/README.md +++ b/scripts/eval-runners/README.md @@ -44,22 +44,41 @@ proven reports `strict` isolation when hard confinement is proven and boundary, prompt fidelity, result capture, or other mandatory control remains incompatible. -The fake runner is deterministic and is the conformance reference. Codex, -OpenCode, and Cline are thin harness-specific adapters. Their native CLI -flags, environment setup, event parsing, authentication injection, and -isolation checks stay inside their own directories. Windows is supported in +The fake runner is deterministic and is the conformance reference. GitHub +Copilot, Codex, OpenCode, and Cline are thin harness-specific adapters. Their +native CLI flags, environment setup, event parsing, authentication injection, +and isolation checks stay inside their own directories. Windows is supported in pragmatic mode when the native CLI satisfies the mandatory controls. -Codex uses `--ask-for-approval never` with `exec --sandbox workspace-write`; -it does not combine explicit sandbox selection with `--approve-for-me`. -OpenCode uses `run --format json --auto` with isolated global/config roots and -preserves repository-owned project configuration; it does not depend on -`OPENCODE_DISABLE_PROJECT_CONFIG` or use `--pure`. Cline uses `--json`, -`--auto-approve true`, `--retries 0`, `--config `, +`github-copilot` with `claude-haiku-4.5` is the Codebelt reference evaluation +configuration: a stable, economical pairing for routine skill comparison. It is +a repository convention, not an Anthropic default, and the model stays +configurable through `execution-profile.json`, so any Copilot-served model can +be selected. Cross-runner and cross-model numbers are never blended into one +score; a paired `with_skill` versus `without_skill` comparison is only +meaningful within one identical runner, model, and configuration stratum. + +GitHub Copilot uses `copilot --prompt --output-format json --model + --allow-all-tools --no-ask-user --no-custom-instructions +--disable-builtin-mcps` with an isolated `COPILOT_HOME`; it passes no +`--resume`, `--continue`, `--session-id`, or `--connect`, and it does not use +the blanket `--yolo`, `--allow-all`, `--allow-all-paths`, or `--allow-all-urls` +switches. It authenticates from a narrow GitHub token (`COPILOT_GITHUB_TOKEN`, +`GH_TOKEN`, or `GITHUB_TOKEN` - for example `gh auth token` or a fine-grained +PAT with Copilot access) rather than importing the ambient `copilot login` +profile, and redacts that token from output with `--secret-env-vars`. Because +Copilot co-mingles login state with behavioral configuration in `COPILOT_HOME`, +the runner requires that separable token and classifies honestly rather than +copying the profile. Codex uses `--ask-for-approval never` with `exec --sandbox +workspace-write`; it does not combine explicit sandbox selection with +`--approve-for-me`. OpenCode uses `run --format json --auto` with isolated +global/config roots and preserves repository-owned project configuration; it +does not depend on `OPENCODE_DISABLE_PROJECT_CONFIG` or use `--pure`. Cline uses +`--json`, `--auto-approve true`, `--retries 0`, `--config `, `--data-dir /.cline/data`, and run-local hooks; it passes no session -id. All three capture an exact observable CLI version and pass only a narrow -provider environment credential when available. None copies a global skill -directory, memory store, plugin set, or normal agent profile into a run. +id. Each captures an exact observable CLI version and passes only a narrow +environment credential when available. None copies a global skill directory, +memory store, plugin set, or normal agent profile into a run. Freebuff is currently documented as planned/blocked. Its supported CLI remains TUI-oriented and does not provide the required one-prompt, noninteractive, diff --git a/scripts/eval-runners/github-copilot/runner.ps1 b/scripts/eval-runners/github-copilot/runner.ps1 new file mode 100644 index 0000000..e802933 --- /dev/null +++ b/scripts/eval-runners/github-copilot/runner.ps1 @@ -0,0 +1,606 @@ +<#! +.SYNOPSIS + GitHub Copilot CLI Eval Runner adapter. + +.DESCRIPTION + This is the only place where GitHub Copilot CLI flags, COPILOT_HOME + handling, non-interactive JSONL event parsing, GitHub-token authentication, + and Copilot isolation limitations are defined. It implements the unchanged + describe/preflight/execute process contract shared by every runner. + + Copilot with claude-haiku-4.5 is the Codebelt reference evaluation + configuration. The reference is a repository convention for economical, + stable comparison; it is not an Anthropic default. The model stays fully + configurable through execution-profile.json, so any Copilot-served model can + be selected. +#> +[CmdletBinding()] +param( + [Parameter(Mandatory = $true, Position = 0)] + [ValidateSet('describe', 'preflight', 'execute')] + [string]$Command, + + [string]$Run, + [string]$Profile +) + +$ErrorActionPreference = 'Stop' +Set-StrictMode -Version Latest +. (Join-Path $PSScriptRoot '..\runner-common.ps1') + +# GitHub Copilot authenticates from a GitHub token, not a model-provider API +# key. These are the supported token variables in precedence order; only the +# present one is forwarded into the isolated worker environment. +$copilotAuthVariables = @('COPILOT_GITHUB_TOKEN', 'GH_TOKEN', 'GITHUB_TOKEN') +# Model routing runs through GitHub Copilot; the profile provider names the +# routing backend, not a direct model vendor. +$copilotProviders = @('github', 'github-copilot', 'copilot') + +$descriptor = [ordered]@{ + schema = (Get-RunnerSchemaNames).Descriptor + protocol_version = (Get-RunnerSchemaNames).Protocol + name = 'github-copilot' + version = '0.9.1' + platforms = @('windows', 'linux', 'macos') + harness = [ordered]@{ name = 'GitHub Copilot CLI'; version = 'unavailable' } + capabilities = [ordered]@{ + fresh_context = 'supported' + isolated_home_config = 'supported' + isolated_working_directory = 'supported' + filesystem_confinement = 'conditional' + ambient_candidate_skill_exclusion = 'supported' + candidate_skill_exposure = 'supported' + prompt_fidelity = 'supported' + model_configuration_lock = 'supported' + response_capture = 'supported' + transcript_event_capture = 'supported' + token_telemetry = 'conditional' + cache_token_telemetry = 'conditional' + tool_call_telemetry = 'conditional' + command_evidence = 'conditional' + file_evidence = 'conditional' + cost_telemetry = 'unsupported' + credential_child_filtering = 'conditional' + native_skill_activation_evidence = 'unsupported' + } + supported_telemetry = @('transcript_event_capture', 'token_telemetry', 'cache_token_telemetry', 'tool_call_telemetry', 'command_evidence', 'file_evidence') + configuration_profiles = @('isolated-default') + tool_profiles = @('default') +} + +function Write-ProtocolError { + param([string]$Message) + + [Console]::Error.WriteLine($Message) + exit 2 +} + +function Resolve-CopilotInputs { + if ([string]::IsNullOrWhiteSpace($Run) -or [string]::IsNullOrWhiteSpace($Profile)) { + throw 'preflight and execute require -Run and -Profile.' + } + return [pscustomobject]@{ + Run = Resolve-RunContract -RunPath $Run + Profile = Resolve-ExecutionProfile -ProfilePath $Profile + } +} + +function Get-CopilotTokenVariable { + foreach ($name in $copilotAuthVariables) { + if (-not [string]::IsNullOrWhiteSpace([Environment]::GetEnvironmentVariable($name))) { + return $name + } + } + return $null +} + +function Test-CopilotLoginProfileExists { + # Existence check only; the file is never read or logged so no credential or + # login value is exposed. The default location is COPILOT_HOME or the + # platform user profile's .copilot directory. + $configuredHome = [Environment]::GetEnvironmentVariable('COPILOT_HOME') + $homeRoot = if ([string]::IsNullOrWhiteSpace($configuredHome)) { + Join-Path ([Environment]::GetFolderPath('UserProfile')) '.copilot' + } else { + $configuredHome + } + return (Test-Path -LiteralPath (Join-Path $homeRoot 'config.json') -PathType Leaf) +} + +function Invoke-CopilotCli { + param( + [Parameter(Mandatory = $true)][object]$CommandInfo, + [Parameter(Mandatory = $true)][string[]]$Arguments, + [Parameter(Mandatory = $true)][object]$Inputs, + [System.Collections.IDictionary]$Environment, + [byte[]]$InputBytes = @(), + [int]$TimeoutSeconds = 60 + ) + + $allArguments = @($CommandInfo.Prefix) + @($Arguments) + return Invoke-RunnerProcess -FileName $CommandInfo.FileName -ArgumentList $allArguments -WorkingDirectory $Inputs.Run.WorkingDirectoryPath -Environment $Environment -InputBytes $InputBytes -TimeoutSeconds $TimeoutSeconds +} + +function Get-CopilotHelpResult { + param( + [Parameter(Mandatory = $true)][object]$CommandInfo, + [Parameter(Mandatory = $true)][object]$Inputs, + [string[]]$Arguments = @('--help') + ) + + $environment = New-RunnerEnvironment -Run $Inputs.Run + return Invoke-CopilotCli -CommandInfo $CommandInfo -Arguments $Arguments -Inputs $Inputs -Environment $environment -TimeoutSeconds 30 +} + +function Resolve-SandboxCommand { + param([Parameter(Mandatory = $true)][string]$Name) + + return Resolve-ExternalCommand -Name $Name +} + +function Get-CopilotDescriptor { + $copy = [ordered]@{} + foreach ($key in $descriptor.Keys) { $copy[$key] = $descriptor[$key] } + $commandInfo = Resolve-ExternalCommand -Name 'copilot' + $version = 'unavailable' + if ($null -ne $commandInfo) { + $observation = Get-ExternalCommandVersion -CommandInfo $commandInfo + $version = [string]$observation.Version + } + $copy.harness = [ordered]@{ name = 'GitHub Copilot CLI'; version = $version } + return $copy +} + +function Get-CopilotPromptText { + param([Parameter(Mandatory = $true)][object]$Inputs) + + # The prompt is delivered verbatim through --prompt. run.json stages UTF-8 + # prompt bytes, so decoding to a string and forwarding it as one argv value + # round-trips the exact characters; prompt_sha256 in the result confirms it. + return [System.Text.UTF8Encoding]::new($false).GetString($Inputs.Run.PromptBytes) +} + +function New-CopilotCliArguments { + param( + [Parameter(Mandatory = $true)][object]$Inputs, + [ValidateSet('windows', 'linux', 'macos', 'unknown')][string]$VisiblePlatform = (Get-PlatformName) + ) + + $directoryArgument = Get-SandboxVisiblePath -HostPath $Inputs.Run.WorkingDirectoryPath -RunRoot $Inputs.Run.RunRoot -Platform $VisiblePlatform + $secretList = ($copilotAuthVariables -join ',') + $arguments = [System.Collections.Generic.List[string]]::new() + # Noninteractive one-shot JSONL run. --allow-all-tools is the narrow switch + # that removes tool-approval prompts without --allow-all-paths or + # --allow-all-urls, so file access stays inside the working directory and + # temp roots and URL/network access is not blanket-granted. --no-ask-user + # keeps the agent from pausing for questions. --no-custom-instructions stops + # ambient AGENTS.md / instruction discovery (including the host repository's) + # from leaking into either arm. --disable-builtin-mcps drops the ambient + # GitHub MCP server. --secret-env-vars redacts the GitHub token from output. + foreach ($argument in @( + '-C', $directoryArgument, + '--model', $Inputs.Profile.Model, + '--output-format', 'json', + '--allow-all-tools', + '--no-ask-user', + '--no-custom-instructions', + '--disable-builtin-mcps', + '--no-color', + '--log-level', 'none', + '--no-auto-update', + ('--secret-env-vars=' + $secretList) + )) { + $arguments.Add([string]$argument) + } + if (-not [string]::IsNullOrWhiteSpace([string]$Inputs.Profile.ReasoningEffort)) { + $arguments.Add('--reasoning-effort') + $arguments.Add([string]$Inputs.Profile.ReasoningEffort) + } + $arguments.Add('--prompt') + $arguments.Add((Get-CopilotPromptText -Inputs $Inputs)) + return @($arguments) +} + +function Get-CopilotCapabilityMap { + param( + [Parameter(Mandatory = $true)][object]$Inputs, + [bool]$HardFilesystemConfinement = $false + ) + + $capabilities = [ordered]@{} + foreach ($capabilityName in @(Get-JsonPropertyNames -Object $descriptor.capabilities)) { + $capabilities[$capabilityName] = [string](Get-JsonProperty -Object $descriptor.capabilities -Name $capabilityName) + } + $capabilities['filesystem_confinement'] = if ($HardFilesystemConfinement) { 'supported' } else { 'unsupported' } + $capabilities['candidate_skill_exposure'] = if ($Inputs.Run.CandidateSkillExposed) { 'supported' } else { 'excluded' } + return $capabilities +} + +function Get-CopilotPreflight { + param([Parameter(Mandatory = $true)][object]$Inputs) + + $checks = [System.Collections.Generic.List[object]]::new() + $reasons = [System.Collections.Generic.List[string]]::new() + $warnings = [System.Collections.Generic.List[string]]::new() + $profile = $Inputs.Profile + $run = $Inputs.Run + $commandInfo = Resolve-ExternalCommand -Name 'copilot' + $platform = Get-PlatformName + $sandboxInfo = if ($platform -eq 'linux') { Resolve-SandboxCommand -Name 'bwrap' } elseif ($platform -eq 'macos') { Resolve-SandboxCommand -Name 'sandbox-exec' } else { $null } + $versionObservation = $null + + if ($profile.Runner -ne 'github-copilot') { + $reasons.Add("execution-profile.json selects '$($profile.Runner)' rather than github-copilot.") + } else { + $checks.Add((New-PreflightCheck -Name 'runner_selection' -Status passed -Detail 'The selected runner is github-copilot.')) + } + if ([string]::IsNullOrWhiteSpace($profile.Provider) -or $profile.Provider.ToLowerInvariant() -notin $copilotProviders) { + $reasons.Add("GitHub Copilot requires provider 'github-copilot' (also accepts 'github' or 'copilot'); received '$($profile.Provider)'.") + } else { + $checks.Add((New-PreflightCheck -Name 'provider' -Status passed -Detail $profile.Provider)) + } + if ([string]::IsNullOrWhiteSpace($profile.Model)) { + $reasons.Add('GitHub Copilot requires a model in execution-profile.json (claude-haiku-4.5 is the Codebelt reference).') + } else { + $checks.Add((New-PreflightCheck -Name 'model' -Status passed -Detail $profile.Model)) + } + if ($profile.ConfigurationProfile -ne 'isolated-default') { + $reasons.Add("configuration_profile '$($profile.ConfigurationProfile)' is unsupported by github-copilot.") + } + if ($profile.ToolProfile -ne 'default') { + $reasons.Add("tool_profile '$($profile.ToolProfile)' is unsupported by github-copilot.") + } + + if ($null -eq $commandInfo) { + $reasons.Add('The GitHub Copilot CLI executable is not available on PATH.') + } else { + $checks.Add((New-PreflightCheck -Name 'harness_executable' -Status passed -Detail $commandInfo.Source)) + try { + $versionObservation = Get-ExternalCommandVersion -CommandInfo $commandInfo -WorkingDirectory $run.WorkingDirectoryPath -Environment (New-RunnerEnvironment -Run $run) -TimeoutSeconds 30 + if (-not $versionObservation.Available) { + $reasons.Add('The GitHub Copilot CLI did not expose an exact observable version through --version.') + $checks.Add((New-PreflightCheck -Name 'harness_version' -Status unavailable -Detail 'copilot --version did not return a usable version string.')) + } else { + $checks.Add((New-PreflightCheck -Name 'harness_version' -Status passed -Detail ([string]$versionObservation.Version))) + } + $help = Get-CopilotHelpResult -CommandInfo $commandInfo -Inputs $Inputs + if ($help.TimedOut -or $help.ExitCode -ne 0) { + $reasons.Add("Copilot --help failed with exit status $($help.ExitCode).") + } else { + $helpText = [string]::Join("`n", @($help.Stdout, $help.Stderr)) + foreach ($flag in @('--prompt', '--output-format', '--model', '--allow-all-tools', '--no-ask-user', '--no-custom-instructions', '--disable-builtin-mcps', '--secret-env-vars')) { + if ($helpText -notmatch [regex]::Escape($flag)) { + $reasons.Add("The installed Copilot CLI does not advertise required flag '$flag'.") + } + } + $visibleForConstruction = if ($platform -eq 'linux' -and $null -ne $sandboxInfo) { 'linux' } else { $platform } + $constructed = New-CopilotCliArguments -Inputs $Inputs -VisiblePlatform $visibleForConstruction + foreach ($forbidden in @('--resume', '-r', '--continue', '--session-id', '--connect', '--yolo', '--allow-all', '--allow-all-paths', '--allow-all-urls')) { + if (@($constructed) -contains $forbidden) { $reasons.Add("The constructed Copilot invocation must not use session-continuation or over-broad permission option '$forbidden'.") } + } + foreach ($required in @('--prompt', '--output-format', '--allow-all-tools', '--no-ask-user', '--no-custom-instructions')) { + if (@($constructed) -notcontains $required) { $reasons.Add("The constructed Copilot invocation must include '$required'.") } + } + $promptCount = @($constructed | Where-Object { $_ -eq '--prompt' -or $_ -eq '-p' }).Count + if ($promptCount -ne 1) { $reasons.Add('The constructed Copilot invocation must deliver the prompt exactly once.') } + if ($reasons.Count -eq 0) { + $checks.Add((New-PreflightCheck -Name 'harness_contract' -Status passed -Detail 'Copilot accepts the constructed noninteractive invocation: --prompt once, --output-format json, --model, --allow-all-tools, --no-ask-user, --no-custom-instructions, --disable-builtin-mcps, and no session continuation.')) + } + } + } catch { + $reasons.Add("Could not inspect Copilot CLI capabilities: $($_.Exception.Message)") + } + } + + $tokenVariable = Get-CopilotTokenVariable + if (-not [string]::IsNullOrWhiteSpace($tokenVariable)) { + $checks.Add((New-PreflightCheck -Name 'authentication' -Status passed -Detail "Authentication is available through the narrow $tokenVariable GitHub token; the ambient Copilot login profile is not copied into the run.")) + } elseif (Test-CopilotLoginProfileExists) { + $reasons.Add('A Copilot login profile exists at the default location, but GitHub Copilot co-mingles login state with behavioral configuration in COPILOT_HOME. The runner requires a separable GitHub token (COPILOT_GITHUB_TOKEN, GH_TOKEN, or GITHUB_TOKEN - for example `gh auth token` or a fine-grained PAT with Copilot access) rather than importing that profile.') + } else { + $reasons.Add('No GitHub Copilot authentication is available. Export COPILOT_GITHUB_TOKEN (or GH_TOKEN/GITHUB_TOKEN) with Copilot access.') + } + + if ($platform -notin @('linux', 'macos')) { + $checks.Add((New-PreflightCheck -Name 'filesystem_confinement' -Status not_applicable -Detail "Platform '$platform' has no configured external hard-confinement mechanism; pragmatic isolation remains available.")) + $warnings.Add("Platform '$platform' has no external hard filesystem confinement in this adapter; execution will report pragmatic isolation.") + } elseif ($null -eq $sandboxInfo) { + $sandboxName = if ($platform -eq 'linux') { 'bwrap' } else { 'sandbox-exec' } + $checks.Add((New-PreflightCheck -Name 'filesystem_confinement' -Status unavailable -Detail "External '$sandboxName' is unavailable; pragmatic isolation remains available.")) + $warnings.Add("External '$sandboxName' was unavailable; execution will report pragmatic isolation.") + } else { + $checks.Add((New-PreflightCheck -Name 'filesystem_confinement' -Status passed -Detail "External $($sandboxInfo.Source) sandbox confines Copilot to the staged run and required system runtime paths.")) + } + + $checks.Add((New-PreflightCheck -Name 'fresh_session' -Status passed -Detail 'The adapter starts one new copilot -p process and passes no --resume, --continue, --session-id, or --connect.')) + $checks.Add((New-PreflightCheck -Name 'ambient_configuration' -Status passed -Detail 'The adapter points COPILOT_HOME at the run''s isolated home, disables built-in MCP servers, and disables ambient custom instructions, so global skills, plugins, MCP config, sessions, memories, and instructions do not load.')) + $checks.Add((New-PreflightCheck -Name 'run_paths' -Status passed -Detail "-C $($run.WorkingDirectoryPath); COPILOT_HOME under $($run.HomeDirectoryPath)")) + $checks.Add((New-PreflightCheck -Name 'prompt_fidelity' -Status passed -Detail 'The exact UTF-8 prompt bytes are delivered once as the --prompt argument value.')) + $checks.Add((New-PreflightCheck -Name 'credential_boundary' -Status passed -Detail 'Only the present GitHub token variable is forwarded; --secret-env-vars redacts it from Copilot output; no login profile or auth file is copied into the run.')) + $warnings.Add('GitHub Copilot inherits the shell environment for shell tools apart from a blocklist; the runner redacts the GitHub token from output with --secret-env-vars but cannot independently prove the child-tool environment filter hides it from every tool path.') + if ($platform -eq 'macos') { + $warnings.Add('macOS sandbox-exec is deprecated by Apple but is used only when present; a future runner revision may replace it with an equivalent supported mechanism.') + } + $promptLength = (Get-CopilotPromptText -Inputs $Inputs).Length + if ($promptLength -gt 30000) { + $warnings.Add('The prompt exceeds 30000 characters; on some hosts a very long --prompt argument can approach the operating-system command-line length limit.') + } + + $hardConfinement = $null -ne $sandboxInfo -and $platform -in @('linux', 'macos') + $capabilities = Get-CopilotCapabilityMap -Inputs $Inputs -HardFilesystemConfinement $hardConfinement + $harnessVersion = if ($null -eq $versionObservation) { 'unavailable' } else { [string]$versionObservation.Version } + $descriptorCopy = [ordered]@{} + foreach ($key in $descriptor.Keys) { $descriptorCopy[$key] = $descriptor[$key] } + $descriptorCopy.harness = [ordered]@{ name = 'GitHub Copilot CLI'; version = $harnessVersion } + $mechanisms = [System.Collections.Generic.List[string]]::new() + foreach ($mechanism in @('copilot --prompt --output-format json', '--allow-all-tools', '--no-ask-user', '--no-custom-instructions', '--disable-builtin-mcps', '--secret-env-vars token redaction', 'isolated COPILOT_HOME', 'isolated HOME/XDG roots', 'narrow GitHub token authentication', 'no session continuation')) { $mechanisms.Add($mechanism) } + if ($hardConfinement) { $mechanisms.Add("external $($sandboxInfo.Source) filesystem sandbox") } else { $mechanisms.Add('pragmatic process/environment isolation without hard filesystem confinement') } + return New-PreflightDocument -Descriptor $descriptorCopy -Profile $profile -Run $run -Compatible ($reasons.Count -eq 0) -Checks @($checks) -Mechanisms @($mechanisms) -ResolvedCapabilities $capabilities -Warnings @($warnings) -Reasons @($reasons) +} + +function New-CopilotEnvironment { + param([Parameter(Mandatory = $true)][object]$Inputs) + + $copilotHome = Join-Path $Inputs.Run.HomeDirectoryPath '.copilot' + New-Item -ItemType Directory -Path $copilotHome -Force | Out-Null + return New-RunnerEnvironment -Run $Inputs.Run -AuthenticationVariables $copilotAuthVariables -Additional @{ + COPILOT_HOME = $copilotHome + COPILOT_AUTO_UPDATE = 'false' + } +} + +function New-CopilotInsideEnvironment { + param( + [Parameter(Mandatory = $true)][object]$Inputs, + [Parameter(Mandatory = $true)][System.Collections.IDictionary]$Environment + ) + + $insideEnvironment = [ordered]@{ + HOME = '/run/home' + USERPROFILE = '/run/home' + XDG_CONFIG_HOME = '/run/home/.config' + XDG_DATA_HOME = '/run/home/.local/share' + XDG_CACHE_HOME = '/run/home/.cache' + TEMP = '/run/home/tmp' + TMP = '/run/home/tmp' + COPILOT_HOME = '/run/home/.copilot' + COPILOT_AUTO_UPDATE = 'false' + PATH = '/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin' + CI = '1' + NO_COLOR = '1' + } + foreach ($authName in $copilotAuthVariables) { + if ($Environment.Contains($authName) -and -not [string]::IsNullOrWhiteSpace([string]$Environment[$authName])) { + $insideEnvironment[$authName] = [string]$Environment[$authName] + } + } + return $insideEnvironment +} + +function Write-CopilotCapture { + param( + [Parameter(Mandatory = $true)][object]$RunData, + [Parameter(Mandatory = $true)][string]$RelativePath, + [Parameter(Mandatory = $true)][AllowEmptyString()][string]$Text + ) + + $path = Join-Path $RunData.Run.RunRoot ($RelativePath -replace '/', [System.IO.Path]::DirectorySeparatorChar) + New-Item -ItemType Directory -Path (Split-Path -Parent $path) -Force | Out-Null + [System.IO.File]::WriteAllText($path, $Text, [System.Text.UTF8Encoding]::new($false)) + return New-ArtifactReference -Run $RunData.Run -Path $RelativePath -Scope run -MediaType (Get-MediaType -Path $RelativePath) +} + +function Add-NullableInt64 { + param([object]$Current, [object]$Value) + + if ($null -eq $Value) { return $Current } + if ($null -eq $Current) { return [int64]$Value } + return ([int64]$Current + [int64]$Value) +} + +function Read-CopilotEvents { + param( + [Parameter(Mandatory = $true)][object]$Parsed, + [Parameter(Mandatory = $true)][AllowEmptyCollection()][System.Collections.Generic.List[string]]$Warnings + ) + + $assistantContents = [System.Collections.Generic.List[string]]::new() + $finalText = $null + $observedModel = $null + $usageInput = $null + $usageOutput = $null + $usageCacheRead = $null + $usageCacheWrite = $null + $usageNumToolCalls = 0 + $usageSeen = $false + $toolStarts = 0 + $sessionError = $null + $eventCounts = @{} + + foreach ($event in @($Parsed.Events)) { + $eventType = [string](Get-JsonProperty -Object $event -Name 'type' -Default '') + if ([string]::IsNullOrWhiteSpace($eventType)) { + $Warnings.Add('Copilot emitted an event without a type; it was ignored.') + continue + } + if ($eventCounts.ContainsKey($eventType)) { $eventCounts[$eventType]++ } else { $eventCounts[$eventType] = 1 } + $data = Get-JsonProperty -Object $event -Name 'data' -Default $null + switch ($eventType) { + 'assistant.message' { + $content = [string](Get-JsonProperty -Object $data -Name 'content' -Default '') + if (-not [string]::IsNullOrWhiteSpace($content)) { + $assistantContents.Add($content) + $finalText = $content + } + $model = [string](Get-JsonProperty -Object $data -Name 'model' -Default '') + if (-not [string]::IsNullOrWhiteSpace($model)) { $observedModel = $model } + } + 'assistant.usage' { + $usageSeen = $true + $usageInput = Add-NullableInt64 -Current $usageInput -Value (Get-JsonProperty -Object $data -Name 'inputTokens' -Default $null) + $usageOutput = Add-NullableInt64 -Current $usageOutput -Value (Get-JsonProperty -Object $data -Name 'outputTokens' -Default $null) + $usageCacheRead = Add-NullableInt64 -Current $usageCacheRead -Value (Get-JsonProperty -Object $data -Name 'cacheReadTokens' -Default $null) + $usageCacheWrite = Add-NullableInt64 -Current $usageCacheWrite -Value (Get-JsonProperty -Object $data -Name 'cacheWriteTokens' -Default $null) + $numToolCalls = Get-JsonProperty -Object $data -Name 'numToolCalls' -Default $null + if ($null -ne $numToolCalls) { $usageNumToolCalls += [int]$numToolCalls } + $model = [string](Get-JsonProperty -Object $data -Name 'model' -Default '') + if (-not [string]::IsNullOrWhiteSpace($model)) { $observedModel = $model } + } + 'tool.execution_start' { $toolStarts++ } + 'session.error' { $sessionError = [string](Get-JsonProperty -Object $data -Name 'message' -Default 'Copilot reported a session error.') } + { $_ -in @('session.start', 'session.info', 'session.idle', 'session.shutdown', 'session.task_complete', 'user.message', 'assistant.message_start', 'assistant.message_delta', 'assistant.turn_start', 'assistant.turn_end', 'assistant.reasoning', 'assistant.tool_call_delta', 'tool.execution_progress', 'tool.execution_partial_result', 'tool.execution_complete', 'command.execute', 'command.completed', 'session.usage_info', 'session.usage_checkpoint') } { } + default { $Warnings.Add("Unknown Copilot event '$eventType' was preserved as a warning.") } + } + } + + if ([string]::IsNullOrWhiteSpace($finalText) -and $assistantContents.Count -gt 0) { + $finalText = [string]::Join("`n", $assistantContents) + } + $toolCalls = if ($toolStarts -gt 0) { $toolStarts } else { $usageNumToolCalls } + + return [pscustomobject]@{ + FinalText = $finalText + ObservedModel = $observedModel + UsageSeen = $usageSeen + UsageInput = $usageInput + UsageOutput = $usageOutput + UsageCacheRead = $usageCacheRead + UsageCacheWrite = $usageCacheWrite + ToolCalls = $toolCalls + SessionError = $sessionError + EventCounts = $eventCounts + } +} + +function Invoke-CopilotExecute { + param([Parameter(Mandatory = $true)][object]$Inputs) + + $preflight = Get-CopilotPreflight -Inputs $Inputs + $started = [DateTime]::UtcNow + $sessionId = [Guid]::NewGuid().ToString('D') + $executionDescriptor = [ordered]@{} + foreach ($key in $descriptor.Keys) { $executionDescriptor[$key] = $descriptor[$key] } + $executionDescriptor.harness = $preflight.harness + if ($preflight.status -ne 'compatible') { + $finished = [DateTime]::UtcNow + return New-ExecutionResult -Descriptor $executionDescriptor -Profile $Inputs.Profile -Run $Inputs.Run -Status incompatible -FinalResponseReason 'preflight_incompatible' -StartedUtc $started.ToString('o') -FinishedUtc $finished.ToString('o') -DurationSeconds ($finished - $started).TotalSeconds -Failure (New-ExecutionFailure -Code 'incompatible' -Message ([string]::Join('; ', @($preflight.reasons)))) -SessionId $sessionId -IsolationCapabilities ([ordered]@{}) -IsolationMechanisms @('preflight-only') -Evidence ([ordered]@{ preflight = $preflight; resume = $false }) -AttemptCount 1 + } + + $commandInfo = Resolve-ExternalCommand -Name 'copilot' + $environment = New-CopilotEnvironment -Inputs $Inputs + $platform = Get-PlatformName + $sandboxInfo = if ($platform -eq 'linux') { Resolve-SandboxCommand -Name 'bwrap' } elseif ($platform -eq 'macos') { Resolve-SandboxCommand -Name 'sandbox-exec' } else { $null } + $hardFilesystem = $null -ne $sandboxInfo -and $platform -in @('linux', 'macos') + $visiblePlatform = if ($hardFilesystem) { $platform } elseif ($platform -eq 'linux') { 'unknown' } else { $platform } + $arguments = New-CopilotCliArguments -Inputs $Inputs -VisiblePlatform $visiblePlatform + + if ($platform -eq 'linux' -and $hardFilesystem) { + $insideEnvironment = New-CopilotInsideEnvironment -Inputs $Inputs -Environment $environment + $sandboxArguments = Get-LinuxEvalSandboxArguments -Inputs $Inputs -CommandInfo $commandInfo -InsideEnvironment $insideEnvironment + $process = Invoke-RunnerProcess -FileName $sandboxInfo.FileName -ArgumentList (@($sandboxArguments) + @($arguments)) -WorkingDirectory $Inputs.Run.WorkingDirectoryPath -Environment $environment -TimeoutSeconds $Inputs.Profile.TimeoutSeconds + } elseif ($platform -eq 'macos' -and $hardFilesystem) { + $sandboxProfile = New-MacosEvalSandboxProfile -Inputs $Inputs -CommandInfo $commandInfo + $sandboxArguments = @('-f', $sandboxProfile, '--', $commandInfo.FileName) + @($commandInfo.Prefix) + @($arguments) + $process = Invoke-RunnerProcess -FileName $sandboxInfo.FileName -ArgumentList $sandboxArguments -WorkingDirectory $Inputs.Run.WorkingDirectoryPath -Environment $environment -TimeoutSeconds $Inputs.Profile.TimeoutSeconds + } else { + $process = Invoke-CopilotCli -CommandInfo $commandInfo -Arguments $arguments -Inputs $Inputs -Environment $environment -TimeoutSeconds $Inputs.Profile.TimeoutSeconds + } + + $stdoutArtifact = Write-CopilotCapture -RunData $Inputs -RelativePath 'evidence/copilot-events.jsonl' -Text $process.Stdout + $stderrArtifact = Write-CopilotCapture -RunData $Inputs -RelativePath 'evidence/copilot-stderr.txt' -Text $process.Stderr + $artifacts = [System.Collections.Generic.List[object]]::new() + $artifacts.Add($stdoutArtifact) + $artifacts.Add($stderrArtifact) + + $warnings = [System.Collections.Generic.List[string]]::new() + $parsed = ConvertFrom-JsonLines -Text $process.Stdout + foreach ($parseError in @($parsed.Errors)) { $warnings.Add("Copilot event parse error: $parseError") } + $parsedEvents = Read-CopilotEvents -Parsed $parsed -Warnings $warnings + + $finalText = $parsedEvents.FinalText + $status = 'completed' + $reason = $null + $failure = $null + $exitStatus = if ($process.TimedOut) { $null } else { [Nullable[int]]$process.ExitCode } + if ($process.TimedOut) { + $status = 'timed_out' + $reason = 'copilot_timeout' + $failure = New-ExecutionFailure -Code 'timed_out' -Message 'Copilot did not finish before timeout_seconds.' + } elseif ($process.ExitCode -ne 0 -or $null -ne $parsedEvents.SessionError) { + $status = 'failed' + $reason = 'copilot_failure' + $message = if ($null -ne $parsedEvents.SessionError) { [string]$parsedEvents.SessionError } else { "Copilot exited with status $($process.ExitCode)." } + $failure = New-ExecutionFailure -Code 'copilot_failure' -Message $message + } elseif ([string]::IsNullOrWhiteSpace($finalText)) { + $warnings.Add('Copilot exited successfully without an assistant message; the final response is unavailable.') + $reason = 'copilot_did_not_return_final_response' + } + + $tokenMetric = if (-not $parsedEvents.UsageSeen) { + New-UnavailableMetric -Reason 'copilot_did_not_expose_usage_events' + } else { + $usageValue = [ordered]@{} + if ($null -ne $parsedEvents.UsageInput) { $usageValue['input_tokens'] = [int64]$parsedEvents.UsageInput } + if ($null -ne $parsedEvents.UsageOutput) { $usageValue['output_tokens'] = [int64]$parsedEvents.UsageOutput } + if ($null -ne $parsedEvents.UsageCacheRead) { $usageValue['cache_read_tokens'] = [int64]$parsedEvents.UsageCacheRead } + if ($null -ne $parsedEvents.UsageCacheWrite) { $usageValue['cache_write_tokens'] = [int64]$parsedEvents.UsageCacheWrite } + if ($usageValue.Count -eq 0) { New-UnavailableMetric -Reason 'copilot_usage_event_had_no_supported_buckets' } else { New-AvailableMetric -Value $usageValue } + } + $telemetry = [ordered]@{ + transcript = New-AvailableMetric -Value ([ordered]@{ artifact = 'evidence/copilot-events.jsonl'; complete = $true }) + tokens = $tokenMetric + tool_calls = New-AvailableMetric -Value ([int]$parsedEvents.ToolCalls) + cost = New-UnavailableMetric -Reason 'copilot_exposes_a_billing_multiplier_not_a_currency_cost' + } + + $capabilities = Get-CopilotCapabilityMap -Inputs $Inputs -HardFilesystemConfinement $hardFilesystem + $mechanisms = [System.Collections.Generic.List[string]]::new() + foreach ($mechanism in @('copilot --prompt --output-format json', '--allow-all-tools', '--no-ask-user', '--no-custom-instructions', '--disable-builtin-mcps', '--secret-env-vars token redaction', 'isolated COPILOT_HOME', 'narrow GitHub token authentication', 'no session continuation')) { $mechanisms.Add($mechanism) } + if ($hardFilesystem) { $mechanisms.Add("external $($sandboxInfo.Source) filesystem sandbox") } else { $mechanisms.Add('pragmatic process/environment isolation without hard filesystem confinement') } + if (-not $hardFilesystem) { $warnings.Add('Hard filesystem confinement was unavailable; the completed arm is reported as pragmatic isolation.') } + $warnings.Add('GitHub Copilot inherits the shell environment for shell tools apart from a blocklist; the runner redacts the GitHub token from output with --secret-env-vars but cannot independently prove the child-tool environment filter hides it from every tool path.') + + $tokenVariable = Get-CopilotTokenVariable + $credentialEvidence = [ordered]@{ + source = if ([string]::IsNullOrWhiteSpace($tokenVariable)) { 'unavailable' } else { 'environment' } + github_token_variable = $tokenVariable + secret_env_var_redaction = @($copilotAuthVariables) + login_profile_copied = $false + auth_file_copied = $false + value_observed = $false + } + $observedModel = if ([string]::IsNullOrWhiteSpace([string]$parsedEvents.ObservedModel)) { $null } else { [string]$parsedEvents.ObservedModel } + $resolvedConfiguration = [ordered]@{ + status = 'accepted_request' + reason = 'Copilot accepted the requested model alias and configuration; it does not expose a distinct backend model snapshot beyond the model it reports in usage events.' + observations = [ordered]@{ + provider = $Inputs.Profile.Provider + model = $Inputs.Profile.Model + reasoning_effort = $Inputs.Profile.ReasoningEffort + observed_model = $observedModel + } + } + + $finished = [DateTime]::UtcNow + $sandboxEvidence = if (-not $hardFilesystem) { 'unavailable' } elseif ($platform -eq 'linux') { 'bwrap' } else { 'sandbox-exec' } + return New-ExecutionResult -Descriptor $executionDescriptor -Profile $Inputs.Profile -Run $Inputs.Run -Status $status -FinalResponse $finalText -FinalResponseReason $reason -StartedUtc $process.StartedUtc.ToString('o') -FinishedUtc $finished.ToString('o') -DurationSeconds $process.DurationSeconds -ExitStatus $exitStatus -Failure $failure -SessionId $sessionId -IsolationCapabilities $capabilities -IsolationMechanisms @($mechanisms) -ResolvedConfiguration $resolvedConfiguration -Telemetry $telemetry -Artifacts @($artifacts) -Warnings @($warnings) -Evidence ([ordered]@{ event_counts = $parsedEvents.EventCounts; observed_model = $observedModel; prompt_delivery = 'argument'; prompt_first_input = $true; resume = $false; stdout_exit_code = $process.ExitCode; sandbox = $sandboxEvidence; credential = $credentialEvidence }) -AttemptCount 1 +} + +try { + [void](Assert-RunnerDescriptor -Descriptor $descriptor) + switch ($Command) { + 'describe' { Write-RunnerJson -Value (Get-CopilotDescriptor) -AsOutput } + 'preflight' { + $inputs = Resolve-CopilotInputs + Write-RunnerJson -Value (Get-CopilotPreflight -Inputs $inputs) -AsOutput + } + 'execute' { + $inputs = Resolve-CopilotInputs + $result = Invoke-CopilotExecute -Inputs $inputs + [void](Assert-ExecutionResult -Result $result) + Write-RunnerJson -Value $result -AsOutput + } + } +} catch { + Write-ProtocolError -Message $_.Exception.Message +} diff --git a/scripts/eval-runners/tests/fixtures/copilot-events.jsonl b/scripts/eval-runners/tests/fixtures/copilot-events.jsonl new file mode 100644 index 0000000..e3b5a1a --- /dev/null +++ b/scripts/eval-runners/tests/fixtures/copilot-events.jsonl @@ -0,0 +1,9 @@ +{"type":"session.start","id":"e1","parentId":null,"timestamp":"2026-01-01T00:00:00.000Z","data":{"sessionId":"fixture-session"}} +{"type":"user.message","id":"e2","parentId":"e1","timestamp":"2026-01-01T00:00:00.100Z","data":{"content":"fixture prompt"}} +{"type":"assistant.message","id":"e3","parentId":"e2","timestamp":"2026-01-01T00:00:01.000Z","data":{"messageId":"m1","model":"claude-haiku-4.5","content":"fixture progress note"}} +{"type":"tool.execution_start","id":"e4","parentId":"e3","timestamp":"2026-01-01T00:00:01.200Z","data":{"callId":"t1","toolName":"str_replace_editor"}} +{"type":"tool.execution_complete","id":"e5","parentId":"e4","timestamp":"2026-01-01T00:00:01.800Z","data":{"callId":"t1","status":"success"}} +{"type":"assistant.message","id":"e6","parentId":"e5","timestamp":"2026-01-01T00:00:02.000Z","data":{"messageId":"m2","model":"claude-haiku-4.5","content":"fixture final response"}} +{"type":"assistant.usage","id":"e7","parentId":"e6","ephemeral":true,"timestamp":"2026-01-01T00:00:02.100Z","data":{"model":"claude-haiku-4.5","inputTokens":12,"outputTokens":4,"cacheReadTokens":3,"numToolCalls":1,"cost":0.2,"finishReason":"stop"}} +{"type":"session.task_complete","id":"e8","parentId":"e7","timestamp":"2026-01-01T00:00:02.200Z","data":{}} +{"type":"future.event.v99","payload":"unknown"} diff --git a/scripts/eval-runners/tests/test-runner-conformance.ps1 b/scripts/eval-runners/tests/test-runner-conformance.ps1 index 8a61bef..42b5e48 100644 --- a/scripts/eval-runners/tests/test-runner-conformance.ps1 +++ b/scripts/eval-runners/tests/test-runner-conformance.ps1 @@ -45,6 +45,10 @@ $recordedOldOpenAi = $env:OPENAI_API_KEY $recordedOldCodexHome = $env:CODEX_HOME $recordedOldGlobalSecret = $env:AGENTIC_GLOBAL_SECRET $recordedOldProjectDisable = $env:OPENCODE_DISABLE_PROJECT_CONFIG +$recordedOldCopilotToken = $env:COPILOT_GITHUB_TOKEN +$recordedOldGhToken = $env:GH_TOKEN +$recordedOldGithubToken = $env:GITHUB_TOKEN +$recordedOldCopilotHome = $env:COPILOT_HOME try { $fakeBin = Join-Path $recordedRoot 'bin' New-Item -ItemType Directory -Path $fakeBin -Force | Out-Null @@ -72,9 +76,15 @@ $record = [ordered]@{ disable_project_config_present = -not [string]::IsNullOrWhiteSpace([Environment]::GetEnvironmentVariable('OPENCODE_DISABLE_PROJECT_CONFIG')) project_config_visible = Test-Path -LiteralPath (Join-Path (Get-Location).Path 'opencode.json') -PathType Leaf stdin_received = $false + prompt_via_arg = ($arguments -contains '--prompt') -or ($arguments -contains '-p') + prompt_arg_count = @($arguments | Where-Object { $_ -eq '--prompt' -or $_ -eq '-p' }).Count + copilot_home = [Environment]::GetEnvironmentVariable('COPILOT_HOME') + custom_instructions_disabled = ($arguments -contains '--no-custom-instructions') + builtin_mcps_disabled = ($arguments -contains '--disable-builtin-mcps') + secret_env_vars_arg = @($arguments | Where-Object { $_ -like '--secret-env-vars=*' }) } if ($arguments -contains '--version') { - $version = switch ($harness) { 'codex' { 'recorded-codex 9.1' } 'opencode' { 'recorded-opencode 9.2' } default { 'recorded-cline 9.3' } } + $version = switch ($harness) { 'codex' { 'recorded-codex 9.1' } 'opencode' { 'recorded-opencode 9.2' } 'copilot' { 'GitHub Copilot CLI recorded-1.0.80' } default { 'recorded-cline 9.3' } } [IO.File]::AppendAllText($logPath, (($record | ConvertTo-Json -Compress) + [Environment]::NewLine), [Text.UTF8Encoding]::new($false)) Write-Output $version exit 0 @@ -83,6 +93,7 @@ if ($arguments -contains '--help') { $help = switch ($harness) { 'codex' { '--ask-for-approval never --ephemeral --ignore-user-config --ignore-rules --json --output-last-message --sandbox --cd --model --config --approve-for-me' } 'opencode' { '--format --dir --model --auto --pure --continue --session' } + 'copilot' { '--prompt --output-format --model --allow-all-tools --no-ask-user --no-custom-instructions --disable-builtin-mcps --no-color --log-level --secret-env-vars --no-auto-update -C --resume --continue --session-id --connect --yolo --allow-all --allow-all-paths --allow-all-urls' } default { '--json --auto-approve --cwd --config --data-dir --hooks-dir --provider --model --thinking --timeout --retries --id' } } [IO.File]::AppendAllText($logPath, (($record | ConvertTo-Json -Compress) + [Environment]::NewLine), [Text.UTF8Encoding]::new($false)) @@ -91,7 +102,7 @@ if ($arguments -contains '--help') { } $stdinText = [Console]::In.ReadToEnd() $record.stdin_received = -not [string]::IsNullOrEmpty($stdinText) -$probeCommand = '$result = [ordered]@{ provider_visible = -not [string]::IsNullOrWhiteSpace([Environment]::GetEnvironmentVariable(''OPENAI_API_KEY'')); auth_file_visible = Test-Path -LiteralPath (Join-Path ([Environment]::GetEnvironmentVariable(''HOME'')) ''.codex/auth.json''); global_secret_visible = -not [string]::IsNullOrWhiteSpace([Environment]::GetEnvironmentVariable(''AGENTIC_GLOBAL_SECRET'')); project_disable_visible = -not [string]::IsNullOrWhiteSpace([Environment]::GetEnvironmentVariable(''OPENCODE_DISABLE_PROJECT_CONFIG'')) }; $result | ConvertTo-Json -Compress' +$probeCommand = '$result = [ordered]@{ provider_visible = -not [string]::IsNullOrWhiteSpace([Environment]::GetEnvironmentVariable(''OPENAI_API_KEY'')); copilot_token_visible = -not [string]::IsNullOrWhiteSpace([Environment]::GetEnvironmentVariable(''COPILOT_GITHUB_TOKEN'')); auth_file_visible = Test-Path -LiteralPath (Join-Path ([Environment]::GetEnvironmentVariable(''HOME'')) ''.codex/auth.json''); global_secret_visible = -not [string]::IsNullOrWhiteSpace([Environment]::GetEnvironmentVariable(''AGENTIC_GLOBAL_SECRET'')); project_disable_visible = -not [string]::IsNullOrWhiteSpace([Environment]::GetEnvironmentVariable(''OPENCODE_DISABLE_PROJECT_CONFIG'')) }; $result | ConvertTo-Json -Compress' $probeInfo = [Diagnostics.ProcessStartInfo]::new() $probeInfo.FileName = (Get-Command pwsh).Source $probeInfo.UseShellExecute = $false @@ -107,6 +118,7 @@ $probeInfo.Environment['PATH'] = [Environment]::GetEnvironmentVariable('PATH') $probeInfo.Environment['HOME'] = [Environment]::GetEnvironmentVariable('HOME') $probeInfo.Environment['USERPROFILE'] = [Environment]::GetEnvironmentVariable('USERPROFILE') if ($harness -ne 'codex') { $probeInfo.Environment['OPENAI_API_KEY'] = [Environment]::GetEnvironmentVariable('OPENAI_API_KEY') } +if ($harness -eq 'copilot') { $probeInfo.Environment['COPILOT_GITHUB_TOKEN'] = [Environment]::GetEnvironmentVariable('COPILOT_GITHUB_TOKEN') } $probe = [Diagnostics.Process]::new() $probe.StartInfo = $probeInfo try { @@ -117,6 +129,7 @@ try { if ($probe.ExitCode -ne 0) { throw "worker credential probe failed: $probeError" } $probeResult = $probeOutput | ConvertFrom-Json $record.worker_provider_visible = [bool]$probeResult.provider_visible + $record.worker_copilot_token_visible = [bool]$probeResult.copilot_token_visible $record.worker_auth_file_visible = [bool]$probeResult.auth_file_visible $record.worker_global_secret_visible = [bool]$probeResult.global_secret_visible $record.worker_project_disable_visible = [bool]$probeResult.project_disable_visible @@ -139,6 +152,15 @@ if ($harness -eq 'codex') { Write-Output '{"type":"text","text":"recorded OpenCode final response"}' Write-Output '{"type":"step_finish","part":{"tokens":{"input":2,"output":3},"cost":0.01}}' Write-Output '{"type":"future.event.v99","payload":"fixture"}' +} elseif ($harness -eq 'copilot') { + Write-Output '{"type":"session.start","id":"e1","parentId":null,"data":{"sessionId":"recorded"}}' + Write-Output '{"type":"assistant.message","id":"e2","parentId":"e1","data":{"messageId":"m1","model":"claude-haiku-4.5","content":"recorded Copilot progress"}}' + Write-Output '{"type":"tool.execution_start","id":"e3","parentId":"e2","data":{"callId":"t1","toolName":"str_replace_editor"}}' + Write-Output '{"type":"tool.execution_complete","id":"e4","parentId":"e3","data":{"callId":"t1","status":"success"}}' + Write-Output '{"type":"assistant.message","id":"e5","parentId":"e4","data":{"messageId":"m2","model":"claude-haiku-4.5","content":"recorded Copilot final response"}}' + Write-Output '{"type":"assistant.usage","id":"e6","parentId":"e5","ephemeral":true,"data":{"model":"claude-haiku-4.5","inputTokens":2,"outputTokens":3,"cacheReadTokens":1,"numToolCalls":1,"cost":0.2}}' + Write-Output '{"type":"session.task_complete","id":"e7","parentId":"e6","data":{}}' + Write-Output '{"type":"future.event.v99","payload":"fixture"}' } else { Write-Output '{"type":"say","say":"text","text":"recorded Cline progress","partial":false}' Write-Output '{"type":"say","say":"tool","name":"read_file","text":"fixture.md","partial":false}' @@ -147,21 +169,26 @@ if ($harness -eq 'codex') { Write-Output '{"type":"future.event.v99","payload":"fixture"}' } '@ - foreach ($harness in @('codex', 'opencode', 'cline')) { + foreach ($harness in @('codex', 'opencode', 'cline', 'copilot')) { [System.IO.File]::WriteAllText((Join-Path $fakeBin "$harness.ps1"), $fakeCli, [System.Text.UTF8Encoding]::new($false)) } $env:PATH = "$fakeBin$([System.IO.Path]::PathSeparator)$recordedOldPath" $env:OPENAI_API_KEY = 'recorded-canary-not-logged' $env:AGENTIC_GLOBAL_SECRET = 'recorded-unrelated-canary-not-logged' $env:OPENCODE_DISABLE_PROJECT_CONFIG = '1' + $env:COPILOT_GITHUB_TOKEN = 'recorded-copilot-canary-not-logged' + $env:GH_TOKEN = $null + $env:GITHUB_TOKEN = $null $recordedProfiles = [ordered]@{} - foreach ($runnerName in @('codex', 'opencode', 'cline')) { + foreach ($runnerName in @('codex', 'opencode', 'cline', 'copilot')) { $profilePath = Join-Path $recordedRoot "$runnerName-profile.json" + $profileProvider = if ($runnerName -eq 'copilot') { 'github-copilot' } else { 'openai' } + $profileModel = if ($runnerName -eq 'copilot') { 'claude-haiku-4.5' } else { 'fixture-model' } Write-TestJson -Path $profilePath -Value ([ordered]@{ schema = (Get-RunnerSchemaNames).Profile - runner = $runnerName - provider = 'openai' - model = 'fixture-model' + runner = if ($runnerName -eq 'copilot') { 'github-copilot' } else { $runnerName } + provider = $profileProvider + model = $profileModel reasoning_effort = 'medium' configuration_profile = 'isolated-default' tool_profile = 'default' @@ -175,11 +202,12 @@ if ($harness -eq 'codex') { $recordedVersion = Get-ExternalCommandVersion -CommandInfo $resolvedRecordedCodex -WorkingDirectory (Join-Path $with.Root 'repo') if (-not $recordedVersion.Available) { throw "recorded Codex --version is not observable (exit=$($recordedVersion.Process.ExitCode), timed_out=$($recordedVersion.Process.TimedOut), stdout='$($recordedVersion.Process.Stdout)', stderr='$($recordedVersion.Process.Stderr)')" } Assert-Equal 'recorded-codex 9.1' $recordedVersion.Version 'recorded Codex exact version helper' - foreach ($runnerName in @('codex', 'opencode', 'cline')) { - $runnerPath = Join-Path $runnerRoot "$runnerName\runner.ps1" + foreach ($runnerName in @('codex', 'opencode', 'cline', 'copilot')) { + $runnerDir = if ($runnerName -eq 'copilot') { 'github-copilot' } else { $runnerName } + $runnerPath = Join-Path $runnerRoot "$runnerDir\runner.ps1" $description = Invoke-AdapterJson -RunnerPath $runnerPath -Command describe -RunPath $with.Path -ProfilePath $recordedProfiles[$runnerName] [void](Assert-RunnerDescriptor -Descriptor $description) - $expectedVersion = switch ($runnerName) { 'codex' { 'recorded-codex 9.1' } 'opencode' { 'recorded-opencode 9.2' } default { 'recorded-cline 9.3' } } + $expectedVersion = switch ($runnerName) { 'codex' { 'recorded-codex 9.1' } 'opencode' { 'recorded-opencode 9.2' } 'copilot' { 'GitHub Copilot CLI recorded-1.0.80' } default { 'recorded-cline 9.3' } } Assert-Equal $expectedVersion $description.harness.version "$runnerName exact describe version" $preflightWith = Invoke-AdapterJson -RunnerPath $runnerPath -Command preflight -RunPath $with.Path -ProfilePath $recordedProfiles[$runnerName] $preflightWithout = Invoke-AdapterJson -RunnerPath $runnerPath -Command preflight -RunPath $without.Path -ProfilePath $recordedProfiles[$runnerName] @@ -211,7 +239,7 @@ if ($harness -eq 'codex') { $logPath = Join-Path $with.Root "repo\$runnerName-fake-cli-log.jsonl" Assert-True (Test-Path -LiteralPath $logPath -PathType Leaf) "$runnerName recorded process log exists" $records = @(Get-Content -LiteralPath $logPath | ForEach-Object { $_ | ConvertFrom-Json }) - $executionRecords = @($records | Where-Object { $_.stdin_received -eq $true }) + $executionRecords = @($records | Where-Object { $_.stdin_received -eq $true -or $_.prompt_via_arg -eq $true }) Assert-Equal 1 $executionRecords.Count "$runnerName one execution process per checked arm" $execution = $executionRecords[0] Assert-True (-not $execution.unrelated_present) "$runnerName does not pass unrelated credential canary" @@ -221,6 +249,8 @@ if ($harness -eq 'codex') { Assert-True (-not $execution.worker_project_disable_visible) "$runnerName worker probe cannot read the parent project-disable variable" if ($runnerName -eq 'codex') { Assert-True (-not $execution.worker_provider_visible) 'Codex shell policy hides the provider API-key variable from the worker probe' + } elseif ($runnerName -eq 'copilot') { + Assert-True $execution.worker_copilot_token_visible 'Copilot GitHub-token child-visibility limitation is recorded by the worker probe' } else { Assert-True $execution.worker_provider_visible "$runnerName credential visibility limitation is recorded by the worker probe" } @@ -237,6 +267,31 @@ if ($harness -eq 'codex') { Assert-True ($args -notcontains '--pure') 'OpenCode preserves repository-owned project configuration' Assert-True ($args -contains '--auto') 'OpenCode is noninteractive' Assert-True $execution.project_config_visible 'OpenCode paired arm retains repository-owned project configuration' + } elseif ($runnerName -eq 'copilot') { + Assert-True ($args -contains '--prompt') 'Copilot delivers the prompt via the --prompt argument' + Assert-Equal 1 $execution.prompt_arg_count 'Copilot delivers the prompt exactly once' + Assert-True (-not $execution.stdin_received) 'Copilot does not read the prompt from stdin' + Assert-True ($args -contains '--output-format' -and $args -contains 'json') 'Copilot uses structured JSONL output' + $modelIndex = [Array]::IndexOf([string[]]$args, '--model') + Assert-Equal 'claude-haiku-4.5' $args[$modelIndex + 1] 'Copilot reference model claude-haiku-4.5 propagates to the CLI invocation' + Assert-True ($args -contains '--allow-all-tools') 'Copilot removes tool-approval prompts without a blanket permission grant' + Assert-True ($args -contains '--no-ask-user') 'Copilot does not pause for interactive questions' + Assert-True ($args -contains '--no-custom-instructions') 'Copilot disables ambient custom instructions' + Assert-True ($args -contains '--disable-builtin-mcps') 'Copilot disables ambient built-in MCP servers' + foreach ($broad in @('--yolo', '--allow-all', '--allow-all-paths', '--allow-all-urls', '--session-id', '--connect', '-r')) { Assert-True ($args -notcontains $broad) "Copilot avoids the over-broad or session option '$broad'" } + Assert-Equal 1 (@($args | Where-Object { $_ -like '--secret-env-vars=*' }).Count) 'Copilot redacts the GitHub token from output with --secret-env-vars' + Assert-True ($execution.custom_instructions_disabled -and $execution.builtin_mcps_disabled) 'Copilot ambient instruction/MCP disabling is observed by the fake CLI' + Assert-True (Test-PathInside -BasePath (Join-Path $with.Root 'home') -CandidatePath ([string]$execution.copilot_home)) 'Copilot COPILOT_HOME is the run''s isolated home' + Assert-Equal 'recorded Copilot final response' $resultWith.final_response.text 'Copilot final response is the last assistant message, not an intermediate one' + Assert-Equal 'claude-haiku-4.5' $resultWith.requested.model 'Copilot requested model is preserved as the Codebelt reference model' + Assert-True ($null -eq $resultWith.resolved.model) 'Copilot does not claim a distinct backend model resolution' + Assert-Equal 'claude-haiku-4.5' $resultWith.evidence.observed_model 'Copilot observed model is captured separately from the requested model' + Assert-Equal 'available' $resultWith.telemetry.tokens.status 'Copilot reports available token telemetry' + Assert-Equal 2 ([int]$resultWith.telemetry.tokens.value.input_tokens) 'Copilot input tokens are parsed from assistant.usage' + Assert-Equal 3 ([int]$resultWith.telemetry.tokens.value.output_tokens) 'Copilot output tokens are parsed from assistant.usage' + Assert-Equal 'available' $resultWith.telemetry.tool_calls.status 'Copilot reports available tool-call telemetry' + Assert-True ([int]$resultWith.telemetry.tool_calls.value -ge 1) 'Copilot parses documented tool.execution events' + Assert-Equal 'unavailable' $resultWith.telemetry.cost.status 'Copilot does not estimate a currency cost' } else { $retryIndex = [Array]::IndexOf([string[]]$args, '--retries') Assert-Equal '0' $args[$retryIndex + 1] 'Cline disables internal retries' @@ -254,7 +309,7 @@ if ($harness -eq 'codex') { $withoutLogPath = Join-Path $without.Root "repo\$runnerName-fake-cli-log.jsonl" Assert-True (Test-Path -LiteralPath $withoutLogPath -PathType Leaf) "$runnerName baseline process log exists" $withoutRecords = @(Get-Content -LiteralPath $withoutLogPath | ForEach-Object { $_ | ConvertFrom-Json }) - Assert-Equal 1 @($withoutRecords | Where-Object { $_.stdin_received -eq $true }).Count "$runnerName baseline has one execution process" + Assert-Equal 1 @($withoutRecords | Where-Object { $_.stdin_received -eq $true -or $_.prompt_via_arg -eq $true }).Count "$runnerName baseline has one execution process" $withoutLogText = [System.IO.File]::ReadAllText($withoutLogPath, [System.Text.UTF8Encoding]::new($false)) Assert-True ($withoutLogText -notmatch 'recorded-canary|recorded-unrelated-canary') "$runnerName baseline log does not contain credential values" } @@ -274,6 +329,26 @@ if ($harness -eq 'codex') { Assert-True (@($fileAuthPreflight.reasons | Where-Object { $_ -match 'auth\.json' }).Count -gt 0) 'Codex file-auth limitation is explicit' $env:OPENAI_API_KEY = 'recorded-canary-not-logged' $env:CODEX_HOME = $recordedOldCodexHome + # GitHub Copilot authentication: with no separable token the runner is fail-closed and never imports the ambient profile. + $copilotEmptyHome = Join-Path $recordedRoot 'copilot-empty-home' + New-Item -ItemType Directory -Path $copilotEmptyHome -Force | Out-Null + $env:COPILOT_GITHUB_TOKEN = $null + $env:GH_TOKEN = $null + $env:GITHUB_TOKEN = $null + $env:COPILOT_HOME = $copilotEmptyHome + $copilotNoAuth = Invoke-AdapterJson -RunnerPath (Join-Path $runnerRoot 'github-copilot\runner.ps1') -Command preflight -RunPath $with.Path -ProfilePath $recordedProfiles['copilot'] + Assert-Equal 'incompatible' $copilotNoAuth.status 'Copilot without a GitHub token is fail-closed' + Assert-True (@($copilotNoAuth.reasons | Where-Object { $_ -match 'COPILOT_GITHUB_TOKEN' }).Count -gt 0) 'Copilot missing-auth reason names the required token variable' + $copilotLoginHome = Join-Path $recordedRoot 'copilot-login-home' + New-Item -ItemType Directory -Path $copilotLoginHome -Force | Out-Null + [System.IO.File]::WriteAllText((Join-Path $copilotLoginHome 'config.json'), '{"loggedInUsers":[{"login":"canary-not-logged"}]}', [System.Text.UTF8Encoding]::new($false)) + $env:COPILOT_HOME = $copilotLoginHome + $copilotLoginOnly = Invoke-AdapterJson -RunnerPath (Join-Path $runnerRoot 'github-copilot\runner.ps1') -Command preflight -RunPath $with.Path -ProfilePath $recordedProfiles['copilot'] + Assert-Equal 'incompatible' $copilotLoginOnly.status 'Copilot login-only profile is not imported as a credential' + Assert-True (@($copilotLoginOnly.reasons | Where-Object { $_ -match 'separable GitHub token' }).Count -gt 0) 'Copilot login-only reason explains the co-mingling limitation' + Assert-True (($copilotLoginOnly | ConvertTo-Json -Depth 100) -notmatch 'canary-not-logged') 'Copilot preflight never surfaces login-profile contents' + $env:COPILOT_GITHUB_TOKEN = 'recorded-copilot-canary-not-logged' + $env:COPILOT_HOME = $recordedOldCopilotHome Write-Output 'Real runner deterministic adapter conformance: PASS' } finally { $env:PATH = $recordedOldPath @@ -281,6 +356,10 @@ if ($harness -eq 'codex') { $env:CODEX_HOME = $recordedOldCodexHome $env:AGENTIC_GLOBAL_SECRET = $recordedOldGlobalSecret $env:OPENCODE_DISABLE_PROJECT_CONFIG = $recordedOldProjectDisable + $env:COPILOT_GITHUB_TOKEN = $recordedOldCopilotToken + $env:GH_TOKEN = $recordedOldGhToken + $env:GITHUB_TOKEN = $recordedOldGithubToken + $env:COPILOT_HOME = $recordedOldCopilotHome if (Test-Path -LiteralPath $recordedRoot) { Remove-Item -LiteralPath $recordedRoot -Recurse -Force } } } @@ -551,13 +630,17 @@ try { Assert-Equal 'recorded-cli 1.2.3' (Get-ObservableVersionFromText "`nrecorded-cli 1.2.3`n") 'observable version capture keeps the exact line' Assert-True ($null -eq (Get-ObservableVersionFromText "`n `n")) 'empty version output has no observable value' - foreach ($fixture in @('codex-events.jsonl', 'opencode-events.jsonl')) { + foreach ($fixture in @('codex-events.jsonl', 'opencode-events.jsonl', 'copilot-events.jsonl')) { $fixturePath = Join-Path $PSScriptRoot "fixtures\$fixture" $parsed = ConvertFrom-JsonLines -Text ([System.IO.File]::ReadAllText($fixturePath, [System.Text.UTF8Encoding]::new($false))) Assert-Equal 0 $parsed.Errors.Count "recorded $fixture has valid JSONL" Assert-True ($parsed.Events.Count -ge 4) "recorded $fixture has events" Assert-True (@($parsed.Events | Where-Object { $_.type -eq 'future.event.v99' }).Count -eq 1) "recorded $fixture includes an unknown event" } + $copilotFixture = ConvertFrom-JsonLines -Text ([System.IO.File]::ReadAllText((Join-Path $PSScriptRoot 'fixtures\copilot-events.jsonl'), [System.Text.UTF8Encoding]::new($false))) + Assert-True (@($copilotFixture.Events | Where-Object { $_.type -eq 'assistant.message' }).Count -ge 1) 'recorded copilot fixture includes documented assistant.message output' + Assert-True (@($copilotFixture.Events | Where-Object { $_.type -eq 'assistant.usage' }).Count -eq 1) 'recorded copilot fixture includes documented assistant.usage output' + Assert-True (@($copilotFixture.Events | Where-Object { $_.type -eq 'tool.execution_start' }).Count -eq 1) 'recorded copilot fixture includes documented tool.execution output' $clineFixture = ConvertFrom-JsonLines -Text ([System.IO.File]::ReadAllText((Join-Path $PSScriptRoot 'fixtures\cline-events.jsonl'), [System.Text.UTF8Encoding]::new($false))) Assert-Equal 0 $clineFixture.Errors.Count 'recorded cline fixture has valid JSONL' Assert-True ($clineFixture.Events.Count -ge 9) 'recorded cline fixture has events' @@ -567,8 +650,10 @@ try { $prepareText = [System.IO.File]::ReadAllText((Join-Path $repoRoot 'scripts\prepare-skill-evals.ps1'), [System.Text.UTF8Encoding]::new($false)) $reportText = [System.IO.File]::ReadAllText((Join-Path $repoRoot 'scripts\generate-eval-report.ps1'), [System.Text.UTF8Encoding]::new($false)) - Assert-True ($prepareText -notmatch '(?i)codex\s+exec|opencode\s+run|cline\s+--') 'portable preparation must not contain harness-specific CLI invocations' - Assert-True ($reportText -notmatch '(?i)codex\s+exec|opencode\s+run|cline\s+--') 'reporting must not contain harness-specific branches' + $bridgeText = [System.IO.File]::ReadAllText((Join-Path $runnerRoot 'bridge-execution-result.ps1'), [System.Text.UTF8Encoding]::new($false)) + Assert-True ($prepareText -notmatch '(?i)codex\s+exec|opencode\s+run|cline\s+--|copilot\s+-p|copilot\s+--prompt|github-copilot') 'portable preparation must not contain harness-specific CLI invocations or runner branches' + Assert-True ($reportText -notmatch '(?i)codex\s+exec|opencode\s+run|cline\s+--|copilot\s+-p|copilot\s+--prompt|github-copilot') 'reporting must not contain harness-specific branches' + Assert-True ($bridgeText -notmatch '(?i)codex\s+exec|opencode\s+run|cline\s+--|copilot\s+-p|copilot\s+--prompt|github-copilot') 'the raw-to-portable bridge must remain runner-neutral' $rawPath = Join-Path $iteration 'conformance\results\with-skill.execution-result.json' $resultPath = Join-Path $iteration 'conformance\results\with-skill.result.json' From e29cbfa77ec5e80dba3499a3e54da60f73e085a3 Mon Sep 17 00:00:00 2001 From: "aicia[bot]" Date: Fri, 21 Aug 2026 21:46:38 +0200 Subject: [PATCH 06/39] =?UTF-8?q?=F0=9F=94=A7=20update=20eval=20infrastruc?= =?UTF-8?q?ture=20for=20github=20copilot=20support?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Extend prepare-skill-evals.ps1 and validate-skill-templates.ps1 to support GitHub Copilot runner configuration and execution profile selection. Enables dynamic runner resolution and integration with the new GitHub Copilot CLI evaluation path. --- scripts/prepare-skill-evals.ps1 | 39 +++++++++++++++------------- scripts/validate-skill-templates.ps1 | 2 +- 2 files changed, 22 insertions(+), 19 deletions(-) diff --git a/scripts/prepare-skill-evals.ps1 b/scripts/prepare-skill-evals.ps1 index eede3f3..def13c9 100644 --- a/scripts/prepare-skill-evals.ps1 +++ b/scripts/prepare-skill-evals.ps1 @@ -8,16 +8,18 @@ harness, provider, and model they choose, then validates the results that come back. Prepare mode writes one directory per eval. The grading key and result stubs stay at the eval-case level, outside - the two hermetic run directories a worker actually sees: + the two isolated run directories a worker actually sees: eval-metadata.json id, name, prompt, expected output, assertions, fixtures, hashes, assumptions results/ one prefilled result stub per configuration - with_skill/ a hermetic run: prompt.md, run.json, repo/ (materialized fixtures), home/, skill// + with_skill/ an isolated run: prompt.md, run.json, repo/ (materialized fixtures), home/, skill// without_skill/ the same run without any skill/ directory and no skill instructions - Each run directory is the worker's sandbox root: repo/ is the working tree, home/ is an isolated profile, and skill/ + Each run directory is the worker's staged root: repo/ is the working tree, home/ is an isolated profile, and skill/ (with_skill only) holds the candidate skill revision. run.json is a harness-neutral contract naming only paths inside - the run directory. Preparation validates the isolation invariants and fails early if a package would let a baseline - reach the skill, let a worker reach the source repository, or stage mismatched fixtures. + the run directory. The selected runner enforces the isolation and reports strict confidence when it also proves hard + filesystem confinement or pragmatic confidence when it does not. Preparation validates the isolation invariants and + fails early if a package would let a baseline reach the skill, stage the source repository into a run, or stage + mismatched fixtures. Collect mode reads a prepared package plus whatever result files were filled in, validates them, and invokes the packaged Anthropic skill-creator aggregator and static eval viewer after writing a deterministic comparison. The @@ -176,10 +178,11 @@ $executionResultSchema = 'codebeltnet/agentic/eval-execution-result/1' $runnerProtocolSchema = 'codebeltnet/agentic/eval-runner-protocol/1' $maxFixtureInlineBytes = 32768 -# A materialized run is hermetic: the harness treats the run directory as the worker's sandbox root, mounts repo/ as +# A materialized run is self-contained: the runner treats the run directory as the worker's staged root, uses repo/ as # the working directory and home/ as the isolated user profile, and exposes skill/ only for a with_skill run. Nothing -# else in the package - the grading key, the paired run, other evals, or results - lives inside a run directory, so a -# worker confined to its run directory cannot reach any of it. +# else in the package - the grading key, the paired run, other evals, or results - is staged inside a run directory, so a +# worker that stays within its run directory is never handed any of it. Hard filesystem confinement is an optional +# confidence signal a runner may add on top; it is not required for this staging boundary. $runDirectoryNames = [ordered]@{ Working = 'repo' Home = 'home' @@ -1266,9 +1269,9 @@ function Invoke-PrepareMode { $repoFiles = @($fixtures | ForEach-Object { $_.RepoRelative } | Sort-Object) - # Materialize both runs. Each run directory is the worker's sandbox root: repo/ is the working tree, home/ is an + # Materialize both runs. Each run directory is the worker's staged root: repo/ is the working tree, home/ is an # isolated profile, and skill/ (with_skill only) holds the candidate. The grading key and results live one level - # up, outside every run directory, so a worker confined to its run directory can never reach them. + # up, outside every run directory, so a worker that stays within its run directory is never handed them. foreach ($configuration in @('with_skill', 'without_skill')) { $runDir = Join-Path $evalDirectory $configuration New-Item -ItemType Directory -Path $runDir -Force | Out-Null @@ -1311,7 +1314,7 @@ function Invoke-PrepareMode { $assumptions = [System.Collections.Generic.List[string]]::new() $assumptions.Add('Run with_skill and without_skill on the same model, same version, and same configuration. Different models measure the model, not the skill.') - $assumptions.Add('Each run is hermetic: launch a fresh worker with its run directory as the sandbox root, its repo/ as the working directory, and its home/ as the isolated profile.') + $assumptions.Add('Each run is isolated: launch a fresh worker with its run directory as the staged root, its repo/ as the working directory, and its home/ as the isolated profile. The runner reports strict confidence when it also proves hard filesystem confinement and pragmatic confidence when it does not.') $assumptions.Add("Both runs share an identical materialized repository. Only the with_skill run exposes the candidate skill under skill/$Skill/.") $notInlinedFixtures = @($fixtures | Where-Object { -not $_.Inlined }) if ($notInlinedFixtures.Count -gt 0) { @@ -1445,7 +1448,7 @@ function Invoke-PrepareMode { 'fresh context', 'isolated HOME/config', 'isolated CWD', - 'filesystem sandbox', + 'staged filesystem/workspace boundary', 'candidate skill exposure', 'transcript capture' ) @@ -1462,7 +1465,7 @@ function Invoke-PrepareMode { $runnerPath = Join-Path $iterationDirectory 'RUN-THIS.prompt.md' Write-Utf8File -Path $runnerPath -Content (New-RunnerPrompt -IterationDirectory $iterationDirectory -IterationNumber $iterationNumber -ManifestEvals @($manifestEvals)) - Write-Host "Prepared $($manifestEvals.Count) eval case(s) for '$Skill' (iteration $iterationNumber) as $($manifestEvals.Count * 2) hermetic run package(s)." + Write-Host "Prepared $($manifestEvals.Count) eval case(s) for '$Skill' (iteration $iterationNumber) as $($manifestEvals.Count * 2) isolated run package(s)." Write-Host "Package: $iterationDirectory" Write-Host '' Write-Host 'Every run is a self-contained directory: repo/ is the working tree, home/ is an isolated' @@ -1514,7 +1517,7 @@ function New-RunnerPrompt { [void]$builder.AppendLine('1. Read `manifest.json` and `execution-profile.json`. If `runner` is null, unavailable, or unsupported, fail clearly; do not guess a default. The profile contains no credentials.') [void]$builder.AppendLine('2. Resolve the selected package-local runner with the resolver. Ask it for `describe` and validate its protocol, descriptor, and capability declarations before running an arm. Do not invent harness-specific CLI commands.') [void]$builder.AppendLine('3. For every eval case, use the exact `run_manifest` path from `manifest.json` and the same profile path. Preflight each arm, then invoke the runner exactly once with `execute`. The runner receives only `run.json` and `execution-profile.json`; it must never receive or inspect expected output, assertions, grading, paired output, benchmark data, or human feedback.') - [void]$builder.AppendLine('4. Keep `with_skill` and `without_skill` in fresh independent processes/sessions. Use the same model, provider, configuration, tools, and limits. The runner must send each `prompt.md` unchanged as the first task input and must enforce the run contract, including the baseline skill exclusion and filesystem boundary.') + [void]$builder.AppendLine('4. Keep `with_skill` and `without_skill` in fresh independent processes/sessions. Use the same model, provider, configuration, tools, and limits. The runner must send each `prompt.md` unchanged as the first task input and must enforce the run contract, including the baseline skill exclusion and the staged filesystem/workspace boundary.') [void]$builder.AppendLine('5. Save the runner''s single normalized JSON response unchanged as the matching `execution_result` path. Preserve the complete final response, status, telemetry, evidence references, hashes, isolation mechanisms, warnings, and compatibility deviations. Do not retry for answer quality. A refusal is a result; timeout, harness failure, and incompatibility are results.') [void]$builder.AppendLine('6. If the runner cannot satisfy a required guarantee, keep the normalized status `incompatible` and stop that arm. Never fall back to the old generic isolated-worker behavior and never substitute a different runner.') [void]$builder.AppendLine() @@ -1562,7 +1565,7 @@ function New-PackageReadme { [void]$builder.AppendLine("- ``$($entry.eval_name)/`` - eval $($entry.eval_id)") } [void]$builder.AppendLine() - [void]$builder.AppendLine('Each eval directory holds the grading key (`eval-metadata.json`), result stubs under `results/`, and two hermetic run directories: `with_skill/` and `without_skill/`. A run directory holds `prompt.md`, a `run.json` contract, a `repo/` working tree materialized from the fixtures, an isolated `home/`, and - for `with_skill` only - a `skill/` directory with the candidate skill. The grading key and results sit outside both run directories, so a worker confined to its run directory never sees them.') + [void]$builder.AppendLine('Each eval directory holds the grading key (`eval-metadata.json`), result stubs under `results/`, and two isolated run directories: `with_skill/` and `without_skill/`. A run directory holds `prompt.md`, a `run.json` contract, a `repo/` working tree materialized from the fixtures, an isolated `home/`, and - for `with_skill` only - a `skill/` directory with the candidate skill. The grading key and results sit outside both run directories, so a worker that stays within its run directory is never handed them.') [void]$builder.AppendLine('The package root also holds `execution-profile.json`, the package-local Eval Runner protocol under `tools/eval-runners/`, and raw `execution-result.json` paths beside the existing result stubs. `run.json` defines what one blind arm must execute; the profile defines with what runner/model/configuration; the selected runner defines how.') [void]$builder.AppendLine('The package also carries the exact Anthropic skill-creator assets used after execution under `tools/skill-creator`: `tools/skill-creator/agents/grader.md`, `tools/skill-creator/agents/comparator.md`, `tools/skill-creator/agents/analyzer.md`, `tools/skill-creator/references/schemas.md`, `tools/skill-creator/scripts/aggregate_benchmark.py`, and `tools/skill-creator/eval-viewer/generate_review.py` plus `tools/skill-creator/eval-viewer/viewer.html`.') [void]$builder.AppendLine() @@ -1570,12 +1573,12 @@ function New-PackageReadme { [void]$builder.AppendLine() [void]$builder.AppendLine('The package guarantees what a generator can: identical materialized repositories for both runs, the candidate skill staged only under `with_skill/skill/`, an empty isolated `home/` per run, and a `run.json` that names only paths inside the run directory. Fixture and skill hashes are recorded so you can prove what each worker received.') [void]$builder.AppendLine() - [void]$builder.AppendLine('The harness must supply the rest at runtime: a fresh context per run, the run directory as the working and config root (working directory `repo/`, HOME `home/`), and a filesystem sandbox that keeps the worker inside its run directory so global skills, global config, the source repository, the paired run, and the grading key stay out of reach. Prompt wording alone does not enforce this; the sandbox does.') + [void]$builder.AppendLine('The harness must supply the rest at runtime: a fresh context per run, the run directory as the working and config root (working directory `repo/`, HOME `home/`), controlled candidate-skill exposure, prompt fidelity, and sufficient response capture. Because global skills, global config, the source repository, the paired run, and the grading key are never staged inside a run directory, a worker that stays within its run directory is not handed them. The selected runner enforces these controls - prompt wording alone does not - and reports strict confidence when it also proves hard filesystem confinement or pragmatic confidence when the mandatory controls hold without it. Hard filesystem confinement is an added confidence signal, not a prerequisite, so Windows and other hosts without a hard sandbox run in pragmatic mode.') [void]$builder.AppendLine() [void]$builder.AppendLine('## How to run') [void]$builder.AppendLine() [void]$builder.AppendLine('1. Read `execution-profile.json`. If no runner is selected, fail clearly instead of guessing. Resolve the selected package-local runner and run `describe`, then `preflight`, for every arm.') - [void]$builder.AppendLine('2. Invoke `execute` exactly once for each `run.json`, preserving its one JSON `execution-result.json` unchanged. The runner must provide a fresh process/session, isolated home/config, isolated CWD, filesystem confinement, baseline skill exclusion, prompt fidelity, model/configuration lock, and complete response capture or return `incompatible`.') + [void]$builder.AppendLine('2. Invoke `execute` exactly once for each `run.json`, preserving its one JSON `execution-result.json` unchanged. The runner must provide a fresh process/session, isolated home/config, isolated CWD, baseline skill exclusion, prompt fidelity, model/configuration lock, and complete response capture, or return `incompatible`. Hard filesystem confinement is not one of these mandatory controls: when the runner proves it the run reports strict isolation, and when it does not the run reports pragmatic isolation.') [void]$builder.AppendLine('3. After all arms complete or fail, run `tools/eval-runners/bridge-execution-result.ps1` for each raw result. It writes the existing `eval-result/2` file and preserves explicit unavailable telemetry. Only then read the grading key, grade with `tools/skill-creator/agents/grader.md`, and run `tools/generate-eval-report.ps1`.') [void]$builder.AppendLine() [void]$builder.AppendLine('`RUN-THIS.prompt.md` is the external Eval Orchestrator handoff. It selects the package-local runner from the profile, invokes the common protocol once per blind arm, bridges raw evidence into the existing result shape, reveals grading material only after execution, and invokes Anthropic skill-creator''s compatible aggregator and static viewer through the package adapter. It never executes an eval prompt in its own context.') @@ -1880,7 +1883,7 @@ function Invoke-CollectMode { [void]$builder.AppendLine() [void]$builder.AppendLine('## Isolation reported') [void]$builder.AppendLine() - [void]$builder.AppendLine('Flags each run''s harness confirmed: fresh context, isolated home, isolated cwd, filesystem sandbox, candidate skill exposure, transcript capture (Y/N, ? unknown). Process-dependent assertions are only gradeable from a run with process evidence.') + [void]$builder.AppendLine('Flags each run''s harness confirmed: fresh context, isolated home, isolated cwd, filesystem confinement (strict when a hard sandbox is proven, otherwise pragmatic), candidate skill exposure, transcript capture (Y/N, ? unknown). Process-dependent assertions are only gradeable from a run with process evidence.') [void]$builder.AppendLine() [void]$builder.AppendLine('| Eval | Configuration | Isolation | Process evidence |') [void]$builder.AppendLine('| --- | --- | --- | --- |') diff --git a/scripts/validate-skill-templates.ps1 b/scripts/validate-skill-templates.ps1 index 0831431..6264054 100644 --- a/scripts/validate-skill-templates.ps1 +++ b/scripts/validate-skill-templates.ps1 @@ -1343,7 +1343,7 @@ Add-ValidationResult -Results $results -Name 'Skill evaluation prepares portable } foreach ($isolationField in @('fresh_context_required', 'isolated_home_required', 'isolated_cwd_required')) { if (-not [bool]$manifest.isolation.$isolationField) { - throw "manifest.isolation.$isolationField must be true so a harness knows the run is hermetic." + throw "manifest.isolation.$isolationField must be true so a harness knows the run requires an isolated context." } } From fe1070cae462c6abd7c03f8bb1234bcc0dec9cb5 Mon Sep 17 00:00:00 2001 From: "aicia[bot]" Date: Fri, 21 Aug 2026 21:46:49 +0200 Subject: [PATCH 07/39] =?UTF-8?q?=F0=9F=93=9D=20document=20github=20copilo?= =?UTF-8?q?t=20reference=20configuration=20and=20eval=20isolation?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Establish GitHub Copilot with claude-haiku-4.5 as the Codebelt reference evaluation configuration for economical and stable comparison across evals. Update eval isolation terminology from 'hermetic' to 'isolated' and clarify Windows support as a first-class pragmatic evaluation target. Document filesystem confinement as a confidence enhancement rather than a universal prerequisite, enabling broader platform coverage. --- AGENTS.md | 6 +++--- README.md | 2 +- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index c3f67fd..ee0f76e 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -76,11 +76,11 @@ Generate the package with the repository script rather than by hand: pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -Skill ``` -It reads `skills//evals/evals.json` and writes one directory per eval into `.bot/-workspace/iteration-/`. The grading key and result stubs stay at the eval-case level, outside the two hermetic run directories a worker actually sees: +It reads `skills//evals/evals.json` and writes one directory per eval into `.bot/-workspace/iteration-/`. The grading key and result stubs stay at the eval-case level, outside the two isolated run directories a worker actually sees: - `eval-metadata.json` — eval id and name, original prompt, expected output, assertions, required fixtures, fixture and skill hashes, and the assumptions needed to reproduce the run. This is the grading key and lives outside every run directory. - `results/` — one prefilled result stub per configuration, also outside the run directories. -- `with_skill/` — a hermetic run directory that is the worker's sandbox root. It holds `prompt.md` (the task with the effective skill instructions inlined, plus the same input context and response contract as the baseline), `run.json` (a harness-neutral contract naming only paths inside the run directory), `repo/` (the fixtures materialized as real files, which is the worker's working directory), an isolated empty `home/`, and `skill//` (the exact candidate skill revision, so nothing falls back to a globally installed copy). +- `with_skill/` — an isolated run directory that is the worker's staged root. It holds `prompt.md` (the task with the effective skill instructions inlined, plus the same input context and response contract as the baseline), `run.json` (a harness-neutral contract naming only paths inside the run directory), `repo/` (the fixtures materialized as real files, which is the worker's working directory), an isolated empty `home/`, and `skill//` (the exact candidate skill revision, so nothing falls back to a globally installed copy). - `without_skill/` — the same run directory without any `skill/` directory and with no skill instructions or mention of the skill under test. Its `repo/` is byte-identical to the with_skill one. At the iteration root it also writes `manifest.json` and `RUN-THIS.prompt.md`, the single prompt that hands the whole package to an agent of the user's choosing. The one-file path requires a harness that can create isolated workers or sessions, each launched from its run directory with `repo/` as the working directory and `home/` as an isolated profile. A plain single-context client runs one prompt file directly per fresh session instead. @@ -134,7 +134,7 @@ Four things still hold while you execute: An agent that prepared a package in this session does not get to turn around and execute it. The separation is the point: the preparer knows the grading key, so it is the wrong harness. This is the only role-based disqualification. -The selected executor has two ordered phases. Its current context may read `RUN-THIS.prompt.md`, `manifest.json`, `execution-profile.json`, and the runner protocol files, but it must not execute an eval prompt itself. In phase one, it resolves the selected runner, validates `describe`, preflights each `run.json`, and invokes `execute` exactly once for every case's fresh `with_skill` and `without_skill` arm. The runner launches each native harness session from its own run directory with `repo/` as the working directory, `home/` as the isolated profile, and the required filesystem boundary. The runner receives only `run.json` and `execution-profile.json`; workers never see the runner, manifest, grading key, sibling results, or orchestration commentary, because all of those live outside the run directory. Never reuse a worker or session between runs. In phase two, after all available execution results are complete or failed, the executor validates and freezes the raw results, bridges them into `eval-result/2`, reads the grading key, follows the packaged `skill-creator` grader guidance, writes the grading evidence, invokes the package adapter so Anthropic's aggregator and eval viewer produce the report, and returns the report path and comparison. It does not ask the user whether to start either phase. +The selected executor has two ordered phases. Its current context may read `RUN-THIS.prompt.md`, `manifest.json`, `execution-profile.json`, and the runner protocol files, but it must not execute an eval prompt itself. In phase one, it resolves the selected runner, validates `describe`, preflights each `run.json`, and invokes `execute` exactly once for every case's fresh `with_skill` and `without_skill` arm. The runner launches each native harness session from its own run directory with `repo/` as the working directory, `home/` as the isolated profile, and the required isolation controls; hard filesystem confinement, when a runner proves it, raises the reported isolation from pragmatic to strict but is not itself a prerequisite. The runner receives only `run.json` and `execution-profile.json`; workers never see the runner, manifest, grading key, sibling results, or orchestration commentary, because all of those live outside the run directory. Never reuse a worker or session between runs. In phase two, after all available execution results are complete or failed, the executor validates and freezes the raw results, bridges them into `eval-result/2`, reads the grading key, follows the packaged `skill-creator` grader guidance, writes the grading evidence, invokes the package adapter so Anthropic's aggregator and eval viewer produce the report, and returns the report path and comparison. It does not ask the user whether to start either phase. The candidate instructions are already inlined in the with_skill run's `prompt.md` and staged under its `skill//` directory; the orchestrator does not load or summarize them for the worker. The baseline run has no `skill/` directory and no candidate instructions, and the orchestrator must not expose the candidate skill through another route, including a globally installed copy. The generated prompt files and the baseline `run.json` also omit the skill name, eval identifiers, and configuration labels so workers receive an ordinary task rather than an announcement that they are under evaluation. diff --git a/README.md b/README.md index 2254e0f..51a29f7 100644 --- a/README.md +++ b/README.md @@ -14,7 +14,7 @@ Another repo rule is intentionally strict: every repo-managed skill ships with i Skill validation is local and deterministic. The Priority 1 **AI/LLM Evaluation Automation Prohibition** in `AGENTS.md` forbids repository preparation, validation, CI, hooks, deterministic tests, automatic fan-out, graders, and completion gates from using an authenticated Copilot, Claude, Codex, Gemini, or other model account. There is no repository opt-in switch. A human-selected external Eval Orchestrator may invoke an explicitly selected package-local Eval Runner for a package it was handed; that boundary never authorizes live model execution in CI or automatic repository workflows. -Evaluation keeps Anthropic's `skill-creator` methodology and portable paired-run conventions while replacing only the execution transport. `run.json` remains the runner-neutral one-arm contract; `execution-profile.json` selects the runner/provider/model/configuration; and `execution-result.json` normalizes one blind execution before the existing `eval-result/2` bridge and reports. The conforming real runners are Codex, OpenCode, and Cline, with a deterministic fake runner used for conformance. The repository prepares the paired candidate and baseline inputs as a portable package and stops; the external Eval Orchestrator resolves, preflights, and invokes the selected Eval Runner once per fresh arm, then grades and reports. Hard filesystem confinement raises reported isolation confidence from pragmatic to strict but is not a universal platform prerequisite; mandatory experimental controls remain fail-closed. Freebuff remains planned/blocked until it exposes a supported noninteractive machine-readable transport. No provider has a privileged runtime role, and native skill activation is not evaluated in v0.9.1. Adding or modifying a skill triggers package preparation automatically, as a completion gate an agent cannot skip: +Evaluation keeps Anthropic's `skill-creator` methodology and portable paired-run conventions while replacing only the execution transport. `run.json` remains the runner-neutral one-arm contract; `execution-profile.json` selects the runner/provider/model/configuration; and `execution-result.json` normalizes one blind execution before the existing `eval-result/2` bridge and reports. The conforming real runners are GitHub Copilot, Codex, OpenCode, and Cline, with a deterministic fake runner used for conformance. GitHub Copilot with `claude-haiku-4.5` is the Codebelt reference evaluation configuration — a repository convention for economical, stable comparison, not an Anthropic default — and the model stays configurable through `execution-profile.json`. The repository prepares the paired candidate and baseline inputs as a portable package and stops; the external Eval Orchestrator resolves, preflights, and invokes the selected Eval Runner once per fresh arm, then grades and reports. Hard filesystem confinement raises reported isolation confidence from pragmatic to strict but is not a universal platform prerequisite, so Windows is a first-class pragmatic target; mandatory experimental controls remain fail-closed. Freebuff remains planned/blocked until it exposes a supported noninteractive machine-readable transport. No provider has a privileged runtime role, and native skill activation is not evaluated in v0.9.1. Adding or modifying a skill triggers package preparation automatically, as a completion gate an agent cannot skip: ```powershell pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -Changed From 3077725a569370ea1ddbe9d27cff3711da69e53f Mon Sep 17 00:00:00 2001 From: "aicia[bot]" Date: Fri, 21 Aug 2026 23:19:14 +0200 Subject: [PATCH 08/39] =?UTF-8?q?=F0=9F=A7=B1=20update=20github=20copilot?= =?UTF-8?q?=20runner=20for=20stdin=20prompt=20delivery?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Modernize the GitHub Copilot eval runner to deliver the prepared prompt through stdin instead of the --prompt argument. This improves byte fidelity and supports larger prompts without hitting command-line length limits. Update authentication handling to follow Copilot's normal precedence: explicit COPILOT_GITHUB_TOKEN/GH_TOKEN/GITHUB_TOKEN environment variables, then OS keychain, then GitHub CLI fallback through GH_CONFIG_DIR. Remove --no-custom-instructions so repository-owned instructions remain visible to both paired arms. Add COPILOT_CACHE_HOME isolation. Refine --secret-env-vars to filter all listed token variables from shell and MCP child environments. Update preflight to report conditional native keychain readiness without requiring a live model request. Extend runner conformance tests to validate stdin prompt delivery with byte-exact verification, multiple token variables, repository instruction visibility, and authentication source detection. --- scripts/eval-runners/README.md | 33 ++- .../eval-runners/github-copilot/runner.ps1 | 197 +++++++++++------ .../tests/test-runner-conformance.ps1 | 207 ++++++++++++++---- 3 files changed, 321 insertions(+), 116 deletions(-) diff --git a/scripts/eval-runners/README.md b/scripts/eval-runners/README.md index 9378fc6..3e889e2 100644 --- a/scripts/eval-runners/README.md +++ b/scripts/eval-runners/README.md @@ -58,26 +58,35 @@ be selected. Cross-runner and cross-model numbers are never blended into one score; a paired `with_skill` versus `without_skill` comparison is only meaningful within one identical runner, model, and configuration stratum. -GitHub Copilot uses `copilot --prompt --output-format json --model - --allow-all-tools --no-ask-user --no-custom-instructions ---disable-builtin-mcps` with an isolated `COPILOT_HOME`; it passes no +GitHub Copilot uses `copilot -C --model +--output-format json --allow-all-tools --no-ask-user --disable-builtin-mcps +--no-color --log-level none --no-auto-update +--secret-env-vars=COPILOT_GITHUB_TOKEN,GH_TOKEN,GITHUB_TOKEN` with the exact +prepared prompt bytes delivered once through stdin. It passes no `--prompt`/`-p`, `--resume`, `--continue`, `--session-id`, or `--connect`, and it does not use the blanket `--yolo`, `--allow-all`, `--allow-all-paths`, or `--allow-all-urls` -switches. It authenticates from a narrow GitHub token (`COPILOT_GITHUB_TOKEN`, -`GH_TOKEN`, or `GITHUB_TOKEN` - for example `gh auth token` or a fine-grained -PAT with Copilot access) rather than importing the ambient `copilot login` -profile, and redacts that token from output with `--secret-env-vars`. Because -Copilot co-mingles login state with behavioral configuration in `COPILOT_HOME`, -the runner requires that separable token and classifies honestly rather than -copying the profile. Codex uses `--ask-for-approval never` with `exec --sandbox +switches. `--allow-all-tools` is a broad tool-approval grant required for +noninteractive execution; it does not disable path or URL verification. +Repository-owned custom instructions remain enabled and are staged identically +in both paired arms. Personal Copilot configuration is excluded by run-local +`COPILOT_HOME`, `COPILOT_CACHE_HOME`, `HOME`, `USERPROFILE`, and XDG roots; +the runner does not copy the normal `.copilot` directory. Authentication follows +Copilot's normal order: explicit `COPILOT_GITHUB_TOKEN`, `GH_TOKEN`, or +`GITHUB_TOKEN`, then the OS credential store, then GitHub CLI fallback through a +host-derived `GH_CONFIG_DIR` when available. `--secret-env-vars` removes every +listed token variable from shell and MCP child environments. Preflight does not +make a model request and therefore reports native keychain/service readiness as +conditional rather than claiming successful remote authentication. Codex uses +`--ask-for-approval never` with `exec --sandbox workspace-write`; it does not combine explicit sandbox selection with `--approve-for-me`. OpenCode uses `run --format json --auto` with isolated global/config roots and preserves repository-owned project configuration; it does not depend on `OPENCODE_DISABLE_PROJECT_CONFIG` or use `--pure`. Cline uses `--json`, `--auto-approve true`, `--retries 0`, `--config `, `--data-dir /.cline/data`, and run-local hooks; it passes no session -id. Each captures an exact observable CLI version and passes only a narrow -environment credential when available. None copies a global skill directory, +id. Each captures an exact observable CLI version and passes only documented +environment credentials when the selected runner supports them. None copies a +global skill directory, memory store, plugin set, or normal agent profile into a run. Freebuff is currently documented as planned/blocked. Its supported CLI remains diff --git a/scripts/eval-runners/github-copilot/runner.ps1 b/scripts/eval-runners/github-copilot/runner.ps1 index e802933..f56066a 100644 --- a/scripts/eval-runners/github-copilot/runner.ps1 +++ b/scripts/eval-runners/github-copilot/runner.ps1 @@ -4,7 +4,7 @@ .DESCRIPTION This is the only place where GitHub Copilot CLI flags, COPILOT_HOME - handling, non-interactive JSONL event parsing, GitHub-token authentication, + handling, non-interactive JSONL event parsing, Copilot authentication, and Copilot isolation limitations are defined. It implements the unchanged describe/preflight/execute process contract shared by every runner. @@ -28,9 +28,9 @@ $ErrorActionPreference = 'Stop' Set-StrictMode -Version Latest . (Join-Path $PSScriptRoot '..\runner-common.ps1') -# GitHub Copilot authenticates from a GitHub token, not a model-provider API -# key. These are the supported token variables in precedence order; only the -# present one is forwarded into the isolated worker environment. +# GitHub Copilot checks these token variables before its OS credential store and +# GitHub CLI fallback. The values are forwarded only to the Copilot process; +# --secret-env-vars removes them from shell and MCP child environments. $copilotAuthVariables = @('COPILOT_GITHUB_TOKEN', 'GH_TOKEN', 'GITHUB_TOKEN') # Model routing runs through GitHub Copilot; the profile provider names the # routing backend, not a direct model vendor. @@ -60,7 +60,7 @@ $descriptor = [ordered]@{ command_evidence = 'conditional' file_evidence = 'conditional' cost_telemetry = 'unsupported' - credential_child_filtering = 'conditional' + credential_child_filtering = 'supported' native_skill_activation_evidence = 'unsupported' } supported_telemetry = @('transcript_event_capture', 'token_telemetry', 'cache_token_telemetry', 'tool_call_telemetry', 'command_evidence', 'file_evidence') @@ -94,17 +94,35 @@ function Get-CopilotTokenVariable { return $null } -function Test-CopilotLoginProfileExists { - # Existence check only; the file is never read or logged so no credential or - # login value is exposed. The default location is COPILOT_HOME or the - # platform user profile's .copilot directory. - $configuredHome = [Environment]::GetEnvironmentVariable('COPILOT_HOME') - $homeRoot = if ([string]::IsNullOrWhiteSpace($configuredHome)) { - Join-Path ([Environment]::GetFolderPath('UserProfile')) '.copilot' - } else { - $configuredHome +function Get-CopilotGhConfigDirectory { + # GH_CONFIG_DIR is an authentication-state exception to the isolated + # Copilot configuration roots. Resolve it from GitHub CLI's documented + # precedence without reading or logging any credential file. + $configured = [Environment]::GetEnvironmentVariable('GH_CONFIG_DIR') + if ([string]::IsNullOrWhiteSpace($configured)) { + $xdgConfig = [Environment]::GetEnvironmentVariable('XDG_CONFIG_HOME') + if (-not [string]::IsNullOrWhiteSpace($xdgConfig)) { + $configured = Join-Path $xdgConfig 'gh' + } elseif ((Get-PlatformName) -eq 'windows') { + $applicationData = [Environment]::GetFolderPath([Environment+SpecialFolder]::ApplicationData) + if (-not [string]::IsNullOrWhiteSpace($applicationData)) { + $configured = Join-Path $applicationData 'GitHub CLI' + } + } else { + $userHome = [Environment]::GetEnvironmentVariable('HOME') + if ([string]::IsNullOrWhiteSpace($userHome)) { + $userHome = [Environment]::GetFolderPath([Environment+SpecialFolder]::UserProfile) + } + if (-not [string]::IsNullOrWhiteSpace($userHome)) { + $configured = Join-Path (Join-Path $userHome '.config') 'gh' + } + } } - return (Test-Path -LiteralPath (Join-Path $homeRoot 'config.json') -PathType Leaf) + + if ([string]::IsNullOrWhiteSpace($configured) -or -not (Test-Path -LiteralPath $configured -PathType Container)) { + return $null + } + return [System.IO.Path]::GetFullPath($configured) } function Invoke-CopilotCli { @@ -151,15 +169,6 @@ function Get-CopilotDescriptor { return $copy } -function Get-CopilotPromptText { - param([Parameter(Mandatory = $true)][object]$Inputs) - - # The prompt is delivered verbatim through --prompt. run.json stages UTF-8 - # prompt bytes, so decoding to a string and forwarding it as one argv value - # round-trips the exact characters; prompt_sha256 in the result confirms it. - return [System.Text.UTF8Encoding]::new($false).GetString($Inputs.Run.PromptBytes) -} - function New-CopilotCliArguments { param( [Parameter(Mandatory = $true)][object]$Inputs, @@ -169,21 +178,22 @@ function New-CopilotCliArguments { $directoryArgument = Get-SandboxVisiblePath -HostPath $Inputs.Run.WorkingDirectoryPath -RunRoot $Inputs.Run.RunRoot -Platform $VisiblePlatform $secretList = ($copilotAuthVariables -join ',') $arguments = [System.Collections.Generic.List[string]]::new() - # Noninteractive one-shot JSONL run. --allow-all-tools is the narrow switch - # that removes tool-approval prompts without --allow-all-paths or - # --allow-all-urls, so file access stays inside the working directory and - # temp roots and URL/network access is not blanket-granted. --no-ask-user - # keeps the agent from pausing for questions. --no-custom-instructions stops - # ambient AGENTS.md / instruction discovery (including the host repository's) - # from leaking into either arm. --disable-builtin-mcps drops the ambient - # GitHub MCP server. --secret-env-vars redacts the GitHub token from output. + # Noninteractive one-shot JSONL run. --allow-all-tools is a broad + # tool-approval grant required for programmatic execution; it does not + # include --allow-all-paths or --allow-all-urls, so normal path and URL + # verification remains active. --no-ask-user keeps the agent from pausing + # for questions. Repository-owned custom instructions remain enabled so + # both paired arms see the staged repository exactly as supplied. Ambient + # Copilot state is excluded by the run-local COPILOT_HOME and environment + # roots. --disable-builtin-mcps drops the built-in GitHub MCP server. + # --secret-env-vars removes the listed credentials from shell and MCP + # child environments. foreach ($argument in @( '-C', $directoryArgument, '--model', $Inputs.Profile.Model, '--output-format', 'json', '--allow-all-tools', '--no-ask-user', - '--no-custom-instructions', '--disable-builtin-mcps', '--no-color', '--log-level', 'none', @@ -196,8 +206,6 @@ function New-CopilotCliArguments { $arguments.Add('--reasoning-effort') $arguments.Add([string]$Inputs.Profile.ReasoningEffort) } - $arguments.Add('--prompt') - $arguments.Add((Get-CopilotPromptText -Inputs $Inputs)) return @($arguments) } @@ -268,7 +276,7 @@ function Get-CopilotPreflight { $reasons.Add("Copilot --help failed with exit status $($help.ExitCode).") } else { $helpText = [string]::Join("`n", @($help.Stdout, $help.Stderr)) - foreach ($flag in @('--prompt', '--output-format', '--model', '--allow-all-tools', '--no-ask-user', '--no-custom-instructions', '--disable-builtin-mcps', '--secret-env-vars')) { + foreach ($flag in @('--output-format', '--model', '--allow-all-tools', '--no-ask-user', '--disable-builtin-mcps', '--secret-env-vars')) { if ($helpText -notmatch [regex]::Escape($flag)) { $reasons.Add("The installed Copilot CLI does not advertise required flag '$flag'.") } @@ -278,13 +286,17 @@ function Get-CopilotPreflight { foreach ($forbidden in @('--resume', '-r', '--continue', '--session-id', '--connect', '--yolo', '--allow-all', '--allow-all-paths', '--allow-all-urls')) { if (@($constructed) -contains $forbidden) { $reasons.Add("The constructed Copilot invocation must not use session-continuation or over-broad permission option '$forbidden'.") } } - foreach ($required in @('--prompt', '--output-format', '--allow-all-tools', '--no-ask-user', '--no-custom-instructions')) { - if (@($constructed) -notcontains $required) { $reasons.Add("The constructed Copilot invocation must include '$required'.") } + foreach ($required in @('--output-format', '--allow-all-tools', '--no-ask-user', '--disable-builtin-mcps', '--secret-env-vars')) { + $present = @($constructed) -contains $required + if ($required -eq '--secret-env-vars') { + $present = $present -or (@($constructed | Where-Object { $_ -like '--secret-env-vars=*' }).Count -gt 0) + } + if (-not $present) { $reasons.Add("The constructed Copilot invocation must include '$required'.") } } - $promptCount = @($constructed | Where-Object { $_ -eq '--prompt' -or $_ -eq '-p' }).Count - if ($promptCount -ne 1) { $reasons.Add('The constructed Copilot invocation must deliver the prompt exactly once.') } + $promptOptionCount = @($constructed | Where-Object { $_ -eq '--prompt' -or $_ -eq '-p' -or $_ -like '--prompt=*' }).Count + if ($promptOptionCount -ne 0) { $reasons.Add('The constructed Copilot invocation must not place the prompt in argv; prompt delivery uses stdin.') } if ($reasons.Count -eq 0) { - $checks.Add((New-PreflightCheck -Name 'harness_contract' -Status passed -Detail 'Copilot accepts the constructed noninteractive invocation: --prompt once, --output-format json, --model, --allow-all-tools, --no-ask-user, --no-custom-instructions, --disable-builtin-mcps, and no session continuation.')) + $checks.Add((New-PreflightCheck -Name 'harness_contract' -Status passed -Detail 'Copilot accepts the constructed noninteractive invocation: stdin prompt delivery, --output-format json, --model, broad --allow-all-tools approval, --no-ask-user, --disable-builtin-mcps, --secret-env-vars, and no session continuation.')) } } } catch { @@ -293,12 +305,17 @@ function Get-CopilotPreflight { } $tokenVariable = Get-CopilotTokenVariable + $ghConfigDirectory = Get-CopilotGhConfigDirectory if (-not [string]::IsNullOrWhiteSpace($tokenVariable)) { - $checks.Add((New-PreflightCheck -Name 'authentication' -Status passed -Detail "Authentication is available through the narrow $tokenVariable GitHub token; the ambient Copilot login profile is not copied into the run.")) - } elseif (Test-CopilotLoginProfileExists) { - $reasons.Add('A Copilot login profile exists at the default location, but GitHub Copilot co-mingles login state with behavioral configuration in COPILOT_HOME. The runner requires a separable GitHub token (COPILOT_GITHUB_TOKEN, GH_TOKEN, or GITHUB_TOKEN - for example `gh auth token` or a fine-grained PAT with Copilot access) rather than importing that profile.') + $checks.Add((New-PreflightCheck -Name 'authentication' -Status passed -Detail "Authentication is available through the explicit $tokenVariable environment variable; Copilot OS-keychain and GitHub CLI state are not copied into the run.")) } else { - $reasons.Add('No GitHub Copilot authentication is available. Export COPILOT_GITHUB_TOKEN (or GH_TOKEN/GITHUB_TOKEN) with Copilot access.') + $ghDetail = if ($null -eq $ghConfigDirectory) { + 'no host GitHub CLI configuration directory was observable' + } else { + 'the host GitHub CLI configuration directory will remain available for Copilot''s documented gh auth token fallback' + } + $checks.Add((New-PreflightCheck -Name 'authentication' -Status unavailable -Detail "No explicit token is present; native Copilot OS-keychain lookup is delegated to the installed CLI and $ghDetail. This preflight does not contact the Copilot service.")) + $warnings.Add('Authentication readiness beyond explicit environment tokens and the observable GitHub CLI fallback cannot be proven without a live Copilot request; preflight remains conditional and does not reject a tokenless native OAuth/keychain configuration.') } if ($platform -notin @('linux', 'macos')) { @@ -312,20 +329,14 @@ function Get-CopilotPreflight { $checks.Add((New-PreflightCheck -Name 'filesystem_confinement' -Status passed -Detail "External $($sandboxInfo.Source) sandbox confines Copilot to the staged run and required system runtime paths.")) } - $checks.Add((New-PreflightCheck -Name 'fresh_session' -Status passed -Detail 'The adapter starts one new copilot -p process and passes no --resume, --continue, --session-id, or --connect.')) - $checks.Add((New-PreflightCheck -Name 'ambient_configuration' -Status passed -Detail 'The adapter points COPILOT_HOME at the run''s isolated home, disables built-in MCP servers, and disables ambient custom instructions, so global skills, plugins, MCP config, sessions, memories, and instructions do not load.')) + $checks.Add((New-PreflightCheck -Name 'fresh_session' -Status passed -Detail 'The adapter starts one new Copilot process, supplies one stdin prompt, and passes no --resume, --continue, --session-id, or --connect.')) + $checks.Add((New-PreflightCheck -Name 'ambient_configuration' -Status passed -Detail 'The adapter points COPILOT_HOME, Copilot cache, HOME, USERPROFILE, and XDG roots at the run''s isolated home, disables built-in MCP servers, and preserves only staged repository-owned custom instructions; personal Copilot skills, plugins, MCP config, sessions, memories, and instructions are not imported.')) $checks.Add((New-PreflightCheck -Name 'run_paths' -Status passed -Detail "-C $($run.WorkingDirectoryPath); COPILOT_HOME under $($run.HomeDirectoryPath)")) - $checks.Add((New-PreflightCheck -Name 'prompt_fidelity' -Status passed -Detail 'The exact UTF-8 prompt bytes are delivered once as the --prompt argument value.')) - $checks.Add((New-PreflightCheck -Name 'credential_boundary' -Status passed -Detail 'Only the present GitHub token variable is forwarded; --secret-env-vars redacts it from Copilot output; no login profile or auth file is copied into the run.')) - $warnings.Add('GitHub Copilot inherits the shell environment for shell tools apart from a blocklist; the runner redacts the GitHub token from output with --secret-env-vars but cannot independently prove the child-tool environment filter hides it from every tool path.') + $checks.Add((New-PreflightCheck -Name 'prompt_fidelity' -Status passed -Detail 'The prepared UTF-8 prompt bytes are supplied once through stdin; the execution fake proves the received bytes match the staged prompt.')) + $checks.Add((New-PreflightCheck -Name 'credential_boundary' -Status passed -Detail 'Only supported authentication state is made available to Copilot; --secret-env-vars removes every listed token variable from shell and MCP child environments; no Copilot profile or credential file is copied.')) if ($platform -eq 'macos') { $warnings.Add('macOS sandbox-exec is deprecated by Apple but is used only when present; a future runner revision may replace it with an equivalent supported mechanism.') } - $promptLength = (Get-CopilotPromptText -Inputs $Inputs).Length - if ($promptLength -gt 30000) { - $warnings.Add('The prompt exceeds 30000 characters; on some hosts a very long --prompt argument can approach the operating-system command-line length limit.') - } - $hardConfinement = $null -ne $sandboxInfo -and $platform -in @('linux', 'macos') $capabilities = Get-CopilotCapabilityMap -Inputs $Inputs -HardFilesystemConfinement $hardConfinement $harnessVersion = if ($null -eq $versionObservation) { 'unavailable' } else { [string]$versionObservation.Version } @@ -333,7 +344,7 @@ function Get-CopilotPreflight { foreach ($key in $descriptor.Keys) { $descriptorCopy[$key] = $descriptor[$key] } $descriptorCopy.harness = [ordered]@{ name = 'GitHub Copilot CLI'; version = $harnessVersion } $mechanisms = [System.Collections.Generic.List[string]]::new() - foreach ($mechanism in @('copilot --prompt --output-format json', '--allow-all-tools', '--no-ask-user', '--no-custom-instructions', '--disable-builtin-mcps', '--secret-env-vars token redaction', 'isolated COPILOT_HOME', 'isolated HOME/XDG roots', 'narrow GitHub token authentication', 'no session continuation')) { $mechanisms.Add($mechanism) } + foreach ($mechanism in @('copilot --output-format json', 'prompt on stdin', '--allow-all-tools broad tool approval', 'path and URL verification preserved (no --allow-all-paths/--allow-all-urls)', '--no-ask-user', 'repository-owned custom instructions preserved', '--disable-builtin-mcps', '--secret-env-vars shell/MCP child filtering', 'isolated COPILOT_HOME and COPILOT_CACHE_HOME', 'isolated HOME/XDG roots', 'OS-keychain authentication delegated to Copilot', 'GitHub CLI authentication fallback through GH_CONFIG_DIR when available', 'no session continuation')) { $mechanisms.Add($mechanism) } if ($hardConfinement) { $mechanisms.Add("external $($sandboxInfo.Source) filesystem sandbox") } else { $mechanisms.Add('pragmatic process/environment isolation without hard filesystem confinement') } return New-PreflightDocument -Descriptor $descriptorCopy -Profile $profile -Run $run -Compatible ($reasons.Count -eq 0) -Checks @($checks) -Mechanisms @($mechanisms) -ResolvedCapabilities $capabilities -Warnings @($warnings) -Reasons @($reasons) } @@ -342,11 +353,19 @@ function New-CopilotEnvironment { param([Parameter(Mandatory = $true)][object]$Inputs) $copilotHome = Join-Path $Inputs.Run.HomeDirectoryPath '.copilot' + $copilotCacheHome = Join-Path $Inputs.Run.HomeDirectoryPath '.copilot-cache' New-Item -ItemType Directory -Path $copilotHome -Force | Out-Null - return New-RunnerEnvironment -Run $Inputs.Run -AuthenticationVariables $copilotAuthVariables -Additional @{ + New-Item -ItemType Directory -Path $copilotCacheHome -Force | Out-Null + $additional = @{ COPILOT_HOME = $copilotHome + COPILOT_CACHE_HOME = $copilotCacheHome COPILOT_AUTO_UPDATE = 'false' } + $ghConfigDirectory = Get-CopilotGhConfigDirectory + if ($null -ne $ghConfigDirectory) { + $additional['GH_CONFIG_DIR'] = $ghConfigDirectory + } + return New-RunnerEnvironment -Run $Inputs.Run -AuthenticationVariables $copilotAuthVariables -Additional $additional } function New-CopilotInsideEnvironment { @@ -364,6 +383,7 @@ function New-CopilotInsideEnvironment { TEMP = '/run/home/tmp' TMP = '/run/home/tmp' COPILOT_HOME = '/run/home/.copilot' + COPILOT_CACHE_HOME = '/run/home/.copilot-cache' COPILOT_AUTO_UPDATE = 'false' PATH = '/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin' CI = '1' @@ -374,9 +394,42 @@ function New-CopilotInsideEnvironment { $insideEnvironment[$authName] = [string]$Environment[$authName] } } + if ($Environment.Contains('GH_CONFIG_DIR') -and -not [string]::IsNullOrWhiteSpace([string]$Environment['GH_CONFIG_DIR'])) { + $insideEnvironment['GH_CONFIG_DIR'] = '/run/gh-config' + } return $insideEnvironment } +function Add-CopilotLinuxAuthenticationMount { + param( + [Parameter(Mandatory = $true)][string[]]$Arguments, + [Parameter(Mandatory = $true)][string]$HostPath + ) + + $result = [System.Collections.Generic.List[string]]::new() + foreach ($argument in $Arguments) { $result.Add([string]$argument) } + $separatorIndex = $result.IndexOf('--') + if ($separatorIndex -lt 0) { + throw 'The Linux Copilot sandbox argument list has no command separator.' + } + $result.Insert($separatorIndex, '/run/gh-config') + $result.Insert($separatorIndex, $HostPath) + $result.Insert($separatorIndex, '--ro-bind') + return @($result) +} + +function Add-CopilotMacosAuthenticationPath { + param( + [Parameter(Mandatory = $true)][string]$ProfilePath, + [Parameter(Mandatory = $true)][string]$HostPath + ) + + $normalized = $HostPath.Replace('\', '/') + $escaped = $normalized.Replace('"', '\"') + $line = '(allow file-read* (subpath "' + $escaped + '"))' + [System.IO.File]::AppendAllText($ProfilePath, "`n$line`n", [System.Text.UTF8Encoding]::new($false)) +} + function Write-CopilotCapture { param( [Parameter(Mandatory = $true)][object]$RunData, @@ -497,13 +550,19 @@ function Invoke-CopilotExecute { if ($platform -eq 'linux' -and $hardFilesystem) { $insideEnvironment = New-CopilotInsideEnvironment -Inputs $Inputs -Environment $environment $sandboxArguments = Get-LinuxEvalSandboxArguments -Inputs $Inputs -CommandInfo $commandInfo -InsideEnvironment $insideEnvironment - $process = Invoke-RunnerProcess -FileName $sandboxInfo.FileName -ArgumentList (@($sandboxArguments) + @($arguments)) -WorkingDirectory $Inputs.Run.WorkingDirectoryPath -Environment $environment -TimeoutSeconds $Inputs.Profile.TimeoutSeconds + if ($environment.Contains('GH_CONFIG_DIR') -and -not [string]::IsNullOrWhiteSpace([string]$environment['GH_CONFIG_DIR'])) { + $sandboxArguments = Add-CopilotLinuxAuthenticationMount -Arguments $sandboxArguments -HostPath ([string]$environment['GH_CONFIG_DIR']) + } + $process = Invoke-RunnerProcess -FileName $sandboxInfo.FileName -ArgumentList (@($sandboxArguments) + @($arguments)) -WorkingDirectory $Inputs.Run.WorkingDirectoryPath -Environment $environment -InputBytes $Inputs.Run.PromptBytes -TimeoutSeconds $Inputs.Profile.TimeoutSeconds } elseif ($platform -eq 'macos' -and $hardFilesystem) { $sandboxProfile = New-MacosEvalSandboxProfile -Inputs $Inputs -CommandInfo $commandInfo + if ($environment.Contains('GH_CONFIG_DIR') -and -not [string]::IsNullOrWhiteSpace([string]$environment['GH_CONFIG_DIR'])) { + Add-CopilotMacosAuthenticationPath -ProfilePath $sandboxProfile -HostPath ([string]$environment['GH_CONFIG_DIR']) + } $sandboxArguments = @('-f', $sandboxProfile, '--', $commandInfo.FileName) + @($commandInfo.Prefix) + @($arguments) - $process = Invoke-RunnerProcess -FileName $sandboxInfo.FileName -ArgumentList $sandboxArguments -WorkingDirectory $Inputs.Run.WorkingDirectoryPath -Environment $environment -TimeoutSeconds $Inputs.Profile.TimeoutSeconds + $process = Invoke-RunnerProcess -FileName $sandboxInfo.FileName -ArgumentList $sandboxArguments -WorkingDirectory $Inputs.Run.WorkingDirectoryPath -Environment $environment -InputBytes $Inputs.Run.PromptBytes -TimeoutSeconds $Inputs.Profile.TimeoutSeconds } else { - $process = Invoke-CopilotCli -CommandInfo $commandInfo -Arguments $arguments -Inputs $Inputs -Environment $environment -TimeoutSeconds $Inputs.Profile.TimeoutSeconds + $process = Invoke-CopilotCli -CommandInfo $commandInfo -Arguments $arguments -Inputs $Inputs -Environment $environment -InputBytes $Inputs.Run.PromptBytes -TimeoutSeconds $Inputs.Profile.TimeoutSeconds } $stdoutArtifact = Write-CopilotCapture -RunData $Inputs -RelativePath 'evidence/copilot-events.jsonl' -Text $process.Stdout @@ -513,7 +572,11 @@ function Invoke-CopilotExecute { $artifacts.Add($stderrArtifact) $warnings = [System.Collections.Generic.List[string]]::new() - $parsed = ConvertFrom-JsonLines -Text $process.Stdout + $parsed = if ([string]::IsNullOrEmpty([string]$process.Stdout)) { + [pscustomobject]@{ Events = @(); Errors = @() } + } else { + ConvertFrom-JsonLines -Text $process.Stdout + } foreach ($parseError in @($parsed.Errors)) { $warnings.Add("Copilot event parse error: $parseError") } $parsedEvents = Read-CopilotEvents -Parsed $parsed -Warnings $warnings @@ -555,16 +618,18 @@ function Invoke-CopilotExecute { $capabilities = Get-CopilotCapabilityMap -Inputs $Inputs -HardFilesystemConfinement $hardFilesystem $mechanisms = [System.Collections.Generic.List[string]]::new() - foreach ($mechanism in @('copilot --prompt --output-format json', '--allow-all-tools', '--no-ask-user', '--no-custom-instructions', '--disable-builtin-mcps', '--secret-env-vars token redaction', 'isolated COPILOT_HOME', 'narrow GitHub token authentication', 'no session continuation')) { $mechanisms.Add($mechanism) } + foreach ($mechanism in @('copilot --output-format json', 'prompt on stdin', '--allow-all-tools broad tool approval', 'path and URL verification preserved (no --allow-all-paths/--allow-all-urls)', '--no-ask-user', 'repository-owned custom instructions preserved', '--disable-builtin-mcps', '--secret-env-vars shell/MCP child filtering', 'isolated COPILOT_HOME and COPILOT_CACHE_HOME', 'isolated HOME/XDG roots', 'OS-keychain authentication delegated to Copilot', 'GitHub CLI authentication fallback through GH_CONFIG_DIR when available', 'no session continuation')) { $mechanisms.Add($mechanism) } if ($hardFilesystem) { $mechanisms.Add("external $($sandboxInfo.Source) filesystem sandbox") } else { $mechanisms.Add('pragmatic process/environment isolation without hard filesystem confinement') } if (-not $hardFilesystem) { $warnings.Add('Hard filesystem confinement was unavailable; the completed arm is reported as pragmatic isolation.') } - $warnings.Add('GitHub Copilot inherits the shell environment for shell tools apart from a blocklist; the runner redacts the GitHub token from output with --secret-env-vars but cannot independently prove the child-tool environment filter hides it from every tool path.') $tokenVariable = Get-CopilotTokenVariable + $ghConfigDirectory = Get-CopilotGhConfigDirectory $credentialEvidence = [ordered]@{ - source = if ([string]::IsNullOrWhiteSpace($tokenVariable)) { 'unavailable' } else { 'environment' } + source = if (-not [string]::IsNullOrWhiteSpace($tokenVariable)) { 'environment' } elseif ($null -ne $ghConfigDirectory) { 'copilot_os_keychain_or_github_cli' } else { 'copilot_os_keychain_or_github_cli_unverified' } github_token_variable = $tokenVariable - secret_env_var_redaction = @($copilotAuthVariables) + secret_env_vars = @($copilotAuthVariables) + secret_env_var_scope = @('shell', 'mcp') + github_cli_config_forwarded = $null -ne $ghConfigDirectory login_profile_copied = $false auth_file_copied = $false value_observed = $false @@ -583,7 +648,7 @@ function Invoke-CopilotExecute { $finished = [DateTime]::UtcNow $sandboxEvidence = if (-not $hardFilesystem) { 'unavailable' } elseif ($platform -eq 'linux') { 'bwrap' } else { 'sandbox-exec' } - return New-ExecutionResult -Descriptor $executionDescriptor -Profile $Inputs.Profile -Run $Inputs.Run -Status $status -FinalResponse $finalText -FinalResponseReason $reason -StartedUtc $process.StartedUtc.ToString('o') -FinishedUtc $finished.ToString('o') -DurationSeconds $process.DurationSeconds -ExitStatus $exitStatus -Failure $failure -SessionId $sessionId -IsolationCapabilities $capabilities -IsolationMechanisms @($mechanisms) -ResolvedConfiguration $resolvedConfiguration -Telemetry $telemetry -Artifacts @($artifacts) -Warnings @($warnings) -Evidence ([ordered]@{ event_counts = $parsedEvents.EventCounts; observed_model = $observedModel; prompt_delivery = 'argument'; prompt_first_input = $true; resume = $false; stdout_exit_code = $process.ExitCode; sandbox = $sandboxEvidence; credential = $credentialEvidence }) -AttemptCount 1 + return New-ExecutionResult -Descriptor $executionDescriptor -Profile $Inputs.Profile -Run $Inputs.Run -Status $status -FinalResponse $finalText -FinalResponseReason $reason -StartedUtc $process.StartedUtc.ToString('o') -FinishedUtc $finished.ToString('o') -DurationSeconds $process.DurationSeconds -ExitStatus $exitStatus -Failure $failure -SessionId $sessionId -IsolationCapabilities $capabilities -IsolationMechanisms @($mechanisms) -ResolvedConfiguration $resolvedConfiguration -Telemetry $telemetry -Artifacts @($artifacts) -Warnings @($warnings) -Evidence ([ordered]@{ event_counts = $parsedEvents.EventCounts; observed_model = $observedModel; prompt_delivery = 'stdin'; prompt_first_input = $true; resume = $false; stdout_exit_code = $process.ExitCode; sandbox = $sandboxEvidence; credential = $credentialEvidence }) -AttemptCount 1 } try { diff --git a/scripts/eval-runners/tests/test-runner-conformance.ps1 b/scripts/eval-runners/tests/test-runner-conformance.ps1 index 42b5e48..509ccab 100644 --- a/scripts/eval-runners/tests/test-runner-conformance.ps1 +++ b/scripts/eval-runners/tests/test-runner-conformance.ps1 @@ -49,6 +49,7 @@ $recordedOldCopilotToken = $env:COPILOT_GITHUB_TOKEN $recordedOldGhToken = $env:GH_TOKEN $recordedOldGithubToken = $env:GITHUB_TOKEN $recordedOldCopilotHome = $env:COPILOT_HOME +$recordedOldGhConfigDir = $env:GH_CONFIG_DIR try { $fakeBin = Join-Path $recordedRoot 'bin' New-Item -ItemType Directory -Path $fakeBin -Force | Out-Null @@ -66,21 +67,49 @@ $logPath = Join-Path (Get-Location).Path ("{0}-fake-cli-log.jsonl" -f $harness) $arguments = @($RemainingArguments | ForEach-Object { [string]$_ }) $authNames = @('OPENAI_API_KEY', 'ANTHROPIC_API_KEY', 'GOOGLE_API_KEY', 'GEMINI_API_KEY', 'OPENROUTER_API_KEY', 'XAI_API_KEY', 'MISTRAL_API_KEY', 'CLINE_API_KEY') $authPresent = @($authNames | Where-Object { -not [string]::IsNullOrWhiteSpace([Environment]::GetEnvironmentVariable($_)) }) +$copilotAuthNames = @('COPILOT_GITHUB_TOKEN', 'GH_TOKEN', 'GITHUB_TOKEN') +$copilotAuthPresent = @($copilotAuthNames | Where-Object { -not [string]::IsNullOrWhiteSpace([Environment]::GetEnvironmentVariable($_)) }) +$copilotHome = [Environment]::GetEnvironmentVariable('COPILOT_HOME') +$repositoryAgentsPath = Join-Path (Get-Location).Path 'AGENTS.md' +$repositoryCopilotInstructionsPath = Join-Path (Get-Location).Path '.github\copilot-instructions.md' +$candidateSkillPath = Join-Path (Split-Path -Parent (Get-Location).Path) 'skill' +$repositoryInstructionMarkerVisible = $false +if (Test-Path -LiteralPath $repositoryCopilotInstructionsPath -PathType Leaf) { + $repositoryInstructionMarkerVisible = [IO.File]::ReadAllText($repositoryCopilotInstructionsPath, [Text.UTF8Encoding]::new($false)).Contains('repo-owned-copilot-instruction') +} +$copilotAuthenticationSource = if ($copilotAuthPresent.Count -gt 0) { + 'explicit_environment' +} elseif (-not [string]::IsNullOrWhiteSpace($copilotHome) -and (Test-Path -LiteralPath (Join-Path $copilotHome 'fixture-os-keychain-available') -PathType Leaf)) { + 'os_keychain' +} elseif (-not [string]::IsNullOrWhiteSpace([Environment]::GetEnvironmentVariable('GH_CONFIG_DIR')) -and (Test-Path -LiteralPath ([Environment]::GetEnvironmentVariable('GH_CONFIG_DIR')) -PathType Container)) { + 'github_cli' +} else { + 'unavailable' +} $record = [ordered]@{ args = $arguments working_directory = (Get-Location).Path home = [Environment]::GetEnvironmentVariable('HOME') userprofile = [Environment]::GetEnvironmentVariable('USERPROFILE') auth_names_present = $authPresent + copilot_auth_names_present = $copilotAuthPresent + copilot_authentication_source = $copilotAuthenticationSource unrelated_present = -not [string]::IsNullOrWhiteSpace([Environment]::GetEnvironmentVariable('AGENTIC_GLOBAL_SECRET')) disable_project_config_present = -not [string]::IsNullOrWhiteSpace([Environment]::GetEnvironmentVariable('OPENCODE_DISABLE_PROJECT_CONFIG')) project_config_visible = Test-Path -LiteralPath (Join-Path (Get-Location).Path 'opencode.json') -PathType Leaf stdin_received = $false - prompt_via_arg = ($arguments -contains '--prompt') -or ($arguments -contains '-p') - prompt_arg_count = @($arguments | Where-Object { $_ -eq '--prompt' -or $_ -eq '-p' }).Count - copilot_home = [Environment]::GetEnvironmentVariable('COPILOT_HOME') + prompt_via_arg = @($arguments | Where-Object { $_ -eq '--prompt' -or $_ -eq '-p' -or $_ -like '--prompt=*' }).Count -gt 0 + prompt_arg_count = @($arguments | Where-Object { $_ -eq '--prompt' -or $_ -eq '-p' -or $_ -like '--prompt=*' }).Count + copilot_home = $copilotHome + copilot_cache_home = [Environment]::GetEnvironmentVariable('COPILOT_CACHE_HOME') + gh_config_dir = [Environment]::GetEnvironmentVariable('GH_CONFIG_DIR') custom_instructions_disabled = ($arguments -contains '--no-custom-instructions') builtin_mcps_disabled = ($arguments -contains '--disable-builtin-mcps') + repository_agents_visible = Test-Path -LiteralPath $repositoryAgentsPath -PathType Leaf + repository_copilot_instructions_visible = Test-Path -LiteralPath $repositoryCopilotInstructionsPath -PathType Leaf + repository_instruction_marker_visible = $repositoryInstructionMarkerVisible + candidate_skill_staged = Test-Path -LiteralPath $candidateSkillPath -PathType Container + ambient_copilot_instructions_visible = if ([string]::IsNullOrWhiteSpace($copilotHome)) { $false } else { Test-Path -LiteralPath (Join-Path $copilotHome 'copilot-instructions.md') -PathType Leaf } secret_env_vars_arg = @($arguments | Where-Object { $_ -like '--secret-env-vars=*' }) } if ($arguments -contains '--version') { @@ -100,9 +129,29 @@ if ($arguments -contains '--help') { Write-Output $help exit 0 } -$stdinText = [Console]::In.ReadToEnd() -$record.stdin_received = -not [string]::IsNullOrEmpty($stdinText) -$probeCommand = '$result = [ordered]@{ provider_visible = -not [string]::IsNullOrWhiteSpace([Environment]::GetEnvironmentVariable(''OPENAI_API_KEY'')); copilot_token_visible = -not [string]::IsNullOrWhiteSpace([Environment]::GetEnvironmentVariable(''COPILOT_GITHUB_TOKEN'')); auth_file_visible = Test-Path -LiteralPath (Join-Path ([Environment]::GetEnvironmentVariable(''HOME'')) ''.codex/auth.json''); global_secret_visible = -not [string]::IsNullOrWhiteSpace([Environment]::GetEnvironmentVariable(''AGENTIC_GLOBAL_SECRET'')); project_disable_visible = -not [string]::IsNullOrWhiteSpace([Environment]::GetEnvironmentVariable(''OPENCODE_DISABLE_PROJECT_CONFIG'')) }; $result | ConvertTo-Json -Compress' +$stdinMemory = [IO.MemoryStream]::new() +[Console]::OpenStandardInput().CopyTo($stdinMemory) +$stdinBytes = $stdinMemory.ToArray() +$expectedPromptPath = Join-Path (Split-Path -Parent (Get-Location).Path) 'prompt.md' +$expectedPromptBytes = if (Test-Path -LiteralPath $expectedPromptPath -PathType Leaf) { [IO.File]::ReadAllBytes($expectedPromptPath) } else { [byte[]]@() } +$stdinExact = $stdinBytes.Length -eq $expectedPromptBytes.Length +if ($stdinExact) { + for ($index = 0; $index -lt $stdinBytes.Length; $index++) { + if ($stdinBytes[$index] -ne $expectedPromptBytes[$index]) { + $stdinExact = $false + break + } + } +} +$stdinHash = [Convert]::ToHexString(([Security.Cryptography.SHA256]::HashData($stdinBytes))).ToLowerInvariant() +$record.stdin_received = $stdinBytes.Length -gt 0 +$record.stdin_delivery_count = if ($stdinBytes.Length -gt 0) { 1 } else { 0 } +$record.stdin_byte_length = $stdinBytes.Length +$record.stdin_sha256 = $stdinHash +$record.stdin_exact = $stdinExact +$record.stdin_expected_sha256 = [Convert]::ToHexString(([Security.Cryptography.SHA256]::HashData($expectedPromptBytes))).ToLowerInvariant() +$record.stdin_utf8_round_trip = ([Text.UTF8Encoding]::new($false, $true).GetString($stdinBytes) -eq [Text.UTF8Encoding]::new($false, $true).GetString($expectedPromptBytes)) +$probeCommand = '$result = [ordered]@{ provider_visible = -not [string]::IsNullOrWhiteSpace([Environment]::GetEnvironmentVariable(''OPENAI_API_KEY'')); copilot_token_visible = -not [string]::IsNullOrWhiteSpace([Environment]::GetEnvironmentVariable(''COPILOT_GITHUB_TOKEN'')); gh_token_visible = -not [string]::IsNullOrWhiteSpace([Environment]::GetEnvironmentVariable(''GH_TOKEN'')); github_token_visible = -not [string]::IsNullOrWhiteSpace([Environment]::GetEnvironmentVariable(''GITHUB_TOKEN'')); auth_file_visible = Test-Path -LiteralPath (Join-Path ([Environment]::GetEnvironmentVariable(''HOME'')) ''.codex/auth.json''); global_secret_visible = -not [string]::IsNullOrWhiteSpace([Environment]::GetEnvironmentVariable(''AGENTIC_GLOBAL_SECRET'')); project_disable_visible = -not [string]::IsNullOrWhiteSpace([Environment]::GetEnvironmentVariable(''OPENCODE_DISABLE_PROJECT_CONFIG'')) }; $result | ConvertTo-Json -Compress' $probeInfo = [Diagnostics.ProcessStartInfo]::new() $probeInfo.FileName = (Get-Command pwsh).Source $probeInfo.UseShellExecute = $false @@ -118,7 +167,6 @@ $probeInfo.Environment['PATH'] = [Environment]::GetEnvironmentVariable('PATH') $probeInfo.Environment['HOME'] = [Environment]::GetEnvironmentVariable('HOME') $probeInfo.Environment['USERPROFILE'] = [Environment]::GetEnvironmentVariable('USERPROFILE') if ($harness -ne 'codex') { $probeInfo.Environment['OPENAI_API_KEY'] = [Environment]::GetEnvironmentVariable('OPENAI_API_KEY') } -if ($harness -eq 'copilot') { $probeInfo.Environment['COPILOT_GITHUB_TOKEN'] = [Environment]::GetEnvironmentVariable('COPILOT_GITHUB_TOKEN') } $probe = [Diagnostics.Process]::new() $probe.StartInfo = $probeInfo try { @@ -130,6 +178,8 @@ try { $probeResult = $probeOutput | ConvertFrom-Json $record.worker_provider_visible = [bool]$probeResult.provider_visible $record.worker_copilot_token_visible = [bool]$probeResult.copilot_token_visible + $record.worker_gh_token_visible = [bool]$probeResult.gh_token_visible + $record.worker_github_token_visible = [bool]$probeResult.github_token_visible $record.worker_auth_file_visible = [bool]$probeResult.auth_file_visible $record.worker_global_secret_visible = [bool]$probeResult.global_secret_visible $record.worker_project_disable_visible = [bool]$probeResult.project_disable_visible @@ -137,6 +187,10 @@ try { $probe.Dispose() } [IO.File]::AppendAllText($logPath, (($record | ConvertTo-Json -Compress) + [Environment]::NewLine), [Text.UTF8Encoding]::new($false)) +if ($harness -eq 'copilot' -and $copilotAuthenticationSource -eq 'unavailable') { + [Console]::Error.WriteLine('deterministic fixture: no Copilot authentication mechanism is available') + exit 17 +} if ($harness -eq 'codex') { $outputIndex = [Array]::IndexOf([string[]]$arguments, '--output-last-message') if ($outputIndex -ge 0 -and $outputIndex + 1 -lt $arguments.Count) { @@ -177,8 +231,17 @@ if ($harness -eq 'codex') { $env:AGENTIC_GLOBAL_SECRET = 'recorded-unrelated-canary-not-logged' $env:OPENCODE_DISABLE_PROJECT_CONFIG = '1' $env:COPILOT_GITHUB_TOKEN = 'recorded-copilot-canary-not-logged' - $env:GH_TOKEN = $null - $env:GITHUB_TOKEN = $null + $env:GH_TOKEN = 'recorded-gh-canary-not-logged' + $env:GITHUB_TOKEN = 'recorded-github-canary-not-logged' + $recordedGhConfig = Join-Path $recordedRoot 'github-cli-auth' + New-Item -ItemType Directory -Path $recordedGhConfig -Force | Out-Null + [System.IO.File]::WriteAllText((Join-Path $recordedGhConfig 'auth-marker.txt'), 'fixture auth state without a credential value', [System.Text.UTF8Encoding]::new($false)) + $env:GH_CONFIG_DIR = $recordedGhConfig + $ambientCopilotHome = Join-Path $recordedRoot 'ambient-copilot-home' + New-Item -ItemType Directory -Path $ambientCopilotHome -Force | Out-Null + [System.IO.File]::WriteAllText((Join-Path $ambientCopilotHome 'copilot-instructions.md'), '# ambient-personal-instruction-not-logged', [System.Text.UTF8Encoding]::new($false)) + [System.IO.File]::WriteAllText((Join-Path $ambientCopilotHome 'config.json'), '{"loggedInUsers":[{"login":"ambient-profile-not-logged"}]}', [System.Text.UTF8Encoding]::new($false)) + $env:COPILOT_HOME = $ambientCopilotHome $recordedProfiles = [ordered]@{} foreach ($runnerName in @('codex', 'opencode', 'cline', 'copilot')) { $profilePath = Join-Path $recordedRoot "$runnerName-profile.json" @@ -215,7 +278,12 @@ if ($harness -eq 'codex') { Assert-Equal 'compatible' $preflightWithout.status "$runnerName without_skill pragmatic preflight" Assert-Equal $expectedVersion $preflightWith.harness.version "$runnerName exact preflight version" Assert-Equal 'pragmatic' $preflightWith.isolation.level "$runnerName pragmatic preflight level" - if ($runnerName -ne 'codex') { + if ($runnerName -eq 'copilot') { + Assert-True (@($preflightWith.checks | Where-Object { $_.name -eq 'authentication' -and $_.status -eq 'passed' }).Count -eq 1) 'Copilot preflight accepts explicit environment authentication' + Assert-True (@($preflightWith.mechanisms | Where-Object { $_ -eq '--allow-all-tools broad tool approval' }).Count -eq 1) 'Copilot preflight describes --allow-all-tools as broad tool approval' + Assert-True (@($preflightWith.mechanisms | Where-Object { $_ -eq 'path and URL verification preserved (no --allow-all-paths/--allow-all-urls)' }).Count -eq 1) 'Copilot preflight records preserved path and URL verification' + } + if ($runnerName -in @('opencode', 'cline')) { Assert-True (@($preflightWith.warnings | Where-Object { $_ -match 'child-tool environment filter' }).Count -gt 0) "$runnerName reports the child credential-filter limitation" } $resultWith = Invoke-AdapterJson -RunnerPath $runnerPath -Command execute -RunPath $with.Path -ProfilePath $recordedProfiles[$runnerName] @@ -242,6 +310,10 @@ if ($harness -eq 'codex') { $executionRecords = @($records | Where-Object { $_.stdin_received -eq $true -or $_.prompt_via_arg -eq $true }) Assert-Equal 1 $executionRecords.Count "$runnerName one execution process per checked arm" $execution = $executionRecords[0] + Assert-True $execution.stdin_received "$runnerName receives a non-empty stdin prompt" + Assert-Equal 1 $execution.stdin_delivery_count "$runnerName delivers one prompt through stdin" + Assert-True $execution.stdin_exact "$runnerName fake CLI received the exact staged prompt bytes" + Assert-True $execution.stdin_utf8_round_trip "$runnerName preserves arbitrary UTF-8 prompt content" Assert-True (-not $execution.unrelated_present) "$runnerName does not pass unrelated credential canary" Assert-True (-not $execution.disable_project_config_present) "$runnerName does not pass ambient project-disable override" Assert-True (-not $execution.worker_auth_file_visible) "$runnerName worker probe cannot read a copied Codex auth file" @@ -250,7 +322,9 @@ if ($harness -eq 'codex') { if ($runnerName -eq 'codex') { Assert-True (-not $execution.worker_provider_visible) 'Codex shell policy hides the provider API-key variable from the worker probe' } elseif ($runnerName -eq 'copilot') { - Assert-True $execution.worker_copilot_token_visible 'Copilot GitHub-token child-visibility limitation is recorded by the worker probe' + Assert-True (-not $execution.worker_copilot_token_visible) 'Copilot secret COPILOT_GITHUB_TOKEN is unavailable to the worker probe' + Assert-True (-not $execution.worker_gh_token_visible) 'Copilot secret GH_TOKEN is unavailable to the worker probe' + Assert-True (-not $execution.worker_github_token_visible) 'Copilot secret GITHUB_TOKEN is unavailable to the worker probe' } else { Assert-True $execution.worker_provider_visible "$runnerName credential visibility limitation is recorded by the worker probe" } @@ -268,20 +342,31 @@ if ($harness -eq 'codex') { Assert-True ($args -contains '--auto') 'OpenCode is noninteractive' Assert-True $execution.project_config_visible 'OpenCode paired arm retains repository-owned project configuration' } elseif ($runnerName -eq 'copilot') { - Assert-True ($args -contains '--prompt') 'Copilot delivers the prompt via the --prompt argument' - Assert-Equal 1 $execution.prompt_arg_count 'Copilot delivers the prompt exactly once' - Assert-True (-not $execution.stdin_received) 'Copilot does not read the prompt from stdin' + Assert-True (@($args | Where-Object { $_ -eq '--prompt' -or $_ -eq '-p' -or $_ -like '--prompt=*' }).Count -eq 0) 'Copilot does not place the prompt in argv' + Assert-Equal 0 $execution.prompt_arg_count 'Copilot has no prompt argument' + Assert-True $execution.stdin_received 'Copilot reads the prompt from stdin' Assert-True ($args -contains '--output-format' -and $args -contains 'json') 'Copilot uses structured JSONL output' $modelIndex = [Array]::IndexOf([string[]]$args, '--model') Assert-Equal 'claude-haiku-4.5' $args[$modelIndex + 1] 'Copilot reference model claude-haiku-4.5 propagates to the CLI invocation' - Assert-True ($args -contains '--allow-all-tools') 'Copilot removes tool-approval prompts without a blanket permission grant' + Assert-True ($args -contains '--allow-all-tools') 'Copilot grants broad tool approval for noninteractive execution' Assert-True ($args -contains '--no-ask-user') 'Copilot does not pause for interactive questions' - Assert-True ($args -contains '--no-custom-instructions') 'Copilot disables ambient custom instructions' + Assert-True ($args -notcontains '--no-custom-instructions') 'Copilot preserves repository-owned custom instructions' Assert-True ($args -contains '--disable-builtin-mcps') 'Copilot disables ambient built-in MCP servers' foreach ($broad in @('--yolo', '--allow-all', '--allow-all-paths', '--allow-all-urls', '--session-id', '--connect', '-r')) { Assert-True ($args -notcontains $broad) "Copilot avoids the over-broad or session option '$broad'" } - Assert-Equal 1 (@($args | Where-Object { $_ -like '--secret-env-vars=*' }).Count) 'Copilot redacts the GitHub token from output with --secret-env-vars' - Assert-True ($execution.custom_instructions_disabled -and $execution.builtin_mcps_disabled) 'Copilot ambient instruction/MCP disabling is observed by the fake CLI' + Assert-Equal 1 (@($args | Where-Object { $_ -like '--secret-env-vars=*' }).Count) 'Copilot filters protected variables with --secret-env-vars' + Assert-Equal 'COPILOT_GITHUB_TOKEN,GH_TOKEN,GITHUB_TOKEN' ([string]($args | Where-Object { $_ -like '--secret-env-vars=*' }) -replace '^--secret-env-vars=', '') 'Copilot protects every forwarded token variable' + Assert-True (-not $execution.custom_instructions_disabled -and $execution.builtin_mcps_disabled) 'Copilot preserves repository instructions while disabling built-in MCPs' + Assert-True ($execution.repository_agents_visible -and $execution.repository_copilot_instructions_visible -and $execution.repository_instruction_marker_visible) 'Copilot sees staged repository-owned instructions' + Assert-True $execution.candidate_skill_staged 'Copilot with_skill arm retains the staged candidate skill independently of repository instructions' + Assert-True (-not $execution.ambient_copilot_instructions_visible) 'Copilot does not see the ambient personal instruction file' + Assert-Equal 'explicit_environment' $execution.copilot_authentication_source 'Copilot uses explicit environment authentication in the token fixture' + Assert-Equal 3 @($execution.copilot_auth_names_present).Count 'Copilot process receives all protected token variables without logging values' + Assert-True (Test-PathInside -BasePath (Join-Path $with.Root 'home') -CandidatePath ([string]$execution.copilot_cache_home)) 'Copilot cache is run-local' Assert-True (Test-PathInside -BasePath (Join-Path $with.Root 'home') -CandidatePath ([string]$execution.copilot_home)) 'Copilot COPILOT_HOME is the run''s isolated home' + Assert-Equal 'stdin' $resultWith.evidence.prompt_delivery 'Copilot result records stdin prompt delivery' + Assert-Equal 'COPILOT_GITHUB_TOKEN' $resultWith.evidence.credential.github_token_variable 'Copilot follows explicit token precedence' + Assert-Equal 'supported' $resultWith.isolation.capabilities.credential_child_filtering 'Copilot documents protected child-environment filtering' + Assert-Equal 'shell,mcp' ([string]::Join(',', @($resultWith.evidence.credential.secret_env_var_scope))) 'Copilot evidence names the documented filtering scope' Assert-Equal 'recorded Copilot final response' $resultWith.final_response.text 'Copilot final response is the last assistant message, not an intermediate one' Assert-Equal 'claude-haiku-4.5' $resultWith.requested.model 'Copilot requested model is preserved as the Codebelt reference model' Assert-True ($null -eq $resultWith.resolved.model) 'Copilot does not claim a distinct backend model resolution' @@ -305,13 +390,23 @@ if ($harness -eq 'codex') { Assert-True ([int]$resultWith.telemetry.tool_calls.value -ge 1) 'Cline parses documented tool events' } $logText = [System.IO.File]::ReadAllText($logPath, [System.Text.UTF8Encoding]::new($false)) - Assert-True ($logText -notmatch 'recorded-canary|recorded-unrelated-canary') "$runnerName logs do not contain credential values" + Assert-True ($logText -notmatch 'recorded-canary|recorded-unrelated-canary|recorded-copilot-canary|recorded-gh-canary|recorded-github-canary') "$runnerName logs do not contain credential values" + Assert-True (($resultWith | ConvertTo-Json -Depth 100) -notmatch 'recorded-canary|recorded-unrelated-canary|recorded-copilot-canary|recorded-gh-canary|recorded-github-canary') "$runnerName result evidence does not contain credential values" $withoutLogPath = Join-Path $without.Root "repo\$runnerName-fake-cli-log.jsonl" Assert-True (Test-Path -LiteralPath $withoutLogPath -PathType Leaf) "$runnerName baseline process log exists" $withoutRecords = @(Get-Content -LiteralPath $withoutLogPath | ForEach-Object { $_ | ConvertFrom-Json }) - Assert-Equal 1 @($withoutRecords | Where-Object { $_.stdin_received -eq $true -or $_.prompt_via_arg -eq $true }).Count "$runnerName baseline has one execution process" + $withoutExecution = @($withoutRecords | Where-Object { $_.stdin_received -eq $true -or $_.prompt_via_arg -eq $true }) + Assert-Equal 1 $withoutExecution.Count "$runnerName baseline has one execution process" $withoutLogText = [System.IO.File]::ReadAllText($withoutLogPath, [System.Text.UTF8Encoding]::new($false)) - Assert-True ($withoutLogText -notmatch 'recorded-canary|recorded-unrelated-canary') "$runnerName baseline log does not contain credential values" + Assert-True ($withoutLogText -notmatch 'recorded-canary|recorded-unrelated-canary|recorded-copilot-canary|recorded-gh-canary|recorded-github-canary') "$runnerName baseline log does not contain credential values" + Assert-True $withoutExecution[0].stdin_exact "$runnerName baseline receives exact prompt bytes" + if ($runnerName -eq 'copilot') { + Assert-True $withoutExecution[0].repository_agents_visible 'Copilot baseline sees the same staged AGENTS.md instruction' + Assert-True $withoutExecution[0].repository_copilot_instructions_visible 'Copilot baseline sees the same staged repository instruction' + Assert-True (-not $withoutExecution[0].candidate_skill_staged) 'Copilot baseline does not receive the candidate skill directory' + Assert-True (-not $withoutExecution[0].ambient_copilot_instructions_visible) 'Copilot baseline excludes the ambient personal instruction' + Assert-True (($resultWithout | ConvertTo-Json -Depth 100) -notmatch 'recorded-canary|recorded-unrelated-canary|recorded-copilot-canary|recorded-gh-canary|recorded-github-canary') 'Copilot baseline result evidence does not contain credential values' + } } $staleCli = $fakeCli.Replace("'opencode' { '--format --dir --model --auto --pure --continue --session' }", "'opencode' { '--format --dir --model --pure --continue --session' }") [System.IO.File]::WriteAllText((Join-Path $fakeBin 'opencode.ps1'), $staleCli, [System.Text.UTF8Encoding]::new($false)) @@ -329,25 +424,57 @@ if ($harness -eq 'codex') { Assert-True (@($fileAuthPreflight.reasons | Where-Object { $_ -match 'auth\.json' }).Count -gt 0) 'Codex file-auth limitation is explicit' $env:OPENAI_API_KEY = 'recorded-canary-not-logged' $env:CODEX_HOME = $recordedOldCodexHome - # GitHub Copilot authentication: with no separable token the runner is fail-closed and never imports the ambient profile. - $copilotEmptyHome = Join-Path $recordedRoot 'copilot-empty-home' - New-Item -ItemType Directory -Path $copilotEmptyHome -Force | Out-Null + # GitHub Copilot authentication: explicit env, OS-keychain, GitHub CLI, and + # no-auth fixtures are all deterministic and contain no credential values. $env:COPILOT_GITHUB_TOKEN = $null $env:GH_TOKEN = $null $env:GITHUB_TOKEN = $null - $env:COPILOT_HOME = $copilotEmptyHome - $copilotNoAuth = Invoke-AdapterJson -RunnerPath (Join-Path $runnerRoot 'github-copilot\runner.ps1') -Command preflight -RunPath $with.Path -ProfilePath $recordedProfiles['copilot'] - Assert-Equal 'incompatible' $copilotNoAuth.status 'Copilot without a GitHub token is fail-closed' - Assert-True (@($copilotNoAuth.reasons | Where-Object { $_ -match 'COPILOT_GITHUB_TOKEN' }).Count -gt 0) 'Copilot missing-auth reason names the required token variable' - $copilotLoginHome = Join-Path $recordedRoot 'copilot-login-home' - New-Item -ItemType Directory -Path $copilotLoginHome -Force | Out-Null - [System.IO.File]::WriteAllText((Join-Path $copilotLoginHome 'config.json'), '{"loggedInUsers":[{"login":"canary-not-logged"}]}', [System.Text.UTF8Encoding]::new($false)) - $env:COPILOT_HOME = $copilotLoginHome - $copilotLoginOnly = Invoke-AdapterJson -RunnerPath (Join-Path $runnerRoot 'github-copilot\runner.ps1') -Command preflight -RunPath $with.Path -ProfilePath $recordedProfiles['copilot'] - Assert-Equal 'incompatible' $copilotLoginOnly.status 'Copilot login-only profile is not imported as a credential' - Assert-True (@($copilotLoginOnly.reasons | Where-Object { $_ -match 'separable GitHub token' }).Count -gt 0) 'Copilot login-only reason explains the co-mingling limitation' - Assert-True (($copilotLoginOnly | ConvertTo-Json -Depth 100) -notmatch 'canary-not-logged') 'Copilot preflight never surfaces login-profile contents' - $env:COPILOT_GITHUB_TOKEN = 'recorded-copilot-canary-not-logged' + $missingGhConfig = Join-Path $recordedRoot 'missing-github-cli-auth' + + # The fixture marker is fake-CLI input only; it models a positive OS + # keychain lookup without naming or reading a real credential-store file. + $copilotKeychainHome = Join-Path $recordedRoot 'copilot-keychain-home' + New-Item -ItemType Directory -Path $copilotKeychainHome -Force | Out-Null + New-Item -ItemType Directory -Path (Join-Path $with.Root 'home\.copilot') -Force | Out-Null + [System.IO.File]::WriteAllText((Join-Path $with.Root 'home\.copilot\fixture-os-keychain-available'), 'fixture marker only', [Text.UTF8Encoding]::new($false)) + $env:COPILOT_HOME = $copilotKeychainHome + $env:GH_CONFIG_DIR = $missingGhConfig + $copilotKeychainPreflight = Invoke-AdapterJson -RunnerPath (Join-Path $runnerRoot 'github-copilot\runner.ps1') -Command preflight -RunPath $with.Path -ProfilePath $recordedProfiles['copilot'] + Assert-Equal 'compatible' $copilotKeychainPreflight.status 'Copilot tokenless OS-keychain authentication remains compatible' + Assert-True (@($copilotKeychainPreflight.checks | Where-Object { $_.name -eq 'authentication' -and $_.status -eq 'unavailable' }).Count -eq 1) 'Copilot preflight leaves native keychain readiness conditional' + Assert-True (@($copilotKeychainPreflight.warnings | Where-Object { $_ -match 'cannot be proven' }).Count -gt 0) 'Copilot preflight explains the unverified keychain/service boundary' + $copilotKeychainResult = Invoke-AdapterJson -RunnerPath (Join-Path $runnerRoot 'github-copilot\runner.ps1') -Command execute -RunPath $with.Path -ProfilePath $recordedProfiles['copilot'] + Assert-Equal 'completed' $copilotKeychainResult.status 'Copilot keychain fixture executes without an exported token' + $keychainRecords = @(Get-Content -LiteralPath (Join-Path $with.Root 'repo\copilot-fake-cli-log.jsonl') | ForEach-Object { $_ | ConvertFrom-Json } | Where-Object { $_.copilot_authentication_source -eq 'os_keychain' }) + Assert-Equal 1 $keychainRecords.Count 'Copilot fake observes the simulated OS-keychain path' + Remove-Item -LiteralPath (Join-Path $with.Root 'home\.copilot\fixture-os-keychain-available') -Force + + $copilotGhFallbackHome = Join-Path $recordedRoot 'copilot-gh-fallback-home' + New-Item -ItemType Directory -Path $copilotGhFallbackHome -Force | Out-Null + $copilotGhConfig = Join-Path $recordedRoot 'copilot-gh-config' + New-Item -ItemType Directory -Path $copilotGhConfig -Force | Out-Null + [System.IO.File]::WriteAllText((Join-Path $copilotGhConfig 'auth-marker.txt'), 'fixture auth state without a credential value', [Text.UTF8Encoding]::new($false)) + $env:COPILOT_HOME = $copilotGhFallbackHome + $env:GH_CONFIG_DIR = $copilotGhConfig + $copilotGhPreflight = Invoke-AdapterJson -RunnerPath (Join-Path $runnerRoot 'github-copilot\runner.ps1') -Command preflight -RunPath $with.Path -ProfilePath $recordedProfiles['copilot'] + Assert-Equal 'compatible' $copilotGhPreflight.status 'Copilot GitHub CLI fallback remains compatible' + $copilotGhResult = Invoke-AdapterJson -RunnerPath (Join-Path $runnerRoot 'github-copilot\runner.ps1') -Command execute -RunPath $with.Path -ProfilePath $recordedProfiles['copilot'] + Assert-Equal 'completed' $copilotGhResult.status 'Copilot GitHub CLI fallback fixture executes without an exported token' + Assert-True $copilotGhResult.evidence.credential.github_cli_config_forwarded 'Copilot records GitHub CLI auth-state forwarding without a credential value' + $ghRecords = @(Get-Content -LiteralPath (Join-Path $with.Root 'repo\copilot-fake-cli-log.jsonl') | ForEach-Object { $_ | ConvertFrom-Json } | Where-Object { $_.copilot_authentication_source -eq 'github_cli' }) + Assert-Equal 1 $ghRecords.Count 'Copilot fake observes the simulated GitHub CLI fallback path' + + $copilotNoAuthHome = Join-Path $recordedRoot 'copilot-no-auth-home' + New-Item -ItemType Directory -Path $copilotNoAuthHome -Force | Out-Null + $env:COPILOT_HOME = $copilotNoAuthHome + $env:GH_CONFIG_DIR = $missingGhConfig + $copilotNoAuthPreflight = Invoke-AdapterJson -RunnerPath (Join-Path $runnerRoot 'github-copilot\runner.ps1') -Command preflight -RunPath $with.Path -ProfilePath $recordedProfiles['copilot'] + Assert-Equal 'compatible' $copilotNoAuthPreflight.status 'Copilot preflight does not require an exported token when native auth is not observable' + Assert-True (@($copilotNoAuthPreflight.warnings | Where-Object { $_ -match 'conditional' }).Count -gt 0) 'Copilot no-auth preflight is explicitly conditional' + $copilotNoAuthResult = Invoke-AdapterJson -RunnerPath (Join-Path $runnerRoot 'github-copilot\runner.ps1') -Command execute -RunPath $with.Path -ProfilePath $recordedProfiles['copilot'] + Assert-Equal 'failed' $copilotNoAuthResult.status 'Copilot no-auth execution failure is captured without a model request' + Assert-Equal 'copilot_os_keychain_or_github_cli_unverified' $copilotNoAuthResult.evidence.credential.source 'Copilot no-auth evidence does not claim authentication' + Assert-True (($copilotNoAuthResult | ConvertTo-Json -Depth 100) -notmatch 'ambient-profile-not-logged|recorded-copilot-canary|recorded-gh-canary|recorded-github-canary') 'Copilot authentication fixtures never expose credential values' $env:COPILOT_HOME = $recordedOldCopilotHome Write-Output 'Real runner deterministic adapter conformance: PASS' } finally { @@ -360,6 +487,7 @@ if ($harness -eq 'codex') { $env:GH_TOKEN = $recordedOldGhToken $env:GITHUB_TOKEN = $recordedOldGithubToken $env:COPILOT_HOME = $recordedOldCopilotHome + $env:GH_CONFIG_DIR = $recordedOldGhConfigDir if (Test-Path -LiteralPath $recordedRoot) { Remove-Item -LiteralPath $recordedRoot -Recurse -Force } } } @@ -412,7 +540,10 @@ function New-TestRun { $homeDirectory = Join-Path $runRoot 'home' New-Item -ItemType Directory -Path $repo,$homeDirectory -Force | Out-Null [System.IO.File]::WriteAllText((Join-Path $homeDirectory 'README.txt'), 'isolated home', [System.Text.UTF8Encoding]::new($false)) - $prompt = "# task`r`n`r`nByte fidelity: Δ and emoji 🚀.`r`n" + New-Item -ItemType Directory -Path (Join-Path $repo '.github') -Force | Out-Null + [System.IO.File]::WriteAllText((Join-Path $repo 'AGENTS.md'), '# repo-owned-agent-instruction', [System.Text.UTF8Encoding]::new($false)) + [System.IO.File]::WriteAllText((Join-Path $repo '.github\copilot-instructions.md'), '# repo-owned-copilot-instruction', [System.Text.UTF8Encoding]::new($false)) + $prompt = "# task`r`n`r`nByte fidelity: Δ and emoji 🚀.`r`n" + ("large-prompt-line-0123456789`r`n" * 4096) [System.IO.File]::WriteAllBytes((Join-Path $runRoot 'prompt.md'), [System.Text.UTF8Encoding]::new($false).GetBytes($prompt)) if ($Configuration -eq 'with_skill') { $skill = Join-Path $runRoot 'skill\candidate' From 6d5c554d2db07fbd47f8d0ebe700d629ec7b3f62 Mon Sep 17 00:00:00 2001 From: "aicia[bot]" Date: Sat, 22 Aug 2026 11:43:15 +0200 Subject: [PATCH 09/39] =?UTF-8?q?=F0=9F=90=9B=20ensure=20evidence=20direct?= =?UTF-8?q?ory=20is=20created=20in=20codex=20runner?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The codex runner writes to the evidence directory but was not ensuring it existed first. Adding defensive directory creation prevents file-not-found errors when the evidence directory hasn't been created yet. --- scripts/eval-runners/codex/runner.ps1 | 1 + 1 file changed, 1 insertion(+) diff --git a/scripts/eval-runners/codex/runner.ps1 b/scripts/eval-runners/codex/runner.ps1 index e83bb4a..d6420ed 100644 --- a/scripts/eval-runners/codex/runner.ps1 +++ b/scripts/eval-runners/codex/runner.ps1 @@ -427,6 +427,7 @@ function Invoke-CodexExecute { $auth = Get-CodexAuthSource -Provider ([string]$Inputs.Profile.Provider) $environment = New-CodexEnvironment -Inputs $Inputs -Auth $auth $lastResponsePath = 'evidence/codex-final.txt' + New-Item -ItemType Directory -Path (Join-Path $Inputs.Run.RunRoot 'evidence') -Force | Out-Null $platform = Get-PlatformName $sandboxInfo = if ($platform -eq 'linux') { Resolve-SandboxCommand -Name 'bwrap' } elseif ($platform -eq 'macos') { Resolve-SandboxCommand -Name 'sandbox-exec' } else { $null } $hardFilesystem = $null -ne $sandboxInfo -and $platform -in @('linux', 'macos') From 5b9bbe5ab9d568fde1499c2523834d01c4957f99 Mon Sep 17 00:00:00 2001 From: "aicia[bot]" Date: Sat, 22 Aug 2026 11:43:26 +0200 Subject: [PATCH 10/39] =?UTF-8?q?=F0=9F=A6=BA=20validate=20output=20direct?= =?UTF-8?q?ory=20exists=20in=20test=20runner=20conformance?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Change from defensive directory creation to explicit validation. The conformance test now requires the output parent directory to already exist and exits with a clear error message if it doesn't, improving error clarity when the directory structure is misconfigured. --- scripts/eval-runners/tests/test-runner-conformance.ps1 | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/scripts/eval-runners/tests/test-runner-conformance.ps1 b/scripts/eval-runners/tests/test-runner-conformance.ps1 index 509ccab..0feceed 100644 --- a/scripts/eval-runners/tests/test-runner-conformance.ps1 +++ b/scripts/eval-runners/tests/test-runner-conformance.ps1 @@ -195,7 +195,11 @@ if ($harness -eq 'codex') { $outputIndex = [Array]::IndexOf([string[]]$arguments, '--output-last-message') if ($outputIndex -ge 0 -and $outputIndex + 1 -lt $arguments.Count) { $outputPath = $arguments[$outputIndex + 1] - New-Item -ItemType Directory -Path (Split-Path -Parent $outputPath) -Force | Out-Null + $outputParent = Split-Path -Parent $outputPath + if (-not (Test-Path -LiteralPath $outputParent -PathType Container)) { + [Console]::Error.WriteLine("recorded Codex requires the output parent directory to exist: $outputParent") + exit 19 + } [IO.File]::WriteAllText($outputPath, 'recorded Codex final response', [Text.UTF8Encoding]::new($false)) } Write-Output '{"type":"thread.started","thread_id":"recorded-thread"}' From db9cda5e104d805b54030366776876338627f69e Mon Sep 17 00:00:00 2001 From: "aicia[bot]" Date: Sat, 22 Aug 2026 12:21:56 +0200 Subject: [PATCH 11/39] =?UTF-8?q?=F0=9F=93=9D=20refactor=20skill=20authori?= =?UTF-8?q?ng=20guidance=20and=20optimize=20descriptions?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Restructured AGENTS.md Skill Authoring section for clarity, brevity, and progressive disclosure. Updated README.md skill catalog to note that descriptions are lean activation metadata. Optimized all 21 repo-managed skill descriptions to be concise, trigger-oriented metadata following the new authoring standard. Changes align description quality with specification requirements and improve skill discoverability. --- AGENTS.md | 152 +++++++++------------ README.md | 2 + skills/agent-smith/SKILL.md | 68 ++++----- skills/dotnet-benchmark/SKILL.md | 2 +- skills/dotnet-change-impact/SKILL.md | 4 +- skills/dotnet-docfx-digest/SKILL.md | 2 +- skills/dotnet-new-app-slnx/SKILL.md | 4 +- skills/dotnet-new-lib-slnx/SKILL.md | 2 +- skills/dotnet-remote-testing/SKILL.md | 4 +- skills/dotnet-segregated-assets/SKILL.md | 2 +- skills/dotnet-strong-name-signing/SKILL.md | 4 +- skills/dotnet-test/SKILL.md | 2 +- skills/git-keep-a-changelog/SKILL.md | 2 +- skills/git-nuget-readme/SKILL.md | 2 +- skills/git-nuget-release-notes/SKILL.md | 2 +- skills/git-remote-release/SKILL.md | 2 +- skills/git-repo-digest/SKILL.md | 2 +- skills/git-visual-commits/SKILL.md | 2 +- skills/git-visual-squash-summary/SKILL.md | 2 +- skills/markdown-illustrator/SKILL.md | 2 +- skills/skill-creator-agnostic/SKILL.md | 3 +- skills/trunk-first-repo/SKILL.md | 4 +- 22 files changed, 127 insertions(+), 144 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index ee0f76e..a41c85b 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -353,126 +353,106 @@ Interim progress updates should describe user-relevant progress, evidence, block - Mention tool/runtime failures only when they block progress, require approval, or change the planned validation - Prefer concise phrasing such as "The first read attempt failed before returning file content; I'm retrying and will report only if that changes the result" -## Anthropic Skill Authoring Reference +## Skill Authoring -Essential conventions from [The Complete Guide to Building Skills for Claude](https://resources.anthropic.com/hubfs/The-Complete-Guide-to-Building-Skill-for-Claude.pdf) (Anthropic, Jan 2026). All skills in this repo must follow these rules. +Skills MUST follow the Agent Skills specification and remain compatible with Anthropic's skill guidance. Repository conventions below may intentionally be stricter than the specification. -### File Structure +### Structure -``` +```text skill-name/ -├── SKILL.md # Required — exact spelling, case-sensitive -├── scripts/ # Optional — executable code (Python, Bash, etc.) -├── references/ # Optional — documentation loaded as needed -└── assets/ # Optional — templates, fonts, icons used in output +├── SKILL.md # Required +├── scripts/ # Optional executable automation +├── references/ # Optional supporting documentation +└── assets/ # Optional templates and output resources ``` -- **No `README.md`** inside the skill folder — all documentation goes in `SKILL.md` or `references/` -- Folder name must be **kebab-case** (no spaces, no underscores, no capitals) -- Folder name must match the `name:` field in YAML frontmatter +- `SKILL.md` MUST use that exact case-sensitive name. +- Skill folders MUST use kebab-case. +- Frontmatter `name` MUST match the folder name. +- Do NOT place `README.md` inside a skill folder. Put skill documentation in `SKILL.md` or `references/`. +- Keep `SKILL.md` focused. Move detailed or rarely needed material to `references/`. +- Keep `SKILL.md` below 5,000 words. -### Progressive Disclosure (Three Levels) +### Progressive Disclosure -| Level | When loaded | Token cost | Content | -|-------|------------|------------|---------| -| **Level 1: Metadata** | Always (at startup) | ~100 tokens | `name` and `description` from YAML frontmatter | -| **Level 2: Instructions** | When skill is triggered | Under 5k tokens | SKILL.md body — workflows, steps, guidance | -| **Level 3: Resources** | As needed | Effectively unlimited | Linked files: scripts, references, assets, FORMS.md | +Design every skill around three levels of context: -Keep SKILL.md under **500 lines / 5,000 words**. Move detailed content to `references/`. Keep references **one level deep** from SKILL.md — nested references cause partial reads. +1. **Metadata:** `name` and `description` are available before activation. +2. **Instructions:** the `SKILL.md` body is loaded when the skill is selected. +3. **Resources:** scripts, references, and assets are loaded or used only when needed. -### YAML Frontmatter +Minimize content at earlier levels. Do not put instructions into metadata merely to advertise skill capabilities. -Required fields: +### Frontmatter ```yaml --- -name: kebab-case-name # max 64 chars, lowercase + numbers + hyphens only -description: > # max 1024 chars, must include WHAT + WHEN + triggers - What it does. Use when user asks to [specific phrases]. +name: skill-name +description: Use when ... --- ``` -Optional fields: +Required: -```yaml -license: MIT # for open-source skills -compatibility: > # max 500 chars — environment requirements - Requires network access and Python 3.10+ -metadata: # custom key-value pairs - author: Company Name - version: 1.0.0 - mcp-server: server-name -``` +- `name`: 1-64 characters, lowercase alphanumeric characters and hyphens only; MUST match the skill directory. +- `description`: 1-1024 characters and MUST communicate both what the skill is for and when it should activate. -**Forbidden**: XML angle brackets (`< >`), names containing "claude" or "anthropic" (reserved). +Optional fields such as `license`, `compatibility`, `metadata`, and supported tool restrictions MAY be used when they provide meaningful runtime or distribution information. -### Description Field — The Most Important Part +### Description Is a Trigger -Structure: `[What it does] + [When to use it] + [Key capabilities]` +Treat `description` as **activation metadata, not documentation**. -```yaml -# ✅ Good — specific, actionable, includes triggers -description: > - Manages Linear project workflows including sprint planning, - task creation, and status tracking. Use when user mentions - "sprint", "Linear tasks", "project planning", or asks to - "create tickets". - -# ❌ Bad — too vague, no triggers -description: Helps with projects. -``` +- SHOULD begin with trigger-oriented language such as `Use when...`. +- SHOULD describe **user intent**, not the skill's implementation. +- SHOULD stay at or below a **300-character soft ceiling**. +- MAY be shorter than 150 characters when that is sufficient. Never pad a description to meet a minimum length. +- MAY exceed 300 characters only when additional wording materially improves trigger precision or recall. +- MUST remain within the 1024-character specification limit. +- SHOULD include distinctive tasks, artifacts, technologies, file types, or domain terms that help discriminate the skill from others. +- SHOULD cover natural paraphrases conceptually rather than stuffing exact trigger phrases or keywords. +- SHOULD add exclusions only when needed to prevent realistic near-miss or overlapping skills from triggering. +- MUST NOT summarize workflows, scripts, implementation details, references, rationale, or every capability of the skill. +- MUST NOT broaden the description merely to advertise functionality. -- Include trigger phrases users would actually say -- Mention file types if relevant -- Add negative triggers to prevent over-triggering: `Do NOT use for simple data exploration` +Prefer: -### Writing Instructions - -- Be **specific and actionable** — `Run scripts/validate.py --input {filename}` not `Validate the data` -- Include **error handling** — common errors, causes, and solutions -- Use **feedback loops** — run validator → fix errors → repeat -- Put **critical instructions at the top** — use `## Critical` or `## Important` headers -- For critical validations, **use scripts over language instructions** — code is deterministic -- Prefer **dynamic defaults over hardcoded values** when the source data is available from the repo, environment, or an official machine-readable feed +```yaml +description: Use when creating or refactoring .NET tests that require deterministic remote or containerized execution across supported test harnesses. +``` -### Skill Categories +Avoid: -| Category | Purpose | Example | -|----------|---------|---------| -| **Document & Asset Creation** | Consistent, high-quality output (docs, code, designs) | `frontend-design`, `docx`, `xlsx` | -| **Workflow Automation** | Multi-step processes with validation gates | `skill-creator`, scaffolding skills | -| **MCP Enhancement** | Workflow guidance layered on top of MCP tool access | `sentry-code-review` | +```yaml +description: Provides comprehensive guidance, scripts, configuration options, troubleshooting procedures, and best practices for running .NET tests remotely using containers and multiple supported test harnesses. +``` -### Common Patterns +Optimize for **trigger precision and recall per character**, not descriptive completeness. -1. **Sequential workflow** — explicit step ordering with dependencies and rollback -2. **Multi-MCP coordination** — phase separation, data passing between services -3. **Iterative refinement** — draft → validate → fix → repeat until quality threshold -4. **Context-aware selection** — decision trees for choosing the right tool/approach -5. **Domain-specific intelligence** — compliance checks, governance, audit trails +### Instructions -### Testing Checklist +Inside `SKILL.md`: -Before shipping a skill, verify: +- Make instructions specific, actionable, and ordered where sequencing matters. +- Put critical constraints near the top. +- Include error handling where failures are predictable. +- Use validation and refinement loops where output quality benefits from iteration. +- Prefer deterministic scripts for repeatable or critical validation rather than lengthy natural-language procedures. +- Prefer values discoverable from the repository, environment, or authoritative machine-readable sources over hardcoded defaults. +- Keep detailed reference material out of the main instruction path. -- [ ] Triggers on obvious tasks -- [ ] Triggers on paraphrased requests -- [ ] Does **not** trigger on unrelated topics -- [ ] Functional tests pass (correct outputs, error handling, edge cases) -- [ ] Performance improves over baseline (fewer messages, fewer errors, fewer tokens) +### Validation -Debug triggering: ask Claude `"When would you use the [skill name] skill?"` — it will quote the description back. +Before shipping or materially changing a skill, verify that it: -### Troubleshooting Quick Reference +- triggers for obvious relevant requests; +- triggers for realistic paraphrases and implicit intent; +- does not trigger for realistic near-miss requests; +- behaves correctly after activation; +- improves the intended outcome compared with not using the skill. -| Symptom | Likely cause | Fix | -|---------|-------------|-----| -| Skill won't upload | `SKILL.md` misspelled or YAML invalid | Exact case `SKILL.md`, check `---` delimiters | -| Skill never triggers | Description too vague | Add trigger phrases, mention file types | -| Skill triggers too often | Description too broad | Add negative triggers, narrow scope | -| Instructions not followed | Too verbose or ambiguous | Shorten, use bullets, move detail to `references/` | -| Slow / degraded responses | Too much content loaded | Keep SKILL.md under 5k words, use progressive disclosure | +When optimizing a description, test both **should-trigger** and **should-not-trigger** cases. Prefer measured trigger behavior over arbitrary description length, and avoid tailoring descriptions to individual evaluation phrases. ## Karpathy Rules diff --git a/README.md b/README.md index 51a29f7..461f81e 100644 --- a/README.md +++ b/README.md @@ -123,6 +123,8 @@ npx skills add https://github.com/codebeltnet/agentic --skill agent-smith ## Available Skills +Each `SKILL.md` description is lean activation metadata. The catalog below explains what happens after a skill is selected. + | Skill | Description | |-------|-------------| | [git-visual-commits](skills/git-visual-commits/SKILL.md) | AI-driven git commit workflow with authoritative routing for `git bot commit`, `git commit`, and `git our commit`, including the exact `Please do a git bot commit yolo` form. It locks the requested identity, treats yolo/auto only as scoped auto-approval modifiers, never as the commit message, and does not hand commit execution to changelog or release-note skills. It uses deterministically validated emoji-first subjects, optional conventional prefixes only on explicit request, full-worktree semantic grouping unless narrowed, a visible multi-file single-category quality gate, commit bodies by default, and post-commit identity/body verification. Multi-file plans that initially collapse to one category also require a visible full-context quality gate; one-file changes keep the fast path. Stack-agnostic. | diff --git a/skills/agent-smith/SKILL.md b/skills/agent-smith/SKILL.md index 8a22b7a..ed86fb2 100644 --- a/skills/agent-smith/SKILL.md +++ b/skills/agent-smith/SKILL.md @@ -1,7 +1,7 @@ --- name: agent-smith -description: > - Use this skill to apply a rigorous, evidence-driven software-craftsmanship standard across an engineering task. Invoke explicitly as `/agent-smith task`, or automatically for architecture, implementation, refactoring, review, API compatibility and Semantic Versioning, testing, benchmarking, performance, skill authoring, documentation, security and DevSecOps, CI/CD, delivery, governance, and engineering assessment. For .NET, also use for IDE or CA diagnostic remediation, EditorConfig cleanup, code-style compliance, informational diagnostics, and `dotnet format` conformance. Perform the requested work, respect repository scope and conventions, validate before completion, and report evidence and risk honestly. Technology-neutral core; specialist guidance loads on demand. Do NOT use for ordinary prose, casual conversation, translation, image generation, or unrelated factual questions. +description: > + Use when the user wants evidence-driven architecture, implementation, refactoring, review, API compatibility/SemVer, testing, performance, skill authoring, documentation, security/DevSecOps, CI/CD, delivery, governance, or .NET IDE/CA and EditorConfig remediation. --- # Agent Smith @@ -14,21 +14,21 @@ The name is a deliberate, understated nod to a relentless *agent* combined with Apply **one coherent engineering standard** across design, implementation, validation, documentation, delivery, and governance. When this skill is active you do not merely advise — you **perform the requested task** to that standard, then validate it and report honestly. -The standard is technology-neutral. Specialist guidance (including .NET, Git, GitHub, CI/CD, REST, and software-supply-chain security) is loaded only when the task calls for it, and is never imposed on work where it does not apply. - -## Critical skill-authoring lock - -When creating, modifying, reviewing, or evaluating a skill, read `references/skill-authoring.md` before editing. Inspect the actual skill, applicable repository instructions, real execution traces, repeated work, and failure evidence before recommending changes; if unavailable, make that inspection the first required step. Always analyze the task graph for safe parallelism and concurrency. State how independent retrieval, execution, validation, and grading can use bounded fan-out; encode it in the skill when useful. Keep dependencies, shared mutations, rate-limited calls, and fragile ordered workflows sequential. - -For non-trivial reusable scripts and deterministic validators bundled with a skill, choose C# and .NET by default in this .NET-first skill collection. Cross-repository portability alone is not a reason to retain or introduce Python, Bash, or PowerShell. Use another language only for a concrete repository/host constraint, vendor SDK, or materially simpler native operation; state the evidence. Resolve the latest supported .NET LTS dynamically from Microsoft's official support policy when compatible repository SDK/target-framework pins or explicit user constraints do not decide. Do not replace a simpler native command with a C# program merely to satisfy the preference. - -## Critical .NET conformance lock - -When the task selects .NET EditorConfig conformance mode, read both `references/dotnet.md` and `references/dotnet-editorconfig-conformance.md` from the activated skill directory before the first formatter command. Do not search for those resources relative to the target repository or improvise the workflow if a required reference cannot be read. - -For informational diagnostics, informational-or-higher conformance, and any conformance task that does not explicitly set a different minimum, every discovery, investigation, retry, and final `dotnet format` command must include both `--severity info` and `--verify-no-changes`. The formatter defaults to `warn` when `--severity` is omitted, which can hide the findings that define success. Keep the resolved severity explicit and identical throughout the remediation loop. `--no-restore` changes restore behaviour only; it never replaces either required flag or proves conformance. - -If a prior mutating formatter pass has produced `Unmerged change from project` annotations, use the bundled `scripts/repair-roslyn-multiproject-artifacts.ps1` from this skill directory. The tool detects the neutral Roslyn multi-project artifact signature and reports the structural pattern independently of diagnostic ID. It currently repairs only the proven `whole-document-namespace-conversion` pattern. Run its default check mode first and use `-Apply` only when every artifact is reported as `recoverable`; an unrecognized or differing candidate prevents all writes. Pattern-specific recovery is not permission to use the formatter in mutating mode. +The standard is technology-neutral. Specialist guidance (including .NET, Git, GitHub, CI/CD, REST, and software-supply-chain security) is loaded only when the task calls for it, and is never imposed on work where it does not apply. + +## Critical skill-authoring lock + +When creating, modifying, reviewing, or evaluating a skill, read `references/skill-authoring.md` before editing. Inspect the actual skill, applicable repository instructions, real execution traces, repeated work, and failure evidence before recommending changes; if unavailable, make that inspection the first required step. Always analyze the task graph for safe parallelism and concurrency. State how independent retrieval, execution, validation, and grading can use bounded fan-out; encode it in the skill when useful. Keep dependencies, shared mutations, rate-limited calls, and fragile ordered workflows sequential. + +For non-trivial reusable scripts and deterministic validators bundled with a skill, choose C# and .NET by default in this .NET-first skill collection. Cross-repository portability alone is not a reason to retain or introduce Python, Bash, or PowerShell. Use another language only for a concrete repository/host constraint, vendor SDK, or materially simpler native operation; state the evidence. Resolve the latest supported .NET LTS dynamically from Microsoft's official support policy when compatible repository SDK/target-framework pins or explicit user constraints do not decide. Do not replace a simpler native command with a C# program merely to satisfy the preference. + +## Critical .NET conformance lock + +When the task selects .NET EditorConfig conformance mode, read both `references/dotnet.md` and `references/dotnet-editorconfig-conformance.md` from the activated skill directory before the first formatter command. Do not search for those resources relative to the target repository or improvise the workflow if a required reference cannot be read. + +For informational diagnostics, informational-or-higher conformance, and any conformance task that does not explicitly set a different minimum, every discovery, investigation, retry, and final `dotnet format` command must include both `--severity info` and `--verify-no-changes`. The formatter defaults to `warn` when `--severity` is omitted, which can hide the findings that define success. Keep the resolved severity explicit and identical throughout the remediation loop. `--no-restore` changes restore behaviour only; it never replaces either required flag or proves conformance. + +If a prior mutating formatter pass has produced `Unmerged change from project` annotations, use the bundled `scripts/repair-roslyn-multiproject-artifacts.ps1` from this skill directory. The tool detects the neutral Roslyn multi-project artifact signature and reports the structural pattern independently of diagnostic ID. It currently repairs only the proven `whole-document-namespace-conversion` pattern. Run its default check mode first and use `-Apply` only when every artifact is reported as `recoverable`; an unrecognized or differing candidate prevents all writes. Pattern-specific recovery is not permission to use the formatter in mutating mode. ## Activation and invocation @@ -74,16 +74,16 @@ A task may select **multiple** modes. Load core principles for every invocation, | Mode | Use when the task involves | Load | |------|----------------------------|------| | Architecture | system design, boundaries, distributed systems, integration, DDD, CQRS, event-driven design, deployment topology, migration | `references/architecture.md` | -| API design & compatibility | public/HTTP APIs, libraries, contracts, serialization, versioning, Semantic Versioning | `references/api-design-and-compatibility.md` | -| Implementation | coding and refactoring | `references/implementation.md` | -| .NET | .NET or C# is relevant | `references/dotnet.md` | -| .NET EditorConfig conformance | the user explicitly requests EditorConfig, code-style, formatter-supported analyzer, informational IDE, or named diagnostic remediation or verification | `references/dotnet.md` + `references/dotnet-editorconfig-conformance.md` | -| Testing | test design/review, regression, functional/integration/contract testing | `references/testing.md` | +| API design & compatibility | public/HTTP APIs, libraries, contracts, serialization, versioning, Semantic Versioning | `references/api-design-and-compatibility.md` | +| Implementation | coding and refactoring | `references/implementation.md` | +| .NET | .NET or C# is relevant | `references/dotnet.md` | +| .NET EditorConfig conformance | the user explicitly requests EditorConfig, code-style, formatter-supported analyzer, informational IDE, or named diagnostic remediation or verification | `references/dotnet.md` + `references/dotnet-editorconfig-conformance.md` | +| Testing | test design/review, regression, functional/integration/contract testing | `references/testing.md` | | Performance | benchmarking, profiling, optimization, latency, throughput, allocation, scalability | `references/performance.md` | | Security & DevSecOps | identity, authorization, secrets, dependencies, pipelines, supply chain, permissions, deployment security | `references/security-and-devsecops.md` | | Delivery & repository engineering | CI/CD, Git, branching, repo structure, releases, automation, containers, deployment | `references/delivery-and-repositories.md` | | Documentation | public API docs, README, architecture docs, guides, release notes, examples, DocFX | `references/documentation.md` | -| Skill authoring | creating, modifying, reviewing, describing, or evaluating agent skills and their bundled resources | `references/skill-authoring.md` | +| Skill authoring | creating, modifying, reviewing, describing, or evaluating agent skills and their bundled resources | `references/skill-authoring.md` | | Governance | policies, standards, compliance, metrics, enterprise repo governance, guardrails | `references/governance.md` | **Load .NET guidance only when .NET or C# is actually relevant.** For non-.NET work, apply the core principles and let local conventions govern language-specific detail. @@ -92,10 +92,10 @@ A task may select **multiple** modes. Load core principles for every invocation, - **Small implementation** (`/agent-smith add validation for an optional config property`): core principles + implementation (+ platform reference if relevant) + testing. Proportional process, no architecture document. - **Benchmark assessment**: core principles + decision framework + performance + implementation + platform reference (e.g. `dotnet.md`) + response contract; agent-handoff template only if delegation is requested. -- **Public API review**: core principles + decision framework + api-design-and-compatibility + implementation + platform reference + documentation + response contract. -- **CI/CD pipeline**: core principles + decision framework + security-and-devsecops + delivery-and-repositories + governance (when policy is involved) + response contract. -- **Skill authoring**: core principles + decision framework (for material choices) + skill authoring + implementation/testing/documentation as applicable. Make concurrency and script-runtime choices explicit. -- **Scoped .NET diagnostic remediation** (`/agent-smith fix the named IDE and CA diagnostics in src/Codebelt.Core`): core principles + implementation + .NET + .NET EditorConfig conformance + testing. Preserve the user-supplied diagnostic IDs and path through discovery, edits, final verification, build, tests, and reporting. +- **Public API review**: core principles + decision framework + api-design-and-compatibility + implementation + platform reference + documentation + response contract. +- **CI/CD pipeline**: core principles + decision framework + security-and-devsecops + delivery-and-repositories + governance (when policy is involved) + response contract. +- **Skill authoring**: core principles + decision framework (for material choices) + skill authoring + implementation/testing/documentation as applicable. Make concurrency and script-runtime choices explicit. +- **Scoped .NET diagnostic remediation** (`/agent-smith fix the named IDE and CA diagnostics in src/Codebelt.Core`): core principles + implementation + .NET + .NET EditorConfig conformance + testing. Preserve the user-supplied diagnostic IDs and path through discovery, edits, final verification, build, tests, and reporting. ## Repository precedence @@ -130,7 +130,7 @@ For each material finding: Issue → Why it matters → Evidence or reasoning ## Response behaviour -**Be concise. Sacrifice grammar for the sake of concision.** Prefer clear fragments when they shorten feedback. Remain respectful and technically defensible. Challenge weak assumptions; preserve good existing decisions; prioritize material issues; avoid empty praise; distinguish recommendation from requirement; avoid exaggerated certainty; explain non-obvious trade-offs. Never omit required evidence, validation limits, blockers, compatibility impact, or material risk. +**Be concise. Sacrifice grammar for the sake of concision.** Prefer clear fragments when they shorten feedback. Remain respectful and technically defensible. Challenge weak assumptions; preserve good existing decisions; prioritize material issues; avoid empty praise; distinguish recommendation from requirement; avoid exaggerated certainty; explain non-obvious trade-offs. Never omit required evidence, validation limits, blockers, compatibility impact, or material risk. For substantial assessments, use the structure in `references/response-contract.md` (Assessment → Findings → Recommendation → Trade-offs → Validation → Actionable handoff). Do not force that structure onto every response. When producing a formal assessment or a delegation prompt, use `references/engineering-assessment-template.md` or `references/agent-handoff-template.md`. @@ -162,7 +162,7 @@ This skill must not: - turn every small task into an architecture exercise; - produce advice without completing the requested work when implementation is possible; - fabricate evidence or claim unperformed validation; -- sacrifice correctness, required evidence, or material context for terseness; +- sacrifice correctness, required evidence, or material context for terseness; - broaden the task without justification; - introduce dependencies or abstractions without demonstrating value. @@ -174,15 +174,15 @@ Load on demand, per the routing table: - `references/decision-framework.md` — structured reasoning for material decisions. - `references/architecture.md` — system design and boundaries. - `references/api-design-and-compatibility.md` — public and HTTP API contracts and versioning. -- `references/implementation.md` — coding and refactoring discipline. -- `references/dotnet.md` — .NET/C#-specific guidance (load only when relevant). -- `references/dotnet-editorconfig-conformance.md` — scope-aware, read-only `dotnet format` discovery and verification for explicit .NET EditorConfig, code-style, and supported analyzer conformance work. -- `references/testing.md` — test design and review. +- `references/implementation.md` — coding and refactoring discipline. +- `references/dotnet.md` — .NET/C#-specific guidance (load only when relevant). +- `references/dotnet-editorconfig-conformance.md` — scope-aware, read-only `dotnet format` discovery and verification for explicit .NET EditorConfig, code-style, and supported analyzer conformance work. +- `references/testing.md` — test design and review. - `references/performance.md` — benchmarking, profiling, optimization. - `references/security-and-devsecops.md` — identity, secrets, dependencies, pipelines, supply chain. - `references/delivery-and-repositories.md` — CI/CD, Git, releases, repository engineering. - `references/documentation.md` — documentation as part of the product. -- `references/skill-authoring.md` — skill design, parallelism, .NET-first scripts, descriptions, and eval loops. +- `references/skill-authoring.md` — skill design, parallelism, .NET-first scripts, descriptions, and eval loops. - `references/governance.md` — policies, standards, and metrics (Intent → Drivers → Metrics → Actions). - `references/response-contract.md` — review severity, finding shape, and assessment structure. - `references/engineering-assessment-template.md` — fill-in template for a formal assessment. diff --git a/skills/dotnet-benchmark/SKILL.md b/skills/dotnet-benchmark/SKILL.md index 8d35f41..be8bc2c 100644 --- a/skills/dotnet-benchmark/SKILL.md +++ b/skills/dotnet-benchmark/SKILL.md @@ -1,7 +1,7 @@ --- name: dotnet-benchmark description: > - Discover, prioritize, and author trustworthy BenchmarkDotNet performance experiments for a .NET type while following codebelt engineering conventions and using the Codebelt.Extensions.BenchmarkDotNet Console runner. Use whenever a user wants to benchmark, micro-benchmark, performance-test, profile, optimize, compare implementations, investigate allocations or contention, or find likely bottlenecks in a .NET type or method. The skill inspects source and usage evidence, ranks high-value operations instead of every public member, selects representative workloads, rejects misleading microbenchmarks, creates or reuses the tuning/ and tooling/ harness, preflights existing-report skips, semantic-preflights workload correctness, validates discovery, and keeps full runs human-initiated. When the user says yolo, it auto-accepts routine defaults and proceeds through safe validation without confirmation churn. + Use when the user wants to design, author, review, or diagnose BenchmarkDotNet experiments for .NET code, compare implementations, measure allocations or contention, or benchmark a specific type or method. Also use to judge whether profiling or load testing is the better instrument. --- # Evidence-Driven .NET Benchmarking diff --git a/skills/dotnet-change-impact/SKILL.md b/skills/dotnet-change-impact/SKILL.md index 47e5b64..be53457 100644 --- a/skills/dotnet-change-impact/SKILL.md +++ b/skills/dotnet-change-impact/SKILL.md @@ -1,7 +1,7 @@ --- name: dotnet-change-impact -description: > - Classifies .NET library or NuGet package changes and recommends the correct release bump: Major, Minor, or Patch. Applies both Semantic Versioning (MAJOR.MINOR.PATCH) and .NET assembly/file versioning (Major.Minor.Build.Revision), grounded in Microsoft’s official .NET library compatibility rules. Use when evaluating the current branch, breaking changes, API diffs, public API changes, dependency updates, TFM/platform support, interface or enum changes, overloads, analyzers, source generators, or binary/source/behavioral/design-time/backwards compatibility. When no explicit change details or compare range are provided, inspects the current Git branch and compares it against the upstream default branch automatically. Always returns structured compatibility reasoning with the recommendation. +description: > + Use when the user wants a Major, Minor, or Patch recommendation for .NET library or NuGet package changes, including API diffs, behavior changes, dependencies, target frameworks, analyzers, or source generators, based on compatibility impact. --- # .NET Change Impact diff --git a/skills/dotnet-docfx-digest/SKILL.md b/skills/dotnet-docfx-digest/SKILL.md index 9fa6aad..3ec78d7 100644 --- a/skills/dotnet-docfx-digest/SKILL.md +++ b/skills/dotnet-docfx-digest/SKILL.md @@ -1,7 +1,7 @@ --- name: dotnet-docfx-digest description: > - Create and maintain developer-friendly DocFX documentation digests for .NET public APIs: repo-wide no-input audits, namespace pages, purpose-first API summaries, extension-member documentation, overwrite files, examples, availability notes, AGENTS.md maintenance, and verification. Use when the user asks to document a .NET API, update DocFX docs, create namespace pages, improve API summaries, add extension-member tables, update XML comments, add examples, maintain DocFX overwrite files, or verify documentation builds. Treat "use dotnet-docfx-digest", "complete missing documentation", and public .NET API changes as automatic triggers. + Use when the user wants to create, repair, audit, or complete DocFX docs for .NET public APIs, or has changed public API that needs namespace pages, XML comments, overwrite files, extension-member tables, examples, or build verification. Exclude private/internal APIs. --- # .NET DocFX Digest Steward diff --git a/skills/dotnet-new-app-slnx/SKILL.md b/skills/dotnet-new-app-slnx/SKILL.md index 3bffe36..625e48f 100644 --- a/skills/dotnet-new-app-slnx/SKILL.md +++ b/skills/dotnet-new-app-slnx/SKILL.md @@ -1,7 +1,7 @@ --- name: dotnet-new-app-slnx -description: > - Scaffold a new .NET standalone application solution following codebelt engineering conventions. Use this skill when the user wants to create a new .NET application — Console, Web, or Worker service. Also use when the user mentions "new app", "new console app", "new web api", "new mvc app", "new razor app", "new web app", "new worker service", "scaffold app", "dotnet new web", "dotnet new webapi", "dotnet new mvc", "dotnet new webapp", "dotnet new worker", "dotnet new console", or wants a .NET application project with CI/CD pipeline, functional tests, and code quality tooling. ALWAYS use this skill when asked to scaffold or create a new .NET application solution. +description: > + Use when the user wants to scaffold a complete codebelt-style .NET application solution (`.slnx`) for Console, Web API, MVC, Razor, empty Web, or Worker hosts, with repository tooling and functional tests. Do not use for a quick throwaway project or application logic. --- # .NET Application Solution Setup (Codebelt Conventions) diff --git a/skills/dotnet-new-lib-slnx/SKILL.md b/skills/dotnet-new-lib-slnx/SKILL.md index 0f464d1..5a54b6e 100644 --- a/skills/dotnet-new-lib-slnx/SKILL.md +++ b/skills/dotnet-new-lib-slnx/SKILL.md @@ -1,7 +1,7 @@ --- name: dotnet-new-lib-slnx description: > - Scaffold a new .NET NuGet library solution following codebelt engineering conventions. Use this skill when the user wants to create a new NuGet library, class library, or reusable .NET package. Also use when the user mentions "new library", "new NuGet package", "scaffold library", "class library solution", "dotnet new classlib", or wants a .NET library project with multi-target frameworks, strong-name signing, NuGet packaging, DocFX documentation, CI/CD pipeline, and code quality tooling. ALWAYS use this skill when asked to scaffold or create a new .NET library solution. + Use when the user wants to scaffold a complete codebelt-style .NET library or NuGet package solution (`.slnx`), including packaging, tests, DocFX, benchmarks, and strong-name signing. Do not use merely to add a library project to an established solution. --- # .NET Library Solution Setup (Codebelt Conventions) diff --git a/skills/dotnet-remote-testing/SKILL.md b/skills/dotnet-remote-testing/SKILL.md index 1d9a698..c9cd53b 100644 --- a/skills/dotnet-remote-testing/SKILL.md +++ b/skills/dotnet-remote-testing/SKILL.md @@ -1,7 +1,7 @@ --- name: dotnet-remote-testing -description: > - Run .NET tests inside a resolved remote Docker environment — Visual Studio's Remote Testing without hand-writing container plumbing. Invoking this skill IS the request: run the tests immediately. Never reply with a menu of options or a questionnaire. Use when asked to remote test, run tests in Docker or a container, target a specific .NET SDK, list or select test environments, or honor an existing testenvironments.json. Honors configured Docker environments, or derives them from Microsoft's live .NET release index using mcr.microsoft.com/dotnet/sdk images, plus codebeltnet/ubuntu-testrunner for multi-targeted repos, via the runner scripts/remote-test.cs. Docker only; WSL and SSH are unsupported. Do NOT use to author or refactor test code, choose a testing framework, generate Dockerfiles, or run tests on the host. +description: > + Use when the user wants to run, list, or plan .NET tests in Docker remote-test environments, including `testenvironments.json` or a requested SDK/container. Do not use to write or refactor tests, create Dockerfiles, use WSL/SSH, or run tests on the host. compatibility: > Requires the .NET 10 SDK or later (`dotnet run --file`), a running Docker daemon, and PowerShell 7+. Zero-config discovery needs network access; a cache enables offline reuse. --- diff --git a/skills/dotnet-segregated-assets/SKILL.md b/skills/dotnet-segregated-assets/SKILL.md index cb2513f..feef224 100644 --- a/skills/dotnet-segregated-assets/SKILL.md +++ b/skills/dotnet-segregated-assets/SKILL.md @@ -1,7 +1,7 @@ --- name: dotnet-segregated-assets description: > - Migrate or configure an ASP.NET Core web application so developers keep authoring static files in the conventional wwwroot while deployed static content is served by Codebelt Static Content Provider (codebeltnet/web-cdn-origin:2.0.0), a separate asset host rather than the web app. Use when asked to segregate static assets, move wwwroot off the web app, stop shipping wwwroot with the app, or reconcile Cuemon App/CDN TagHelpers with a segregated topology. Reuse existing Cuemon or project abstractions, distinguish App assets from shared CDN assets, preserve Static Web Assets, and verify publish/local invariants deterministically. Do NOT use to build a general-purpose CDN or migrate non-ASP.NET static sites. + Use when the user wants an ASP.NET Core app to keep authoring static files in `wwwroot` while serving deployed app assets from `codebeltnet/web-cdn-origin`, including Cuemon `app-*`/`cdn-*` migration. Do not use for general CDN design or non-ASP.NET sites. compatibility: > Requires the .NET SDK 10+ and PowerShell 7+. NuGet.org access is required when plan resolves an existing Cuemon package reference. Docker is optional (only for the local origin). CI guidance targets GitHub Actions, which is the assumed delivery surface. --- diff --git a/skills/dotnet-strong-name-signing/SKILL.md b/skills/dotnet-strong-name-signing/SKILL.md index 634d786..543d600 100644 --- a/skills/dotnet-strong-name-signing/SKILL.md +++ b/skills/dotnet-strong-name-signing/SKILL.md @@ -1,7 +1,7 @@ --- name: dotnet-strong-name-signing -description: > - Generate a strong name key (.snk) file for signing .NET assemblies using pure .NET cryptography — no Visual Studio Developer PowerShell or sn.exe required. Works in any terminal. Use this skill when the user wants to create a strong name key, generate an .snk file, sign .NET assemblies, or mentions "strong-name", "snk", "AssemblyOriginatorKeyFile", "SignAssembly", or asks how to sign a .NET library. Also use when scaffolding .NET libraries or NuGet packages that need assembly signing. ALWAYS use this skill when asked to generate or create a strong name key file. +description: > + Use when the user wants to generate a `.snk` strong-name key, configure `SignAssembly` or `AssemblyOriginatorKeyFile`, or sign a .NET assembly, library, or NuGet package. Do not use for Authenticode, code-signing certificates, or secret-management tasks. --- # Strong Name Signing for .NET diff --git a/skills/dotnet-test/SKILL.md b/skills/dotnet-test/SKILL.md index c095854..869d159 100644 --- a/skills/dotnet-test/SKILL.md +++ b/skills/dotnet-test/SKILL.md @@ -1,7 +1,7 @@ --- name: dotnet-test description: > - Move .NET xUnit test projects onto Codebelt's entrypoint-owned test hosts, replacing Microsoft's WebApplicationFactory and hand-rolled host plumbing with WebApplicationTestFactory, WebApplicationTest, ApplicationTestFactory, and ApplicationTest — for ASP.NET Core, console, and worker applications alike. Invoking this skill IS the request: inspect the repository and refactor immediately, never opening with a menu, a capability list, or a questionnaire. Use for WebApplicationFactory migration, xUnit v2-to-v3 modernization, Microsoft Testing Platform adoption, managed fixtures, reusable functional-test harnesses, in-process console or worker tests, and unit-test bootstrap. Preserve behavior, test names, and package ownership, then validate restore/build/test. Do NOT use for NUnit/MSTest-only work, production refactoring without a test-project goal, or process-launching end-to-end harnesses. + Use when the user wants to bootstrap or refactor .NET xUnit tests onto Codebelt entrypoint-owned hosts, migrate `WebApplicationFactory`, modernize xUnit v2 to v3 or Microsoft Testing Platform, or add web, console, or worker fixtures. Exclude NUnit/MSTest and process-based end-to-end harnesses. compatibility: > Requires .NET SDK, PowerShell 7+, and network access to NuGet for dynamic package resolution. --- diff --git a/skills/git-keep-a-changelog/SKILL.md b/skills/git-keep-a-changelog/SKILL.md index c151fd0..3b182fc 100644 --- a/skills/git-keep-a-changelog/SKILL.md +++ b/skills/git-keep-a-changelog/SKILL.md @@ -1,7 +1,7 @@ --- name: git-keep-a-changelog description: > - Create or update CHANGELOG.md from git history using Keep a Changelog 1.1.0 style. Use when the user explicitly asks to create or update a changelog, draft release notes, prepare or finalize a release changelog, or requests a SemVer-aware release summary. Treat `ready to release` and `rtr` as triggers only in a versioned release context. Treat `yolo` and `auto` only as autonomy modifiers after explicit changelog or release-note intent; they are never standalone triggers. Never select this skill for `git bot commit yolo`, `git commit auto`, or another commit-execution request unless the user also explicitly asks to update the changelog or release notes. Reads full commit bodies and diffs, isolates branch history, includes pending changes automatically only in scoped yolo or auto mode, and writes curated surviving base-to-HEAD outcomes for review. + Use when the user wants to create or update `CHANGELOG.md`, follow Keep a Changelog, or finalize a versioned changelog. Treat `ready to release` or `rtr` as triggers only with version context. Do not trigger for GitHub releases, NuGet package notes, commit execution, or bare `yolo`/`auto`. compatibility: > Requires Git and PowerShell 7+ for deterministic branch-scope resolution. --- diff --git a/skills/git-nuget-readme/SKILL.md b/skills/git-nuget-readme/SKILL.md index 79c187c..94a6b4f 100644 --- a/skills/git-nuget-readme/SKILL.md +++ b/skills/git-nuget-readme/SKILL.md @@ -1,7 +1,7 @@ --- name: git-nuget-readme description: > - Create or update a NuGet package README.md from git history and real .NET project metadata for repositories that ship a package from `src/`. Use this skill whenever the user asks to write a package README, refresh NuGet-facing docs, improve the repo README for a library, summarize the current branch into README copy, or make a package more compelling to adopt on NuGet. Treat requests like "update the README for this package", "write a NuGet README from git", "refresh the library README", "make this NuGet package easier to pick", or "generate a devex-friendly README for this assembly" as automatic triggers. The skill discovers the advertised packable project, grounds the README in real package and source metadata, preserves honest claims, and writes forthcoming, adoption-friendly copy instead of generic marketing fluff. + Use when the user wants to create or refresh a NuGet-facing `README.md` for a .NET package, grounded in the packable `src/` project and current repository changes. Do not use for general repository docs, DocFX API pages, changelogs, or release notes. --- # Git NuGet README diff --git a/skills/git-nuget-release-notes/SKILL.md b/skills/git-nuget-release-notes/SKILL.md index b91002e..2b49771 100644 --- a/skills/git-nuget-release-notes/SKILL.md +++ b/skills/git-nuget-release-notes/SKILL.md @@ -1,7 +1,7 @@ --- name: git-nuget-release-notes description: > - Create or update per-package NuGet release notes from git history for .NET repositories that store cumulative `.nuget/{ProjectName}/PackageReleaseNotes.txt` files. Use this skill whenever the user asks for NuGet release notes, `PackageReleaseNotes.txt`, per-assembly or per-package release notes, or wants git commits turned into package release notes instead of a repo-wide changelog. Treat requests like "update PackageReleaseNotes.txt", "write NuGet release notes from git", "summarize this release per assembly", or "create missing package release notes under .nuget" as automatic triggers. The skill discovers packable `src/` projects, resolves concrete release version and availability per package, creates missing files when needed, preserves cumulative newest-first history, reduces each package to its surviving base-to-HEAD delta before using history as context, and avoids raw commit-log dumps or unsupported claims. + Use when the user wants per-package NuGet release notes in cumulative `.nuget/{ProjectName}/PackageReleaseNotes.txt` files, including creating missing files for packable `src/` projects. Do not use for repository `CHANGELOG.md`, GitHub releases, or package README work. --- # Git NuGet Release Notes diff --git a/skills/git-remote-release/SKILL.md b/skills/git-remote-release/SKILL.md index 1610c65..7409e03 100644 --- a/skills/git-remote-release/SKILL.md +++ b/skills/git-remote-release/SKILL.md @@ -1,7 +1,7 @@ --- name: git-remote-release description: > - Generate GitHub release notes by summarizing all commits and pull requests between two Git tags, branches, or the current branch and the upstream default branch. Use when the user asks to write release notes, generate release notes, draft a GitHub release, create release notes from tags, summarize changes between versions, summarize the current branch, or provides a GitHub compare URL. Trigger phrases: "release notes", "generate release notes", "what changed between", "summarize changes from v1 to v2", "GitHub release", "summarize this branch", compare URLs like "github.com/owner/repo/compare/v1...v2". When no explicit input is given, detects the current branch and compares against the upstream default branch automatically. + Use when the user wants GitHub release notes or a human-readable change summary from commits and pull requests between tags, branches, or a GitHub compare URL. Do not use for `CHANGELOG.md`, NuGet `PackageReleaseNotes.txt`, commit messages, or squash summaries. --- # Git Remote Release diff --git a/skills/git-repo-digest/SKILL.md b/skills/git-repo-digest/SKILL.md index f1e0dc0..0d41847 100644 --- a/skills/git-repo-digest/SKILL.md +++ b/skills/git-repo-digest/SKILL.md @@ -1,7 +1,7 @@ --- name: git-repo-digest description: > - Generate source-grounded repository digest markdown from deterministic local evidence bundles. Use when the user asks to create, refresh, or complete repo/package digests, family or project overview pages, .bot/digests output, digest workspace workflows, or result/Index.md plus result/{PackageName}.md files for any repository URL. The skill runs its bundled .NET file-based evidence generator over a git clone, separates authoritative XML evidence from Markdown prompts and reading aids, writes package digests first, then writes the overview from completed package digests, and enforces complete-read grounding and no-invention rules even when file output is capped. + Use when the user wants source-grounded repository or package digest Markdown from a repository URL or existing `.bot/digests` workspace, especially `result/Index.md` and per-package pages. Do not use for README, DocFX, changelog, release-note, or one-off prose summaries. --- # Git Repo Digest diff --git a/skills/git-visual-commits/SKILL.md b/skills/git-visual-commits/SKILL.md index d5664f1..d8f4eb5 100644 --- a/skills/git-visual-commits/SKILL.md +++ b/skills/git-visual-commits/SKILL.md @@ -1,7 +1,7 @@ --- name: git-visual-commits description: > - Execute the structured git commit workflow whenever the user says `git bot commit`, `git commit`, or `git our commit`; asks the agent to commit or stage changes; or asks to write or review a commit message. Treat `Please do a git bot commit yolo` and equivalent wording as an authoritative invocation of this skill: select bot identity, enable auto-approval, and never treat `yolo` as the message or route the request to changelog or release-note skills. Treat commit wording as an automatic trigger for this skill, not as a casual hint. `yolo` and `auto` are modifiers only inside an explicit commit request and never standalone triggers. Apply full-worktree semantic grouping unless narrowed, validated emoji-first lowercase subjects, conventional prefixes only on explicit request, and post-commit identity and body verification. + Use when the user asks to stage or commit changes, write or review a commit message, or says `git bot commit`, `git commit`, or `git our commit`. Treat commit wording as an automatic trigger for this skill, not as a casual hint. `yolo` and `auto` only modify an explicit commit request. --- # Git Visual Commits diff --git a/skills/git-visual-squash-summary/SKILL.md b/skills/git-visual-squash-summary/SKILL.md index 71c5ddf..3e6fe6d 100644 --- a/skills/git-visual-squash-summary/SKILL.md +++ b/skills/git-visual-squash-summary/SKILL.md @@ -1,7 +1,7 @@ --- name: git-visual-squash-summary description: > - Turn many commits into a curated grouped squash summary for squash-and-merge contexts. Use when the user asks to squash a branch, summarize PR commits, or clean up history. Defaults to full feature branch against base (not tracking remote), includes all authors unless narrowed, and acts immediately—the skill is read-only with no permission-seeking. Returns grouped lines only, resolving the cumulative diff to drop reverted churn, preserving identifiers and overlap, and avoiding changelog wording. A bare invocation is a complete request: run git commands immediately and return summary lines, never an instruction recap or permission question. + Use when the user wants a read-only, ready-to-paste grouped squash-and-merge or PR branch summary from a commit range or the current feature branch. Do not use to mutate history, write changelog or release notes, execute commits, or summarize a single commit message. --- # Git Visual Squash Summary diff --git a/skills/markdown-illustrator/SKILL.md b/skills/markdown-illustrator/SKILL.md index 22329df..f0af52c 100644 --- a/skills/markdown-illustrator/SKILL.md +++ b/skills/markdown-illustrator/SKILL.md @@ -1,7 +1,7 @@ --- name: markdown-illustrator description: > - Turn a markdown document into a visualization-first chat response consisting of one Visual Brief and one high-quality diffuser prompt generated with best-effort reasoning. Use when the user references a .md file and wants a hero image, cover image, visual digest, keynote opener, illustration, or diffuser prompt, especially for requests like "turn roadmap.md into a keynote opener image" or "create a visual digest for onboarding-notes.md". Default to zero follow-up questions, no file creation, and no style/theme/model menus; infer a compact visual strategy from the request and document, and only honor extra specificity when the user explicitly asks for a named model, aesthetic, or visual treatment such as whiteboard or blackboard. + Use when the user references a Markdown file and wants one hero, cover, keynote-opener, visual-digest, or illustration prompt derived from it, including whiteboard, blackboard, or named image-model styling. Do not trigger for image generation itself or general Markdown summarization. --- # Markdown Illustrator diff --git a/skills/skill-creator-agnostic/SKILL.md b/skills/skill-creator-agnostic/SKILL.md index e8ac43a..4f68913 100644 --- a/skills/skill-creator-agnostic/SKILL.md +++ b/skills/skill-creator-agnostic/SKILL.md @@ -1,7 +1,8 @@ --- name: skill-creator-agnostic description: > - DEPRECATED — no longer maintained and scheduled for removal in 1.0.0. Retained only for backward compatibility. Do not use for new skill creation, modification, or benchmarking. Use Anthropic's `skill-creator` directly and apply this repository's skill-authoring rules from `AGENTS.md` instead. + DEPRECATED — no longer maintained and scheduled for removal in 1.0.0. Use only when the user explicitly invokes `skill-creator-agnostic` or asks about its status. Redirect skill creation, modification, evaluation, and benchmarking to Anthropic's `skill-creator` plus repository `AGENTS.md`. +disable-model-invocation: true --- # Skill Creator Agnostic diff --git a/skills/trunk-first-repo/SKILL.md b/skills/trunk-first-repo/SKILL.md index a1168d6..d1aabb0 100644 --- a/skills/trunk-first-repo/SKILL.md +++ b/skills/trunk-first-repo/SKILL.md @@ -1,7 +1,7 @@ --- name: trunk-first-repo -description: > - Initialize a folder as a git repository following scaled trunk-based development. Sets up an empty main branch (seed commit only), creates a versioned feature branch, pushes main before feature branches, and enforces a PR-first workflow where content only reaches main through pull requests. Use this skill when the user wants to initialize a git repo, set up a new repository, start a project with proper git workflow, safely push the first trunk-first branches later with "push remote", or mentions "trunk-based", "PR workflow", "branch protection", "git init", or wants to follow GitHub PR best practices. ALWAYS use this skill when asked to initialize or set up a git repository. +description: > + Use when the user wants to initialize a new Git repository with a scaled trunk-based, PR-first workflow, including an empty seeded `main`, a versioned feature branch, branch protection, or a guarded first remote push. Do not use for commits or history cleanup in an existing repository. --- # Trunk-First Repo From dd24b6f6f7a6b7d45a684e20e33ff9d493991445 Mon Sep 17 00:00:00 2001 From: "aicia[bot]" Date: Sat, 22 Aug 2026 13:38:05 +0200 Subject: [PATCH 12/39] =?UTF-8?q?=E2=99=BB=EF=B8=8F=20enhance=20git-keep-a?= =?UTF-8?q?-changelog=20with=20layered=20capability=20classification?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Added Layered Capability Classification section to distinguish classification of independently selectable adapters and runners from their parent directory or framework. This ensures that new child adapters are correctly classified as Added rather than grouped with pre-existing framework changes. Added comprehensive eval case covering framework refinement, new adapter introduction, defect repair, and layered state classification to validate the enhanced guidance. --- skills/git-keep-a-changelog/SKILL.md | 11 +++++++++++ skills/git-keep-a-changelog/evals/evals.json | 14 ++++++++++++++ 2 files changed, 25 insertions(+) diff --git a/skills/git-keep-a-changelog/SKILL.md b/skills/git-keep-a-changelog/SKILL.md index 3b182fc..2ec032b 100644 --- a/skills/git-keep-a-changelog/SKILL.md +++ b/skills/git-keep-a-changelog/SKILL.md @@ -80,6 +80,17 @@ Reduce first. Interpret second. Summarize last. Establish the classification baseline at the user-facing release-entity boundary, not independently for every changed file. For a repo-managed skill, the entity is the skill capability together with its dedicated files and inseparable registration, catalog, documentation, validation, and eval wiring. If that entity is absent at the base and present at `HEAD`, its introduction is `Added`; intermediate commits that refine, fix, document, or validate it cannot create `Changed` or `Fixed` outcomes for that same new entity. A change to a separately pre-existing shared capability remains its own outcome and is classified from its own base state. +### Layered Capability Classification + +Do not use a top-level directory or the first framework commit as the only release entity. Classify at the smallest independently selectable user-facing boundary. For layered eval tooling, the protocol/framework and each selectable runner, CLI, TUI, or harness adapter can have different release states. + +- A child adapter absent at the resolved base and present at `HEAD` is `Added` even when its parent directory already existed at the base or was introduced earlier in the branch. Run the entity resolver for the parent and each independent child path when the diff supports that decomposition. +- Keep implementation, fixtures, conformance tests, and wiring with the capability they introduce. Do not repeat an adapter in a parent `Added` bullet and again in `Changed` as “added to the lineup.” +- Refinements to a pre-existing adapter or framework are `Changed` or `Fixed` from their final delta. A defect repaired before first release of a base-absent capability remains part of that capability's `Added` outcome. +- A planned, blocked, or unsupported CLI/TUI/harness is not support and must not be listed as an `Added` adapter. + +Use the final state, not commit verbs: newly usable execution support belongs under `Added`; changes to existing runner, orchestration, report, telemetry, or package behavior belong under `Changed`; and distinct supported repairs belong under `Fixed`. For example, a new framework followed by Cline or GitHub Copilot adapters gets `Added` outcomes for the framework and adapters, while changes to an existing Codex adapter are classified separately. + 1. Inspect cumulative manifest and version deltas across `diff_range`. 2. Inspect the cumulative base-to-`HEAD` diff. 3. Inspect any approved pending worktree changes that are part of the draft. diff --git a/skills/git-keep-a-changelog/evals/evals.json b/skills/git-keep-a-changelog/evals/evals.json index 4d080ab..c9504aa 100644 --- a/skills/git-keep-a-changelog/evals/evals.json +++ b/skills/git-keep-a-changelog/evals/evals.json @@ -251,6 +251,20 @@ "Does not reinterpret yolo as a release or changelog trigger", "Defers the request to the git-visual-commits workflow" ] + }, + { + "id": 22, + "prompt": "Create a deterministic temp git repo outside the current repository under `$env:TEMP`, then use git-keep-a-changelog there. Start from a tagged base release containing `CHANGELOG.md`, an existing `scripts/eval-runners/README.md`, an existing `scripts/eval-runners/codex/runner.ps1`, an existing report generator, and an existing conformance test. On branch `v1.1.0/eval-runner-support`, add the shared execution contracts and package bridge, add `scripts/eval-runners/cline/runner.ps1` and `scripts/eval-runners/github-copilot/runner.ps1` with their fixtures and documentation, refine the existing report telemetry and Codex command behavior, add a `freebuff-readiness.md` file that says its TUI transport is planned and blocked, and repair a distinct existing Codex evidence-directory defect. Include commits from multiple contributors. Update `CHANGELOG.md` and stop after the edit.", + "expected_output": "The changelog separates the new execution framework and independently selectable Cline and GitHub Copilot adapters as Added outcomes, classifies changes to the pre-existing report/Codex behavior separately, records the supported Codex repair when justified, does not repeat the new adapters under Changed, and does not claim the planned Freebuff TUI is supported.", + "expectations": [ + "Does not treat the top-level scripts/eval-runners directory as the only release entity when independently selectable adapters have distinct final states", + "Runs or applies path-backed base-versus-HEAD classification to the shared framework and the Cline and GitHub Copilot adapter boundaries", + "Places first-time Cline and GitHub Copilot support under Added even when the parent eval-runner framework already existed at the base or earlier in the branch", + "Classifies refinements to the pre-existing report generator or Codex adapter from their surviving base-to-HEAD behavior rather than calling them new support", + "Does not repeat a newly supported adapter in both Added and Changed or describe it merely as added to an existing lineup under Changed", + "Does not claim the planned or blocked Freebuff TUI is a supported runner or a completed Added capability", + "Keeps the changelog grounded in the final state and includes contributions from every selected author" + ] } ] } From d215906bb3c78746792c5fbed154aaa845ec6210 Mon Sep 17 00:00:00 2001 From: "aicia[bot]" Date: Sat, 22 Aug 2026 13:38:12 +0200 Subject: [PATCH 13/39] =?UTF-8?q?=F0=9F=A6=BA=20add=20validator=20checks?= =?UTF-8?q?=20for=20layered=20capability=20classification?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Extended git-keep-a-changelog skill validation to verify the presence and correctness of the new Layered Capability Classification section and its guidance on adapter/runner boundary classification. Added checks for section presence, guidance on child adapter states, refinement classification, and avoidance of repeat classification patterns. --- scripts/validate-skill-templates.ps1 | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/scripts/validate-skill-templates.ps1 b/scripts/validate-skill-templates.ps1 index 6264054..80a2816 100644 --- a/scripts/validate-skill-templates.ps1 +++ b/scripts/validate-skill-templates.ps1 @@ -2164,6 +2164,11 @@ Add-ValidationResult -Results $results -Name 'Git keep a changelog skill updates Assert-Contains -Name 'git-keep-a-changelog/scripts/resolve-release-entity.ps1' -Content $entityResolver -Needle "'Added'" Assert-Contains -Name 'git-keep-a-changelog/scripts/resolve-release-entity.ps1' -Content $entityResolver -Needle "'Unchanged'" Assert-Contains -Name 'git-keep-a-changelog/scripts/test-resolve-release-entity.ps1' -Content $entityResolverTests -Needle "Assert-Classification -EntityPath 'skills/dotnet-test' -Expected 'Added'" + Assert-Contains -Name 'git-keep-a-changelog/SKILL.md' -Content $skill -Needle '### Layered Capability Classification' + Assert-Contains -Name 'git-keep-a-changelog/SKILL.md' -Content $skill -Needle 'Do not use a top-level directory or the first framework commit as the only release entity.' + Assert-Contains -Name 'git-keep-a-changelog/SKILL.md' -Content $skill -Needle 'A child adapter absent at the resolved base and present at `HEAD` is `Added`' + Assert-Contains -Name 'git-keep-a-changelog/SKILL.md' -Content $skill -Needle 'Do not repeat an adapter in a parent `Added` bullet and again in `Changed`' + Assert-Contains -Name 'git-keep-a-changelog/SKILL.md' -Content $skill -Needle 'A planned, blocked, or unsupported CLI/TUI/harness is not support' Assert-Contains -Name 'git-keep-a-changelog/evals/evals.json' -Content $evals -Needle 'Updates CHANGELOG.md directly instead of only drafting notes in chat' Assert-Contains -Name 'git-keep-a-changelog/evals/evals.json' -Content $evals -Needle 'Reads full commit subjects and bodies before writing the release entry' @@ -2208,6 +2213,8 @@ Add-ValidationResult -Results $results -Name 'Git summary skills reduce ranges t Assert-Contains -Name 'git-keep-a-changelog/evals/evals.json' -Content $changelogEvals -Needle 'Does not add a Security or other section entry when the final diff contradicts the commit message claim' Assert-Contains -Name 'git-keep-a-changelog/evals/evals.json' -Content $changelogEvals -Needle 'Does not create a Changed section or Changed bullet for dotnet-test refinements made before its first release' Assert-Contains -Name 'git-keep-a-changelog/evals/evals.json' -Content $changelogEvals -Needle 'Does not preserve the earlier draft bullet as a frozen baseline that forces later refinements into `Changed`' + Assert-Contains -Name 'git-keep-a-changelog/evals/evals.json' -Content $changelogEvals -Needle 'Does not treat the top-level scripts/eval-runners directory as the only release entity when independently selectable adapters have distinct final states' + Assert-Contains -Name 'git-keep-a-changelog/evals/evals.json' -Content $changelogEvals -Needle 'Does not claim the planned or blocked Freebuff TUI is a supported runner or a completed Added capability' Assert-Contains -Name 'git-nuget-release-notes/SKILL.md' -Content $nugetSkill -Needle 'History is evidence; the resulting state is truth.' Assert-Contains -Name 'git-nuget-release-notes/SKILL.md' -Content $nugetSkill -Needle 'Classify each user-facing package capability from whether it existed at the resolved base' From 736cd893e996b0d2d541142a3572656ff716cd1d Mon Sep 17 00:00:00 2001 From: "aicia[bot]" Date: Sat, 22 Aug 2026 13:38:20 +0200 Subject: [PATCH 14/39] =?UTF-8?q?=F0=9F=92=AC=20update=20changelog=20for?= =?UTF-8?q?=20v0.9.1=20release?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Regenerated [0.9.1] release entry from current git state (2026-08-22) to capture all commits through HEAD, including GitHub Copilot CLI runner support, Cline runner addition, skill description optimization, and AGENTS.md authoring guidance restructuring. Updated release highlight, Added/Changed/Fixed sections, and compare link to reflect the complete v0.9.1 release scope. --- CHANGELOG.md | 19 +++++++++++++++---- 1 file changed, 15 insertions(+), 4 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 64730ea..7403b5e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,20 +4,30 @@ All notable changes to this project will be documented in this file. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). -## [0.9.1] - 2026-08-21 +## [0.9.1] - 2026-08-22 -This patch release adds the harness-agnostic Eval Runner execution boundary without changing the paired evaluation methodology or existing report schemas. Prepared packages now carry `execution-profile.json`, package-local runner protocol tools, and normalized `execution-result.json` evidence; the deterministic fake runner is the conformance reference, with Codex and OpenCode as the initial real adapters. Repository automation remains model-free; only a human-directed external Eval Orchestrator may invoke the selected runner, and unsupported isolation fails closed. +This patch release adds harness-agnostic Eval Runner execution boundary infrastructure without changing the paired evaluation methodology or existing report schemas, while optimizing skill descriptions and refactoring repository-level authoring guidance. Prepared packages now carry `execution-profile.json`, package-local runner protocol tools, and normalized `execution-result.json` evidence. The deterministic fake runner is the conformance reference, with Codex, GitHub Copilot CLI, Cline, and OpenCode as supported real adapters. Repository automation remains model-free; only a human-directed external Eval Orchestrator may invoke the selected runner, and unsupported isolation fails closed. ### Added -- `scripts/eval-runners/` with the common `describe`/`preflight`/`execute` process contract, execution-profile and execution-result schemas, deterministic fake runner, Codex adapter, OpenCode adapter, artifact/hash validation, and bridge into the existing `eval-result/2` result shape, +- `scripts/eval-runners/` with the common `describe`/`preflight`/`execute` process contract, execution-profile and execution-result schemas, deterministic fake runner, Codex adapter, GitHub Copilot CLI adapter, Cline adapter, OpenCode adapter, runner resolution, artifact/hash validation, and bridge into the existing `eval-result/2` result shape, +- GitHub Copilot CLI as a supported Eval Runner with authentication handling (GitHub tokens, OS keychain, CLI fallback), JSONL-based event output parsing, stdin-based prompt delivery for byte fidelity, repository instruction visibility, and conformance tests covering token management and authentication source detection, - deterministic fake-runner conformance coverage for fresh paired sessions, prompt fidelity, isolation boundaries, candidate-skill exposure, status normalization, unavailable telemetry, event warnings, artifact references, and report compatibility, - runner-aware package preparation that reuses `run.json`, keeps runner selection outside `evals/evals.json`, and preserves Anthropic-compatible benchmark/report artifacts. ### Changed - `AGENTS.md`, `README.md`, and `CONTRIBUTING.md` now distinguish the Eval Runner, Eval Orchestrator, Grader, and Human Reviewer and clarify that runner execution is an explicit external-handoff boundary rather than repository automation, -- report timing output omits unavailable duration and token telemetry instead of writing zero placeholders. +- All 21 repo-managed skill descriptions refactored to lean, trigger-oriented activation metadata following progressive-disclosure principles and specification compliance, +- `AGENTS.md` Skill Authoring section restructured for clarity, brevity, and progressive disclosure of form fields, asset handling, and dynamic defaults, +- report timing output omits unavailable duration and token telemetry instead of writing zero placeholders, +- Cline and GitHub Copilot runners added to the eval runner lineup alongside existing Codex and OpenCode support, +- Runner conformance tests enhanced with additional event fixtures and isolation capability assessment. + +### Fixed + +- Codex runner now ensures the evidence directory is created before writing output files, preventing file-not-found errors, +- Test runner conformance validation now requires the output parent directory to exist upfront with explicit error reporting when the directory structure is misconfigured. ## [0.9.0] - 2026-08-20 @@ -627,6 +637,7 @@ This is a minor release that introduces two complementary git workflow skills, e - Improved scaffold fidelity with hidden `.bot` asset preservation, explicit UTF-8 and BOM handling, and checks aimed at preventing mojibake or incomplete generated output. +[0.9.1]: https://github.com/codebeltnet/agentic/compare/v0.9.0...v0.9.1 [0.9.0]: https://github.com/codebeltnet/agentic/compare/v0.8.2...v0.9.0 [0.8.2]: https://github.com/codebeltnet/agentic/compare/v0.8.1...v0.8.2 [0.8.1]: https://github.com/codebeltnet/agentic/compare/v0.8.0...v0.8.1 From 60412e43b543299d20af6b6a456581292db952bc Mon Sep 17 00:00:00 2001 From: "aicia[bot]" Date: Sat, 22 Aug 2026 17:59:52 +0200 Subject: [PATCH 15/39] =?UTF-8?q?=F0=9F=93=9D=20clarify=20repository=20pol?= =?UTF-8?q?icy=20and=20evaluation=20methodology?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Updated repository guidelines, contributor expectations, and evaluation process documentation to distinguish Eval Runners, Orchestrators, and Graders while clarifying the explicit external-handoff boundary. All 21 repo-managed skill descriptions refactored for trigger-oriented activation metadata following progressive disclosure. Improved section organization in AGENTS.md for skill authoring, form handling, and dynamic defaults. --- AGENTS.md | 22 ++++++++++++---------- CONTRIBUTING.md | 10 +++++----- README.md | 6 +++--- 3 files changed, 20 insertions(+), 18 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index a41c85b..94efa77 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -40,24 +40,26 @@ This rule is Priority 1. If another repository rule, skill, test, or completion Anthropic's `skill-creator` owns the evaluation methodology this repository uses: define evals, run each task once with the skill and once without it, hold the model, the environment, the task, and the inputs constant, then compare. Keep that experimental design. Only the execution transport changes here. -Where `skill-creator` says to spawn with-skill and baseline subagents in the same turn, this repository prepares a portable evaluation package and stops. The package keeps the existing paired methodology: `run.json` defines what one blind arm executes, `execution-profile.json` selects the runner/model/configuration, and the Eval Runner defines how its harness satisfies the contract. The user selects the runner, provider, and model, then hands `RUN-THIS.prompt.md` to an external Eval Orchestrator. That orchestrator preflights and invokes one fresh runner process per arm, bridges raw `execution-result.json` evidence into the existing result shape, grades only after execution, invokes the packaged Anthropic `skill-creator` aggregator and static viewer, and returns the finished reports. Preparation, collection, validation, and reporting remain deterministic and never invoke a model. +Where `skill-creator` says to spawn with-skill and baseline subagents in the same turn, this repository prepares a portable evaluation package and stops. The package keeps the existing paired methodology: `run.json` defines what one blind arm executes, `execution-profile.json` selects the runner/model/configuration, and the Eval Runner defines how its harness satisfies the contract. Before an execution-ready `RUN-THIS.prompt.md` is emitted, the user-facing preparation flow resolves a Harness + Model choice; the portable profile stores the internal runner id and the opaque runner-native model selector. The external Eval Orchestrator never chooses runner or model policy. It preflights and invokes one fresh runner process per arm, bridges raw `execution-result.json` evidence into the existing result shape, grades only after execution, invokes the packaged Anthropic `skill-creator` aggregator and static viewer, and returns the finished reports. Preparation, collection, validation, and reporting remain deterministic and never invoke a model. ### Asking for an eval `eval `, `evaluate `, `eval this skill`, `prepare evals for `, and `evaluate using the existing evals` are all requests for this workflow. Treat them as instructions to prepare the package, never to run it, and never as a request to write new eval cases unless the user asks for that too. -Run the script immediately when asked. Do not reply with a plan, a menu of options, or a question about which harness or model the user wants; the harness and model are chosen after the package exists, by the user, outside this repository. +Resolve the execution configuration before running the package preparation script. In an interactive agent session, offer Codebelt Reference first (`GitHub Copilot CLI` + `claude-haiku-4.5`) and verify that model through `scripts/Get-HarnessModels.ps1`; if it is unavailable, show the current discovered Copilot models and ask for a replacement. For manual selection, ask for Harness, discover current models for that harness with `scripts/Get-HarnessModels.ps1`, then pass the resulting runner/model pair to the preparation script. Cline and OpenCode discovery is free-only; GitHub Copilot and Codex discovery lists all currently available models. Never guess stale model ids, silently switch harnesses, or generate an execution-ready package with a null runner or model. ``` -pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -Skill dotnet-test +pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -Skill dotnet-test -Runner github-copilot -Model claude-haiku-4.5 ``` `eval` with no skill named, or `eval changed`, means the whole changed set: ``` -pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -Changed +pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -Changed -Runner github-copilot -Model claude-haiku-4.5 ``` +Use `-CodebeltReference` only when the script should perform the dynamic Copilot catalog check itself and fail if `claude-haiku-4.5` is no longer present. For noninteractive direct script use, omitting both `-Runner/-Model` and `-CodebeltReference` is an error whenever a package would be generated. + ### Handing the package over Every package contains `RUN-THIS.prompt.md`, one instruction that drives the whole thing. It makes the user-selected external agent the Eval Orchestrator, Grader, and report producer. The orchestrator resolves the selected Eval Runner, preflights it, invokes it once for every blind `with_skill` and `without_skill` arm, records normalized results and available metrics, grades only after collection, writes the grading fields, and generates the static report without executing an eval prompt in its own context. @@ -70,10 +72,10 @@ The normal path ends in the external evaluator: after all workers finish, it rea ### Prepare, do not execute -Generate the package with the repository script rather than by hand: +Generate the package with the repository script rather than by hand after resolving the Harness + Model choice: ``` -pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -Skill +pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -Skill -Runner -Model ``` It reads `skills//evals/evals.json` and writes one directory per eval into `.bot/-workspace/iteration-/`. The grading key and result stubs stay at the eval-case level, outside the two isolated run directories a worker actually sees: @@ -96,7 +98,7 @@ Adding or modifying any repo-managed skill triggers this workflow. It is not som After the final skill edit is in place, run: ``` -pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -Changed +pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -Changed -Runner -Model ``` It resolves every repo-managed skill this branch changed, uncommitted work included, and prepares a package for each. With no skill changed it says so and exits clean, which satisfies the gate. @@ -154,7 +156,7 @@ A meaningful A/B result requires both configurations to run on the same model, t ### Result handoff -An externally produced result comes back identified by eval id, configuration (`with_skill` or `without_skill`), model and provider, and the produced output. It may also carry the transcript, duration, total tokens, tool-call count, output files, and notes. The user can hand it over as filled-in `results/*.result.json` files, or state it in chat and let the agent fill them in. +An externally produced result comes back identified by eval id, configuration (`with_skill` or `without_skill`), runner-native model, harness, and the produced output. It may also carry the transcript, duration, total tokens, tool-call count, output files, and notes. The user can hand it over as filled-in `results/*.result.json` files, or state it in chat and let the agent fill them in. Which artifact transfer happens depends on where the harness ran, and `RUN-THIS.prompt.md` tells it to close either way. A harness sharing a disk with the package writes the result files, grading, `benchmark.json`, `benchmark.md`, the first-party `report.html`, and the exact upstream `skill-creator-report.html` itself and reports the first-party report path. A harness that does not - a different product, a browser, or a sandbox - ends with one paste-ready block carrying the package path and every completed result object, including grading, plus the reports as file artifacts when supported. A repository session can use `-CollectResults` only as a fallback for transferred results that lack the report artifacts. "Bring the results back" means those artifacts, never a prose recap of how the runs went. @@ -185,7 +187,7 @@ Every repo-managed skill must include its own `evals/evals.json` file at `skills - To compare a skill against a baseline, prepare a package with **Portable Eval Handoff** and hand `RUN-THIS.prompt.md` to the user; the repository agent never runs the prompts, while the user-directed external executor runs, grades, and reports the paired comparison - Deterministic scaffold/template skills must keep local deterministic validators as well; evals supplement validators, they do not replace them -If you add a new skill or modify an existing repo-managed skill, update that skill's `evals/evals.json` and run `pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -Changed` before considering the work complete. Do not commit temp workspaces, benchmark outputs, or generated review files into this repository unless the user explicitly asks for checked-in artifacts. +If you add a new skill or modify an existing repo-managed skill, update that skill's `evals/evals.json` and run `pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -Changed -Runner -Model ` before considering the work complete. Use `-CodebeltReference` instead only after its dynamic Copilot model check passes. Do not commit temp workspaces, benchmark outputs, or generated review files into this repository unless the user explicitly asks for checked-in artifacts. ## Git Identity @@ -315,7 +317,7 @@ Before any completion message, reread the skill instructions and the current con For script-backed workflows, creating or editing files is not enough on its own. If a skill requires deterministic maintenance or verification commands, run them before completion and report their concrete outcome. For `dotnet-docfx-digest`, `scripts/agents.cs` and `scripts/docfx.cs --build-api-model --validate-samples --verify-docfx-build` are blocking completion gates whenever the skill or task summary says they are required. -Whenever a repo-managed skill was edited, two gates apply in a fixed order. `pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -Changed` runs first and prepares the eval packages for the changed skills, reporting the prompt paths. `scripts/sync-skill-install.ps1` runs last, because every other step can still change a file. Report the actual output of both; an earlier run in the same session satisfies neither. See [Eval preparation is a completion gate](#eval-preparation-is-a-completion-gate) and [Local Install Sync](#local-install-sync). +Whenever a repo-managed skill was edited, two gates apply in a fixed order. `pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -Changed -Runner -Model ` (or `-CodebeltReference` after dynamic availability verification) runs first and prepares the eval packages for the changed skills, reporting the prompt paths. `scripts/sync-skill-install.ps1` runs last, because every other step can still change a file. Report the actual output of both; an earlier run in the same session satisfies neither. See [Eval preparation is a completion gate](#eval-preparation-is-a-completion-gate) and [Local Install Sync](#local-install-sync). ## User Input UX diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index e075545..3d1dc3d 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -89,22 +89,22 @@ Aim for 3–5 evals that cover distinct scenarios: happy path, edge cases, and c Evals are prepared, not executed, from this repository. Adding or modifying a repo-managed skill requires preparing the packages for every skill the branch touched, which is a completion gate rather than an optional extra: ```console -pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -Changed +pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -Changed -Runner github-copilot -Model claude-haiku-4.5 ``` Run it after the last skill edit and before `scripts/sync-skill-install.ps1`, which stays last. For a single skill on demand, use: ```console -pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -Skill +pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -Skill -Runner -Model ``` -The script writes `.bot/-workspace/iteration-/` with one directory per eval. Each holds the grading key `eval-metadata.json` and result stubs under `results/` at the eval-case level, plus two paired run directories, `with_skill/` and `without_skill/`. A run directory is the worker's run root: `prompt.md`, a `run.json` contract, a `repo/` working tree materialized from the fixtures, an isolated `home/`, and - for `with_skill` only - a `skill//` copy of the candidate. The grading key and results sit outside both run directories. At the root it writes `manifest.json`, `execution-profile.json`, the package-local Eval Runner protocol, the package report adapter, the exact Anthropic skill-creator grader/aggregator/viewer assets, and `RUN-THIS.prompt.md`, the one prompt you hand to the external Eval Orchestrator. That orchestrator resolves and preflights the selected runner, invokes its common `describe`/`preflight`/`execute` surface once per fresh blind arm, stores raw `execution-result.json`, bridges the results, grades only after execution, and runs the adapter, which invokes `aggregate_benchmark.py` and `eval-viewer/generate_review.py --static`. It never runs an eval prompt in the coordinator context and never reuses a worker. Both worker prompts carry the same task, materialized repository, and response contract; only the operating instructions and the presence of `skill/` differ, and neither prompt identifies itself as an eval. `.gitignore` covers `.bot/*`, so nothing there reaches git. The script refuses an `-OutputRoot` inside the repository but outside `.bot/`; pass an explicit temp path when the harness does not need repository-local storage. +Before running the script, choose a Harness + Model. Use `scripts/Get-HarnessModels.ps1 -Runner ` to list current selectors; Cline and OpenCode are free-only, while GitHub Copilot and Codex list all currently available models. The Codebelt Reference shortcut is GitHub Copilot CLI + `claude-haiku-4.5`; `-CodebeltReference` verifies that the model still exists and fails instead of silently substituting a different model. The script writes `.bot/-workspace/iteration-/` with one directory per eval. Each holds the grading key `eval-metadata.json` and result stubs under `results/` at the eval-case level, plus two paired run directories, `with_skill/` and `without_skill/`. A run directory is the worker's run root: `prompt.md`, a `run.json` contract, a `repo/` working tree materialized from the fixtures, an isolated `home/`, and - for `with_skill` only - a `skill//` copy of the candidate. The grading key and results sit outside both run directories. At the root it writes `manifest.json`, `execution-profile.json`, the package-local Eval Runner protocol, the package report adapter, the exact Anthropic skill-creator grader/aggregator/viewer assets, and `RUN-THIS.prompt.md`, the one prompt you hand to the external Eval Orchestrator. That orchestrator resolves and preflights the selected runner, invokes its common `describe`/`preflight`/`execute` surface once per fresh blind arm, stores raw `execution-result.json`, bridges the results, grades only after execution, and runs the adapter, which invokes `aggregate_benchmark.py` and `eval-viewer/generate_review.py --static`. It never runs an eval prompt in the coordinator context, never chooses runner/model policy, and never reuses a worker. Both worker prompts carry the same task, materialized repository, and response contract; only the operating instructions and the presence of `skill/` differ, and neither prompt identifies itself as an eval. `.gitignore` covers `.bot/*`, so nothing there reaches git. The script refuses an `-OutputRoot` inside the repository but outside `.bot/`; pass an explicit temp path when the harness does not need repository-local storage. Repository preparation, validation, CI, hooks, deterministic tests, and automatic completion gates never run those prompts or invoke a model. That boundary is the Priority 1 rule in `AGENTS.md`, and preparing a prompt is not permission to execute one. A human-selected external Eval Orchestrator handed a specific package may invoke the selected package-local Eval Runner; this explicit handoff boundary does not weaken the repository prohibition or authorize CI/live evals. Run both configurations on the same model, same version, and same configuration. A with-skill run on one model against a baseline on another measures the model as much as the skill and is not a skill-effectiveness result. -Record each external runner result in the matching `execution-result.json` first. It must carry the protocol/schema, opaque run and fresh session ids, status, complete final response or explicit unavailability, runner/harness identity, requested and resolved execution selection, timestamps and duration, exit/failure state, prompt/run/profile hashes, resolved isolation mechanisms, warnings, and artifact references. Include token, cache, cost, tool, command, file, and transcript evidence only when the harness exposes it; unavailable values remain explicitly unavailable and are never estimated. The deterministic bridge then writes the existing `results/*.result.json` shape, after which grading may add `grading[].passed` and evidence. Assertions about tool, shell, or file behavior are only gradeable from a run that captured that evidence. If the results were transferred without those report artifacts, validate and compare with: +Record each external runner result in the matching `execution-result.json` first. It must carry the protocol/schema, opaque run and fresh session ids, status, complete final response or explicit unavailability, runner/harness identity, requested and resolved model selection, timestamps and duration, exit/failure state, prompt/run/profile hashes, resolved isolation mechanisms, warnings, and artifact references. Include token, cache, cost, tool, command, file, and transcript evidence only when the harness exposes it; unavailable values remain explicitly unavailable and are never estimated. The deterministic bridge then writes the existing `results/*.result.json` shape, after which grading may add `grading[].passed` and evidence. Assertions about tool, shell, or file behavior are only gradeable from a run that captured that evidence. If the results were transferred without those report artifacts, validate and compare with: ```console pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -CollectResults @@ -147,7 +147,7 @@ pwsh -NoProfile -File ./scripts/validate-skill-templates.ps1 -Ref HEAD - [ ] At least one eval in `evals/evals.json` - [ ] The skill's `evals/evals.json` exists and its `skill_name` matches the folder/frontmatter name - [ ] Any optional `files` entries in `evals/evals.json` point to real fixture files under the same skill folder -- [ ] `pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -Changed` was run after the last skill edit, and the prepared prompt paths were reported +- [ ] `pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -Changed -Runner -Model ` or `-CodebeltReference` was run after the last skill edit, and the prepared prompt paths were reported - [ ] If an external evaluation was run, each result includes the producing model and the package contains the first-party `report.html`, exact upstream `skill-creator-report.html`, `benchmark.json`, and `benchmark.md`; use `-CollectResults` only when transferred results need the repository-side fallback - [ ] `scripts/validate-skill-templates.ps1` passes for the current working tree when changing scaffold or template behavior - [ ] If CI is enabled for the branch, the GitHub Actions validation job passes too diff --git a/README.md b/README.md index 461f81e..7ee721e 100644 --- a/README.md +++ b/README.md @@ -14,13 +14,13 @@ Another repo rule is intentionally strict: every repo-managed skill ships with i Skill validation is local and deterministic. The Priority 1 **AI/LLM Evaluation Automation Prohibition** in `AGENTS.md` forbids repository preparation, validation, CI, hooks, deterministic tests, automatic fan-out, graders, and completion gates from using an authenticated Copilot, Claude, Codex, Gemini, or other model account. There is no repository opt-in switch. A human-selected external Eval Orchestrator may invoke an explicitly selected package-local Eval Runner for a package it was handed; that boundary never authorizes live model execution in CI or automatic repository workflows. -Evaluation keeps Anthropic's `skill-creator` methodology and portable paired-run conventions while replacing only the execution transport. `run.json` remains the runner-neutral one-arm contract; `execution-profile.json` selects the runner/provider/model/configuration; and `execution-result.json` normalizes one blind execution before the existing `eval-result/2` bridge and reports. The conforming real runners are GitHub Copilot, Codex, OpenCode, and Cline, with a deterministic fake runner used for conformance. GitHub Copilot with `claude-haiku-4.5` is the Codebelt reference evaluation configuration — a repository convention for economical, stable comparison, not an Anthropic default — and the model stays configurable through `execution-profile.json`. The repository prepares the paired candidate and baseline inputs as a portable package and stops; the external Eval Orchestrator resolves, preflights, and invokes the selected Eval Runner once per fresh arm, then grades and reports. Hard filesystem confinement raises reported isolation confidence from pragmatic to strict but is not a universal platform prerequisite, so Windows is a first-class pragmatic target; mandatory experimental controls remain fail-closed. Freebuff remains planned/blocked until it exposes a supported noninteractive machine-readable transport. No provider has a privileged runtime role, and native skill activation is not evaluated in v0.9.1. Adding or modifying a skill triggers package preparation automatically, as a completion gate an agent cannot skip: +Evaluation keeps Anthropic's `skill-creator` methodology and portable paired-run conventions while replacing only the execution transport. `run.json` remains the runner-neutral one-arm contract; `execution-profile.json` selects the runner/model/configuration; and `execution-result.json` normalizes one blind execution before the existing `eval-result/2` bridge and reports. The user-facing preparation flow asks for Harness + Model before emitting `RUN-THIS.prompt.md`; the portable profile stores the internal runner id and the opaque runner-native model selector, with no provider field. The conforming real runners are GitHub Copilot, Codex, OpenCode, and Cline, with a deterministic fake runner used for conformance. GitHub Copilot CLI with `claude-haiku-4.5` is the Codebelt Reference evaluation configuration — a repository convention for economical, stable comparison, not an Anthropic default — and its availability is verified against the current Copilot model catalog before automatic selection. Cline and OpenCode discovery lists only currently free models; GitHub Copilot and Codex discovery lists all currently available models. The repository prepares the paired candidate and baseline inputs as a portable package and stops; the external Eval Orchestrator resolves, preflights, and invokes the selected Eval Runner once per fresh arm, then grades and reports. Hard filesystem confinement raises reported isolation confidence from pragmatic to strict but is not a universal platform prerequisite, so Windows is a first-class pragmatic target; mandatory experimental controls remain fail-closed. Freebuff remains planned/blocked until it exposes a supported noninteractive machine-readable transport. Native skill activation is not evaluated in v0.9.1. Adding or modifying a skill triggers package preparation automatically, as a completion gate an agent cannot skip: ```powershell -pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -Changed +pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -Changed -Runner github-copilot -Model claude-haiku-4.5 ``` -That resolves every skill the branch changed and prepares a package for each. `-Skill ` prepares one on demand. Packages land in the gitignored `.bot/-workspace/`, so a harness that refuses to work outside the repository folder can still reach them without anything entering the working tree. +That resolves every skill the branch changed and prepares a package for each. `-Skill ` prepares one on demand, but execution selection must already be resolved; direct noninteractive use without `-Runner/-Model` or `-CodebeltReference` fails before a handoff is generated. `scripts/Get-HarnessModels.ps1 -Runner ` lists current model selectors for the selected harness. Use `-CodebeltReference` only when you want the script to verify the current Copilot catalog and select GitHub Copilot CLI + `claude-haiku-4.5`; if that model is absent, the script fails and prints the current choices rather than substituting another model. Packages land in the gitignored `.bot/-workspace/`, so a harness that refuses to work outside the repository folder can still reach them without anything entering the working tree. Each eval becomes a directory holding the grading key (`eval-metadata.json` with the expected output, assertions, and fixture and skill hashes) and prefilled result stubs, plus two paired run directories. `with_skill/` is a self-contained run root: a `prompt.md` with the effective skill instructions inlined, a `run.json` contract naming only paths inside the run, a `repo/` working tree materialized from the fixtures as real files, an isolated empty `home/`, and a `skill//` copy of the exact candidate revision. `without_skill/` is the same run with a byte-identical `repo/`, no `skill/` directory, and no mention of the skill. The grading key and results sit outside both run directories, so workers are not intentionally given them. Neither prompt identifies itself as an eval or names its configuration. `RUN-THIS.prompt.md` makes the user-selected agent the Eval Orchestrator: it reads the profile, resolves the selected runner, validates `describe`, preflights each `run.json`, invokes one fresh runner process per arm, stores raw `execution-result.json`, bridges it into the existing result shape, reveals the grading key only after execution, grades, and invokes the package adapter. The adapter stages the results into Anthropic's upstream benchmark workspace, runs `aggregate_benchmark.py`, writes the exact upstream `skill-creator-report.html`, and writes a first-party `report.html` with paired outputs, expected outcomes, assertion evidence, telemetry, transcripts, and downloadable feedback, plus `benchmark.json` and `benchmark.md`. Missing telemetry is displayed as unavailable rather than estimated. A runner that cannot satisfy the mandatory experimental controls returns `incompatible`; lack of hard filesystem confinement downgrades the result to pragmatic isolation. `-CollectResults ` remains a deterministic fallback for transferred artifacts without reports. Packages land in gitignored `.bot/` storage by default and are not committed. From e05337dc5e2aab363b0d81cab865bf645c686e3f Mon Sep 17 00:00:00 2001 From: "aicia[bot]" Date: Sat, 22 Aug 2026 18:00:01 +0200 Subject: [PATCH 16/39] =?UTF-8?q?=F0=9F=92=AC=20update=20changelog=20for?= =?UTF-8?q?=20evaluation=20infrastructure=20improvements?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Added entry documenting the harness model resolution workflow, removal of the redundant provider field from execution-profile.json, and introduction of scripts/Get-HarnessModels.ps1 for current model discovery with Codebelt Reference verification and platform-specific filtering. --- CHANGELOG.md | 1 + 1 file changed, 1 insertion(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 7403b5e..1a7da4f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,7 @@ This patch release adds harness-agnostic Eval Runner execution boundary infrastr ### Changed +- Eval preparation now resolves Harness + Model before writing `RUN-THIS.prompt.md`, removes the redundant portable `provider` field from `execution-profile.json` and result reporting, treats model selectors as runner-native opaque strings, and adds `scripts/Get-HarnessModels.ps1` for current model discovery with Codebelt Reference verification and free-only Cline/OpenCode filtering, - `AGENTS.md`, `README.md`, and `CONTRIBUTING.md` now distinguish the Eval Runner, Eval Orchestrator, Grader, and Human Reviewer and clarify that runner execution is an explicit external-handoff boundary rather than repository automation, - All 21 repo-managed skill descriptions refactored to lean, trigger-oriented activation metadata following progressive-disclosure principles and specification compliance, - `AGENTS.md` Skill Authoring section restructured for clarity, brevity, and progressive disclosure of form fields, asset handling, and dynamic defaults, From 7b3bc553b76c77c07471d61a0fe084739ddb5ce1 Mon Sep 17 00:00:00 2001 From: "aicia[bot]" Date: Sat, 22 Aug 2026 18:00:12 +0200 Subject: [PATCH 17/39] =?UTF-8?q?=F0=9F=94=A8=20update=20evaluation=20runn?= =?UTF-8?q?er=20infrastructure=20and=20contracts?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Modernized eval runner implementations across all harness types (GitHub Copilot, Codex, OpenCode, Cline) with updated contract schemas. Added scripts/Get-HarnessModels.ps1 for discovering current model selectors per harness with Codebelt Reference verification and platform-specific filtering. Improved prepare-skill-evals.ps1 to resolve Harness + Model before package generation, updated validation and conformance testing, and refined execution-profile.json to remove redundant provider field while treating model selectors as runner-native opaque strings. --- scripts/Get-HarnessModels.ps1 | 498 ++++++++++++++++++ scripts/eval-report-template.html | 4 +- scripts/eval-runners/README.md | 48 +- .../eval-runners/bridge-execution-result.ps1 | 6 - scripts/eval-runners/cline/runner.ps1 | 49 +- scripts/eval-runners/codex/runner.ps1 | 19 +- .../contracts/execution-profile.schema.json | 6 +- .../contracts/execution-result.schema.json | 37 +- .../contracts/preflight-result.schema.json | 18 + .../eval-runners/github-copilot/runner.ps1 | 147 +++--- scripts/eval-runners/opencode/runner.ps1 | 51 +- scripts/eval-runners/runner-common.ps1 | 20 +- .../tests/test-runner-conformance.ps1 | 78 ++- scripts/generate-eval-report.ps1 | 8 - scripts/prepare-skill-evals.ps1 | 160 +++++- scripts/validate-skill-templates.ps1 | 121 ++++- 16 files changed, 1053 insertions(+), 217 deletions(-) create mode 100644 scripts/Get-HarnessModels.ps1 diff --git a/scripts/Get-HarnessModels.ps1 b/scripts/Get-HarnessModels.ps1 new file mode 100644 index 0000000..93a5f7a --- /dev/null +++ b/scripts/Get-HarnessModels.ps1 @@ -0,0 +1,498 @@ +<# +.SYNOPSIS + Lists current model selectors for a supported eval harness. + +.DESCRIPTION + Discovers runner-native model selectors without executing model requests. GitHub Copilot and Codex return every + model the harness exposes. Cline and OpenCode return only models whose current catalog metadata proves free + availability. Discovery failures are local to the selected harness and never fall back to stale hardcoded catalogs. + +.PARAMETER Runner + Internal Eval Runner id: github-copilot, codex, opencode, or cline. + +.PARAMETER CatalogPath + Optional deterministic catalog fixture used by tests. When supplied, no harness command is invoked. + +.PARAMETER RequireModel + Optional model selector that must exist in the discovered list. + +.PARAMETER Refresh + For harnesses that support explicit catalog refresh, request a refresh before listing. This never executes a model + prompt. +#> +[CmdletBinding()] +param( + [Parameter(Mandatory = $true)] + [ValidateSet('github-copilot', 'codex', 'opencode', 'cline')] + [string]$Runner, + + [string]$CatalogPath, + + [string]$RequireModel, + + [switch]$Refresh +) + +$ErrorActionPreference = 'Stop' +Set-StrictMode -Version Latest + +$runnerCommon = Join-Path $PSScriptRoot 'eval-runners/runner-common.ps1' +. $runnerCommon + +$utf8NoBom = [System.Text.UTF8Encoding]::new($false) + +function Get-HarnessDisplayName { + param([Parameter(Mandatory = $true)][string]$RunnerName) + + switch ($RunnerName) { + 'github-copilot' { return 'GitHub Copilot CLI' } + 'codex' { return 'Codex CLI' } + 'opencode' { return 'OpenCode' } + 'cline' { return 'Cline' } + default { return $RunnerName } + } +} + +function Get-PolicyName { + param([Parameter(Mandatory = $true)][string]$RunnerName) + + if ($RunnerName -in @('cline', 'opencode')) { + return 'free' + } + return 'all' +} + +function Read-CatalogJson { + param([Parameter(Mandatory = $true)][string]$Path) + + if (-not (Test-Path -LiteralPath $Path -PathType Leaf)) { + throw "Catalog fixture '$Path' does not exist." + } + return [System.IO.File]::ReadAllText((Resolve-Path -LiteralPath $Path).Path, $utf8NoBom) | ConvertFrom-Json +} + +function Get-FirstPropertyValue { + param( + [object]$Object, + [string[]]$Names + ) + + foreach ($name in $Names) { + $value = Get-JsonProperty -Object $Object -Name $name -Default $null + if ($null -ne $value -and -not [string]::IsNullOrWhiteSpace([string]$value)) { + return [string]$value + } + } + return $null +} + +function Get-NumericPropertyValues { + param([object]$Object) + + $values = [System.Collections.Generic.List[double]]::new() + if ($null -eq $Object) { + return @() + } + + if ($Object -is [System.Collections.IEnumerable] -and -not ($Object -is [string]) -and -not ($Object -is [System.Collections.IDictionary]) -and -not ($Object -is [pscustomobject])) { + foreach ($item in $Object) { + foreach ($child in @(Get-NumericPropertyValues -Object $item)) { $values.Add([double]$child) } + } + return @($values) + } + + if ($Object -is [System.Collections.IDictionary]) { + $propertyNames = @($Object.Keys) + foreach ($name in $propertyNames) { + $value = $Object[$name] + if ($value -is [int] -or $value -is [long] -or $value -is [double] -or $value -is [decimal]) { + $values.Add([double]$value) + } elseif ($null -ne $value -and -not ($value -is [string]) -and -not ($value -is [System.ValueType]) -and ($value -is [System.Collections.IDictionary] -or $value -is [pscustomobject] -or $value -is [System.Collections.IEnumerable])) { + foreach ($child in @(Get-NumericPropertyValues -Object $value)) { $values.Add([double]$child) } + } + } + return @($values) + } + + foreach ($property in @($Object.PSObject.Properties)) { + $value = $property.Value + if ($value -is [int] -or $value -is [long] -or $value -is [double] -or $value -is [decimal]) { + $values.Add([double]$value) + } elseif ($null -ne $value -and -not ($value -is [string]) -and -not ($value -is [System.ValueType]) -and ($value -is [System.Collections.IDictionary] -or $value -is [pscustomobject] -or $value -is [System.Collections.IEnumerable])) { + foreach ($child in @(Get-NumericPropertyValues -Object $value)) { $values.Add([double]$child) } + } + } + return @($values) +} + +function Get-ModelAvailability { + param([object]$Model) + + $explicit = Get-FirstPropertyValue -Object $Model -Names @('availability', 'billing', 'usageCostDisplay') + if (-not [string]::IsNullOrWhiteSpace($explicit)) { + $normalized = $explicit.ToLowerInvariant() + if ($normalized -eq 'free') { return 'free' } + if ($normalized -in @('paid', 'subscription', 'metered')) { return 'paid' } + } + + foreach ($name in @('free', 'isFree')) { + $value = Get-JsonProperty -Object $Model -Name $name -Default $null + if ($null -eq $value) { + continue + } + if ([bool]$value) { return 'free' } + return 'paid' + } + + foreach ($propertyName in @('pricing', 'cost')) { + $price = Get-JsonProperty -Object $Model -Name $propertyName -Default $null + $numbers = @(Get-NumericPropertyValues -Object $price) + if ($numbers.Count -gt 0) { + if (@($numbers | Where-Object { [double]$_ -gt 0 }).Count -gt 0) { + return 'paid' + } + return 'free' + } + } + + return 'unknown' +} + +function Test-TextModel { + param([object]$Model) + + $operation = [string](Get-JsonProperty -Object $Model -Name 'operation' -Default '') + if (-not [string]::IsNullOrWhiteSpace($operation) -and $operation -notin @('language', 'chat', 'completion')) { + return $false + } + return $true +} + +function ConvertTo-ModelChoice { + param( + [Parameter(Mandatory = $true)][object]$Model, + [Parameter(Mandatory = $true)][string]$RunnerName, + [string]$Source, + [string]$ExplicitSelector + ) + + if (-not (Test-TextModel -Model $Model)) { + return $null + } + + $id = if ([string]::IsNullOrWhiteSpace($ExplicitSelector)) { + Get-FirstPropertyValue -Object $Model -Names @('id', 'slug', 'model', 'modelId', 'model_id') + } else { + $ExplicitSelector + } + if ([string]::IsNullOrWhiteSpace($id)) { + return $null + } + + $provider = Get-FirstPropertyValue -Object $Model -Names @('providerID', 'providerId', 'provider') + if ($RunnerName -in @('cline', 'opencode') -and $id -notmatch '/' -and -not [string]::IsNullOrWhiteSpace($provider)) { + $id = "$provider/$id" + } + + $displayName = Get-FirstPropertyValue -Object $Model -Names @('display_name', 'displayName', 'name') + if ([string]::IsNullOrWhiteSpace($displayName)) { + $displayName = $id + } + + return [pscustomobject][ordered]@{ + id = $id + display_name = $displayName + availability = Get-ModelAvailability -Model $Model + source = $Source + } +} + +function ConvertTo-ModelChoices { + param( + [Parameter(Mandatory = $true)][object]$Catalog, + [Parameter(Mandatory = $true)][string]$RunnerName, + [Parameter(Mandatory = $true)][string]$Source + ) + + $rawModels = [System.Collections.Generic.List[object]]::new() + if ($Catalog -is [array]) { + foreach ($model in $Catalog) { $rawModels.Add($model) } + } elseif (Test-JsonProperty -Object $Catalog -Name 'models') { + foreach ($model in @($Catalog.models)) { $rawModels.Add($model) } + } else { + foreach ($propertyName in @(Get-JsonPropertyNames -Object $Catalog)) { + $value = Get-JsonProperty -Object $Catalog -Name $propertyName -Default $null + if ($null -ne $value -and @($value.PSObject.Properties).Count -gt 0) { + if (-not (Test-JsonProperty -Object $value -Name 'id')) { + $value | Add-Member -NotePropertyName id -NotePropertyValue $propertyName -Force + } + $rawModels.Add($value) + } + } + } + + $choices = [System.Collections.Generic.List[object]]::new() + foreach ($model in $rawModels) { + $choice = ConvertTo-ModelChoice -Model $model -RunnerName $RunnerName -Source $Source + if ($null -ne $choice) { + $choices.Add($choice) + } + } + $seen = @{} + $deduped = [System.Collections.Generic.List[object]]::new() + foreach ($choice in @($choices | Sort-Object id)) { + if (-not $seen.ContainsKey([string]$choice.id)) { + $seen[[string]$choice.id] = $true + $deduped.Add($choice) + } + } + return @($deduped) +} + +function Select-ModelsByPolicy { + param( + [object[]]$Models, + [Parameter(Mandatory = $true)][string]$RunnerName + ) + + $policy = Get-PolicyName -RunnerName $RunnerName + if ($policy -eq 'free') { + $freeModels = @($Models | Where-Object { [string]$_.availability -eq 'free' }) + if ($freeModels.Count -eq 0) { + throw "No free $((Get-HarnessDisplayName -RunnerName $RunnerName)) models are currently available from discovery. Choose another harness or update the harness catalog." + } + return @($freeModels) + } + + return @($Models) +} + +function Invoke-JsonCommand { + param( + [Parameter(Mandatory = $true)][object]$CommandInfo, + [Parameter(Mandatory = $true)][string[]]$Arguments, + [int]$TimeoutSeconds = 60 + ) + + $work = Join-Path ([System.IO.Path]::GetTempPath()) ('agentic-model-discovery-' + [Guid]::NewGuid().ToString('N')) + New-Item -ItemType Directory -Path $work -Force | Out-Null + try { + $environment = New-RunnerProbeEnvironment + foreach ($name in @('HOME', 'USERPROFILE', 'APPDATA', 'LOCALAPPDATA', 'XDG_CONFIG_HOME')) { + $value = [Environment]::GetEnvironmentVariable($name) + if (-not [string]::IsNullOrWhiteSpace($value)) { + $environment[$name] = $value + } + } + $process = Invoke-RunnerProcess -FileName $CommandInfo.FileName -ArgumentList (@($CommandInfo.Prefix) + @($Arguments)) -WorkingDirectory $work -Environment $environment -TimeoutSeconds $TimeoutSeconds + if ($process.TimedOut) { + throw "Command '$($CommandInfo.Source)' timed out during model discovery." + } + if ($process.ExitCode -ne 0) { + $detail = [string]::Join("`n", @($process.Stdout, $process.Stderr)).Trim() + throw "Command '$($CommandInfo.Source) $($Arguments -join ' ')' failed during model discovery with exit code $($process.ExitCode). $detail" + } + return [pscustomobject]@{ Stdout = $process.Stdout; Stderr = $process.Stderr } + } finally { + if (Test-Path -LiteralPath $work) { + Remove-Item -LiteralPath $work -Recurse -Force -ErrorAction SilentlyContinue + } + } +} + +function ConvertFrom-OpenCodeTextCatalog { + param([Parameter(Mandatory = $true)][string]$Text) + + $models = [System.Collections.Generic.List[object]]::new() + $lines = $Text -split "`r?`n" + $currentSelector = $null + $buffer = [System.Collections.Generic.List[string]]::new() + $depth = 0 + foreach ($line in $lines) { + $trimmed = $line.Trim() + if ([string]::IsNullOrWhiteSpace($trimmed)) { + continue + } + if ($depth -eq 0 -and $trimmed -match '^[^\s/]+/.+$') { + $currentSelector = $trimmed + continue + } + if ($trimmed.StartsWith('{') -or $depth -gt 0) { + $buffer.Add($line) + $depth += ([regex]::Matches($line, '\{')).Count + $depth -= ([regex]::Matches($line, '\}')).Count + if ($depth -le 0 -and $buffer.Count -gt 0) { + $json = [string]::Join("`n", @($buffer)) + $object = $json | ConvertFrom-Json + $choice = ConvertTo-ModelChoice -Model $object -RunnerName 'opencode' -Source 'opencode models --verbose' -ExplicitSelector $currentSelector + if ($null -ne $choice) { $models.Add($choice) } + $buffer.Clear() + $depth = 0 + $currentSelector = $null + } + } + } + return @($models) +} + +function Resolve-ClineModelsModulePath { + $command = Resolve-ExternalCommand -Name 'cline' + if ($null -eq $command) { + throw 'Cline CLI executable is not available on PATH.' + } + + $source = [string]$command.Source + $directory = Split-Path -Parent $source + $candidates = @( + (Join-Path $directory 'node_modules/cline/node_modules/@cline/llms/dist/models.js'), + (Join-Path $directory '../lib/node_modules/cline/node_modules/@cline/llms/dist/models.js'), + (Join-Path $directory '../node_modules/cline/node_modules/@cline/llms/dist/models.js') + ) + foreach ($candidate in $candidates) { + $full = [System.IO.Path]::GetFullPath($candidate) + if (Test-Path -LiteralPath $full -PathType Leaf) { + return $full + } + } + + throw 'Cline model registry module was not found under the installed CLI package.' +} + +function Resolve-CopilotSdkPath { + $command = Resolve-ExternalCommand -Name 'copilot' + if ($null -eq $command) { + throw 'GitHub Copilot CLI executable is not available on PATH.' + } + + $source = [string]$command.Source + $directory = Split-Path -Parent $source + $candidates = [System.Collections.Generic.List[string]]::new() + $optionalRoot = Join-Path $directory 'node_modules/@github/copilot/node_modules/@github' + if (Test-Path -LiteralPath $optionalRoot -PathType Container) { + foreach ($package in @(Get-ChildItem -LiteralPath $optionalRoot -Directory -Filter 'copilot-*' -Force)) { + $candidates.Add((Join-Path $package.FullName 'sdk/index.js')) + } + } + $candidates.Add((Join-Path $directory 'node_modules/@github/copilot/node_modules/@github/copilot-win32-x64/sdk/index.js')) + $candidates.Add((Join-Path $directory 'node_modules/@github/copilot/node_modules/@github/copilot-win32-arm64/sdk/index.js')) + $candidates.Add((Join-Path $directory 'node_modules/@github/copilot/node_modules/@github/copilot-linux-x64/sdk/index.js')) + $candidates.Add((Join-Path $directory 'node_modules/@github/copilot/node_modules/@github/copilot-linux-arm64/sdk/index.js')) + $candidates.Add((Join-Path $directory 'node_modules/@github/copilot/node_modules/@github/copilot-darwin-x64/sdk/index.js')) + $candidates.Add((Join-Path $directory 'node_modules/@github/copilot/node_modules/@github/copilot-darwin-arm64/sdk/index.js')) + foreach ($candidate in $candidates) { + $full = [System.IO.Path]::GetFullPath($candidate) + if (Test-Path -LiteralPath $full -PathType Leaf) { + return $full + } + } + + throw 'GitHub Copilot CLI does not expose a package-local SDK model listing surface in this installation.' +} + +function Get-CodexModels { + $command = Resolve-ExternalCommand -Name 'codex' + if ($null -eq $command) { + throw 'Codex CLI executable is not available on PATH.' + } + $result = Invoke-JsonCommand -CommandInfo $command -Arguments @('debug', 'models') -TimeoutSeconds 60 + $catalog = $result.Stdout | ConvertFrom-Json + return ConvertTo-ModelChoices -Catalog $catalog -RunnerName 'codex' -Source 'codex debug models' +} + +function Get-OpenCodeModels { + $command = Resolve-ExternalCommand -Name 'opencode' + if ($null -eq $command) { + throw 'OpenCode CLI executable is not available on PATH.' + } + $arguments = @('models', 'opencode', '--verbose') + if ($Refresh) { $arguments += '--refresh' } + $result = Invoke-JsonCommand -CommandInfo $command -Arguments $arguments -TimeoutSeconds 180 + return ConvertFrom-OpenCodeTextCatalog -Text $result.Stdout +} + +function Get-ClineModels { + $modulePath = Resolve-ClineModelsModulePath + $node = Resolve-ExternalCommand -Name 'node' + if ($null -eq $node) { + throw 'Node.js is required to read the Cline model registry.' + } + + $script = @' +import { pathToFileURL } from "node:url"; +const modulePath = process.argv[1]; +const mod = await import(pathToFileURL(modulePath).href); +const providerIds = await mod.getProviderIds(); +const models = []; +for (const providerId of providerIds) { + const providerModels = await mod.getModelsForProvider(providerId); + for (const [key, model] of Object.entries(providerModels || {})) { + models.push({ + id: model.id || key, + providerID: providerId, + name: model.name || model.id || key, + pricing: model.pricing, + operation: model.operation, + capabilities: model.capabilities + }); + } +} +console.log(JSON.stringify({ models })); +'@ + $result = Invoke-JsonCommand -CommandInfo $node -Arguments @('--input-type=module', '-e', $script, $modulePath) -TimeoutSeconds 60 + $catalog = $result.Stdout | ConvertFrom-Json + return ConvertTo-ModelChoices -Catalog $catalog -RunnerName 'cline' -Source '@cline/llms model registry' +} + +function Get-CopilotModels { + $sdkPath = Resolve-CopilotSdkPath + $node = Resolve-ExternalCommand -Name 'node' + if ($null -eq $node) { + throw 'Node.js is required to read the GitHub Copilot CLI model registry.' + } + + $script = @' +import { pathToFileURL } from "node:url"; +const sdkPath = process.argv[1]; +const mod = await import(pathToFileURL(sdkPath).href); +const ids = Array.isArray(mod.HELP_VISIBLE_MODELS) + ? mod.HELP_VISIBLE_MODELS + : Object.keys(mod.SUPPORTED_MODELS || {}); +console.log(JSON.stringify({ models: ids.map((id) => ({ id, name: id, operation: "language" })) })); +'@ + $result = Invoke-JsonCommand -CommandInfo $node -Arguments @('--input-type=module', '-e', $script, $sdkPath) -TimeoutSeconds 90 + $catalog = $result.Stdout | ConvertFrom-Json + return ConvertTo-ModelChoices -Catalog $catalog -RunnerName 'github-copilot' -Source 'GitHub Copilot CLI help-visible model catalog' +} + +try { + $rawModels = if (-not [string]::IsNullOrWhiteSpace($CatalogPath)) { + ConvertTo-ModelChoices -Catalog (Read-CatalogJson -Path $CatalogPath) -RunnerName $Runner -Source $CatalogPath + } else { + switch ($Runner) { + 'github-copilot' { Get-CopilotModels } + 'codex' { Get-CodexModels } + 'opencode' { Get-OpenCodeModels } + 'cline' { Get-ClineModels } + } + } + $models = @(Select-ModelsByPolicy -Models @($rawModels) -RunnerName $Runner) + if ($models.Count -eq 0) { + throw "No models were returned for $((Get-HarnessDisplayName -RunnerName $Runner))." + } + + if (-not [string]::IsNullOrWhiteSpace($RequireModel) -and @($models | Where-Object { [string]$_.id -eq $RequireModel }).Count -eq 0) { + $available = [string]::Join(', ', @($models | Select-Object -ExpandProperty id)) + throw "Required model '$RequireModel' was not returned by current $((Get-HarnessDisplayName -RunnerName $Runner)) discovery. Available models: $available" + } + + [ordered]@{ + schema = 'codebeltnet/agentic/harness-models/1' + runner = $Runner + harness = Get-HarnessDisplayName -RunnerName $Runner + policy = Get-PolicyName -RunnerName $Runner + models = @($models) + } | ConvertTo-Json -Depth 20 +} catch { + [Console]::Error.WriteLine($_.Exception.Message) + exit 2 +} diff --git a/scripts/eval-report-template.html b/scripts/eval-report-template.html index e5b6588..4881def 100644 --- a/scripts/eval-report-template.html +++ b/scripts/eval-report-template.html @@ -165,7 +165,7 @@ if (!run) return '
' + runLabel(config) + '
Run not recorded.
'; const summary = gradeSummary(run); const rate = passRate(run); - const meta = [run.model, run.provider, run.harness].filter(Boolean).join(' · ') || 'Runtime details not recorded'; + const meta = [run.model, run.harness].filter(Boolean).join(' · ') || 'Runtime details not recorded'; let html = '
' + runLabel(config) + '' + esc(meta) + '
'; html += '
'; html += metric('Turns', run.metrics.turns, config); @@ -303,7 +303,7 @@ document.getElementById('eval-select').addEventListener('change', e => { current = Number(e.target.value); renderOutputs(); }); document.addEventListener('keydown', e => { if (e.target.tagName === 'TEXTAREA') return; if (e.key === 'ArrowLeft') document.getElementById('prev').click(); if (e.key === 'ArrowRight') document.getElementById('next').click(); }); document.getElementById('skill-name').textContent = DATA.skill_name || 'skill'; - document.getElementById('run-meta').textContent = [DATA.metadata && DATA.metadata.model, DATA.metadata && DATA.metadata.provider, DATA.metadata && DATA.metadata.completed_runs + '/' + DATA.metadata.expected_runs + ' runs'].filter(Boolean).join(' · '); + document.getElementById('run-meta').textContent = [DATA.metadata && DATA.metadata.model, DATA.metadata && DATA.metadata.completed_runs + '/' + DATA.metadata.expected_runs + ' runs'].filter(Boolean).join(' · '); for (let i = 0; i < (DATA.evals || []).length; i++) { const option = document.createElement('option'); option.value = String(i); option.textContent = (i + 1) + '. ' + DATA.evals[i].name; document.getElementById('eval-select').appendChild(option); } renderOutputs(); renderBenchmark(); renderSkill(); diff --git a/scripts/eval-runners/README.md b/scripts/eval-runners/README.md index 3e889e2..2559817 100644 --- a/scripts/eval-runners/README.md +++ b/scripts/eval-runners/README.md @@ -16,8 +16,11 @@ working directory, isolated home, staged candidate skill, and required experimental controls. Its `filesystemIsolationRequired` and `mustNotReadOutsideSandbox` fields describe the staged worker-facing package boundary; they do not claim that the host has a hard OS filesystem sandbox. -`execution-profile.json` selects the runner and -execution configuration. It contains no credentials or secrets. +`execution-profile.json` selects the runner, runner-native model selector, and +execution configuration. The model string is opaque to the portable layer: a +runner may pass it through unchanged or split it internally when its native CLI +requires separate provider/model arguments. The profile contains no credentials, +secrets, or portable provider field. `execution-result.json` normalizes one blind execution and keeps grading separate from raw evidence. @@ -50,13 +53,13 @@ native CLI flags, environment setup, event parsing, authentication injection, and isolation checks stay inside their own directories. Windows is supported in pragmatic mode when the native CLI satisfies the mandatory controls. -`github-copilot` with `claude-haiku-4.5` is the Codebelt reference evaluation +`github-copilot` with `claude-haiku-4.5` is the Codebelt Reference evaluation configuration: a stable, economical pairing for routine skill comparison. It is -a repository convention, not an Anthropic default, and the model stays -configurable through `execution-profile.json`, so any Copilot-served model can -be selected. Cross-runner and cross-model numbers are never blended into one -score; a paired `with_skill` versus `without_skill` comparison is only -meaningful within one identical runner, model, and configuration stratum. +a repository convention, not an Anthropic default, and preparation verifies that +the model still appears in the current Copilot catalog before selecting it. +Cross-runner and cross-model numbers are never blended into one score; a paired +`with_skill` versus `without_skill` comparison is only meaningful within one +identical runner, model, and configuration stratum. GitHub Copilot uses `copilot -C --model --output-format json --allow-all-tools --no-ask-user --disable-builtin-mcps @@ -70,24 +73,27 @@ noninteractive execution; it does not disable path or URL verification. Repository-owned custom instructions remain enabled and are staged identically in both paired arms. Personal Copilot configuration is excluded by run-local `COPILOT_HOME`, `COPILOT_CACHE_HOME`, `HOME`, `USERPROFILE`, and XDG roots; -the runner does not copy the normal `.copilot` directory. Authentication follows -Copilot's normal order: explicit `COPILOT_GITHUB_TOKEN`, `GH_TOKEN`, or -`GITHUB_TOKEN`, then the OS credential store, then GitHub CLI fallback through a -host-derived `GH_CONFIG_DIR` when available. `--secret-env-vars` removes every +the runner does not copy the normal `.copilot` directory. Authentication prefers +explicit `COPILOT_GITHUB_TOKEN`, `GH_TOKEN`, or `GITHUB_TOKEN`; when none is +present, the trusted runner may resolve `gh auth token` outside the worker and +inject only that token as a protected environment variable. Host `GH_CONFIG_DIR` +is never forwarded into the evaluated worker. `--secret-env-vars` removes every listed token variable from shell and MCP child environments. Preflight does not make a model request and therefore reports native keychain/service readiness as conditional rather than claiming successful remote authentication. Codex uses `--ask-for-approval never` with `exec --sandbox workspace-write`; it does not combine explicit sandbox selection with -`--approve-for-me`. OpenCode uses `run --format json --auto` with isolated -global/config roots and preserves repository-owned project configuration; it -does not depend on `OPENCODE_DISABLE_PROJECT_CONFIG` or use `--pure`. Cline uses -`--json`, `--auto-approve true`, `--retries 0`, `--config `, -`--data-dir /.cline/data`, and run-local hooks; it passes no session -id. Each captures an exact observable CLI version and passes only documented -environment credentials when the selected runner supports them. None copies a -global skill directory, -memory store, plugin set, or normal agent profile into a run. +`--approve-for-me`. OpenCode uses `run --format json --auto --model +` with isolated global/config roots and preserves +repository-owned project configuration; it does not depend on +`OPENCODE_DISABLE_PROJECT_CONFIG` or use `--pure`. Cline expects a +`provider/model` selector in the profile and derives its native `--provider` +and `--model` arguments inside the adapter. Each captures an exact observable +CLI version and passes only documented environment credentials when the selected +runner supports them. None copies a global skill directory, memory store, plugin +set, or normal agent profile into a run. + +Model discovery lives in `scripts/Get-HarnessModels.ps1`. It uses the current local harness catalog where available: Copilot through the installed CLI SDK help-visible model list, Codex through `codex debug models`, OpenCode through `opencode models opencode --verbose`, and Cline through the installed `@cline/llms` registry. Cline and OpenCode discovery returns only models with current metadata proving free availability; zero free models is a clear local failure, not a fallback to paid models. Freebuff is currently documented as planned/blocked. Its supported CLI remains TUI-oriented and does not provide the required one-prompt, noninteractive, diff --git a/scripts/eval-runners/bridge-execution-result.ps1 b/scripts/eval-runners/bridge-execution-result.ps1 index b7aa352..8774dd3 100644 --- a/scripts/eval-runners/bridge-execution-result.ps1 +++ b/scripts/eval-runners/bridge-execution-result.ps1 @@ -181,11 +181,8 @@ try { $existingGrading = @(Get-ExistingGrading -ResultPath $resultPath) $caps = Get-JsonProperty -Object $raw.isolation -Name 'capabilities' -Default ([ordered]@{}) $requestedModel = [string](Get-JsonProperty -Object $raw.requested -Name 'model' -Default '') - $requestedProvider = [string](Get-JsonProperty -Object $raw.requested -Name 'provider' -Default '') $resolvedModelValue = Get-JsonProperty -Object $raw.resolved -Name 'model' -Default $null - $resolvedProviderValue = Get-JsonProperty -Object $raw.resolved -Name 'provider' -Default $null $resolvedModel = if ($null -eq $resolvedModelValue) { '' } else { [string]$resolvedModelValue } - $resolvedProvider = if ($null -eq $resolvedProviderValue) { '' } else { [string]$resolvedProviderValue } $resolutionStatus = [string](Get-JsonProperty -Object $raw.resolved -Name 'status' -Default 'unavailable') $resolutionReason = [string](Get-JsonProperty -Object $raw.resolved -Name 'reason' -Default '') $notes.Add("configuration_resolution=$resolutionStatus") @@ -197,11 +194,8 @@ try { eval_name = $runData.EvalName configuration = $runData.Mode model = if ([string]::IsNullOrWhiteSpace($resolvedModel)) { $requestedModel } else { $resolvedModel } - provider = if ([string]::IsNullOrWhiteSpace($resolvedProvider)) { $requestedProvider } else { $resolvedProvider } requested_model = $requestedModel - requested_provider = $requestedProvider resolved_model = $resolvedModel - resolved_provider = $resolvedProvider configuration_resolution_status = $resolutionStatus configuration_resolution_reason = $resolutionReason harness = "$(Get-JsonProperty -Object $raw.harness -Name 'name' -Default 'unknown') $(Get-JsonProperty -Object $raw.harness -Name 'version' -Default '')".Trim() diff --git a/scripts/eval-runners/cline/runner.ps1 b/scripts/eval-runners/cline/runner.ps1 index 22109aa..8fb1cdb 100644 --- a/scripts/eval-runners/cline/runner.ps1 +++ b/scripts/eval-runners/cline/runner.ps1 @@ -96,6 +96,21 @@ function Get-ClineDescriptor { return $copy } +function Resolve-ClineModelSelector { + param([string]$Model) + + if ([string]::IsNullOrWhiteSpace($Model)) { + return [pscustomobject]@{ Provider = $null; Model = $null; Valid = $false } + } + + $parts = $Model.Split([char[]]@('/'), 2, [System.StringSplitOptions]::None) + if ($parts.Count -lt 2 -or [string]::IsNullOrWhiteSpace($parts[0]) -or [string]::IsNullOrWhiteSpace($parts[1])) { + return [pscustomobject]@{ Provider = $null; Model = $Model; Valid = $false } + } + + return [pscustomobject]@{ Provider = $parts[0]; Model = $parts[1]; Valid = $true } +} + function New-ClineEnvironment { param([Parameter(Mandatory = $true)][object]$Inputs) @@ -108,8 +123,10 @@ function New-ClineEnvironment { foreach ($directory in @($clineRoot, $dataDirectory, $settingsDirectory, $sandboxDataDirectory, $teamDataDirectory, $hooksDirectory)) { New-Item -ItemType Directory -Path $directory -Force | Out-Null } + $selector = Resolve-ClineModelSelector -Model ([string]$Inputs.Profile.Model) + $authVariables = @(if ($selector.Valid) { Get-ProviderAuthenticationVariables -Provider $selector.Provider }) return [pscustomobject]@{ - Environment = New-RunnerEnvironment -Run $Inputs.Run -AuthenticationVariables @(Get-ProviderAuthenticationVariables -Provider ([string]$Inputs.Profile.Provider)) -Additional @{ + Environment = New-RunnerEnvironment -Run $Inputs.Run -AuthenticationVariables $authVariables -Additional @{ CLINE_DATA_DIR = $dataDirectory CLINE_SANDBOX_DATA_DIR = $sandboxDataDirectory CLINE_HOOKS_DIR = $hooksDirectory @@ -147,7 +164,9 @@ function Get-ClineInsideEnvironment { CI = '1' NO_COLOR = '1' } - foreach ($name in @(Get-ProviderAuthenticationVariables -Provider ([string]$Inputs.Profile.Provider))) { + $selector = Resolve-ClineModelSelector -Model ([string]$Inputs.Profile.Model) + $authVariables = @(if ($selector.Valid) { Get-ProviderAuthenticationVariables -Provider $selector.Provider }) + foreach ($name in $authVariables) { if ($EnvironmentData.Environment.Contains($name) -and -not [string]::IsNullOrWhiteSpace([string]$EnvironmentData.Environment[$name])) { $inside[$name] = [string]$EnvironmentData.Environment[$name] } @@ -166,8 +185,9 @@ function New-ClineCliArguments { $configPath = Get-SandboxVisiblePath -HostPath $EnvironmentData.ConfigPath -RunRoot $Inputs.Run.RunRoot -Platform $VisiblePlatform $dataRoot = Get-SandboxVisiblePath -HostPath $EnvironmentData.DataDirectory -RunRoot $Inputs.Run.RunRoot -Platform $VisiblePlatform $hooksDirectory = Get-SandboxVisiblePath -HostPath $EnvironmentData.HooksDirectory -RunRoot $Inputs.Run.RunRoot -Platform $VisiblePlatform + $selector = Resolve-ClineModelSelector -Model ([string]$Inputs.Profile.Model) $arguments = [System.Collections.Generic.List[string]]::new() - foreach ($argument in @('--json', '--auto-approve', 'true', '--cwd', $workingDirectory, '--config', $configPath, '--data-dir', $dataRoot, '--hooks-dir', $hooksDirectory, '--provider', $Inputs.Profile.Provider, '--model', $Inputs.Profile.Model, '--retries', '0', '--timeout', [string]$Inputs.Profile.TimeoutSeconds)) { + foreach ($argument in @('--json', '--auto-approve', 'true', '--cwd', $workingDirectory, '--config', $configPath, '--data-dir', $dataRoot, '--hooks-dir', $hooksDirectory, '--provider', $selector.Provider, '--model', $selector.Model, '--retries', '0', '--timeout', [string]$Inputs.Profile.TimeoutSeconds)) { $arguments.Add([string]$argument) } if (-not [string]::IsNullOrWhiteSpace([string]$Inputs.Profile.ReasoningEffort)) { @@ -210,15 +230,14 @@ function Get-ClinePreflight { } else { $checks.Add((New-PreflightCheck -Name 'runner_selection' -Status passed -Detail 'The selected runner is cline.')) } - if ([string]::IsNullOrWhiteSpace($profile.Provider)) { - $reasons.Add('Cline requires a provider in execution-profile.json.') - } else { - $checks.Add((New-PreflightCheck -Name 'provider' -Status passed -Detail $profile.Provider)) - } + $selector = Resolve-ClineModelSelector -Model ([string]$profile.Model) if ([string]::IsNullOrWhiteSpace($profile.Model)) { $reasons.Add('Cline requires a model in execution-profile.json.') + } elseif (-not $selector.Valid) { + $reasons.Add("Cline requires a runner-native model selector in 'provider/model' form; received '$($profile.Model)'.") } else { $checks.Add((New-PreflightCheck -Name 'model' -Status passed -Detail $profile.Model)) + $checks.Add((New-PreflightCheck -Name 'native_model_selector' -Status passed -Detail "--provider $($selector.Provider); --model $($selector.Model)")) } if ($profile.ConfigurationProfile -ne 'isolated-default') { $reasons.Add("configuration_profile '$($profile.ConfigurationProfile)' is unsupported by cline.") } if ($profile.ToolProfile -ne 'default') { $reasons.Add("tool_profile '$($profile.ToolProfile)' is unsupported by cline.") } @@ -260,13 +279,15 @@ function Get-ClinePreflight { } } - $authVariables = @(Get-ProviderAuthenticationVariables -Provider ([string]$profile.Provider)) + $authVariables = @(if ($selector.Valid) { Get-ProviderAuthenticationVariables -Provider $selector.Provider }) $authVariable = $null foreach ($name in $authVariables) { if (-not [string]::IsNullOrWhiteSpace([Environment]::GetEnvironmentVariable($name))) { $authVariable = $name; break } } - if ([string]::IsNullOrWhiteSpace($authVariable)) { - $reasons.Add("No narrow provider authentication environment variable is available for '$($profile.Provider)'; ambient Cline auth profiles are not copied into an eval run.") + if ($authVariables.Count -eq 0) { + $checks.Add((New-PreflightCheck -Name 'authentication' -Status not_applicable -Detail 'No runner-known provider API-key environment variable is required for this Cline model selector.')) + } elseif ([string]::IsNullOrWhiteSpace($authVariable)) { + $reasons.Add("No narrow provider authentication environment variable is available for model provider '$($selector.Provider)'; ambient Cline auth profiles are not copied into an eval run.") } else { $checks.Add((New-PreflightCheck -Name 'authentication' -Status passed -Detail "Provider credential will be passed only as $authVariable.")) } @@ -498,15 +519,17 @@ function Invoke-ClineExecute { $mechanisms = [System.Collections.Generic.List[string]]::new() foreach ($mechanism in @('cline --json', '--auto-approve true', '--retries 0', 'no --id session resume', 'run-local HOME', 'run-local Cline data/config/hooks directories', 'prompt on stdin')) { $mechanisms.Add($mechanism) } if ($hardFilesystem) { $mechanisms.Add("external $($sandboxInfo.Source) filesystem sandbox") } else { $mechanisms.Add('pragmatic process/environment isolation without hard filesystem confinement'); $warnings.Add('Hard filesystem confinement was unavailable; the completed arm is reported as pragmatic isolation.') } + $nativeSelector = Resolve-ClineModelSelector -Model ([string]$Inputs.Profile.Model) $credentialEvidence = [ordered]@{ - provider_environment_variables = @(Get-ProviderAuthenticationVariables -Provider ([string]$Inputs.Profile.Provider)) + model_provider = $nativeSelector.Provider + provider_environment_variables = @(if ($nativeSelector.Valid) { Get-ProviderAuthenticationVariables -Provider $nativeSelector.Provider }) unrelated_environment_excluded = $true child_tool_visibility = 'not_exposed_by_runner_environment; Cline child filtering is not independently observable' value_observed = $false } $sessionResultId = if ([string]::IsNullOrWhiteSpace($returnedSessionId)) { $sessionId } else { $returnedSessionId } $sandboxEvidence = if (-not $hardFilesystem) { 'unavailable' } elseif ($platform -eq 'linux') { 'bwrap' } else { 'sandbox-exec' } - return New-ExecutionResult -Descriptor $executionDescriptor -Profile $Inputs.Profile -Run $Inputs.Run -Status $status -FinalResponse $finalText -FinalResponseReason $reason -StartedUtc $process.StartedUtc.ToString('o') -FinishedUtc $finished.ToString('o') -DurationSeconds $process.DurationSeconds -ExitStatus $exitStatus -Failure $failure -SessionId $sessionResultId -IsolationCapabilities $capabilities -IsolationMechanisms @($mechanisms) -ResolvedConfiguration ([ordered]@{ status = 'accepted_request'; reason = 'Cline accepted the requested provider, model, thinking, and configuration but did not expose concrete backend resolution.'; observations = [ordered]@{ provider = $Inputs.Profile.Provider; model = $Inputs.Profile.Model; reasoning_effort = $Inputs.Profile.ReasoningEffort; retries = 0 } }) -Telemetry $telemetry -Artifacts @($artifacts) -Warnings @($warnings) -Evidence ([ordered]@{ event_counts = $eventCounts; commands = @($commands); files = @($files); prompt_first_input = $true; resume = $false; session_id_supplied = $false; retry_argument = 0; sandbox = $sandboxEvidence; credential = $credentialEvidence }) -AttemptCount 1 + return New-ExecutionResult -Descriptor $executionDescriptor -Profile $Inputs.Profile -Run $Inputs.Run -Status $status -FinalResponse $finalText -FinalResponseReason $reason -StartedUtc $process.StartedUtc.ToString('o') -FinishedUtc $finished.ToString('o') -DurationSeconds $process.DurationSeconds -ExitStatus $exitStatus -Failure $failure -SessionId $sessionResultId -IsolationCapabilities $capabilities -IsolationMechanisms @($mechanisms) -ResolvedConfiguration ([ordered]@{ status = 'accepted_request'; reason = 'Cline accepted the requested runner-native model selector, thinking, and configuration but did not expose concrete backend resolution.'; observations = [ordered]@{ model = $Inputs.Profile.Model; native_provider = $nativeSelector.Provider; native_model = $nativeSelector.Model; reasoning_effort = $Inputs.Profile.ReasoningEffort; retries = 0 } }) -Telemetry $telemetry -Artifacts @($artifacts) -Warnings @($warnings) -Evidence ([ordered]@{ event_counts = $eventCounts; commands = @($commands); files = @($files); prompt_first_input = $true; resume = $false; session_id_supplied = $false; retry_argument = 0; sandbox = $sandboxEvidence; credential = $credentialEvidence }) -AttemptCount 1 } try { diff --git a/scripts/eval-runners/codex/runner.ps1 b/scripts/eval-runners/codex/runner.ps1 index d6420ed..bf102e9 100644 --- a/scripts/eval-runners/codex/runner.ps1 +++ b/scripts/eval-runners/codex/runner.ps1 @@ -70,9 +70,7 @@ function Resolve-CodexInputs { } function Get-CodexAuthSource { - param([Parameter(Mandatory = $true)][string]$Provider) - - $authVariables = @(Get-ProviderAuthenticationVariables -Provider $Provider) + $authVariables = @(Get-ProviderAuthenticationVariables -Provider 'openai') foreach ($name in $authVariables) { if (-not [string]::IsNullOrWhiteSpace([Environment]::GetEnvironmentVariable($name))) { return [pscustomobject]@{ Kind = 'environment'; Name = $name; Path = $null } @@ -196,11 +194,6 @@ function Get-CodexPreflight { } else { $checks.Add((New-PreflightCheck -Name 'runner_selection' -Status passed -Detail 'The selected runner is codex.')) } - if ([string]::IsNullOrWhiteSpace($profile.Provider) -or $profile.Provider.ToLowerInvariant() -notin @('openai', 'chatgpt')) { - $reasons.Add("Codex requires provider 'openai' or 'chatgpt'; received '$($profile.Provider)'.") - } else { - $checks.Add((New-PreflightCheck -Name 'provider' -Status passed -Detail $profile.Provider)) - } if ([string]::IsNullOrWhiteSpace($profile.Model)) { $reasons.Add('Codex requires a model in execution-profile.json.') } else { @@ -260,7 +253,7 @@ function Get-CodexPreflight { } } - $auth = Get-CodexAuthSource -Provider ([string]$profile.Provider) + $auth = Get-CodexAuthSource if ($auth.Kind -eq 'missing') { $reasons.Add('No narrow Codex provider API-key environment variable is available.') } elseif ($auth.Kind -eq 'file_unsupported') { @@ -304,7 +297,7 @@ function New-CodexEnvironment { $codexHome = Join-Path $Inputs.Run.HomeDirectoryPath '.codex' New-Item -ItemType Directory -Path $codexHome -Force | Out-Null - $environment = New-RunnerEnvironment -Run $Inputs.Run -AuthenticationVariables @(Get-ProviderAuthenticationVariables -Provider ([string]$Inputs.Profile.Provider)) -Additional @{ CODEX_HOME = $codexHome } + $environment = New-RunnerEnvironment -Run $Inputs.Run -AuthenticationVariables @(Get-ProviderAuthenticationVariables -Provider 'openai') -Additional @{ CODEX_HOME = $codexHome } if ($Auth.Kind -ne 'environment') { throw 'Codex execution requires a provider environment credential; file credentials are not safe to expose in the worker HOME.' } @@ -350,7 +343,7 @@ function Get-LinuxCodexSandboxArguments { CI = '1' NO_COLOR = '1' } - foreach ($authName in @(Get-ProviderAuthenticationVariables -Provider ([string]$Inputs.Profile.Provider))) { + foreach ($authName in @(Get-ProviderAuthenticationVariables -Provider 'openai')) { if ($Environment.Contains($authName) -and -not [string]::IsNullOrWhiteSpace([string]$Environment[$authName])) { $insideEnvironment[$authName] = [string]$Environment[$authName] } @@ -424,7 +417,7 @@ function Invoke-CodexExecute { } $commandInfo = Resolve-ExternalCommand -Name 'codex' - $auth = Get-CodexAuthSource -Provider ([string]$Inputs.Profile.Provider) + $auth = Get-CodexAuthSource $environment = New-CodexEnvironment -Inputs $Inputs -Auth $auth $lastResponsePath = 'evidence/codex-final.txt' New-Item -ItemType Directory -Path (Join-Path $Inputs.Run.RunRoot 'evidence') -Force | Out-Null @@ -550,7 +543,7 @@ function Invoke-CodexExecute { child_tool_visibility = 'codex_shell_environment_policy_inherit_none' value_observed = $false } - return New-ExecutionResult -Descriptor $executionDescriptor -Profile $Inputs.Profile -Run $Inputs.Run -Status $status -FinalResponse $finalText -FinalResponseReason $reason -StartedUtc $process.StartedUtc.ToString('o') -FinishedUtc $finished.ToString('o') -DurationSeconds $process.DurationSeconds -ExitStatus $exitStatus -Failure $failure -SessionId $sessionResultId -IsolationCapabilities $capabilities -IsolationMechanisms @($mechanisms) -ResolvedConfiguration ([ordered]@{ status = 'accepted_request'; reason = 'Codex accepted the requested provider, model, and configuration but did not expose concrete backend resolution.'; observations = [ordered]@{ provider = $Inputs.Profile.Provider; model = $Inputs.Profile.Model; reasoning_effort = $Inputs.Profile.ReasoningEffort } }) -Telemetry $telemetry -Artifacts @($artifacts) -Warnings @($warnings) -Evidence ([ordered]@{ thread_id = $threadId; event_counts = $eventCounts; commands = @($commands); files = @($files); prompt_first_input = $true; resume = $false; stdout_exit_code = $process.ExitCode; sandbox = $sandboxEvidence; output_last_message_argument = (Get-SandboxVisiblePath -HostPath (Join-Path $Inputs.Run.RunRoot ($lastResponsePath -replace '/', [System.IO.Path]::DirectorySeparatorChar)) -RunRoot $Inputs.Run.RunRoot -Platform $visiblePlatform); credential = $credentialEvidence }) -AttemptCount 1 + return New-ExecutionResult -Descriptor $executionDescriptor -Profile $Inputs.Profile -Run $Inputs.Run -Status $status -FinalResponse $finalText -FinalResponseReason $reason -StartedUtc $process.StartedUtc.ToString('o') -FinishedUtc $finished.ToString('o') -DurationSeconds $process.DurationSeconds -ExitStatus $exitStatus -Failure $failure -SessionId $sessionResultId -IsolationCapabilities $capabilities -IsolationMechanisms @($mechanisms) -ResolvedConfiguration ([ordered]@{ status = 'accepted_request'; reason = 'Codex accepted the requested model and configuration but did not expose concrete backend resolution.'; observations = [ordered]@{ model = $Inputs.Profile.Model; reasoning_effort = $Inputs.Profile.ReasoningEffort } }) -Telemetry $telemetry -Artifacts @($artifacts) -Warnings @($warnings) -Evidence ([ordered]@{ thread_id = $threadId; event_counts = $eventCounts; commands = @($commands); files = @($files); prompt_first_input = $true; resume = $false; stdout_exit_code = $process.ExitCode; sandbox = $sandboxEvidence; output_last_message_argument = (Get-SandboxVisiblePath -HostPath (Join-Path $Inputs.Run.RunRoot ($lastResponsePath -replace '/', [System.IO.Path]::DirectorySeparatorChar)) -RunRoot $Inputs.Run.RunRoot -Platform $visiblePlatform); credential = $credentialEvidence }) -AttemptCount 1 } try { diff --git a/scripts/eval-runners/contracts/execution-profile.schema.json b/scripts/eval-runners/contracts/execution-profile.schema.json index 91aac6b..5fd745d 100644 --- a/scripts/eval-runners/contracts/execution-profile.schema.json +++ b/scripts/eval-runners/contracts/execution-profile.schema.json @@ -7,7 +7,6 @@ "required": [ "schema", "runner", - "provider", "model", "reasoning_effort", "configuration_profile", @@ -17,9 +16,8 @@ ], "properties": { "schema": { "const": "codebeltnet/agentic/eval-execution-profile/1" }, - "runner": { "type": ["string", "null"], "pattern": "^[a-z0-9][a-z0-9-]*$" }, - "provider": { "type": ["string", "null"], "minLength": 1 }, - "model": { "type": ["string", "null"], "minLength": 1 }, + "runner": { "type": "string", "pattern": "^[a-z0-9][a-z0-9-]*$" }, + "model": { "type": "string", "minLength": 1 }, "reasoning_effort": { "type": ["string", "null"], "minLength": 1 }, "configuration_profile": { "type": "string", "minLength": 1 }, "tool_profile": { "type": "string", "minLength": 1 }, diff --git a/scripts/eval-runners/contracts/execution-result.schema.json b/scripts/eval-runners/contracts/execution-result.schema.json index 258133c..161607e 100644 --- a/scripts/eval-runners/contracts/execution-result.schema.json +++ b/scripts/eval-runners/contracts/execution-result.schema.json @@ -52,17 +52,48 @@ "reason": { "type": "string" } } }, - "requested": { "type": "object" }, + "requested": { + "type": "object", + "additionalProperties": false, + "required": [ + "model", + "reasoning_effort", + "configuration_profile", + "tool_profile", + "timeout_seconds" + ], + "properties": { + "model": { "type": ["string", "null"] }, + "reasoning_effort": { "type": ["string", "null"] }, + "configuration_profile": { "type": "string" }, + "tool_profile": { "type": "string" }, + "timeout_seconds": { "type": "integer" } + } + }, "resolved": { "type": "object", "properties": { - "provider": { "type": ["string", "null"] }, "model": { "type": ["string", "null"] }, "reasoning_effort": { "type": ["string", "null"] }, "configuration_profile": { "type": ["string", "null"] }, "tool_profile": { "type": ["string", "null"] }, "status": { "enum": ["unavailable", "accepted_request", "resolved"] }, - "accepted": { "type": "object" } + "accepted": { + "type": "object", + "additionalProperties": false, + "required": [ + "model", + "reasoning_effort", + "configuration_profile", + "tool_profile" + ], + "properties": { + "model": { "type": ["string", "null"] }, + "reasoning_effort": { "type": ["string", "null"] }, + "configuration_profile": { "type": "string" }, + "tool_profile": { "type": "string" } + } + } } }, "duration_seconds": { "type": "number", "minimum": 0 }, diff --git a/scripts/eval-runners/contracts/preflight-result.schema.json b/scripts/eval-runners/contracts/preflight-result.schema.json index c3351ca..b942e9c 100644 --- a/scripts/eval-runners/contracts/preflight-result.schema.json +++ b/scripts/eval-runners/contracts/preflight-result.schema.json @@ -23,6 +23,24 @@ "schema": { "const": "codebeltnet/agentic/eval-runner-preflight/1" }, "protocol_version": { "const": "codebeltnet/agentic/eval-runner-protocol/1" }, "status": { "enum": ["compatible", "incompatible"] }, + "requested": { + "type": "object", + "additionalProperties": false, + "required": [ + "model", + "reasoning_effort", + "configuration_profile", + "tool_profile", + "timeout_seconds" + ], + "properties": { + "model": { "type": ["string", "null"] }, + "reasoning_effort": { "type": ["string", "null"] }, + "configuration_profile": { "type": "string" }, + "tool_profile": { "type": "string" }, + "timeout_seconds": { "type": "integer" } + } + }, "checks": { "type": "array" }, "resolved_capabilities": { "type": "object" }, "isolation": { diff --git a/scripts/eval-runners/github-copilot/runner.ps1 b/scripts/eval-runners/github-copilot/runner.ps1 index f56066a..b5c21e0 100644 --- a/scripts/eval-runners/github-copilot/runner.ps1 +++ b/scripts/eval-runners/github-copilot/runner.ps1 @@ -32,9 +32,6 @@ Set-StrictMode -Version Latest # GitHub CLI fallback. The values are forwarded only to the Copilot process; # --secret-env-vars removes them from shell and MCP child environments. $copilotAuthVariables = @('COPILOT_GITHUB_TOKEN', 'GH_TOKEN', 'GITHUB_TOKEN') -# Model routing runs through GitHub Copilot; the profile provider names the -# routing backend, not a direct model vendor. -$copilotProviders = @('github', 'github-copilot', 'copilot') $descriptor = [ordered]@{ schema = (Get-RunnerSchemaNames).Descriptor @@ -125,6 +122,68 @@ function Get-CopilotGhConfigDirectory { return [System.IO.Path]::GetFullPath($configured) } +function Get-CopilotGitHubCliToken { + $gh = Resolve-ExternalCommand -Name 'gh' + if ($null -eq $gh) { + return $null + } + + $environment = New-RunnerProbeEnvironment + foreach ($name in @('HOME', 'USERPROFILE', 'APPDATA', 'LOCALAPPDATA', 'XDG_CONFIG_HOME', 'GH_CONFIG_DIR')) { + $value = [Environment]::GetEnvironmentVariable($name) + if (-not [string]::IsNullOrWhiteSpace($value)) { + $environment[$name] = $value + } + } + + $probeDirectory = Join-Path ([System.IO.Path]::GetTempPath()) ('agentic-gh-token-probe-' + [Guid]::NewGuid().ToString('N')) + New-Item -ItemType Directory -Path $probeDirectory -Force | Out-Null + try { + $process = Invoke-RunnerProcess -FileName $gh.FileName -ArgumentList (@($gh.Prefix) + @('auth', 'token')) -WorkingDirectory $probeDirectory -Environment $environment -TimeoutSeconds 30 + if ($process.TimedOut -or $process.ExitCode -ne 0) { + return $null + } + $token = ([string]$process.Stdout).Trim() + if ([string]::IsNullOrWhiteSpace($token)) { + return $null + } + return $token + } finally { + if (Test-Path -LiteralPath $probeDirectory) { + Remove-Item -LiteralPath $probeDirectory -Recurse -Force -ErrorAction SilentlyContinue + } + } +} + +function Resolve-CopilotAuthentication { + $tokenVariable = Get-CopilotTokenVariable + if (-not [string]::IsNullOrWhiteSpace($tokenVariable)) { + return [pscustomobject]@{ + Source = 'environment' + TokenVariable = $tokenVariable + TokenValue = $null + GitHubCliTokenResolved = $false + } + } + + $githubCliToken = Get-CopilotGitHubCliToken + if (-not [string]::IsNullOrWhiteSpace($githubCliToken)) { + return [pscustomobject]@{ + Source = 'github_cli_token' + TokenVariable = 'GH_TOKEN' + TokenValue = $githubCliToken + GitHubCliTokenResolved = $true + } + } + + return [pscustomobject]@{ + Source = 'copilot_os_keychain_or_github_cli_unverified' + TokenVariable = $null + TokenValue = $null + GitHubCliTokenResolved = $false + } +} + function Invoke-CopilotCli { param( [Parameter(Mandatory = $true)][object]$CommandInfo, @@ -242,11 +301,6 @@ function Get-CopilotPreflight { } else { $checks.Add((New-PreflightCheck -Name 'runner_selection' -Status passed -Detail 'The selected runner is github-copilot.')) } - if ([string]::IsNullOrWhiteSpace($profile.Provider) -or $profile.Provider.ToLowerInvariant() -notin $copilotProviders) { - $reasons.Add("GitHub Copilot requires provider 'github-copilot' (also accepts 'github' or 'copilot'); received '$($profile.Provider)'.") - } else { - $checks.Add((New-PreflightCheck -Name 'provider' -Status passed -Detail $profile.Provider)) - } if ([string]::IsNullOrWhiteSpace($profile.Model)) { $reasons.Add('GitHub Copilot requires a model in execution-profile.json (claude-haiku-4.5 is the Codebelt reference).') } else { @@ -304,17 +358,13 @@ function Get-CopilotPreflight { } } - $tokenVariable = Get-CopilotTokenVariable - $ghConfigDirectory = Get-CopilotGhConfigDirectory - if (-not [string]::IsNullOrWhiteSpace($tokenVariable)) { - $checks.Add((New-PreflightCheck -Name 'authentication' -Status passed -Detail "Authentication is available through the explicit $tokenVariable environment variable; Copilot OS-keychain and GitHub CLI state are not copied into the run.")) + $authState = Resolve-CopilotAuthentication + if ($authState.Source -eq 'environment') { + $checks.Add((New-PreflightCheck -Name 'authentication' -Status passed -Detail "Authentication is available through the explicit $($authState.TokenVariable) environment variable; Copilot OS-keychain and GitHub CLI state are not copied into the run.")) + } elseif ($authState.Source -eq 'github_cli_token') { + $checks.Add((New-PreflightCheck -Name 'authentication' -Status passed -Detail 'GitHub CLI fallback resolved a token in the trusted runner; only a protected token environment variable will be passed to Copilot.')) } else { - $ghDetail = if ($null -eq $ghConfigDirectory) { - 'no host GitHub CLI configuration directory was observable' - } else { - 'the host GitHub CLI configuration directory will remain available for Copilot''s documented gh auth token fallback' - } - $checks.Add((New-PreflightCheck -Name 'authentication' -Status unavailable -Detail "No explicit token is present; native Copilot OS-keychain lookup is delegated to the installed CLI and $ghDetail. This preflight does not contact the Copilot service.")) + $checks.Add((New-PreflightCheck -Name 'authentication' -Status unavailable -Detail 'No explicit token is present and GitHub CLI fallback did not yield a token; native Copilot OS-keychain lookup is delegated to the installed CLI. This preflight does not contact the Copilot service.')) $warnings.Add('Authentication readiness beyond explicit environment tokens and the observable GitHub CLI fallback cannot be proven without a live Copilot request; preflight remains conditional and does not reject a tokenless native OAuth/keychain configuration.') } @@ -344,7 +394,7 @@ function Get-CopilotPreflight { foreach ($key in $descriptor.Keys) { $descriptorCopy[$key] = $descriptor[$key] } $descriptorCopy.harness = [ordered]@{ name = 'GitHub Copilot CLI'; version = $harnessVersion } $mechanisms = [System.Collections.Generic.List[string]]::new() - foreach ($mechanism in @('copilot --output-format json', 'prompt on stdin', '--allow-all-tools broad tool approval', 'path and URL verification preserved (no --allow-all-paths/--allow-all-urls)', '--no-ask-user', 'repository-owned custom instructions preserved', '--disable-builtin-mcps', '--secret-env-vars shell/MCP child filtering', 'isolated COPILOT_HOME and COPILOT_CACHE_HOME', 'isolated HOME/XDG roots', 'OS-keychain authentication delegated to Copilot', 'GitHub CLI authentication fallback through GH_CONFIG_DIR when available', 'no session continuation')) { $mechanisms.Add($mechanism) } + foreach ($mechanism in @('copilot --output-format json', 'prompt on stdin', '--allow-all-tools broad tool approval', 'path and URL verification preserved (no --allow-all-paths/--allow-all-urls)', '--no-ask-user', 'repository-owned custom instructions preserved', '--disable-builtin-mcps', '--secret-env-vars shell/MCP child filtering', 'isolated COPILOT_HOME and COPILOT_CACHE_HOME', 'isolated HOME/XDG roots', 'OS-keychain authentication delegated to Copilot', 'GitHub CLI fallback token resolved by the trusted runner when needed', 'no host GH_CONFIG_DIR exposed to the worker', 'no session continuation')) { $mechanisms.Add($mechanism) } if ($hardConfinement) { $mechanisms.Add("external $($sandboxInfo.Source) filesystem sandbox") } else { $mechanisms.Add('pragmatic process/environment isolation without hard filesystem confinement') } return New-PreflightDocument -Descriptor $descriptorCopy -Profile $profile -Run $run -Compatible ($reasons.Count -eq 0) -Checks @($checks) -Mechanisms @($mechanisms) -ResolvedCapabilities $capabilities -Warnings @($warnings) -Reasons @($reasons) } @@ -361,9 +411,10 @@ function New-CopilotEnvironment { COPILOT_CACHE_HOME = $copilotCacheHome COPILOT_AUTO_UPDATE = 'false' } - $ghConfigDirectory = Get-CopilotGhConfigDirectory - if ($null -ne $ghConfigDirectory) { - $additional['GH_CONFIG_DIR'] = $ghConfigDirectory + $tokenVariable = Get-CopilotTokenVariable + $authState = Resolve-CopilotAuthentication + if ($authState.Source -eq 'github_cli_token') { + $additional[$authState.TokenVariable] = $authState.TokenValue } return New-RunnerEnvironment -Run $Inputs.Run -AuthenticationVariables $copilotAuthVariables -Additional $additional } @@ -394,42 +445,9 @@ function New-CopilotInsideEnvironment { $insideEnvironment[$authName] = [string]$Environment[$authName] } } - if ($Environment.Contains('GH_CONFIG_DIR') -and -not [string]::IsNullOrWhiteSpace([string]$Environment['GH_CONFIG_DIR'])) { - $insideEnvironment['GH_CONFIG_DIR'] = '/run/gh-config' - } return $insideEnvironment } -function Add-CopilotLinuxAuthenticationMount { - param( - [Parameter(Mandatory = $true)][string[]]$Arguments, - [Parameter(Mandatory = $true)][string]$HostPath - ) - - $result = [System.Collections.Generic.List[string]]::new() - foreach ($argument in $Arguments) { $result.Add([string]$argument) } - $separatorIndex = $result.IndexOf('--') - if ($separatorIndex -lt 0) { - throw 'The Linux Copilot sandbox argument list has no command separator.' - } - $result.Insert($separatorIndex, '/run/gh-config') - $result.Insert($separatorIndex, $HostPath) - $result.Insert($separatorIndex, '--ro-bind') - return @($result) -} - -function Add-CopilotMacosAuthenticationPath { - param( - [Parameter(Mandatory = $true)][string]$ProfilePath, - [Parameter(Mandatory = $true)][string]$HostPath - ) - - $normalized = $HostPath.Replace('\', '/') - $escaped = $normalized.Replace('"', '\"') - $line = '(allow file-read* (subpath "' + $escaped + '"))' - [System.IO.File]::AppendAllText($ProfilePath, "`n$line`n", [System.Text.UTF8Encoding]::new($false)) -} - function Write-CopilotCapture { param( [Parameter(Mandatory = $true)][object]$RunData, @@ -550,15 +568,9 @@ function Invoke-CopilotExecute { if ($platform -eq 'linux' -and $hardFilesystem) { $insideEnvironment = New-CopilotInsideEnvironment -Inputs $Inputs -Environment $environment $sandboxArguments = Get-LinuxEvalSandboxArguments -Inputs $Inputs -CommandInfo $commandInfo -InsideEnvironment $insideEnvironment - if ($environment.Contains('GH_CONFIG_DIR') -and -not [string]::IsNullOrWhiteSpace([string]$environment['GH_CONFIG_DIR'])) { - $sandboxArguments = Add-CopilotLinuxAuthenticationMount -Arguments $sandboxArguments -HostPath ([string]$environment['GH_CONFIG_DIR']) - } $process = Invoke-RunnerProcess -FileName $sandboxInfo.FileName -ArgumentList (@($sandboxArguments) + @($arguments)) -WorkingDirectory $Inputs.Run.WorkingDirectoryPath -Environment $environment -InputBytes $Inputs.Run.PromptBytes -TimeoutSeconds $Inputs.Profile.TimeoutSeconds } elseif ($platform -eq 'macos' -and $hardFilesystem) { $sandboxProfile = New-MacosEvalSandboxProfile -Inputs $Inputs -CommandInfo $commandInfo - if ($environment.Contains('GH_CONFIG_DIR') -and -not [string]::IsNullOrWhiteSpace([string]$environment['GH_CONFIG_DIR'])) { - Add-CopilotMacosAuthenticationPath -ProfilePath $sandboxProfile -HostPath ([string]$environment['GH_CONFIG_DIR']) - } $sandboxArguments = @('-f', $sandboxProfile, '--', $commandInfo.FileName) + @($commandInfo.Prefix) + @($arguments) $process = Invoke-RunnerProcess -FileName $sandboxInfo.FileName -ArgumentList $sandboxArguments -WorkingDirectory $Inputs.Run.WorkingDirectoryPath -Environment $environment -InputBytes $Inputs.Run.PromptBytes -TimeoutSeconds $Inputs.Profile.TimeoutSeconds } else { @@ -618,18 +630,18 @@ function Invoke-CopilotExecute { $capabilities = Get-CopilotCapabilityMap -Inputs $Inputs -HardFilesystemConfinement $hardFilesystem $mechanisms = [System.Collections.Generic.List[string]]::new() - foreach ($mechanism in @('copilot --output-format json', 'prompt on stdin', '--allow-all-tools broad tool approval', 'path and URL verification preserved (no --allow-all-paths/--allow-all-urls)', '--no-ask-user', 'repository-owned custom instructions preserved', '--disable-builtin-mcps', '--secret-env-vars shell/MCP child filtering', 'isolated COPILOT_HOME and COPILOT_CACHE_HOME', 'isolated HOME/XDG roots', 'OS-keychain authentication delegated to Copilot', 'GitHub CLI authentication fallback through GH_CONFIG_DIR when available', 'no session continuation')) { $mechanisms.Add($mechanism) } + foreach ($mechanism in @('copilot --output-format json', 'prompt on stdin', '--allow-all-tools broad tool approval', 'path and URL verification preserved (no --allow-all-paths/--allow-all-urls)', '--no-ask-user', 'repository-owned custom instructions preserved', '--disable-builtin-mcps', '--secret-env-vars shell/MCP child filtering', 'isolated COPILOT_HOME and COPILOT_CACHE_HOME', 'isolated HOME/XDG roots', 'OS-keychain authentication delegated to Copilot', 'GitHub CLI fallback token resolved by the trusted runner when needed', 'no host GH_CONFIG_DIR exposed to the worker', 'no session continuation')) { $mechanisms.Add($mechanism) } if ($hardFilesystem) { $mechanisms.Add("external $($sandboxInfo.Source) filesystem sandbox") } else { $mechanisms.Add('pragmatic process/environment isolation without hard filesystem confinement') } if (-not $hardFilesystem) { $warnings.Add('Hard filesystem confinement was unavailable; the completed arm is reported as pragmatic isolation.') } - $tokenVariable = Get-CopilotTokenVariable - $ghConfigDirectory = Get-CopilotGhConfigDirectory + $authState = Resolve-CopilotAuthentication $credentialEvidence = [ordered]@{ - source = if (-not [string]::IsNullOrWhiteSpace($tokenVariable)) { 'environment' } elseif ($null -ne $ghConfigDirectory) { 'copilot_os_keychain_or_github_cli' } else { 'copilot_os_keychain_or_github_cli_unverified' } - github_token_variable = $tokenVariable + source = $authState.Source + github_token_variable = $authState.TokenVariable secret_env_vars = @($copilotAuthVariables) secret_env_var_scope = @('shell', 'mcp') - github_cli_config_forwarded = $null -ne $ghConfigDirectory + github_cli_token_resolved = [bool]$authState.GitHubCliTokenResolved + github_cli_config_forwarded = $false login_profile_copied = $false auth_file_copied = $false value_observed = $false @@ -639,7 +651,6 @@ function Invoke-CopilotExecute { status = 'accepted_request' reason = 'Copilot accepted the requested model alias and configuration; it does not expose a distinct backend model snapshot beyond the model it reports in usage events.' observations = [ordered]@{ - provider = $Inputs.Profile.Provider model = $Inputs.Profile.Model reasoning_effort = $Inputs.Profile.ReasoningEffort observed_model = $observedModel diff --git a/scripts/eval-runners/opencode/runner.ps1 b/scripts/eval-runners/opencode/runner.ps1 index ab2971e..92adc75 100644 --- a/scripts/eval-runners/opencode/runner.ps1 +++ b/scripts/eval-runners/opencode/runner.ps1 @@ -106,6 +106,19 @@ function Get-OpenCodeAuthVariable { return $null } +function Get-OpenCodeModelProvider { + param([string]$Model) + + if ([string]::IsNullOrWhiteSpace($Model) -or $Model -notmatch '/') { + return $null + } + $parts = $Model.Split([char[]]@('/'), 2, [System.StringSplitOptions]::None) + if ($parts.Count -lt 2 -or [string]::IsNullOrWhiteSpace($parts[0]) -or [string]::IsNullOrWhiteSpace($parts[1])) { + return $null + } + return $parts[0] +} + function Resolve-SandboxCommand { param([Parameter(Mandatory = $true)][string]$Name) @@ -130,11 +143,10 @@ function New-OpenCodeCliArguments { [Parameter(Mandatory = $true)][object]$Inputs, [ValidateSet('windows', 'linux', 'macos', 'unknown')][string]$VisiblePlatform = (Get-PlatformName) ) - $directoryArgument = Get-SandboxVisiblePath -HostPath $Inputs.Run.WorkingDirectoryPath -RunRoot $Inputs.Run.RunRoot -Platform $VisiblePlatform - $model = "{0}/{1}" -f $Inputs.Profile.Provider, $Inputs.Profile.Model + $directoryArgument = Get-SandboxVisiblePath -HostPath $Inputs.Run.WorkingDirectoryPath -RunRoot $Inputs.Run.RunRoot -Platform $VisiblePlatform $arguments = [System.Collections.Generic.List[string]]::new() - foreach ($argument in @('run', '--format', 'json', '--dir', $directoryArgument, '--model', $model, '--auto')) { + foreach ($argument in @('run', '--format', 'json', '--dir', $directoryArgument, '--model', $Inputs.Profile.Model, '--auto')) { $arguments.Add([string]$argument) } if (-not [string]::IsNullOrWhiteSpace([string]$Inputs.Profile.ReasoningEffort)) { @@ -177,15 +189,10 @@ function Get-OpenCodePreflight { } else { $checks.Add((New-PreflightCheck -Name 'runner_selection' -Status passed -Detail 'The selected runner is opencode.')) } - if ([string]::IsNullOrWhiteSpace($profile.Provider)) { - $reasons.Add('OpenCode requires a provider in execution-profile.json.') - } else { - $checks.Add((New-PreflightCheck -Name 'provider' -Status passed -Detail $profile.Provider)) - } if ([string]::IsNullOrWhiteSpace($profile.Model)) { $reasons.Add('OpenCode requires a model in execution-profile.json.') } else { - $checks.Add((New-PreflightCheck -Name 'model' -Status passed -Detail ("{0}/{1}" -f $profile.Provider, $profile.Model))) + $checks.Add((New-PreflightCheck -Name 'model' -Status passed -Detail $profile.Model)) } if ($profile.ConfigurationProfile -ne 'isolated-default') { $reasons.Add("configuration_profile '$($profile.ConfigurationProfile)' is unsupported by opencode.") @@ -229,9 +236,13 @@ function Get-OpenCodePreflight { } } - $authVariable = if ([string]::IsNullOrWhiteSpace($profile.Provider)) { $null } else { Get-OpenCodeAuthVariable -Provider ([string]$profile.Provider) } - if ([string]::IsNullOrWhiteSpace($authVariable)) { - $reasons.Add("No narrow provider authentication environment variable is available for '$($profile.Provider)'. OpenCode global auth profiles are not copied into an eval run.") + $modelProvider = Get-OpenCodeModelProvider -Model ([string]$profile.Model) + $authVariable = if ([string]::IsNullOrWhiteSpace($modelProvider)) { $null } else { Get-OpenCodeAuthVariable -Provider $modelProvider } + $knownAuthVariables = @(if (-not [string]::IsNullOrWhiteSpace($modelProvider)) { Get-ProviderAuthenticationVariables -Provider $modelProvider }) + if ($knownAuthVariables.Count -eq 0) { + $checks.Add((New-PreflightCheck -Name 'authentication' -Status not_applicable -Detail 'No runner-known provider API-key environment variable is required for this OpenCode model selector.')) + } elseif ([string]::IsNullOrWhiteSpace($authVariable)) { + $reasons.Add("No narrow provider authentication environment variable is available for model provider '$modelProvider'. OpenCode global auth profiles are not copied into an eval run.") } else { $checks.Add((New-PreflightCheck -Name 'authentication' -Status passed -Detail "Provider credential will be passed only as $authVariable.")) } @@ -273,7 +284,9 @@ function New-OpenCodeEnvironment { New-Item -ItemType Directory -Path $configDirectory -Force | Out-Null $configPath = Join-Path $configDirectory 'opencode.json' [System.IO.File]::WriteAllText($configPath, '{}', [System.Text.UTF8Encoding]::new($false)) - return New-RunnerEnvironment -Run $Inputs.Run -AuthenticationVariables @(Get-ProviderAuthenticationVariables -Provider ([string]$Inputs.Profile.Provider)) -Additional @{ + $modelProvider = Get-OpenCodeModelProvider -Model ([string]$Inputs.Profile.Model) + $authVariables = @(if (-not [string]::IsNullOrWhiteSpace($modelProvider)) { Get-ProviderAuthenticationVariables -Provider $modelProvider }) + return New-RunnerEnvironment -Run $Inputs.Run -AuthenticationVariables $authVariables -Additional @{ OPENCODE_CONFIG_DIR = $configDirectory OPENCODE_CONFIG = $configPath OPENCODE_DISABLE_AUTOUPDATE = '1' @@ -321,7 +334,9 @@ function Get-LinuxSandboxArguments { CI = '1' NO_COLOR = '1' } - foreach ($authName in @(Get-ProviderAuthenticationVariables -Provider ([string]$Inputs.Profile.Provider))) { + $modelProvider = Get-OpenCodeModelProvider -Model ([string]$Inputs.Profile.Model) + $authVariables = @(if (-not [string]::IsNullOrWhiteSpace($modelProvider)) { Get-ProviderAuthenticationVariables -Provider $modelProvider }) + foreach ($authName in $authVariables) { if ($Environment.Contains($authName) -and -not [string]::IsNullOrWhiteSpace([string]$Environment[$authName])) { $insideEnvironment[$authName] = [string]$Environment[$authName] } @@ -398,7 +413,7 @@ function Invoke-OpenCodeExecute { $sandboxInfo = if ($platform -eq 'linux') { Resolve-SandboxCommand -Name 'bwrap' } elseif ($platform -eq 'macos') { Resolve-SandboxCommand -Name 'sandbox-exec' } else { $null } $hardFilesystem = $null -ne $sandboxInfo -and $platform -in @('linux', 'macos') $visiblePlatform = if ($hardFilesystem) { $platform } elseif ($platform -eq 'linux') { 'unknown' } else { $platform } - $model = "{0}/{1}" -f $Inputs.Profile.Provider, $Inputs.Profile.Model + $model = [string]$Inputs.Profile.Model $arguments = New-OpenCodeCliArguments -Inputs $Inputs -VisiblePlatform $visiblePlatform if ($platform -eq 'linux' -and $hardFilesystem) { @@ -485,14 +500,16 @@ function Invoke-OpenCodeExecute { foreach ($mechanism in @('opencode run --format json', '--auto', 'isolated OPENCODE_CONFIG_DIR', 'isolated OPENCODE_CONFIG', 'isolated HOME/XDG roots', 'repository-owned project configuration preserved', 'prompt on stdin', 'no session continuation')) { $mechanisms.Add($mechanism) } if ($hardFilesystem) { $mechanisms.Add("external $($sandboxInfo.Source) filesystem sandbox") } else { $mechanisms.Add('pragmatic process/environment isolation without hard filesystem confinement'); $warnings.Add('Hard filesystem confinement was unavailable; the completed arm is reported as pragmatic isolation.') } $sandboxEvidence = if (-not $hardFilesystem) { 'unavailable' } elseif ($platform -eq 'linux') { 'bwrap' } else { 'sandbox-exec' } - $credentialNames = @(Get-ProviderAuthenticationVariables -Provider ([string]$Inputs.Profile.Provider)) + $modelProvider = Get-OpenCodeModelProvider -Model ([string]$Inputs.Profile.Model) + $credentialNames = @(if (-not [string]::IsNullOrWhiteSpace($modelProvider)) { Get-ProviderAuthenticationVariables -Provider $modelProvider }) $credentialEvidence = [ordered]@{ + model_provider = $modelProvider provider_environment_variables = $credentialNames unrelated_environment_excluded = $true child_tool_visibility = 'provider_credential_may_be_visible_to_native_child_tools; no supported child filter is exposed' value_observed = $false } - return New-ExecutionResult -Descriptor $executionDescriptor -Profile $Inputs.Profile -Run $Inputs.Run -Status $status -FinalResponse $finalText -FinalResponseReason $reason -StartedUtc $process.StartedUtc.ToString('o') -FinishedUtc $process.FinishedUtc.ToString('o') -DurationSeconds $process.DurationSeconds -ExitStatus $exitStatus -Failure $failure -SessionId $sessionId -IsolationCapabilities $capabilities -IsolationMechanisms @($mechanisms) -ResolvedConfiguration ([ordered]@{ status = 'accepted_request'; reason = 'OpenCode accepted the requested provider, model, and configuration but did not expose concrete backend resolution.'; observations = [ordered]@{ provider = $Inputs.Profile.Provider; model = $Inputs.Profile.Model; reasoning_effort = $Inputs.Profile.ReasoningEffort } }) -Telemetry $telemetry -Artifacts @($artifacts) -Warnings @($warnings) -Evidence ([ordered]@{ event_counts = $eventCounts; commands = @($commands); prompt_first_input = $true; resume = $false; model_argument = $model; sandbox = $sandboxEvidence; project_configuration = 'repository_owned_project_config_preserved'; disable_project_config_environment = $false; credential = $credentialEvidence }) -AttemptCount 1 + return New-ExecutionResult -Descriptor $executionDescriptor -Profile $Inputs.Profile -Run $Inputs.Run -Status $status -FinalResponse $finalText -FinalResponseReason $reason -StartedUtc $process.StartedUtc.ToString('o') -FinishedUtc $process.FinishedUtc.ToString('o') -DurationSeconds $process.DurationSeconds -ExitStatus $exitStatus -Failure $failure -SessionId $sessionId -IsolationCapabilities $capabilities -IsolationMechanisms @($mechanisms) -ResolvedConfiguration ([ordered]@{ status = 'accepted_request'; reason = 'OpenCode accepted the requested runner-native model selector and configuration but did not expose concrete backend resolution.'; observations = [ordered]@{ model = $Inputs.Profile.Model; reasoning_effort = $Inputs.Profile.ReasoningEffort } }) -Telemetry $telemetry -Artifacts @($artifacts) -Warnings @($warnings) -Evidence ([ordered]@{ event_counts = $eventCounts; commands = @($commands); prompt_first_input = $true; resume = $false; model_argument = $model; sandbox = $sandboxEvidence; project_configuration = 'repository_owned_project_config_preserved'; disable_project_config_environment = $false; credential = $credentialEvidence }) -AttemptCount 1 } try { diff --git a/scripts/eval-runners/runner-common.ps1 b/scripts/eval-runners/runner-common.ps1 index 253f0da..98d38ad 100644 --- a/scripts/eval-runners/runner-common.ps1 +++ b/scripts/eval-runners/runner-common.ps1 @@ -371,7 +371,7 @@ function Resolve-ExecutionProfile { } Assert-ProfileHasNoSecrets -Profile $profile - $allowedProperties = @('schema', 'runner', 'provider', 'model', 'reasoning_effort', 'configuration_profile', 'tool_profile', 'timeout_seconds', 'concurrency') + $allowedProperties = @('schema', 'runner', 'model', 'reasoning_effort', 'configuration_profile', 'tool_profile', 'timeout_seconds', 'concurrency') foreach ($propertyName in @(Get-JsonPropertyNames -Object $profile)) { if ($allowedProperties -notcontains $propertyName) { throw "execution-profile.json contains unsupported field '$propertyName'." @@ -391,17 +391,18 @@ function Resolve-ExecutionProfile { } $runnerValue = [string](Get-JsonProperty -Object $profile -Name 'runner' -Default '') - $providerValue = [string](Get-JsonProperty -Object $profile -Name 'provider' -Default '') $modelValue = [string](Get-JsonProperty -Object $profile -Name 'model' -Default '') if (-not [string]::IsNullOrWhiteSpace($runnerValue) -and $runnerValue -notmatch '^[a-z0-9][a-z0-9-]*$') { throw 'execution-profile.json runner must be a safe lowercase runner name.' } + if ([string]::IsNullOrWhiteSpace($runnerValue) -or [string]::IsNullOrWhiteSpace($modelValue)) { + throw 'execution-profile.json must declare non-empty runner and model before a runner can execute.' + } return [pscustomobject]@{ Path = $resolvedProfilePath Profile = $profile Hash = Get-Sha256HexFromFile -Path $resolvedProfilePath Runner = if ([string]::IsNullOrWhiteSpace($runnerValue)) { $null } else { $runnerValue } - Provider = if ([string]::IsNullOrWhiteSpace($providerValue)) { $null } else { $providerValue } Model = if ([string]::IsNullOrWhiteSpace($modelValue)) { $null } else { $modelValue } ReasoningEffort = if ([string]::IsNullOrWhiteSpace([string]$profile.reasoning_effort)) { $null } else { [string]$profile.reasoning_effort } ConfigurationProfile = [string]$profile.configuration_profile @@ -497,7 +498,6 @@ function New-PreflightDocument { harness = $Descriptor.harness run = [ordered]@{ eval_id = $Run.EvalId; eval_name = $Run.EvalName; configuration = $Run.Mode } requested = [ordered]@{ - provider = $Profile.Provider model = $Profile.Model reasoning_effort = $Profile.ReasoningEffort configuration_profile = $Profile.ConfigurationProfile @@ -595,7 +595,6 @@ function New-ExecutionResult { } $resolved = [ordered]@{ - provider = $null model = $null reasoning_effort = $null configuration_profile = $null @@ -603,7 +602,6 @@ function New-ExecutionResult { status = 'unavailable' reason = 'harness_only_confirmed_the_requested_configuration' accepted = [ordered]@{ - provider = $Profile.Provider model = $Profile.Model reasoning_effort = $Profile.ReasoningEffort configuration_profile = $Profile.ConfigurationProfile @@ -613,7 +611,7 @@ function New-ExecutionResult { if ($null -ne $ResolvedConfiguration) { $resolved.status = [string](Get-JsonProperty -Object $ResolvedConfiguration -Name 'status' -Default 'resolved') $resolved.reason = Get-JsonProperty -Object $ResolvedConfiguration -Name 'reason' -Default $null - foreach ($name in @('provider', 'model', 'reasoning_effort', 'configuration_profile', 'tool_profile')) { + foreach ($name in @('model', 'reasoning_effort', 'configuration_profile', 'tool_profile')) { $value = Get-JsonProperty -Object $ResolvedConfiguration -Name $name -Default $null if ($null -ne $value) { $resolved[$name] = $value } } @@ -644,7 +642,6 @@ function New-ExecutionResult { runner = [ordered]@{ name = [string]$Descriptor.name; version = [string]$Descriptor.version } harness = $Descriptor.harness requested = [ordered]@{ - provider = $Profile.Provider model = $Profile.Model reasoning_effort = $Profile.ReasoningEffort configuration_profile = $Profile.ConfigurationProfile @@ -746,6 +743,13 @@ function Assert-ExecutionResult { if (-not (Test-JsonProperty -Object $Result.resolved -Name 'accepted')) { throw 'execution-result.json resolved must preserve the requested configuration as accepted evidence.' } + $hasPortableProvider = + (Test-JsonProperty -Object $Result.requested -Name 'provider') -or + (Test-JsonProperty -Object $Result.resolved -Name 'provider') -or + (Test-JsonProperty -Object $Result.resolved.accepted -Name 'provider') + if ($hasPortableProvider) { + throw 'execution-result.json must not expose provider in portable requested/resolved configuration fields.' + } foreach ($hashField in @('prompt_sha256', 'run_json_sha256', 'profile_sha256')) { if (-not (Test-Sha256 -Value ([string]$Result.input.$hashField))) { throw "execution-result.json input.$hashField must be a SHA-256 value." diff --git a/scripts/eval-runners/tests/test-runner-conformance.ps1 b/scripts/eval-runners/tests/test-runner-conformance.ps1 index 0feceed..d7c0553 100644 --- a/scripts/eval-runners/tests/test-runner-conformance.ps1 +++ b/scripts/eval-runners/tests/test-runner-conformance.ps1 @@ -230,6 +230,22 @@ if ($harness -eq 'codex') { foreach ($harness in @('codex', 'opencode', 'cline', 'copilot')) { [System.IO.File]::WriteAllText((Join-Path $fakeBin "$harness.ps1"), $fakeCli, [System.Text.UTF8Encoding]::new($false)) } + $fakeGh = @' +[CmdletBinding()] +param([Parameter(ValueFromRemainingArguments = $true)][string[]]$RemainingArguments) +if ($RemainingArguments.Count -eq 2 -and $RemainingArguments[0] -eq 'auth' -and $RemainingArguments[1] -eq 'token') { + $config = [Environment]::GetEnvironmentVariable('GH_CONFIG_DIR') + if (-not [string]::IsNullOrWhiteSpace($config) -and (Test-Path -LiteralPath (Join-Path $config 'auth-marker.txt') -PathType Leaf)) { + Write-Output 'recorded-gh-fallback-token-not-logged' + exit 0 + } + [Console]::Error.WriteLine('not logged in') + exit 1 +} +[Console]::Error.WriteLine('unsupported gh fixture command') +exit 2 +'@ + [System.IO.File]::WriteAllText((Join-Path $fakeBin 'gh.ps1'), $fakeGh, [System.Text.UTF8Encoding]::new($false)) $env:PATH = "$fakeBin$([System.IO.Path]::PathSeparator)$recordedOldPath" $env:OPENAI_API_KEY = 'recorded-canary-not-logged' $env:AGENTIC_GLOBAL_SECRET = 'recorded-unrelated-canary-not-logged' @@ -249,12 +265,15 @@ if ($harness -eq 'codex') { $recordedProfiles = [ordered]@{} foreach ($runnerName in @('codex', 'opencode', 'cline', 'copilot')) { $profilePath = Join-Path $recordedRoot "$runnerName-profile.json" - $profileProvider = if ($runnerName -eq 'copilot') { 'github-copilot' } else { 'openai' } - $profileModel = if ($runnerName -eq 'copilot') { 'claude-haiku-4.5' } else { 'fixture-model' } + $profileModel = switch ($runnerName) { + 'copilot' { 'claude-haiku-4.5' } + 'codex' { 'gpt-5.6-luna' } + 'opencode' { 'opencode/muse-spark-1.2-contributor-free' } + 'cline' { 'deepseek/deepseek-v4-flash' } + } Write-TestJson -Path $profilePath -Value ([ordered]@{ schema = (Get-RunnerSchemaNames).Profile runner = if ($runnerName -eq 'copilot') { 'github-copilot' } else { $runnerName } - provider = $profileProvider model = $profileModel reasoning_effort = 'medium' configuration_profile = 'isolated-default' @@ -330,7 +349,7 @@ if ($harness -eq 'codex') { Assert-True (-not $execution.worker_gh_token_visible) 'Copilot secret GH_TOKEN is unavailable to the worker probe' Assert-True (-not $execution.worker_github_token_visible) 'Copilot secret GITHUB_TOKEN is unavailable to the worker probe' } else { - Assert-True $execution.worker_provider_visible "$runnerName credential visibility limitation is recorded by the worker probe" + Assert-True (-not $execution.worker_provider_visible) "$runnerName free-model fixture does not require a provider API key" } $args = @($execution.args) foreach ($forbidden in @('--continue', '--session', '--resume')) { Assert-True ($args -notcontains $forbidden) "$runnerName does not pass '$forbidden'" } @@ -339,12 +358,16 @@ if ($harness -eq 'codex') { Assert-True ($args -contains 'never') 'Codex approval policy is never' Assert-True ($args -contains '--sandbox' -and $args -contains 'workspace-write') 'Codex retains workspace-write sandbox' Assert-True ($args -notcontains '--approve-for-me') 'Codex avoids the conflicting approve-for-me flag' + $modelIndex = [Array]::IndexOf([string[]]$args, '--model') + Assert-Equal 'gpt-5.6-luna' $args[$modelIndex + 1] 'Codex opaque model selector propagates to the CLI invocation' $outputIndex = [Array]::IndexOf([string[]]$args, '--output-last-message') Assert-Equal (Join-Path $with.Root 'evidence\codex-final.txt') $args[$outputIndex + 1] 'Codex output path is host-visible on Windows' } elseif ($runnerName -eq 'opencode') { Assert-True ($args -notcontains '--pure') 'OpenCode preserves repository-owned project configuration' Assert-True ($args -contains '--auto') 'OpenCode is noninteractive' Assert-True $execution.project_config_visible 'OpenCode paired arm retains repository-owned project configuration' + $modelIndex = [Array]::IndexOf([string[]]$args, '--model') + Assert-Equal 'opencode/muse-spark-1.2-contributor-free' $args[$modelIndex + 1] 'OpenCode opaque model selector propagates to the CLI invocation' } elseif ($runnerName -eq 'copilot') { Assert-True (@($args | Where-Object { $_ -eq '--prompt' -or $_ -eq '-p' -or $_ -like '--prompt=*' }).Count -eq 0) 'Copilot does not place the prompt in argv' Assert-Equal 0 $execution.prompt_arg_count 'Copilot has no prompt argument' @@ -365,10 +388,12 @@ if ($harness -eq 'codex') { Assert-True (-not $execution.ambient_copilot_instructions_visible) 'Copilot does not see the ambient personal instruction file' Assert-Equal 'explicit_environment' $execution.copilot_authentication_source 'Copilot uses explicit environment authentication in the token fixture' Assert-Equal 3 @($execution.copilot_auth_names_present).Count 'Copilot process receives all protected token variables without logging values' + Assert-True ([string]::IsNullOrWhiteSpace([string]$execution.gh_config_dir)) 'Copilot explicit-token path does not forward host GH_CONFIG_DIR' Assert-True (Test-PathInside -BasePath (Join-Path $with.Root 'home') -CandidatePath ([string]$execution.copilot_cache_home)) 'Copilot cache is run-local' Assert-True (Test-PathInside -BasePath (Join-Path $with.Root 'home') -CandidatePath ([string]$execution.copilot_home)) 'Copilot COPILOT_HOME is the run''s isolated home' Assert-Equal 'stdin' $resultWith.evidence.prompt_delivery 'Copilot result records stdin prompt delivery' Assert-Equal 'COPILOT_GITHUB_TOKEN' $resultWith.evidence.credential.github_token_variable 'Copilot follows explicit token precedence' + Assert-True (-not $resultWith.evidence.credential.github_cli_config_forwarded) 'Copilot result records that GH_CONFIG_DIR was not forwarded with an explicit token' Assert-Equal 'supported' $resultWith.isolation.capabilities.credential_child_filtering 'Copilot documents protected child-environment filtering' Assert-Equal 'shell,mcp' ([string]::Join(',', @($resultWith.evidence.credential.secret_env_var_scope))) 'Copilot evidence names the documented filtering scope' Assert-Equal 'recorded Copilot final response' $resultWith.final_response.text 'Copilot final response is the last assistant message, not an intermediate one' @@ -386,6 +411,10 @@ if ($harness -eq 'codex') { Assert-Equal '0' $args[$retryIndex + 1] 'Cline disables internal retries' Assert-True ($args -notcontains '--id') 'Cline does not resume a session' Assert-True ($args -contains '--json') 'Cline uses structured output' + $providerIndex = [Array]::IndexOf([string[]]$args, '--provider') + Assert-Equal 'deepseek' $args[$providerIndex + 1] 'Cline derives its native provider argument from the opaque selector' + $modelIndex = [Array]::IndexOf([string[]]$args, '--model') + Assert-Equal 'deepseek-v4-flash' $args[$modelIndex + 1] 'Cline derives its native model argument from the opaque selector' $configIndex = [Array]::IndexOf([string[]]$args, '--config') Assert-True ($args[$configIndex + 1] -match '(?i)[\\/]\.cline$') 'Cline uses the documented isolated config root' $dataIndex = [Array]::IndexOf([string[]]$args, '--data-dir') @@ -394,22 +423,22 @@ if ($harness -eq 'codex') { Assert-True ([int]$resultWith.telemetry.tool_calls.value -ge 1) 'Cline parses documented tool events' } $logText = [System.IO.File]::ReadAllText($logPath, [System.Text.UTF8Encoding]::new($false)) - Assert-True ($logText -notmatch 'recorded-canary|recorded-unrelated-canary|recorded-copilot-canary|recorded-gh-canary|recorded-github-canary') "$runnerName logs do not contain credential values" - Assert-True (($resultWith | ConvertTo-Json -Depth 100) -notmatch 'recorded-canary|recorded-unrelated-canary|recorded-copilot-canary|recorded-gh-canary|recorded-github-canary') "$runnerName result evidence does not contain credential values" + Assert-True ($logText -notmatch 'recorded-canary|recorded-unrelated-canary|recorded-copilot-canary|recorded-gh-canary|recorded-github-canary|recorded-gh-fallback-token') "$runnerName logs do not contain credential values" + Assert-True (($resultWith | ConvertTo-Json -Depth 100) -notmatch 'recorded-canary|recorded-unrelated-canary|recorded-copilot-canary|recorded-gh-canary|recorded-github-canary|recorded-gh-fallback-token') "$runnerName result evidence does not contain credential values" $withoutLogPath = Join-Path $without.Root "repo\$runnerName-fake-cli-log.jsonl" Assert-True (Test-Path -LiteralPath $withoutLogPath -PathType Leaf) "$runnerName baseline process log exists" $withoutRecords = @(Get-Content -LiteralPath $withoutLogPath | ForEach-Object { $_ | ConvertFrom-Json }) $withoutExecution = @($withoutRecords | Where-Object { $_.stdin_received -eq $true -or $_.prompt_via_arg -eq $true }) Assert-Equal 1 $withoutExecution.Count "$runnerName baseline has one execution process" $withoutLogText = [System.IO.File]::ReadAllText($withoutLogPath, [System.Text.UTF8Encoding]::new($false)) - Assert-True ($withoutLogText -notmatch 'recorded-canary|recorded-unrelated-canary|recorded-copilot-canary|recorded-gh-canary|recorded-github-canary') "$runnerName baseline log does not contain credential values" + Assert-True ($withoutLogText -notmatch 'recorded-canary|recorded-unrelated-canary|recorded-copilot-canary|recorded-gh-canary|recorded-github-canary|recorded-gh-fallback-token') "$runnerName baseline log does not contain credential values" Assert-True $withoutExecution[0].stdin_exact "$runnerName baseline receives exact prompt bytes" if ($runnerName -eq 'copilot') { Assert-True $withoutExecution[0].repository_agents_visible 'Copilot baseline sees the same staged AGENTS.md instruction' Assert-True $withoutExecution[0].repository_copilot_instructions_visible 'Copilot baseline sees the same staged repository instruction' Assert-True (-not $withoutExecution[0].candidate_skill_staged) 'Copilot baseline does not receive the candidate skill directory' Assert-True (-not $withoutExecution[0].ambient_copilot_instructions_visible) 'Copilot baseline excludes the ambient personal instruction' - Assert-True (($resultWithout | ConvertTo-Json -Depth 100) -notmatch 'recorded-canary|recorded-unrelated-canary|recorded-copilot-canary|recorded-gh-canary|recorded-github-canary') 'Copilot baseline result evidence does not contain credential values' + Assert-True (($resultWithout | ConvertTo-Json -Depth 100) -notmatch 'recorded-canary|recorded-unrelated-canary|recorded-copilot-canary|recorded-gh-canary|recorded-github-canary|recorded-gh-fallback-token') 'Copilot baseline result evidence does not contain credential values' } } $staleCli = $fakeCli.Replace("'opencode' { '--format --dir --model --auto --pure --continue --session' }", "'opencode' { '--format --dir --model --pure --continue --session' }") @@ -464,9 +493,10 @@ if ($harness -eq 'codex') { Assert-Equal 'compatible' $copilotGhPreflight.status 'Copilot GitHub CLI fallback remains compatible' $copilotGhResult = Invoke-AdapterJson -RunnerPath (Join-Path $runnerRoot 'github-copilot\runner.ps1') -Command execute -RunPath $with.Path -ProfilePath $recordedProfiles['copilot'] Assert-Equal 'completed' $copilotGhResult.status 'Copilot GitHub CLI fallback fixture executes without an exported token' - Assert-True $copilotGhResult.evidence.credential.github_cli_config_forwarded 'Copilot records GitHub CLI auth-state forwarding without a credential value' - $ghRecords = @(Get-Content -LiteralPath (Join-Path $with.Root 'repo\copilot-fake-cli-log.jsonl') | ForEach-Object { $_ | ConvertFrom-Json } | Where-Object { $_.copilot_authentication_source -eq 'github_cli' }) - Assert-Equal 1 $ghRecords.Count 'Copilot fake observes the simulated GitHub CLI fallback path' + Assert-True $copilotGhResult.evidence.credential.github_cli_token_resolved 'Copilot records GitHub CLI token fallback without storing the token value' + Assert-True (-not $copilotGhResult.evidence.credential.github_cli_config_forwarded) 'Copilot GitHub CLI fallback does not forward host GH_CONFIG_DIR' + $ghRecords = @(Get-Content -LiteralPath (Join-Path $with.Root 'repo\copilot-fake-cli-log.jsonl') | ForEach-Object { $_ | ConvertFrom-Json } | Where-Object { $_.stdin_received -eq $true -and @($_.copilot_auth_names_present).Count -eq 1 -and @($_.copilot_auth_names_present) -contains 'GH_TOKEN' -and [string]::IsNullOrWhiteSpace([string]$_.gh_config_dir) }) + Assert-Equal 1 $ghRecords.Count 'Copilot fake observes only the protected GH_TOKEN produced by trusted GitHub CLI fallback' $copilotNoAuthHome = Join-Path $recordedRoot 'copilot-no-auth-home' New-Item -ItemType Directory -Path $copilotNoAuthHome -Force | Out-Null @@ -607,7 +637,6 @@ try { Write-TestJson -Path $profilePath -Value ([ordered]@{ schema = (Get-RunnerSchemaNames).Profile runner = 'fake' - provider = 'fixture-provider' model = 'fixture-model' reasoning_effort = 'high' configuration_profile = 'isolated-default' @@ -619,7 +648,6 @@ try { Write-TestJson -Path $unsupportedProfilePath -Value ([ordered]@{ schema = (Get-RunnerSchemaNames).Profile runner = 'fake' - provider = 'fixture-provider' model = 'fixture-model' reasoning_effort = $null configuration_profile = 'unsupported' @@ -627,6 +655,18 @@ try { timeout_seconds = 30 concurrency = 1 }) + $legacyProviderProfilePath = Join-Path $iteration 'legacy-provider-profile.json' + Write-TestJson -Path $legacyProviderProfilePath -Value ([ordered]@{ + schema = (Get-RunnerSchemaNames).Profile + runner = 'fake' + provider = 'fixture-provider' + model = 'fixture-model' + reasoning_effort = 'high' + configuration_profile = 'isolated-default' + tool_profile = 'default' + timeout_seconds = 30 + concurrency = 1 + }) $with = New-TestRun -IterationDirectory $iteration -Configuration with_skill $without = New-TestRun -IterationDirectory $iteration -Configuration without_skill @@ -637,6 +677,7 @@ try { Assert-Equal 'fake' $descriptor.name 'descriptor identity' Assert-Equal (Get-RunnerSchemaNames).Protocol $descriptor.protocol_version 'descriptor protocol' Assert-Throws { Assert-RunnerDescriptor -Descriptor ([pscustomobject]@{ schema = $descriptor.schema; protocol_version = 'changed'; name = 'fake' }) } 'changed protocol must fail descriptor validation' + Assert-Throws { Resolve-ExecutionProfile -ProfilePath $legacyProviderProfilePath } 'execution profile rejects the removed provider field' $unsupported = Invoke-Fake -FakePath $fakePath -Command preflight -Run $with.Path -Profile $unsupportedProfilePath Assert-Equal 'incompatible' $unsupported.status 'unsupported capability/profile must fail during preflight' @@ -650,7 +691,7 @@ try { Assert-True $result.session.fresh 'fresh session flag' Assert-True (-not $result.session.resumed) 'resume must be false' Assert-Equal 1 $result.attempt_count 'answer-quality retry is forbidden' - Assert-Equal 'fixture-provider' $result.requested.provider 'provider must pass unchanged' + Assert-True ($result.requested.PSObject.Properties.Name -notcontains 'provider') 'portable execution result must not expose provider' Assert-Equal 'fixture-model' $result.requested.model 'model must pass unchanged' Assert-Equal 'high' $result.requested.reasoning_effort 'reasoning effort must pass unchanged' Assert-Equal 'isolated-default' $result.requested.configuration_profile 'configuration profile must pass unchanged' @@ -786,9 +827,9 @@ try { $prepareText = [System.IO.File]::ReadAllText((Join-Path $repoRoot 'scripts\prepare-skill-evals.ps1'), [System.Text.UTF8Encoding]::new($false)) $reportText = [System.IO.File]::ReadAllText((Join-Path $repoRoot 'scripts\generate-eval-report.ps1'), [System.Text.UTF8Encoding]::new($false)) $bridgeText = [System.IO.File]::ReadAllText((Join-Path $runnerRoot 'bridge-execution-result.ps1'), [System.Text.UTF8Encoding]::new($false)) - Assert-True ($prepareText -notmatch '(?i)codex\s+exec|opencode\s+run|cline\s+--|copilot\s+-p|copilot\s+--prompt|github-copilot') 'portable preparation must not contain harness-specific CLI invocations or runner branches' - Assert-True ($reportText -notmatch '(?i)codex\s+exec|opencode\s+run|cline\s+--|copilot\s+-p|copilot\s+--prompt|github-copilot') 'reporting must not contain harness-specific branches' - Assert-True ($bridgeText -notmatch '(?i)codex\s+exec|opencode\s+run|cline\s+--|copilot\s+-p|copilot\s+--prompt|github-copilot') 'the raw-to-portable bridge must remain runner-neutral' + Assert-True ($prepareText -notmatch '(?i)codex\s+exec|opencode\s+run|cline\s+--|copilot\s+-p|copilot\s+--prompt|Profile\.Provider') 'portable preparation must not contain harness-specific CLI invocations or provider-field branches' + Assert-True ($reportText -notmatch '(?i)codex\s+exec|opencode\s+run|cline\s+--|copilot\s+-p|copilot\s+--prompt|Profile\.Provider') 'reporting must not contain harness-specific or provider-field branches' + Assert-True ($bridgeText -notmatch '(?i)codex\s+exec|opencode\s+run|cline\s+--|copilot\s+-p|copilot\s+--prompt|Profile\.Provider') 'the raw-to-portable bridge must remain runner-neutral' $rawPath = Join-Path $iteration 'conformance\results\with-skill.execution-result.json' $resultPath = Join-Path $iteration 'conformance\results\with-skill.result.json' @@ -802,7 +843,7 @@ try { Assert-Equal 'codebeltnet/agentic/eval-result/2' $portable.schema 'bridge preserves existing result schema' Assert-Equal 'completed' $portable.execution_status 'bridge carries execution status' Assert-Equal 'fixture-model' $portable.model 'bridge carries resolved model' - Assert-Equal 'fixture-provider' $portable.provider 'bridge carries resolved provider' + Assert-True ($portable.PSObject.Properties.Name -notcontains 'provider') 'bridge removes provider from portable result' Assert-True ($null -eq $portable.total_tokens) 'bridge keeps unavailable total tokens unavailable' Assert-Equal 0 $portable.tool_calls 'bridge carries available tool-call count' Assert-True $portable.isolation.transcript_captured 'bridge carries transcript availability' @@ -813,7 +854,6 @@ try { $acceptedBridgeResult = $bridgeResult | ConvertTo-Json -Depth 100 | ConvertFrom-Json $acceptedBridgeResult.resolved.status = 'accepted_request' - $acceptedBridgeResult.resolved.provider = $null $acceptedBridgeResult.resolved.model = $null $acceptedBridgeResult.resolved.reason = 'fixture accepted the requested alias without exposing backend resolution.' Write-TestJson -Path $rawPath -Value $acceptedBridgeResult diff --git a/scripts/generate-eval-report.ps1 b/scripts/generate-eval-report.ps1 index a0123be..c28bfe8 100644 --- a/scripts/generate-eval-report.ps1 +++ b/scripts/generate-eval-report.ps1 @@ -400,11 +400,8 @@ function Get-ReportRun { configuration = $Configuration feedback_key = "eval-$EvalId-$Configuration" model = [string](Get-Property -Object $Result -Name 'model' -Default '') - provider = [string](Get-Property -Object $Result -Name 'provider' -Default '') requested_model = [string](Get-Property -Object $Result -Name 'requested_model' -Default '') - requested_provider = [string](Get-Property -Object $Result -Name 'requested_provider' -Default '') resolved_model = [string](Get-Property -Object $Result -Name 'resolved_model' -Default '') - resolved_provider = [string](Get-Property -Object $Result -Name 'resolved_provider' -Default '') configuration_resolution_status = [string](Get-Property -Object $Result -Name 'configuration_resolution_status' -Default '') configuration_resolution_reason = [string](Get-Property -Object $Result -Name 'configuration_resolution_reason' -Default '') harness = [string](Get-Property -Object $Result -Name 'harness' -Default '') @@ -465,7 +462,6 @@ function Write-FirstPartyReport { $evals = [System.Collections.Generic.List[object]]::new() $allModels = [System.Collections.Generic.List[string]]::new() - $allProviders = [System.Collections.Generic.List[string]]::new() $completedRuns = 0 foreach ($entry in @($Manifest.evals)) { $evalDirectory = Join-Path $IterationPath ([string]$entry.directory) @@ -483,9 +479,6 @@ function Write-FirstPartyReport { if (-not [string]::IsNullOrWhiteSpace([string]$run.model) -and -not $allModels.Contains([string]$run.model)) { $allModels.Add([string]$run.model) } - if (-not [string]::IsNullOrWhiteSpace([string]$run.provider) -and -not $allProviders.Contains([string]$run.provider)) { - $allProviders.Add([string]$run.provider) - } } else { $runMap[$configuration] = $null } @@ -502,7 +495,6 @@ function Write-FirstPartyReport { $metadata = [ordered]@{ model = if ($allModels.Count -gt 0) { $allModels -join ', ' } else { $null } - provider = if ($allProviders.Count -gt 0) { $allProviders -join ', ' } else { $null } completed_runs = $completedRuns expected_runs = @($Manifest.evals).Count * 2 generated_utc = [string](Get-Property -Object $Manifest -Name 'generated_utc' -Default '') diff --git a/scripts/prepare-skill-evals.ps1 b/scripts/prepare-skill-evals.ps1 index def13c9..7663f36 100644 --- a/scripts/prepare-skill-evals.ps1 +++ b/scripts/prepare-skill-evals.ps1 @@ -5,7 +5,7 @@ .DESCRIPTION This script computes and prints. It never executes a prompt, never spawns an agent, and never calls a model. It turns skills//evals/evals.json into a paste-ready evaluation package that a human can run in whatever - harness, provider, and model they choose, then validates the results that come back. + harness and model they choose, then validates the results that come back. Prepare mode writes one directory per eval. The grading key and result stubs stay at the eval-case level, outside the two isolated run directories a worker actually sees: @@ -45,14 +45,17 @@ Overwrite an existing iteration directory. .PARAMETER Runner - Optional runner name written to execution-profile.json. If omitted, the profile remains unselected and an external - orchestrator must fail clearly rather than guess a runner. - -.PARAMETER Provider - Optional provider name written to execution-profile.json. + Required package-local Eval Runner id written to execution-profile.json when -CodebeltReference is not used. .PARAMETER Model - Optional model identifier written to execution-profile.json. + Required runner-native model selector written to execution-profile.json when -CodebeltReference is not used. + +.PARAMETER CodebeltReference + Resolve the Codebelt reference configuration by discovering current GitHub Copilot CLI models and selecting + claude-haiku-4.5 only when it is still available. + +.PARAMETER ModelCatalogPath + Optional deterministic catalog JSON used by the model discovery helper. Intended for tests and offline validation. .PARAMETER ReasoningEffort Optional runner-supported reasoning/effort setting written to execution-profile.json. @@ -83,10 +86,10 @@ This is the fallback for results that were not finalized by the external evaluator. .EXAMPLE - pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -Skill dotnet-strong-name-signing + pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -Skill dotnet-strong-name-signing -Runner github-copilot -Model claude-haiku-4.5 .EXAMPLE - pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -Changed + pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -Changed -CodebeltReference .EXAMPLE pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -CollectResults $env:TEMP/dotnet-strong-name-signing-workspace/iteration-1 @@ -126,11 +129,15 @@ param( [Parameter(ParameterSetName = 'Prepare')] [Parameter(ParameterSetName = 'Changed')] - [string]$Provider, + [string]$Model, [Parameter(ParameterSetName = 'Prepare')] [Parameter(ParameterSetName = 'Changed')] - [string]$Model, + [switch]$CodebeltReference, + + [Parameter(ParameterSetName = 'Prepare')] + [Parameter(ParameterSetName = 'Changed')] + [string]$ModelCatalogPath, [Parameter(ParameterSetName = 'Prepare')] [Parameter(ParameterSetName = 'Changed')] @@ -381,6 +388,91 @@ function Assert-WorkspaceLocation { } } +function Get-HarnessName { + param([Parameter(Mandatory = $true)][string]$RunnerName) + + switch ($RunnerName) { + 'github-copilot' { return 'GitHub Copilot CLI' } + 'codex' { return 'Codex CLI' } + 'opencode' { return 'OpenCode' } + 'cline' { return 'Cline' } + 'fake' { return 'Deterministic fake runner' } + default { return $RunnerName } + } +} + +function Get-SupportedRunnerIds { + param([Parameter(Mandatory = $true)][string]$RepoRoot) + + $runnerRoot = Join-Path $RepoRoot 'scripts/eval-runners' + if (-not (Test-Path -LiteralPath $runnerRoot -PathType Container)) { + return @() + } + + return @(Get-ChildItem -LiteralPath $runnerRoot -Directory -Force | + Where-Object { Test-Path -LiteralPath (Join-Path $_.FullName 'runner.ps1') -PathType Leaf } | + Sort-Object Name | + ForEach-Object { $_.Name }) +} + +function Resolve-ExecutionSelection { + param([Parameter(Mandatory = $true)][string]$RepoRoot) + + $referenceRunner = 'github-copilot' + $referenceModel = 'claude-haiku-4.5' + $supportedRunners = @(Get-SupportedRunnerIds -RepoRoot $RepoRoot) + $supportedText = if ($supportedRunners.Count -gt 0) { $supportedRunners -join ', ' } else { '(none found)' } + + if ($CodebeltReference -and (-not [string]::IsNullOrWhiteSpace($Runner) -or -not [string]::IsNullOrWhiteSpace($Model))) { + throw 'Choose either -CodebeltReference or an explicit -Runner/-Model pair, not both.' + } + + if ($CodebeltReference) { + if ($supportedRunners -notcontains $referenceRunner) { + throw "Codebelt Reference requires runner '$referenceRunner', but it is unavailable. Supported runner IDs: $supportedText." + } + $discoveryScript = Join-Path $RepoRoot 'scripts/Get-HarnessModels.ps1' + if (-not (Test-Path -LiteralPath $discoveryScript -PathType Leaf)) { + throw "Cannot resolve Codebelt Reference because '$discoveryScript' is missing." + } + + $arguments = @('-Runner', $referenceRunner, '-RequireModel', $referenceModel) + if (-not [string]::IsNullOrWhiteSpace($ModelCatalogPath)) { + $arguments += @('-CatalogPath', $ModelCatalogPath) + } + $discoveryOutput = & pwsh -NoProfile -File $discoveryScript @arguments 2>&1 + if ($LASTEXITCODE -ne 0) { + throw "Codebelt Reference requires $referenceRunner + $referenceModel, but current model discovery could not verify it. $($discoveryOutput -join [Environment]::NewLine)" + } + + return [pscustomobject]@{ + Runner = $referenceRunner + Model = $referenceModel + Harness = Get-HarnessName -RunnerName $referenceRunner + Preset = 'Codebelt Reference' + } + } + + $hasRunner = -not [string]::IsNullOrWhiteSpace($Runner) + $hasModel = -not [string]::IsNullOrWhiteSpace($Model) + if (-not $hasRunner -and -not $hasModel) { + throw "Evaluation preparation requires a resolved Harness + Model before RUN-THIS.prompt.md can be generated. Pass -Runner and -Model, or use -CodebeltReference after verifying the current catalog. Supported runner IDs: $supportedText." + } + if ($hasRunner -ne $hasModel) { + throw 'Runner/model selection is atomic: pass both -Runner and -Model, or neither when no package will be generated.' + } + if ($supportedRunners -notcontains $Runner) { + throw "Unsupported runner '$Runner'. Supported runner IDs: $supportedText." + } + + return [pscustomobject]@{ + Runner = $Runner + Model = $Model + Harness = Get-HarnessName -RunnerName $Runner + Preset = 'Custom' + } +} + function Get-EvalName { param([object]$EvalEntry) @@ -572,7 +664,6 @@ function New-ResultStub { eval_name = $EvalName configuration = $Configuration model = '' - provider = '' harness = '' executed_utc = '' output = '' @@ -612,11 +703,12 @@ function New-ResultStub { } function New-ExecutionProfile { + param([Parameter(Mandatory = $true)][object]$ExecutionSelection) + return [ordered]@{ schema = $executionProfileSchema - runner = if ([string]::IsNullOrWhiteSpace($Runner)) { $null } else { $Runner } - provider = if ([string]::IsNullOrWhiteSpace($Provider)) { $null } else { $Provider } - model = if ([string]::IsNullOrWhiteSpace($Model)) { $null } else { $Model } + runner = $ExecutionSelection.Runner + model = $ExecutionSelection.Model reasoning_effort = if ([string]::IsNullOrWhiteSpace($ReasoningEffort)) { $null } else { $ReasoningEffort } configuration_profile = $ConfigurationProfile tool_profile = $ToolProfile @@ -1185,6 +1277,8 @@ function Invoke-PrepareMode { throw "No evals selected for '$Skill'." } + $executionSelection = Resolve-ExecutionSelection -RepoRoot $repoRoot + $workspaceRoot = if ([string]::IsNullOrWhiteSpace($OutputRoot)) { Join-Path (Join-Path $repoRoot '.bot') "$Skill-workspace" } else { @@ -1219,7 +1313,7 @@ function Invoke-PrepareMode { $skillCreatorSourcePath = Resolve-SkillCreatorSourcePath -RequestedPath $null [void](Copy-SkillCreatorEvalTools -IterationDirectory $iterationDirectory -SourcePath $skillCreatorSourcePath) [void](Copy-EvalRunnerTools -RepoRoot $repoRoot -IterationDirectory $iterationDirectory) - ConvertTo-JsonFile -Path (Join-Path $iterationDirectory 'execution-profile.json') -Value (New-ExecutionProfile) + ConvertTo-JsonFile -Path (Join-Path $iterationDirectory 'execution-profile.json') -Value (New-ExecutionProfile -ExecutionSelection $executionSelection) $skillText = [System.IO.File]::ReadAllText($skillMarkdownPath, $utf8NoBom) $skillBody = if ($skillText -match '(?ms)\A---\r?\n.*?\r?\n---\r?\n(?.*)\z') { $Matches['body'] } else { $skillText } @@ -1414,6 +1508,12 @@ function Invoke-PrepareMode { generated_utc = $generatedUtc configurations = @('with_skill', 'without_skill') execution = 'runner_handoff' + execution_selection = [ordered]@{ + harness = $executionSelection.Harness + runner = $executionSelection.Runner + model = $executionSelection.Model + preset = $executionSelection.Preset + } runner_prompt = 'RUN-THIS.prompt.md' execution_profile = 'execution-profile.json' runner_protocol = $runnerProtocolSchema @@ -1465,6 +1565,19 @@ function Invoke-PrepareMode { $runnerPath = Join-Path $iterationDirectory 'RUN-THIS.prompt.md' Write-Utf8File -Path $runnerPath -Content (New-RunnerPrompt -IterationDirectory $iterationDirectory -IterationNumber $iterationNumber -ManifestEvals @($manifestEvals)) + Write-Host 'Evaluation package prepared.' + Write-Host '' + Write-Host 'Execution:' + Write-Host " Harness: $($executionSelection.Harness)" + Write-Host " Runner: $($executionSelection.Runner)" + Write-Host " Model: $($executionSelection.Model)" + if (-not [string]::IsNullOrWhiteSpace([string]$executionSelection.Preset)) { + Write-Host " Preset: $($executionSelection.Preset)" + } + Write-Host '' + Write-Host "Cases: $($manifestEvals.Count)" + Write-Host "Arms: $($manifestEvals.Count * 2)" + Write-Host '' Write-Host "Prepared $($manifestEvals.Count) eval case(s) for '$Skill' (iteration $iterationNumber) as $($manifestEvals.Count * 2) isolated run package(s)." Write-Host "Package: $iterationDirectory" Write-Host '' @@ -1508,16 +1621,16 @@ function New-RunnerPrompt { [void]$builder.AppendLine("Profile: $profilePath") [void]$builder.AppendLine("Runner resolver: $resolverPath") [void]$builder.AppendLine() - [void]$builder.AppendLine('This is a runner-aware package. `run.json` is the existing portable one-arm contract: it defines the prompt, staged files, working directory, isolated home, candidate-skill exposure, and required isolation. `execution-profile.json` selects the runner/provider/model/configuration and carries the execution limits. The selected runner defines how its harness satisfies the contract.') + [void]$builder.AppendLine('This is a runner-aware package. `run.json` is the existing portable one-arm contract: it defines the prompt, staged files, working directory, isolated home, candidate-skill exposure, and required isolation. `execution-profile.json` selects the runner/model/configuration and carries the execution limits. The selected runner defines how its harness satisfies the contract.') [void]$builder.AppendLine() [void]$builder.AppendLine('A human selected the external orchestrator and authorized this handoff. Repository preparation, validation, CI, hooks, and automatic completion gates remain model-free. Do not substitute a generic worker, another runner, or an improvised isolation scheme if the selected runner is unavailable or incompatible.') [void]$builder.AppendLine() [void]$builder.AppendLine('## Phase 1: execute blind arms') [void]$builder.AppendLine() - [void]$builder.AppendLine('1. Read `manifest.json` and `execution-profile.json`. If `runner` is null, unavailable, or unsupported, fail clearly; do not guess a default. The profile contains no credentials.') + [void]$builder.AppendLine('1. Read `manifest.json` and `execution-profile.json`. If `runner` or `model` is null, unavailable, or unsupported, fail clearly and list the supported package-local runner IDs; do not guess a default. The profile contains no credentials.') [void]$builder.AppendLine('2. Resolve the selected package-local runner with the resolver. Ask it for `describe` and validate its protocol, descriptor, and capability declarations before running an arm. Do not invent harness-specific CLI commands.') [void]$builder.AppendLine('3. For every eval case, use the exact `run_manifest` path from `manifest.json` and the same profile path. Preflight each arm, then invoke the runner exactly once with `execute`. The runner receives only `run.json` and `execution-profile.json`; it must never receive or inspect expected output, assertions, grading, paired output, benchmark data, or human feedback.') - [void]$builder.AppendLine('4. Keep `with_skill` and `without_skill` in fresh independent processes/sessions. Use the same model, provider, configuration, tools, and limits. The runner must send each `prompt.md` unchanged as the first task input and must enforce the run contract, including the baseline skill exclusion and the staged filesystem/workspace boundary.') + [void]$builder.AppendLine('4. Keep `with_skill` and `without_skill` in fresh independent processes/sessions. Use the same runner-native model selector, configuration, tools, and limits. The runner must send each `prompt.md` unchanged as the first task input and must enforce the run contract, including the baseline skill exclusion and the staged filesystem/workspace boundary.') [void]$builder.AppendLine('5. Save the runner''s single normalized JSON response unchanged as the matching `execution_result` path. Preserve the complete final response, status, telemetry, evidence references, hashes, isolation mechanisms, warnings, and compatibility deviations. Do not retry for answer quality. A refusal is a result; timeout, harness failure, and incompatibility are results.') [void]$builder.AppendLine('6. If the runner cannot satisfy a required guarantee, keep the normalized status `incompatible` and stop that arm. Never fall back to the old generic isolated-worker behavior and never substitute a different runner.') [void]$builder.AppendLine() @@ -1540,7 +1653,7 @@ function New-RunnerPrompt { [void]$builder.AppendLine('3. Write only `grading[].text`, `grading[].passed`, and `grading[].evidence` for grading. Do not alter raw execution results. Use null for genuinely unavailable judgement and leave missing arms visibly missing.') [void]$builder.AppendLine(('4. Run the existing package report adapter now: `pwsh -NoProfile -File "' + $reportPath + '" -IterationDirectory "' + $IterationDirectory + '"`. It remains the bridge to Anthropic skill-creator''s grader-compatible aggregator and viewer; do not replace it with harness-specific reporting. The packaged compatibility tools remain `scripts/aggregate_benchmark.py` and `eval-viewer/generate_review.py`.')) [void]$builder.AppendLine() - [void]$builder.AppendLine('The completion artifacts are `report.html`, `skill-creator-report.html`, `benchmark.json`, and `benchmark.md` at the package root. Return their absolute paths, completed and missing arm counts, runner/provider/model identity, and a concise evidence-backed summary. If the package cannot be written from the external environment, return one paste-ready block containing the completed result objects and report artifacts.') + [void]$builder.AppendLine('The completion artifacts are `report.html`, `skill-creator-report.html`, `benchmark.json`, and `benchmark.md` at the package root. Return their absolute paths, completed and missing arm counts, runner/model identity, and a concise evidence-backed summary. If the package cannot be written from the external environment, return one paste-ready block containing the completed result objects and report artifacts.') [void]$builder.AppendLine() [void]$builder.AppendLine("This package contains $(@($ManifestEvals).Count) eval case(s), each with paired `with_skill` and `without_skill` runs. The human reviewer remains the final evaluator.") return $builder.ToString() @@ -1557,7 +1670,7 @@ function New-PackageReadme { $builder = [System.Text.StringBuilder]::new() [void]$builder.AppendLine("# Eval package: $SkillName (iteration $IterationNumber)") [void]$builder.AppendLine() - [void]$builder.AppendLine('Prepared by `scripts/prepare-skill-evals.ps1` in `codebeltnet/agentic`. Nothing in this package was executed. `execution-profile.json` selects the user-chosen Eval Runner, provider, model, and limits; the external Eval Orchestrator runs both configurations, grades them, and generates the report.') + [void]$builder.AppendLine('Prepared by `scripts/prepare-skill-evals.ps1` in `codebeltnet/agentic`. Nothing in this package was executed. `execution-profile.json` selects the user-chosen Eval Runner, runner-native model, and limits; the external Eval Orchestrator runs both configurations, grades them, and generates the report.') [void]$builder.AppendLine() [void]$builder.AppendLine('## What is here') [void]$builder.AppendLine() @@ -1577,7 +1690,7 @@ function New-PackageReadme { [void]$builder.AppendLine() [void]$builder.AppendLine('## How to run') [void]$builder.AppendLine() - [void]$builder.AppendLine('1. Read `execution-profile.json`. If no runner is selected, fail clearly instead of guessing. Resolve the selected package-local runner and run `describe`, then `preflight`, for every arm.') + [void]$builder.AppendLine('1. Read `execution-profile.json`. If `runner` or `model` is missing, fail clearly instead of guessing. Resolve the selected package-local runner and run `describe`, then `preflight`, for every arm.') [void]$builder.AppendLine('2. Invoke `execute` exactly once for each `run.json`, preserving its one JSON `execution-result.json` unchanged. The runner must provide a fresh process/session, isolated home/config, isolated CWD, baseline skill exclusion, prompt fidelity, model/configuration lock, and complete response capture, or return `incompatible`. Hard filesystem confinement is not one of these mandatory controls: when the runner proves it the run reports strict isolation, and when it does not the run reports pragmatic isolation.') [void]$builder.AppendLine('3. After all arms complete or fail, run `tools/eval-runners/bridge-execution-result.ps1` for each raw result. It writes the existing `eval-result/2` file and preserves explicit unavailable telemetry. Only then read the grading key, grade with `tools/skill-creator/agents/grader.md`, and run `tools/generate-eval-report.ps1`.') [void]$builder.AppendLine() @@ -1591,7 +1704,7 @@ function New-PackageReadme { [void]$builder.AppendLine() [void]$builder.AppendLine('Fill in each `results/*.result.json`:') [void]$builder.AppendLine() - [void]$builder.AppendLine('- `model`, `provider`, `harness` - what actually ran it, as specifically as you know') + [void]$builder.AppendLine('- `model`, `harness` - what actually ran it, as specifically as you know') [void]$builder.AppendLine('- `executed_utc` - when') [void]$builder.AppendLine('- `output` - the produced output, or a summary plus paths in `output_files`') [void]$builder.AppendLine('- `transcript`, `shell_commands`, `files_read`, `files_written`, `exit_status`, `duration_seconds`, `total_tokens`, `tool_calls` - include the values the harness exposes; omit unavailable values rather than estimating them') @@ -1815,7 +1928,6 @@ function Invoke-CollectMode { $observed[$configuration] = [pscustomobject]@{ Model = $model - Provider = [string](Get-JsonProperty -Object $result -Name 'provider' -Default '') Graded = $graded.Count Passed = $passed Total = $total diff --git a/scripts/validate-skill-templates.ps1 b/scripts/validate-skill-templates.ps1 index 80a2816..163ca4a 100644 --- a/scripts/validate-skill-templates.ps1 +++ b/scripts/validate-skill-templates.ps1 @@ -1269,7 +1269,10 @@ Add-ValidationResult -Results $results -Name 'Skill evaluation prepares portable Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle 'the same model, the same version, and the same configuration' Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle 'a baseline handed the answer key is not a baseline' Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle 'repository automation remains deterministic and never invokes a model.' - Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle 'pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -Skill ' + Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle 'Resolve the execution configuration before running the package preparation script.' + Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle 'scripts/Get-HarnessModels.ps1' + Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle 'Cline and OpenCode discovery is free-only' + Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle 'pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -Skill -Runner -Model ' Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle 'pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -CollectResults ' Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle '### Handing the package over' Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle '### Executing a package you were handed' @@ -1284,7 +1287,7 @@ Add-ValidationResult -Results $results -Name 'Skill evaluation prepares portable Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle 'The user asked for eval results, not a second workflow decision.' Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle '### Asking for an eval' Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle '`eval `, `evaluate `' - Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle 'Run the script immediately when asked. Do not reply with a plan, a menu of options' + Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle 'Resolve the execution configuration before running the package preparation script.' Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle '### Eval preparation is a completion gate' Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle 'Adding or modifying any repo-managed skill triggers this workflow.' Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle 'pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -Changed' @@ -1293,7 +1296,8 @@ Add-ValidationResult -Results $results -Name 'Skill evaluation prepares portable Assert-Contains -Name 'README.md' -Content $readme -Needle 'a completion gate an agent cannot skip' Assert-Contains -Name 'CONTRIBUTING.md' -Content $contributing -Needle 'pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -Changed' Assert-Contains -Name 'README.md' -Content $readme -Needle 'prepares the paired candidate and baseline inputs as a portable package and stops' - Assert-Contains -Name 'CONTRIBUTING.md' -Content $contributing -Needle 'pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -Skill ' + Assert-Contains -Name 'CONTRIBUTING.md' -Content $contributing -Needle 'pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -Skill -Runner -Model ' + Assert-Contains -Name 'CONTRIBUTING.md' -Content $contributing -Needle 'Before running the script, choose a Harness + Model.' Assert-NotContains -Name 'CONTRIBUTING.md' -Content $contributing -Needle 'run-skill-benchmark.ps1' Assert-Contains -Name 'scripts/prepare-skill-evals.ps1' -Content $prepare -Needle 'Eval packages inside this repository must live under .bot/.' Assert-Contains -Name 'scripts/prepare-skill-evals.ps1' -Content $prepare -Needle 'git does not ignore it' @@ -1306,10 +1310,98 @@ Add-ValidationResult -Results $results -Name 'Skill evaluation prepares portable } $scriptPath = Join-Path $repoRoot 'scripts/prepare-skill-evals.ps1' + $modelDiscoveryPath = Join-Path $repoRoot 'scripts/Get-HarnessModels.ps1' $packageRoot = Join-Path ([System.IO.Path]::GetTempPath()) ('agentic-eval-package-' + [Guid]::NewGuid().ToString('N')) $taskMarker = "`n# Task`n" try { - $prepareOutput = & pwsh -NoProfile -File $scriptPath -Skill 'dotnet-strong-name-signing' -OutputRoot $packageRoot 2>&1 + $catalogPath = Join-Path $packageRoot 'fake-model-catalog.json' + New-Item -ItemType Directory -Path $packageRoot -Force | Out-Null + [System.IO.File]::WriteAllText($catalogPath, (@' +{ + "models": [ + { "id": "claude-haiku-4.5", "display_name": "Claude Haiku 4.5", "availability": "paid", "operation": "language" }, + { "id": "gpt-5.6-luna", "display_name": "GPT-5.6 Luna", "availability": "paid", "operation": "language" }, + { "id": "deepseek/deepseek-v4-flash", "display_name": "DeepSeek V4 Flash", "availability": "free", "operation": "language" }, + { "id": "paid-model", "display_name": "Paid Model", "availability": "paid", "operation": "language" }, + { "id": "unknown-model", "display_name": "Unknown Model", "availability": "unknown", "operation": "language" }, + { "id": "opencode/muse-spark-1.2-contributor-free", "display_name": "Muse Spark 1.2", "cost": { "input": 0, "output": 0, "cache": { "read": 0, "write": 0 } }, "operation": "language" } + ] +} +'@), $utf8NoBom) + + $copilotDiscovery = (& pwsh -NoProfile -File $modelDiscoveryPath -Runner 'github-copilot' -CatalogPath $catalogPath 2>&1) + if ($LASTEXITCODE -ne 0) { throw "Get-HarnessModels.ps1 failed for Copilot fixture: $($copilotDiscovery -join [Environment]::NewLine)" } + $copilotModels = ($copilotDiscovery -join [Environment]::NewLine) | ConvertFrom-Json + if (@($copilotModels.models).Count -ne 6) { throw 'Copilot discovery must return all available fixture models.' } + + $codexDiscovery = (& pwsh -NoProfile -File $modelDiscoveryPath -Runner 'codex' -CatalogPath $catalogPath 2>&1) + if ($LASTEXITCODE -ne 0) { throw "Get-HarnessModels.ps1 failed for Codex fixture: $($codexDiscovery -join [Environment]::NewLine)" } + $codexModels = ($codexDiscovery -join [Environment]::NewLine) | ConvertFrom-Json + if (@($codexModels.models).Count -ne 6) { throw 'Codex discovery must return all available fixture models.' } + + foreach ($runnerName in @('cline', 'opencode')) { + $discoveryOutput = & pwsh -NoProfile -File $modelDiscoveryPath -Runner $runnerName -CatalogPath $catalogPath 2>&1 + if ($LASTEXITCODE -ne 0) { throw "Get-HarnessModels.ps1 failed for ${runnerName}: $($discoveryOutput -join [Environment]::NewLine)" } + $discovery = ($discoveryOutput -join [Environment]::NewLine) | ConvertFrom-Json + $ids = @($discovery.models | ForEach-Object { [string]$_.id }) + if ($ids -notcontains 'deepseek/deepseek-v4-flash' -or $ids -notcontains 'opencode/muse-spark-1.2-contributor-free') { + throw "$runnerName discovery must retain free fixture model selectors." + } + if ($ids -contains 'paid-model' -or $ids -contains 'unknown-model') { + throw "$runnerName discovery must not include paid or unknown-availability models." + } + } + + $paidCatalogPath = Join-Path $packageRoot 'paid-model-catalog.json' + [System.IO.File]::WriteAllText($paidCatalogPath, (@' +{ + "models": [ + { "id": "paid-model", "display_name": "Paid Model", "availability": "paid", "operation": "language" }, + { "id": "unknown-model", "display_name": "Unknown Model", "availability": "unknown", "operation": "language" } + ] +} +'@), $utf8NoBom) + $noFreeOutput = & pwsh -NoProfile -File $modelDiscoveryPath -Runner 'opencode' -CatalogPath $paidCatalogPath 2>&1 + if ($LASTEXITCODE -eq 0 -or ($noFreeOutput -join ' ') -notmatch 'No free OpenCode models') { + throw 'OpenCode discovery must fail clearly when free discovery returns zero models.' + } + + $missingCatalogOutput = & pwsh -NoProfile -File $modelDiscoveryPath -Runner 'codex' -CatalogPath (Join-Path $packageRoot 'missing-catalog.json') 2>&1 + if ($LASTEXITCODE -eq 0 -or ($missingCatalogOutput -join ' ') -notmatch 'does not exist') { + throw 'Discovery failures must remain local and must not invent fallback models.' + } + + $referenceOutput = & pwsh -NoProfile -File $modelDiscoveryPath -Runner 'github-copilot' -CatalogPath $catalogPath -RequireModel 'claude-haiku-4.5' 2>&1 + if ($LASTEXITCODE -ne 0) { throw "Codebelt Reference fixture should resolve: $($referenceOutput -join [Environment]::NewLine)" } + $missingReferenceOutput = & pwsh -NoProfile -File $modelDiscoveryPath -Runner 'github-copilot' -CatalogPath $paidCatalogPath -RequireModel 'claude-haiku-4.5' 2>&1 + if ($LASTEXITCODE -eq 0 -or ($missingReferenceOutput -join ' ') -notmatch 'Required model') { + throw 'Codebelt Reference discovery must fail instead of silently substituting a model.' + } + + $referencePackageRoot = Join-Path $packageRoot 'reference-package' + $referencePrepareOutput = & pwsh -NoProfile -File $scriptPath -Skill 'dotnet-strong-name-signing' -Eval 1 -OutputRoot $referencePackageRoot -CodebeltReference -ModelCatalogPath $catalogPath 2>&1 + if ($LASTEXITCODE -ne 0) { + throw "prepare-skill-evals.ps1 -CodebeltReference failed against the fake current catalog: $($referencePrepareOutput -join [Environment]::NewLine)" + } + $referenceProfile = [System.IO.File]::ReadAllText((Join-Path $referencePackageRoot 'iteration-1\execution-profile.json'), $utf8NoBom) | ConvertFrom-Json + if ([string]$referenceProfile.runner -ne 'github-copilot' -or [string]$referenceProfile.model -ne 'claude-haiku-4.5') { + throw 'Codebelt Reference preparation must write github-copilot + claude-haiku-4.5 atomically.' + } + + $missingSelectionRoot = Join-Path ([System.IO.Path]::GetTempPath()) ('agentic-eval-missing-selection-' + [Guid]::NewGuid().ToString('N')) + $missingSelectionOutput = & pwsh -NoProfile -File $scriptPath -Skill 'dotnet-strong-name-signing' -OutputRoot $missingSelectionRoot 2>&1 + if ($LASTEXITCODE -eq 0) { + throw 'prepare-skill-evals.ps1 must refuse to generate RUN-THIS.prompt.md without a resolved runner/model selection.' + } + if (($missingSelectionOutput -join ' ') -notmatch 'requires a resolved Harness \+ Model') { + throw 'prepare-skill-evals.ps1 must explain that Harness + Model selection is required before handoff generation.' + } + if (Test-Path -LiteralPath $missingSelectionRoot) { + Remove-Item -LiteralPath $missingSelectionRoot -Recurse -Force + throw 'prepare-skill-evals.ps1 must not create an unresolved eval package.' + } + + $prepareOutput = & pwsh -NoProfile -File $scriptPath -Skill 'dotnet-strong-name-signing' -OutputRoot $packageRoot -Runner 'github-copilot' -Model 'claude-haiku-4.5' 2>&1 if ($LASTEXITCODE -ne 0) { throw "prepare-skill-evals.ps1 failed: $($prepareOutput -join [Environment]::NewLine)" } @@ -1355,7 +1447,7 @@ Add-ValidationResult -Results $results -Name 'Skill evaluation prepares portable foreach ($needle in @( 'START NOW. You are the external Eval Orchestrator', 'Do not execute evaluation prompts in the current agent context.', - 'execution-profile.json` selects the runner/provider/model/configuration', + 'execution-profile.json` selects the runner/model/configuration', 'invoke the runner exactly once with `execute`', 'never receive or inspect expected output, assertions, grading, paired output, benchmark data, or human feedback', 'Never fall back to the old generic isolated-worker behavior', @@ -1375,10 +1467,13 @@ Add-ValidationResult -Results $results -Name 'Skill evaluation prepares portable foreach ($forbidden in @( 'this context has read the runner instructions and can no longer produce a clean run', '## If you can only hold one context', - 'If you truly cannot, this package is not for you' + 'If you truly cannot, this package is not for you', + 'Choose evaluation configuration', + 'Codebelt Reference', + 'discover current models' )) { if ($runner.Contains($forbidden)) { - throw "RUN-THIS.prompt.md must not contain the refusal path '$forbidden'." + throw "RUN-THIS.prompt.md must not contain forbidden handoff text '$forbidden'." } } foreach ($entry in @($manifest.evals)) { @@ -1539,12 +1634,17 @@ Add-ValidationResult -Results $results -Name 'Skill evaluation prepares portable } $profilePath = Join-Path $iterationDirectory ([string]$manifest.execution_profile) $profile = [System.IO.File]::ReadAllText($profilePath, $utf8NoBom) | ConvertFrom-Json - foreach ($profileField in @('schema', 'runner', 'provider', 'model', 'reasoning_effort', 'configuration_profile', 'tool_profile', 'timeout_seconds', 'concurrency')) { + foreach ($profileField in @('schema', 'runner', 'model', 'reasoning_effort', 'configuration_profile', 'tool_profile', 'timeout_seconds', 'concurrency')) { if ($profile.PSObject.Properties.Name -notcontains $profileField) { throw "execution-profile.json must declare '$profileField'." } } + if ($profile.PSObject.Properties.Name -contains 'provider') { + throw 'execution-profile.json must not declare provider.' + } if ([string]$profile.schema -ne 'codebeltnet/agentic/eval-execution-profile/1' -or + [string]::IsNullOrWhiteSpace([string]$profile.runner) -or + [string]::IsNullOrWhiteSpace([string]$profile.model) -or [int]$profile.timeout_seconds -lt 1 -or [int]$profile.concurrency -lt 1) { throw 'execution-profile.json has an invalid schema or execution limit.' } @@ -1585,7 +1685,6 @@ Add-ValidationResult -Results $results -Name 'Skill evaluation prepares portable $resultPath = Join-Path (Join-Path $firstEvalDirectory 'results') $resultFile $result = [System.IO.File]::ReadAllText($resultPath, $utf8NoBom) | ConvertFrom-Json $result.model = 'validator-model' - $result.provider = 'validator-provider' $result.harness = 'validator-harness' $result.executed_utc = '2026-01-01T00:00:00Z' $result.output = 'validator output' @@ -1648,7 +1747,7 @@ Add-ValidationResult -Results $results -Name 'Skill evaluation prepares portable } $insideRepo = Join-Path $repoRoot 'agentic-eval-isolation-check' - $isolationOutput = & pwsh -NoProfile -File $scriptPath -Skill 'dotnet-strong-name-signing' -OutputRoot $insideRepo 2>&1 + $isolationOutput = & pwsh -NoProfile -File $scriptPath -Skill 'dotnet-strong-name-signing' -OutputRoot $insideRepo -Runner 'github-copilot' -Model 'claude-haiku-4.5' 2>&1 if ($LASTEXITCODE -eq 0) { throw 'prepare-skill-evals.ps1 must refuse an output root inside this repository but outside .bot/.' } @@ -1663,7 +1762,7 @@ Add-ValidationResult -Results $results -Name 'Skill evaluation prepares portable # .bot/ is the sanctioned in-repository home, and it only works while git ignores it. $botRoot = Join-Path (Join-Path $repoRoot '.bot') 'agentic-eval-bot-check' try { - $botOutput = & pwsh -NoProfile -File $scriptPath -Skill 'dotnet-strong-name-signing' -OutputRoot $botRoot 2>&1 + $botOutput = & pwsh -NoProfile -File $scriptPath -Skill 'dotnet-strong-name-signing' -OutputRoot $botRoot -Runner 'github-copilot' -Model 'claude-haiku-4.5' 2>&1 if ($LASTEXITCODE -ne 0) { throw "prepare-skill-evals.ps1 must accept an output root under .bot/: $($botOutput -join [Environment]::NewLine)" } From b0f395f9082ab44d47338e111af86201b15ddb5c Mon Sep 17 00:00:00 2001 From: "aicia[bot]" Date: Sun, 23 Aug 2026 22:11:43 +0200 Subject: [PATCH 18/39] =?UTF-8?q?=F0=9F=94=A8=20refine=20eval=20infrastruc?= =?UTF-8?q?ture=20with=20result=20bridging=20and=20manifest?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Added manifest-paths.ps1 and bridge-manifest-results.ps1 for safe manifest path resolution and result artifact bridging. Updated prepare-skill-evals.ps1 with improved error handling and result collection. Enhanced generate-eval-report.ps1 for report generation and grading workflow. Expanded validate-skill-templates.ps1 with comprehensive fixture validation. Improved conformance testing in test-runner-conformance.ps1 to cover all eval runner contracts. --- .../eval-runners/bridge-execution-result.ps1 | 12 +- .../eval-runners/bridge-manifest-results.ps1 | 104 +++++ scripts/eval-runners/manifest-paths.ps1 | 355 ++++++++++++++++++ .../tests/test-runner-conformance.ps1 | 124 +++++- scripts/generate-eval-report.ps1 | 119 ++++-- scripts/prepare-skill-evals.ps1 | 100 +++-- scripts/validate-skill-templates.ps1 | 64 +++- 7 files changed, 795 insertions(+), 83 deletions(-) create mode 100644 scripts/eval-runners/bridge-manifest-results.ps1 create mode 100644 scripts/eval-runners/manifest-paths.ps1 diff --git a/scripts/eval-runners/bridge-execution-result.ps1 b/scripts/eval-runners/bridge-execution-result.ps1 index 8774dd3..51dfc43 100644 --- a/scripts/eval-runners/bridge-execution-result.ps1 +++ b/scripts/eval-runners/bridge-execution-result.ps1 @@ -96,14 +96,14 @@ function Get-ExistingGrading { param([Parameter(Mandatory = $true)][string]$ResultPath) if (-not (Test-Path -LiteralPath $ResultPath -PathType Leaf)) { - return @() + throw "Manifest-declared result stub '$ResultPath' does not exist; the bridge will not create a new grading-less result file." } - try { - $existing = Read-RunnerJson -Path $ResultPath - return @(Get-JsonProperty -Object $existing -Name 'grading' -Default @()) - } catch { - return @() + + $existing = Read-RunnerJson -Path $ResultPath + if (-not (Test-JsonProperty -Object $existing -Name 'grading')) { + throw "Manifest-declared result stub '$ResultPath' is missing its grading array." } + return @(Get-JsonProperty -Object $existing -Name 'grading' -Default @()) } try { diff --git a/scripts/eval-runners/bridge-manifest-results.ps1 b/scripts/eval-runners/bridge-manifest-results.ps1 new file mode 100644 index 0000000..b096d78 --- /dev/null +++ b/scripts/eval-runners/bridge-manifest-results.ps1 @@ -0,0 +1,104 @@ +<#! +.SYNOPSIS + Bridges every available execution result using only manifest-declared arm paths. + +.DESCRIPTION + Reads manifest.json, validates the exact run_manifest, execution_result, and result + paths for every arm, rejects result-like shadow files, and invokes the existing + one-arm bridge without reconstructing any filename from an arm or mode name. + No model, runner, or grader is started by this helper. +#> +[CmdletBinding()] +param( + [Parameter(Mandatory = $true)] + [string]$IterationDirectory, + + [switch]$RequireComplete +) + +$ErrorActionPreference = 'Stop' +Set-StrictMode -Version Latest + +. (Join-Path $PSScriptRoot 'manifest-paths.ps1') + +try { + $iterationPath = (Resolve-Path -LiteralPath $IterationDirectory -ErrorAction Stop).Path + $manifestPath = Join-Path $iterationPath 'manifest.json' + if (-not (Test-Path -LiteralPath $manifestPath -PathType Leaf)) { + throw "Prepared iteration is missing manifest.json at '$iterationPath'." + } + + $manifest = Read-RunnerJson -Path $manifestPath + $records = @(Get-ManifestRunRecords -IterationDirectory $iterationPath -Manifest $manifest) + $shadows = @(Get-ManifestShadowResultFiles -Records $records) + if ($shadows.Count -gt 0) { + $messages = @($shadows | ForEach-Object { + "$($_.EvalName)/$($_.Configuration) has unreferenced result-like sibling '$($_.Path)'; use the exact manifest result '$($_.CanonicalPath)'." + }) + throw ($messages -join [Environment]::NewLine) + } + + $oneArmBridge = Join-Path $PSScriptRoot 'bridge-execution-result.ps1' + if (-not (Test-Path -LiteralPath $oneArmBridge -PathType Leaf)) { + throw "Package-local one-arm bridge is missing at '$oneArmBridge'." + } + + $bridged = [System.Collections.Generic.List[string]]::new() + $missing = [System.Collections.Generic.List[string]]::new() + foreach ($record in $records) { + if (-not (Test-Path -LiteralPath $record.ExecutionResultPath -PathType Leaf)) { + $missing.Add("$($record.EvalName)/$($record.Configuration)") + continue + } + + $alreadyBridged = $false + try { + $existingResult = Read-RunnerJson -Path $record.ResultPath + $rawResult = Read-RunnerJson -Path $record.ExecutionResultPath + $expectedExecutionFile = [System.IO.Path]::GetRelativePath($record.EvalDirectory, $record.ExecutionResultPath).Replace('\', '/') + $alreadyBridged = @('completed', 'failed', 'timed_out', 'cancelled', 'incompatible') -contains [string](Get-JsonProperty -Object $rawResult -Name 'status' -Default '') -and + [string](Get-JsonProperty -Object $existingResult -Name 'execution_status' -Default '') -eq [string](Get-JsonProperty -Object $rawResult -Name 'status' -Default '') -and + [string](Get-JsonProperty -Object $existingResult -Name 'execution_result_file' -Default '') -eq $expectedExecutionFile -and + -not [string]::IsNullOrWhiteSpace([string](Get-JsonProperty -Object $existingResult -Name 'execution_run_id' -Default '')) + } catch { + $alreadyBridged = $false + } + + if ($alreadyBridged) { + $bridged.Add("$($record.EvalName)/$($record.Configuration)") + continue + } + + $bridgeOutput = & pwsh -NoProfile -File $oneArmBridge ` + -Run $record.RunManifestPath ` + -ExecutionResult $record.ExecutionResultPath ` + -Result $record.ResultPath 2>&1 + if ($LASTEXITCODE -ne 0) { + throw "$($record.EvalName)/$($record.Configuration) bridge failed for manifest paths run='$($record.RunManifestRelative)', execution='$($record.ExecutionResultRelative)', result='$($record.ResultRelative)': $([string]::Join(' ', @($bridgeOutput)))" + } + $bridged.Add("$($record.EvalName)/$($record.Configuration)") + } + + $validation = Test-ManifestResults ` + -IterationDirectory $iterationPath ` + -Manifest $manifest ` + -Records $records ` + -RequireComplete:$RequireComplete + if (-not $validation.Success) { + throw ([string]::Join([Environment]::NewLine, @($validation.Errors))) + } + + Write-RunnerJson -Value ([ordered]@{ + schema = 'codebeltnet/agentic/eval-manifest-bridge/1' + iteration = $iterationPath + expected_arms = $validation.ExpectedArmCount + bridged_arms = $validation.BridgedResults + terminal_execution_results = $validation.TerminalExecutionResults + missing_execution_results = @($missing) + complete = $validation.Complete + warnings = @($validation.Warnings) + }) -AsOutput +} catch { + [Console]::Error.WriteLine($_.Exception.Message) + exit 2 +} diff --git a/scripts/eval-runners/manifest-paths.ps1 b/scripts/eval-runners/manifest-paths.ps1 new file mode 100644 index 0000000..1c98816 --- /dev/null +++ b/scripts/eval-runners/manifest-paths.ps1 @@ -0,0 +1,355 @@ +Set-StrictMode -Version Latest +$ErrorActionPreference = 'Stop' + +. (Join-Path $PSScriptRoot 'runner-common.ps1') + +function Resolve-ManifestDeclaredPath { + param( + [Parameter(Mandatory = $true)][string]$IterationDirectory, + [Parameter(Mandatory = $true)][string]$RelativePath, + [Parameter(Mandatory = $true)][string]$FieldName, + [ValidateSet('Any', 'File', 'Directory')][string]$Kind = 'Any', + [switch]$RequireExists + ) + + Assert-SafeRelativePath -RelativePath $RelativePath -FieldName $FieldName + $resolvedIteration = (Resolve-Path -LiteralPath $IterationDirectory -ErrorAction Stop).Path + $candidate = [System.IO.Path]::GetFullPath((Join-Path $resolvedIteration ($RelativePath -replace '/', [System.IO.Path]::DirectorySeparatorChar))) + if (-not (Test-PathInside -BasePath $resolvedIteration -CandidatePath $candidate)) { + throw "$FieldName resolves outside the prepared iteration package." + } + + $exists = switch ($Kind) { + 'File' { Test-Path -LiteralPath $candidate -PathType Leaf } + 'Directory' { Test-Path -LiteralPath $candidate -PathType Container } + default { Test-Path -LiteralPath $candidate } + } + if ($RequireExists -and -not $exists) { + throw "$FieldName '$RelativePath' does not exist in the prepared iteration package." + } + + if ($exists) { + $resolvedCandidate = (Resolve-Path -LiteralPath $candidate -ErrorAction Stop).Path + if (-not (Test-PathInside -BasePath $resolvedIteration -CandidatePath $resolvedCandidate)) { + throw "$FieldName resolves through a link outside the prepared iteration package." + } + return $resolvedCandidate + } + + return $candidate +} + +function Get-ManifestConfigurations { + param([Parameter(Mandatory = $true)][object]$Manifest) + + $configurations = @(Get-JsonProperty -Object $Manifest -Name 'configurations' -Default @()) + if ($configurations.Count -eq 0) { + throw 'manifest.json must declare at least one configuration arm.' + } + + $seen = [System.Collections.Generic.HashSet[string]]::new([System.StringComparer]::OrdinalIgnoreCase) + $result = [System.Collections.Generic.List[string]]::new() + foreach ($value in $configurations) { + $configuration = [string]$value + if ([string]::IsNullOrWhiteSpace($configuration)) { + throw 'manifest.json contains an empty configuration arm.' + } + if (-not $seen.Add($configuration)) { + throw "manifest.json declares configuration arm '$configuration' more than once." + } + $result.Add($configuration) + } + + return @($result) +} + +function Get-ManifestRunRecords { + param( + [Parameter(Mandatory = $true)][string]$IterationDirectory, + [Parameter(Mandatory = $true)][object]$Manifest + ) + + $records = [System.Collections.Generic.List[object]]::new() + $seenPaths = @{} + $configurations = @(Get-ManifestConfigurations -Manifest $Manifest) + $evals = @(Get-JsonProperty -Object $Manifest -Name 'evals' -Default @()) + if ($evals.Count -eq 0) { + throw 'manifest.json must declare at least one eval case.' + } + + foreach ($entry in $evals) { + $evalName = [string](Get-JsonProperty -Object $entry -Name 'eval_name' -Default '') + $evalId = [int](Get-JsonProperty -Object $entry -Name 'eval_id' -Default 0) + $directoryRelative = [string](Get-JsonProperty -Object $entry -Name 'directory' -Default '') + $metadataRelative = [string](Get-JsonProperty -Object $entry -Name 'metadata' -Default '') + if ([string]::IsNullOrWhiteSpace($evalName) -or $evalId -lt 1) { + throw 'Each manifest eval entry must declare a non-empty eval_name and positive eval_id.' + } + + $evalDirectory = Resolve-ManifestDeclaredPath ` + -IterationDirectory $IterationDirectory ` + -RelativePath $directoryRelative ` + -FieldName "$evalName.directory" ` + -Kind Directory ` + -RequireExists + $metadataPath = Resolve-ManifestDeclaredPath ` + -IterationDirectory $IterationDirectory ` + -RelativePath $metadataRelative ` + -FieldName "$evalName.metadata" ` + -Kind File ` + -RequireExists + $runContainer = Get-JsonProperty -Object $entry -Name 'runs' -Default $null + if ($null -eq $runContainer) { + throw "$evalName manifest entry must declare runs." + } + + foreach ($configuration in $configurations) { + $runEntry = Get-JsonProperty -Object $runContainer -Name $configuration -Default $null + if ($null -eq $runEntry) { + throw "$evalName manifest entry is missing runs.$configuration." + } + + $declaredPaths = [ordered]@{} + foreach ($field in @('run_manifest', 'execution_result', 'result')) { + if (-not (Test-JsonProperty -Object $runEntry -Name $field)) { + throw "$evalName/$configuration manifest entry must declare '$field'." + } + $relative = [string](Get-JsonProperty -Object $runEntry -Name $field -Default '') + if ([string]::IsNullOrWhiteSpace($relative)) { + throw "$evalName/$configuration manifest field '$field' must be non-empty." + } + $declaredPaths[$field] = $relative + } + + $runManifestPath = Resolve-ManifestDeclaredPath ` + -IterationDirectory $IterationDirectory ` + -RelativePath $declaredPaths.run_manifest ` + -FieldName "$evalName/$configuration.run_manifest" ` + -Kind File ` + -RequireExists + $runManifest = Read-RunnerJson -Path $runManifestPath + $runEvalId = [int](Get-JsonProperty -Object $runManifest -Name 'evalId' -Default 0) + $runEvalName = [string](Get-JsonProperty -Object $runManifest -Name 'evalName' -Default '') + $runMode = [string](Get-JsonProperty -Object $runManifest -Name 'mode' -Default '') + if ($runEvalId -ne $evalId -or $runEvalName -ne $evalName -or $runMode -ne $configuration) { + throw "$evalName/$configuration run_manifest '$($declaredPaths.run_manifest)' identifies evalId='$runEvalId', evalName='$runEvalName', mode='$runMode'; it does not match the manifest arm." + } + $executionResultPath = Resolve-ManifestDeclaredPath ` + -IterationDirectory $IterationDirectory ` + -RelativePath $declaredPaths.execution_result ` + -FieldName "$evalName/$configuration.execution_result" ` + -Kind File + $resultPath = Resolve-ManifestDeclaredPath ` + -IterationDirectory $IterationDirectory ` + -RelativePath $declaredPaths.result ` + -FieldName "$evalName/$configuration.result" ` + -Kind File ` + -RequireExists + + $mode = [string](Get-JsonProperty -Object $runEntry -Name 'mode' -Default '') + if (-not [string]::IsNullOrWhiteSpace($mode) -and $mode -ne $configuration) { + throw "$evalName/$configuration manifest mode '$mode' does not match its arm key." + } + + foreach ($path in @( + [pscustomobject]@{ Field = 'run_manifest'; FullPath = $runManifestPath } + [pscustomobject]@{ Field = 'execution_result'; FullPath = $executionResultPath } + [pscustomobject]@{ Field = 'result'; FullPath = $resultPath } + )) { + $pathKey = [System.IO.Path]::GetFullPath($path.FullPath) + if ($seenPaths.ContainsKey($pathKey)) { + $previous = $seenPaths[$pathKey] + throw "$evalName/$configuration.$($path.Field) duplicates $($previous.EvalName)/$($previous.Configuration).$($previous.Field)." + } + $seenPaths[$pathKey] = [pscustomobject]@{ + EvalName = $evalName + Configuration = $configuration + Field = $path.Field + } + } + + $records.Add([pscustomobject]@{ + EvalId = $evalId + EvalName = $evalName + Configuration = $configuration + EvalDirectory = $evalDirectory + MetadataPath = $metadataPath + RunEntry = $runEntry + RunManifestRelative = $declaredPaths.run_manifest + ExecutionResultRelative = $declaredPaths.execution_result + ResultRelative = $declaredPaths.result + RunManifestPath = $runManifestPath + ExecutionResultPath = $executionResultPath + ResultPath = $resultPath + }) + } + } + + return @($records) +} + +function Get-ManifestShadowResultFiles { + param([Parameter(Mandatory = $true)][object[]]$Records) + + $shadows = [System.Collections.Generic.List[object]]::new() + $groups = @($Records | Group-Object { Split-Path -Parent ([string]$_.ResultPath) }) + foreach ($group in $groups) { + $canonical = @($group.Group) + $canonicalPaths = @{} + $canonicalNames = @{} + foreach ($record in $canonical) { + $fullPath = [System.IO.Path]::GetFullPath([string]$record.ResultPath) + $canonicalPaths[$fullPath] = $record + $normalizedName = ([System.IO.Path]::GetFileName($fullPath)).ToLowerInvariant() -replace '[-_]', '' + $canonicalNames[$normalizedName] = $record + } + + $resultDirectory = [string]$group.Name + if (-not (Test-Path -LiteralPath $resultDirectory -PathType Container)) { + continue + } + foreach ($candidate in @(Get-ChildItem -LiteralPath $resultDirectory -File -Filter '*.result.json' -Force)) { + $candidatePath = [System.IO.Path]::GetFullPath($candidate.FullName) + if ($canonicalPaths.ContainsKey($candidatePath)) { + continue + } + $normalizedName = $candidate.Name.ToLowerInvariant() -replace '[-_]', '' + if ($canonicalNames.ContainsKey($normalizedName)) { + $record = $canonicalNames[$normalizedName] + $shadows.Add([pscustomobject]@{ + Path = $candidatePath + CanonicalPath = [string]$record.ResultPath + EvalName = [string]$record.EvalName + Configuration = [string]$record.Configuration + }) + } + } + } + + return @($shadows) +} + +function Test-ManifestResults { + param( + [Parameter(Mandatory = $true)][string]$IterationDirectory, + [Parameter(Mandatory = $true)][object]$Manifest, + [object[]]$Records, + [switch]$RequireComplete + ) + + $manifestRecords = if ($null -eq $Records -or $Records.Count -eq 0) { + @(Get-ManifestRunRecords -IterationDirectory $IterationDirectory -Manifest $Manifest) + } else { + @($Records) + } + $errors = [System.Collections.Generic.List[string]]::new() + $warnings = [System.Collections.Generic.List[string]]::new() + $terminalStatuses = @('completed', 'failed', 'timed_out', 'cancelled', 'incompatible') + $terminalExecutionResults = 0 + $bridgedResults = 0 + + foreach ($shadow in @(Get-ManifestShadowResultFiles -Records $manifestRecords)) { + $errors.Add("$($shadow.EvalName)/$($shadow.Configuration) has an unreferenced result-like sibling '$($shadow.Path)'; the manifest canonical result is '$($shadow.CanonicalPath)'.") + } + + foreach ($record in $manifestRecords) { + $rawExists = Test-Path -LiteralPath $record.ExecutionResultPath -PathType Leaf + $canonicalExists = Test-Path -LiteralPath $record.ResultPath -PathType Leaf + if (-not $canonicalExists) { + $errors.Add("$($record.EvalName)/$($record.Configuration) is missing its manifest-declared result '$($record.ResultRelative)'.") + continue + } + + $raw = $null + $rawStatus = '' + if ($rawExists) { + try { + $raw = Read-RunnerJson -Path $record.ExecutionResultPath + $rawStatus = [string](Get-JsonProperty -Object $raw -Name 'status' -Default '') + } catch { + $errors.Add("$($record.EvalName)/$($record.Configuration) manifest-declared execution result '$($record.ExecutionResultRelative)' is invalid: $($_.Exception.Message)") + continue + } + + if ($terminalStatuses -notcontains $rawStatus) { + $errors.Add("$($record.EvalName)/$($record.Configuration) execution result has non-terminal status '$rawStatus'.") + } else { + $terminalExecutionResults++ + } + } elseif ($RequireComplete) { + $errors.Add("$($record.EvalName)/$($record.Configuration) is missing its manifest-declared execution result '$($record.ExecutionResultRelative)'.") + } else { + $warnings.Add("$($record.EvalName)/$($record.Configuration) has no execution result at the exact manifest path '$($record.ExecutionResultRelative)'.") + } + + $canonical = $null + try { + $canonical = Read-RunnerJson -Path $record.ResultPath + } catch { + $errors.Add("$($record.EvalName)/$($record.Configuration) manifest-declared result '$($record.ResultRelative)' is invalid: $($_.Exception.Message)") + continue + } + + if ([string]$canonical.configuration -ne $record.Configuration) { + $errors.Add("$($record.EvalName)/$($record.Configuration) canonical result declares configuration '$($canonical.configuration)'.") + } + if ([int]$canonical.eval_id -ne $record.EvalId) { + $errors.Add("$($record.EvalName)/$($record.Configuration) canonical result declares eval_id '$($canonical.eval_id)'.") + } + + $metadata = $null + try { + $metadata = Read-RunnerJson -Path $record.MetadataPath + } catch { + $errors.Add("$($record.EvalName) manifest-declared metadata '$($record.MetadataPath)' is invalid: $($_.Exception.Message)") + } + $assertionCount = if ($null -eq $metadata) { -1 } else { @((Get-JsonProperty -Object $metadata -Name 'assertions' -Default @())).Count } + $grading = @(Get-JsonProperty -Object $canonical -Name 'grading' -Default @()) + if ($assertionCount -ge 0 -and $grading.Count -ne $assertionCount) { + $errors.Add("$($record.EvalName)/$($record.Configuration) canonical result grading count $($grading.Count) does not match the $assertionCount manifest assertions.") + } + + $canonicalStatus = [string](Get-JsonProperty -Object $canonical -Name 'execution_status' -Default '') + if ($rawExists -and $terminalStatuses -contains $rawStatus) { + if ($canonicalStatus -eq 'unrun') { + $errors.Add("$($record.EvalName)/$($record.Configuration) has a terminal execution result but its canonical manifest result remains unrun.") + } elseif ($canonicalStatus -ne $rawStatus) { + $errors.Add("$($record.EvalName)/$($record.Configuration) canonical execution_status '$canonicalStatus' does not match raw status '$rawStatus'.") + } + + foreach ($field in @('model', 'harness', 'execution_status', 'execution_run_id', 'execution_result_file')) { + if (-not (Test-JsonProperty -Object $canonical -Name $field) -or [string]::IsNullOrWhiteSpace([string](Get-JsonProperty -Object $canonical -Name $field -Default ''))) { + $errors.Add("$($record.EvalName)/$($record.Configuration) canonical result is missing populated '$field'.") + } + } + $expectedExecutionFile = [System.IO.Path]::GetRelativePath($record.EvalDirectory, $record.ExecutionResultPath).Replace('\', '/') + $actualExecutionFile = [string](Get-JsonProperty -Object $canonical -Name 'execution_result_file' -Default '') + if ($actualExecutionFile -ne $expectedExecutionFile) { + $errors.Add("$($record.EvalName)/$($record.Configuration) canonical execution_result_file '$actualExecutionFile' does not match the manifest execution_result path '$($record.ExecutionResultRelative)'.") + } + if ($canonicalStatus -eq $rawStatus -and $grading.Count -eq $assertionCount) { + $bridgedResults++ + } + } elseif (-not $rawExists -and $RequireComplete -and $canonicalStatus -ne 'unrun') { + $errors.Add("$($record.EvalName)/$($record.Configuration) canonical result is populated without a manifest-declared execution result proving the bridge.") + } + } + + if ($RequireComplete -and $terminalExecutionResults -ne $manifestRecords.Count) { + $errors.Add("Completion gate expected $($manifestRecords.Count) terminal execution results but found $terminalExecutionResults.") + } + if ($RequireComplete -and $bridgedResults -ne $manifestRecords.Count) { + $errors.Add("Completion gate expected $($manifestRecords.Count) bridged canonical results but found $bridgedResults.") + } + + return [pscustomobject]@{ + Records = @($manifestRecords) + Errors = @($errors) + Warnings = @($warnings) + ExpectedArmCount = $manifestRecords.Count + TerminalExecutionResults = $terminalExecutionResults + BridgedResults = $bridgedResults + Complete = $errors.Count -eq 0 -and $terminalExecutionResults -eq $manifestRecords.Count -and $bridgedResults -eq $manifestRecords.Count + Success = $errors.Count -eq 0 + } +} diff --git a/scripts/eval-runners/tests/test-runner-conformance.ps1 b/scripts/eval-runners/tests/test-runner-conformance.ps1 index d7c0553..9596d76 100644 --- a/scripts/eval-runners/tests/test-runner-conformance.ps1 +++ b/scripts/eval-runners/tests/test-runner-conformance.ps1 @@ -17,6 +17,7 @@ Set-StrictMode -Version Latest $runnerRoot = (Resolve-Path (Join-Path $PSScriptRoot '..')).Path $repoRoot = (Resolve-Path (Join-Path $PSScriptRoot '..\..\..')).Path . (Join-Path $runnerRoot 'runner-common.ps1') +. (Join-Path $runnerRoot 'manifest-paths.ps1') function Assert-True { param([bool]$Condition, [string]$Message) @@ -565,7 +566,8 @@ function Invoke-Fake { function New-TestRun { param( [string]$IterationDirectory, - [ValidateSet('with_skill', 'without_skill')][string]$Configuration + [ValidateSet('with_skill', 'without_skill')][string]$Configuration, + [string]$EvalName = 'conformance' ) $evalDirectory = Join-Path $IterationDirectory 'conformance' @@ -592,7 +594,7 @@ function New-TestRun { $run = [ordered]@{ schema = (Get-RunnerSchemaNames).Run evalId = 1 - evalName = 'conformance' + evalName = $EvalName skillName = if ($Configuration -eq 'with_skill') { 'candidate' } else { $null } iteration = 1 mode = $Configuration @@ -827,13 +829,27 @@ try { $prepareText = [System.IO.File]::ReadAllText((Join-Path $repoRoot 'scripts\prepare-skill-evals.ps1'), [System.Text.UTF8Encoding]::new($false)) $reportText = [System.IO.File]::ReadAllText((Join-Path $repoRoot 'scripts\generate-eval-report.ps1'), [System.Text.UTF8Encoding]::new($false)) $bridgeText = [System.IO.File]::ReadAllText((Join-Path $runnerRoot 'bridge-execution-result.ps1'), [System.Text.UTF8Encoding]::new($false)) + $manifestBridgeText = [System.IO.File]::ReadAllText((Join-Path $runnerRoot 'bridge-manifest-results.ps1'), [System.Text.UTF8Encoding]::new($false)) Assert-True ($prepareText -notmatch '(?i)codex\s+exec|opencode\s+run|cline\s+--|copilot\s+-p|copilot\s+--prompt|Profile\.Provider') 'portable preparation must not contain harness-specific CLI invocations or provider-field branches' Assert-True ($reportText -notmatch '(?i)codex\s+exec|opencode\s+run|cline\s+--|copilot\s+-p|copilot\s+--prompt|Profile\.Provider') 'reporting must not contain harness-specific or provider-field branches' Assert-True ($bridgeText -notmatch '(?i)codex\s+exec|opencode\s+run|cline\s+--|copilot\s+-p|copilot\s+--prompt|Profile\.Provider') 'the raw-to-portable bridge must remain runner-neutral' + Assert-True ($prepareText.Contains('bridge-manifest-results.ps1')) 'handoff preparation must use the deterministic package-level manifest bridge' + Assert-True ($prepareText.Contains('runs..run_manifest') -and $prepareText.Contains('runs..execution_result') -and $prepareText.Contains('runs..result')) 'handoff preparation must require every exact manifest arm path' + Assert-True ($prepareText.Contains('Do not derive, normalize, rename, hyphenate, underscore, or otherwise reconstruct any run, execution-result, or result path.')) 'handoff preparation must prohibit reconstructed paths' + Assert-True ($prepareText -notmatch '') 'handoff preparation must not expose an unconstrained result-file placeholder' + Assert-True ($reportText -notmatch 'function Get-ResultPath') 'reporting must not contain a configuration-derived result path helper' + Assert-True ($manifestBridgeText.Contains('Get-ManifestRunRecords') -and $manifestBridgeText.Contains('$record.ResultPath')) 'package-level bridge must resolve exact manifest records' + Assert-True ($manifestBridgeText -notmatch 'with[-_]skill\.result\.json|without[-_]skill\.result\.json') 'package-level bridge must not encode arm-derived result filenames' $rawPath = Join-Path $iteration 'conformance\results\with-skill.execution-result.json' $resultPath = Join-Path $iteration 'conformance\results\with-skill.result.json' New-Item -ItemType Directory -Path (Split-Path -Parent $rawPath) -Force | Out-Null + Write-TestJson -Path $resultPath -Value ([ordered]@{ + schema = (Get-RunnerSchemaNames).PortableResult + eval_id = 1 + configuration = 'with_skill' + grading = @([ordered]@{ text = 'preserved assertion'; passed = $null; evidence = '' }) + }) $bridgeResult = Invoke-Fake -FakePath $fakePath -Command execute -Run $with.Path -Profile $profilePath Write-TestJson -Path $rawPath -Value $bridgeResult $bridgePath = Join-Path $runnerRoot 'bridge-execution-result.ps1' @@ -851,6 +867,110 @@ try { Assert-Equal 'verified' $portable.isolation.status 'bridge carries isolation verification status' Assert-True (@($portable.isolation.mechanisms).Count -gt 0) 'bridge carries isolation mechanisms' Assert-True (@($portable.output_files).Count -gt 0) 'bridge carries confined evidence paths' + Assert-Equal 'preserved assertion' $portable.grading[0].text 'bridge preserves the canonical grading entry' + Assert-True ($null -eq $portable.grading[0].passed) 'bridge preserves the canonical grading state before grading' + + $manifestPackage = Join-Path $iteration 'manifest-path-regression' + $manifestEval = Join-Path $manifestPackage 'conformance' + New-Item -ItemType Directory -Path $manifestEval -Force | Out-Null + $manifestWith = New-TestRun -IterationDirectory $manifestPackage -Configuration with_skill -EvalName 'manifest-path-regression' + $manifestWithout = New-TestRun -IterationDirectory $manifestPackage -Configuration without_skill -EvalName 'manifest-path-regression' + $manifestMetadataPath = Join-Path $manifestEval 'eval-metadata.json' + Write-TestJson -Path $manifestMetadataPath -Value ([ordered]@{ + schema = 'codebeltnet/agentic/eval-metadata/2' + eval_id = 1 + eval_name = 'manifest-path-regression' + assertions = @('preserved assertion', 'completed execution is bridged') + }) + $manifestWithResult = Join-Path $manifestEval 'results\with-skill.result.json' + $manifestWithoutResult = Join-Path $manifestEval 'results\without-skill.result.json' + $manifestWithExecution = Join-Path $manifestEval 'results\with-skill.execution-result.json' + $manifestWithoutExecution = Join-Path $manifestEval 'results\without-skill.execution-result.json' + foreach ($resultPathForStub in @($manifestWithResult, $manifestWithoutResult)) { + $configurationForStub = if ($resultPathForStub -eq $manifestWithResult) { 'with_skill' } else { 'without_skill' } + Write-TestJson -Path $resultPathForStub -Value ([ordered]@{ + schema = (Get-RunnerSchemaNames).PortableResult + eval_id = 1 + configuration = $configurationForStub + execution_status = 'unrun' + grading = @( + [ordered]@{ text = 'preserved assertion'; passed = $null; evidence = '' } + [ordered]@{ text = 'completed execution is bridged'; passed = $null; evidence = '' } + ) + }) + } + $manifest = [ordered]@{ + schema = 'codebeltnet/agentic/eval-package/2' + configurations = @('with_skill', 'without_skill') + evals = @([ordered]@{ + eval_id = 1 + eval_name = 'manifest-path-regression' + directory = 'conformance' + metadata = 'conformance/eval-metadata.json' + runs = [ordered]@{ + with_skill = [ordered]@{ + mode = 'with_skill' + run_manifest = 'conformance/with_skill/run.json' + execution_result = 'conformance/results/with-skill.execution-result.json' + result = 'conformance/results/with-skill.result.json' + } + without_skill = [ordered]@{ + mode = 'without_skill' + run_manifest = 'conformance/without_skill/run.json' + execution_result = 'conformance/results/without-skill.execution-result.json' + result = 'conformance/results/without-skill.result.json' + } + } + }) + } + Write-TestJson -Path (Join-Path $manifestPackage 'manifest.json') -Value $manifest + Copy-Item -LiteralPath $profilePath -Destination (Join-Path $manifestPackage 'execution-profile.json') -Force + $manifestWithExecutionResult = Invoke-Fake -FakePath $fakePath -Command execute -Run $manifestWith.Path -Profile $profilePath + $manifestWithoutExecutionResult = Invoke-Fake -FakePath $fakePath -Command execute -Run $manifestWithout.Path -Profile $profilePath + Write-TestJson -Path $manifestWithExecution -Value $manifestWithExecutionResult + Write-TestJson -Path $manifestWithoutExecution -Value $manifestWithoutExecutionResult + + $manifestObject = Get-Content -LiteralPath (Join-Path $manifestPackage 'manifest.json') -Raw | ConvertFrom-Json + $preBridgeValidation = Test-ManifestResults -IterationDirectory $manifestPackage -Manifest $manifestObject + Assert-True (-not $preBridgeValidation.Success) 'terminal execution plus an unrun canonical result fails validation before bridging' + Assert-True (@($preBridgeValidation.Errors | Where-Object { $_ -match 'remains unrun' }).Count -gt 0) 'pre-bridge validation reports the canonical unrun result' + + $shadowPath = Join-Path $manifestEval 'results\with_skill.result.json' + Write-TestJson -Path $shadowPath -Value ([ordered]@{ + schema = (Get-RunnerSchemaNames).PortableResult + eval_id = 1 + configuration = 'with_skill' + execution_status = 'completed' + grading = @() + }) + $manifestBridgePath = Join-Path $runnerRoot 'bridge-manifest-results.ps1' + $shadowOutput = & pwsh -NoProfile -File $manifestBridgePath -IterationDirectory $manifestPackage -RequireComplete 2>&1 + $shadowExitCode = $LASTEXITCODE + Assert-True ($shadowExitCode -ne 0) 'manifest bridge rejects an unreferenced underscore shadow result' + Assert-True (([string]::Join(' ', @($shadowOutput))) -match 'unreferenced result-like sibling') 'shadow rejection explains the manifest collision' + $canonicalBeforeBridge = Get-Content -LiteralPath $manifestWithResult -Raw | ConvertFrom-Json + Assert-Equal 'unrun' $canonicalBeforeBridge.execution_status 'shadow result is never selected as the canonical result' + Remove-Item -LiteralPath $shadowPath -Force + + $manifestBridgeOutput = & pwsh -NoProfile -File $manifestBridgePath -IterationDirectory $manifestPackage -RequireComplete 2>&1 + if ($LASTEXITCODE -ne 0) { throw "manifest path bridge failed: $([string]::Join(' ', @($manifestBridgeOutput)))" } + $canonicalWith = Get-Content -LiteralPath $manifestWithResult -Raw | ConvertFrom-Json + Assert-Equal 'completed' $canonicalWith.execution_status 'manifest bridge populates the canonical hyphen result' + Assert-Equal 'fixture-model' $canonicalWith.model 'manifest bridge carries the model to the canonical result' + Assert-Equal 'deterministic-fake 1' $canonicalWith.harness 'manifest bridge carries the harness to the canonical result' + Assert-True (-not [string]::IsNullOrWhiteSpace([string]$canonicalWith.output)) 'manifest bridge carries output to the canonical result' + Assert-Equal 'results/with-skill.execution-result.json' $canonicalWith.execution_result_file 'manifest bridge records the exact manifest execution path' + Assert-Equal 2 @($canonicalWith.grading).Count 'manifest bridge preserves the canonical grading count' + Assert-Equal 'preserved assertion' $canonicalWith.grading[0].text 'manifest bridge preserves the canonical grading text' + + $canonicalWith.grading[0].passed = $true + $canonicalWith.grading[0].evidence = 'graded after the first bridge' + Write-TestJson -Path $manifestWithResult -Value $canonicalWith + $repeatBridgeOutput = & pwsh -NoProfile -File $manifestBridgePath -IterationDirectory $manifestPackage -RequireComplete 2>&1 + if ($LASTEXITCODE -ne 0) { throw "repeat manifest path bridge failed: $([string]::Join(' ', @($repeatBridgeOutput)))" } + $canonicalAfterRepeat = Get-Content -LiteralPath $manifestWithResult -Raw | ConvertFrom-Json + Assert-True ([bool]$canonicalAfterRepeat.grading[0].passed) 'repeat manifest bridge preserves completed grading' + Assert-Equal 'graded after the first bridge' $canonicalAfterRepeat.grading[0].evidence 'repeat manifest bridge preserves grading evidence' $acceptedBridgeResult = $bridgeResult | ConvertTo-Json -Depth 100 | ConvertFrom-Json $acceptedBridgeResult.resolved.status = 'accepted_request' diff --git a/scripts/generate-eval-report.ps1 b/scripts/generate-eval-report.ps1 index c28bfe8..2e3bd7f 100644 --- a/scripts/generate-eval-report.ps1 +++ b/scripts/generate-eval-report.ps1 @@ -26,6 +26,10 @@ .PARAMETER SkillCreatorPath Optional skill-creator installation or package-local tools/skill-creator path. The package-local path is the default so a prepared package remains self-contained after preparation. + +.PARAMETER RequireComplete + Fail before reporting success unless every manifest-declared arm has a terminal execution result and a validated + canonical result bridged from that exact path. #> [CmdletBinding()] param( @@ -38,7 +42,9 @@ param( [string]$BenchmarkMarkdownPath, - [string]$SkillCreatorPath + [string]$SkillCreatorPath, + + [switch]$RequireComplete ) $ErrorActionPreference = 'Stop' @@ -46,6 +52,8 @@ Set-StrictMode -Version Latest $utf8NoBom = [System.Text.UTF8Encoding]::new($false) +. (Join-Path $PSScriptRoot 'eval-runners/manifest-paths.ps1') + function Read-JsonFile { param([string]$Path) @@ -177,16 +185,6 @@ function Invoke-PythonScript { } } -function Get-ResultPath { - param( - [string]$EvalDirectory, - [string]$Configuration - ) - - $fileName = if ($Configuration -eq 'with_skill') { 'with-skill.result.json' } else { 'without-skill.result.json' } - return Join-Path (Join-Path $EvalDirectory 'results') $fileName -} - function Copy-RecordedOutputFiles { param( [object]$Result, @@ -428,12 +426,14 @@ function Get-ReportRun { function Get-ReportSkillStats { param( [object]$Manifest, + [object[]]$ManifestRecords, [string]$IterationPath ) $skillRoot = $null - foreach ($entry in @($Manifest.evals)) { - $candidate = Join-Path (Join-Path (Join-Path $IterationPath ([string]$entry.directory)) 'with_skill') ("skill/$($Manifest.skill_name)") + foreach ($record in @($ManifestRecords | Where-Object { [string]$_.Configuration -eq 'with_skill' })) { + $runPackageDirectory = Split-Path -Parent ([string]$record.RunManifestPath) + $candidate = Join-Path $runPackageDirectory ("skill/$($Manifest.skill_name)") if (Test-Path -LiteralPath $candidate -PathType Container) { $skillRoot = $candidate break @@ -455,6 +455,8 @@ function Get-ReportSkillStats { function Write-FirstPartyReport { param( [object]$Manifest, + [object[]]$ManifestRecords, + [object]$Validation, [string]$IterationPath, [string]$OutputPath, [object]$Benchmark @@ -462,20 +464,30 @@ function Write-FirstPartyReport { $evals = [System.Collections.Generic.List[object]]::new() $allModels = [System.Collections.Generic.List[string]]::new() - $completedRuns = 0 + $completedRuns = [int]$Validation.BridgedResults foreach ($entry in @($Manifest.evals)) { - $evalDirectory = Join-Path $IterationPath ([string]$entry.directory) - $metadata = Read-JsonFile -Path (Join-Path $evalDirectory 'eval-metadata.json') + $entryRecords = @($ManifestRecords | Where-Object { [int]$_.EvalId -eq [int]$entry.eval_id }) + if ($entryRecords.Count -eq 0) { + throw "$($entry.eval_name) does not have manifest-declared arm paths." + } + $evalDirectory = [string]$entryRecords[0].EvalDirectory + $metadata = Read-JsonFile -Path ([string]$entryRecords[0].MetadataPath) $runMap = [ordered]@{} $assertions = @($metadata.assertions | ForEach-Object { [string]$_ }) foreach ($configuration in @('with_skill', 'without_skill')) { - $resultPath = Get-ResultPath -EvalDirectory $evalDirectory -Configuration $configuration + $records = @($ManifestRecords | Where-Object { + [int]$_.EvalId -eq [int]$entry.eval_id -and [string]$_.Configuration -eq $configuration + }) + if ($records.Count -ne 1) { + throw "$($entry.eval_name)/$configuration does not have exactly one manifest-declared result path." + } + $runRecord = $records[0] + $resultPath = [string]$runRecord.ResultPath $result = if (Test-Path -LiteralPath $resultPath) { Read-JsonFile -Path $resultPath } else { $null } if ($null -ne $result) { - $run = Get-ReportRun -Result $result -Configuration $configuration -EvalName ([string]$entry.eval_name) -EvalId ([int]$metadata.eval_id) -Assertions $assertions -RunPackageDirectory (Join-Path $evalDirectory $configuration) -EvalDirectory $evalDirectory -IterationPath $IterationPath + $runPackageDirectory = Split-Path -Parent ([string]$runRecord.RunManifestPath) + $run = Get-ReportRun -Result $result -Configuration $configuration -EvalName ([string]$entry.eval_name) -EvalId ([int]$metadata.eval_id) -Assertions $assertions -RunPackageDirectory $runPackageDirectory -EvalDirectory $evalDirectory -IterationPath $IterationPath $runMap[$configuration] = $run - $executionStatus = [string](Get-Property -Object $result -Name 'execution_status' -Default '') - if (-not [string]::IsNullOrWhiteSpace([string]$run.output) -or @($run.output_files).Count -gt 0 -or ($executionStatus -and $executionStatus -ne 'unrun')) { $completedRuns++ } if (-not [string]::IsNullOrWhiteSpace([string]$run.model) -and -not $allModels.Contains([string]$run.model)) { $allModels.Add([string]$run.model) } @@ -503,7 +515,7 @@ function Write-FirstPartyReport { skill_name = [string]$Manifest.skill_name iteration = [int]$Manifest.iteration metadata = $metadata - skill = Get-ReportSkillStats -Manifest $Manifest -IterationPath $IterationPath + skill = Get-ReportSkillStats -Manifest $Manifest -ManifestRecords $ManifestRecords -IterationPath $IterationPath evals = @($evals) benchmark = $Benchmark } @@ -588,6 +600,7 @@ function Write-UpstreamGrading { function New-UpstreamWorkspace { param( [object]$Manifest, + [object[]]$ManifestRecords, [string]$IterationPath, [string]$WorkspacePath ) @@ -596,8 +609,12 @@ function New-UpstreamWorkspace { $workspaceEntries = [System.Collections.Generic.List[object]]::new() foreach ($entry in @($Manifest.evals)) { - $evalDirectory = Join-Path $IterationPath ([string]$entry.directory) - $metadata = Read-JsonFile -Path (Join-Path $evalDirectory 'eval-metadata.json') + $entryRecords = @($ManifestRecords | Where-Object { [int]$_.EvalId -eq [int]$entry.eval_id }) + if ($entryRecords.Count -eq 0) { + throw "$($entry.eval_name) does not have manifest-declared arm paths." + } + $evalDirectory = [string]$entryRecords[0].EvalDirectory + $metadata = Read-JsonFile -Path ([string]$entryRecords[0].MetadataPath) $evalFolder = Join-Path $WorkspacePath ("eval-{0}-{1}" -f $entry.eval_id, (Get-SafeSegment -Value ([string]$entry.eval_name))) New-Item -ItemType Directory -Path $evalFolder -Force | Out-Null @@ -618,7 +635,14 @@ function New-UpstreamWorkspace { New-Item -ItemType Directory -Path $outputsDirectory -Force | Out-Null Write-JsonFile -Path (Join-Path $configurationDirectory 'eval_metadata.json') -Value $upstreamMetadata - $resultPath = Get-ResultPath -EvalDirectory $evalDirectory -Configuration $configuration + $records = @($ManifestRecords | Where-Object { + [int]$_.EvalId -eq [int]$entry.eval_id -and [string]$_.Configuration -eq $configuration + }) + if ($records.Count -ne 1) { + throw "$($entry.eval_name)/$configuration does not have exactly one manifest-declared result path." + } + $runRecord = $records[0] + $resultPath = [string]$runRecord.ResultPath if (-not (Test-Path -LiteralPath $resultPath)) { continue } @@ -639,7 +663,8 @@ function New-UpstreamWorkspace { Write-JsonFile -Path (Join-Path $outputsDirectory 'isolation.json') -Value $isolation } - Copy-RecordedOutputFiles -Result $result -RunPackageDirectory (Join-Path $evalDirectory $configuration) -EvalDirectory $evalDirectory -IterationPath $IterationPath -OutputDirectory $outputsDirectory + $runPackageDirectory = Split-Path -Parent ([string]$runRecord.RunManifestPath) + Copy-RecordedOutputFiles -Result $result -RunPackageDirectory $runPackageDirectory -EvalDirectory $evalDirectory -IterationPath $IterationPath -OutputDirectory $outputsDirectory Write-UpstreamGrading -Result $result -RunDirectory $runDirectory -Assertions @($metadata.assertions | ForEach-Object { [string]$_ }) } } @@ -649,10 +674,21 @@ function New-UpstreamWorkspace { $iterationPath = (Resolve-Path -LiteralPath $IterationDirectory).Path $manifest = Read-JsonFile -Path (Join-Path $iterationPath 'manifest.json') +$manifestRecords = @(Get-ManifestRunRecords -IterationDirectory $iterationPath -Manifest $manifest) +$validation = Test-ManifestResults -IterationDirectory $iterationPath -Manifest $manifest -Records $manifestRecords +if (-not $validation.Success) { + throw ([string]::Join([Environment]::NewLine, @($validation.Errors))) +} +foreach ($warning in @($validation.Warnings)) { + Write-Host "[WARN] $warning" +} +if ($RequireComplete -and -not $validation.Complete) { + throw "Evaluation completion gate failed: expected $($validation.ExpectedArmCount) bridged terminal arms, found $($validation.BridgedResults)." +} $skillCreatorPathResolved = Resolve-SkillCreatorPath -RequestedPath $SkillCreatorPath $pythonCommand = Resolve-PythonCommand $workspacePath = Join-Path $iterationPath '.skill-creator-report' -$workspaceEntries = New-UpstreamWorkspace -Manifest $manifest -IterationPath $iterationPath -WorkspacePath $workspacePath +$workspaceEntries = New-UpstreamWorkspace -Manifest $manifest -ManifestRecords $manifestRecords -IterationPath $iterationPath -WorkspacePath $workspacePath $aggregatePath = Join-Path $skillCreatorPathResolved 'scripts/aggregate_benchmark.py' $viewerPath = Join-Path $skillCreatorPathResolved 'eval-viewer/generate_review.py' @@ -662,10 +698,25 @@ Invoke-PythonScript -PythonCommand $pythonCommand -ScriptPath $aggregatePath -Ar $benchmarkWorkspacePath = Join-Path $workspacePath 'benchmark.json' $benchmark = Read-JsonFile -Path $benchmarkWorkspacePath +$benchmarkRuns = @(Get-Property -Object $benchmark -Name 'runs' -Default @()) +if ($RequireComplete) { + if ($benchmarkRuns.Count -eq 0) { + throw 'Evaluation completion gate failed: benchmark output contains zero completed runs.' + } + if ($benchmarkRuns.Count -ne $validation.ExpectedArmCount -or $benchmarkRuns.Count -ne $validation.BridgedResults) { + throw "Evaluation completion gate failed: benchmark completed-run count $($benchmarkRuns.Count) does not match expected bridged count $($validation.BridgedResults) of $($validation.ExpectedArmCount)." + } +} $models = [System.Collections.Generic.List[string]]::new() foreach ($entry in @($manifest.evals)) { foreach ($configuration in @('with_skill', 'without_skill')) { - $resultPath = Get-ResultPath -EvalDirectory (Join-Path $iterationPath ([string]$entry.directory)) -Configuration $configuration + $records = @($manifestRecords | Where-Object { + [int]$_.EvalId -eq [int]$entry.eval_id -and [string]$_.Configuration -eq $configuration + }) + if ($records.Count -ne 1) { + throw "$($entry.eval_name)/$configuration does not have exactly one manifest-declared result path." + } + $resultPath = [string]$records[0].ResultPath if (Test-Path -LiteralPath $resultPath) { $model = [string](Get-Property -Object (Read-JsonFile -Path $resultPath) -Name 'model' -Default '') if (-not [string]::IsNullOrWhiteSpace($model) -and -not $models.Contains($model)) { @@ -705,7 +756,15 @@ $viewerArguments = @( ) Invoke-PythonScript -PythonCommand $pythonCommand -ScriptPath $viewerPath -Arguments $viewerArguments -Write-FirstPartyReport -Manifest $manifest -IterationPath $iterationPath -OutputPath $htmlOutputPath -Benchmark $benchmark +Write-FirstPartyReport -Manifest $manifest -ManifestRecords $manifestRecords -Validation $validation -IterationPath $iterationPath -OutputPath $htmlOutputPath -Benchmark $benchmark + +if ($RequireComplete) { + foreach ($output in @($benchmarkOutputPath, $benchmarkMarkdownOutputPath, $upstreamHtmlOutputPath, $htmlOutputPath)) { + if (-not (Test-Path -LiteralPath $output -PathType Leaf) -or (Get-Item -LiteralPath $output).Length -eq 0) { + throw "Evaluation completion gate failed: report artifact '$output' is missing or empty." + } + } +} Write-Host "Anthropic skill-creator tools: $skillCreatorPathResolved" Write-Host "Anthropic viewer template: $viewerTemplatePath" @@ -713,3 +772,7 @@ Write-Host "Wrote $benchmarkOutputPath" Write-Host "Wrote $benchmarkMarkdownOutputPath" Write-Host "Wrote $upstreamHtmlOutputPath" Write-Host "Wrote $htmlOutputPath" +Write-Host "Manifest-declared terminal arms: $($validation.BridgedResults)/$($validation.ExpectedArmCount)" +if (-not $validation.Complete) { + Write-Host 'This is a partial report; evaluation completion was not asserted.' +} diff --git a/scripts/prepare-skill-evals.ps1 b/scripts/prepare-skill-evals.ps1 index 7663f36..8f125a6 100644 --- a/scripts/prepare-skill-evals.ps1 +++ b/scripts/prepare-skill-evals.ps1 @@ -159,7 +159,7 @@ param( [Parameter(ParameterSetName = 'Prepare')] [Parameter(ParameterSetName = 'Changed')] [ValidateRange(1, 128)] - [int]$Concurrency = 1, + [int]$Concurrency = 16, [Parameter(ParameterSetName = 'Collect', Mandatory = $true)] [string]$CollectResults @@ -176,6 +176,8 @@ $utf8NoBom = [System.Text.UTF8Encoding]::new($false) [Console]::OutputEncoding = $utf8NoBom $OutputEncoding = $utf8NoBom +. (Join-Path $PSScriptRoot 'eval-runners/manifest-paths.ps1') + $packageSchema = 'codebeltnet/agentic/eval-package/2' $metadataSchema = 'codebeltnet/agentic/eval-metadata/2' $resultSchema = 'codebeltnet/agentic/eval-result/2' @@ -1611,7 +1613,7 @@ function New-RunnerPrompt { $builder = [System.Text.StringBuilder]::new() $profilePath = Join-Path $IterationDirectory 'execution-profile.json' $resolverPath = Join-Path $IterationDirectory "$evalRunnerToolRelativePath/resolve-runner.ps1" - $bridgePath = Join-Path $IterationDirectory "$evalRunnerToolRelativePath/bridge-execution-result.ps1" + $manifestBridgePath = Join-Path $IterationDirectory "$evalRunnerToolRelativePath/bridge-manifest-results.ps1" $reportPath = Join-Path $IterationDirectory $reportToolRelativePath [void]$builder.AppendLine('# Execute, grade, and report this evaluation package') [void]$builder.AppendLine() @@ -1629,9 +1631,9 @@ function New-RunnerPrompt { [void]$builder.AppendLine() [void]$builder.AppendLine('1. Read `manifest.json` and `execution-profile.json`. If `runner` or `model` is null, unavailable, or unsupported, fail clearly and list the supported package-local runner IDs; do not guess a default. The profile contains no credentials.') [void]$builder.AppendLine('2. Resolve the selected package-local runner with the resolver. Ask it for `describe` and validate its protocol, descriptor, and capability declarations before running an arm. Do not invent harness-specific CLI commands.') - [void]$builder.AppendLine('3. For every eval case, use the exact `run_manifest` path from `manifest.json` and the same profile path. Preflight each arm, then invoke the runner exactly once with `execute`. The runner receives only `run.json` and `execution-profile.json`; it must never receive or inspect expected output, assertions, grading, paired output, benchmark data, or human feedback.') + [void]$builder.AppendLine('3. For every eval arm, read the exact `run_manifest`, `execution_result`, and `result` fields from `manifest.json` at `runs..run_manifest`, `runs..execution_result`, and `runs..result`. Bind them without editing the strings: `runPath = exact run_manifest`, `executionPath = exact execution_result`, and `resultPath = exact result`.') [void]$builder.AppendLine('4. Keep `with_skill` and `without_skill` in fresh independent processes/sessions. Use the same runner-native model selector, configuration, tools, and limits. The runner must send each `prompt.md` unchanged as the first task input and must enforce the run contract, including the baseline skill exclusion and the staged filesystem/workspace boundary.') - [void]$builder.AppendLine('5. Save the runner''s single normalized JSON response unchanged as the matching `execution_result` path. Preserve the complete final response, status, telemetry, evidence references, hashes, isolation mechanisms, warnings, and compatibility deviations. Do not retry for answer quality. A refusal is a result; timeout, harness failure, and incompatibility are results.') + [void]$builder.AppendLine('5. Save the runner''s single normalized JSON response unchanged to that arm''s exact `executionPath`. Do not derive, normalize, rename, hyphenate, underscore, or otherwise reconstruct any run, execution-result, or result path. A runner execute operation writes only to the exact manifest-declared `execution_result` path. Preserve the complete final response, status, telemetry, evidence references, hashes, isolation mechanisms, warnings, and compatibility deviations. Do not retry for answer quality. A refusal is a result; timeout, harness failure, and incompatibility are results.') [void]$builder.AppendLine('6. If the runner cannot satisfy a required guarantee, keep the normalized status `incompatible` and stop that arm. Never fall back to the old generic isolated-worker behavior and never substitute a different runner.') [void]$builder.AppendLine() [void]$builder.AppendLine('The package-local process surface is:') @@ -1641,17 +1643,18 @@ function New-RunnerPrompt { [void]$builder.AppendLine("runner.ps1 preflight -Run `"`" -Profile `"$profilePath`"") [void]$builder.AppendLine("runner.ps1 execute -Run `"`" -Profile `"$profilePath`"") [void]$builder.AppendLine('```') - [void]$builder.AppendLine('Use the resolver output to locate `runner.ps1`; `` is data from the profile, not a branch in this orchestration contract. The runner command must be invoked once per arm and its stdout must remain one JSON execution result.') + [void]$builder.AppendLine('Use the resolver output to locate `runner.ps1`; `` is data from the profile, not a branch in this orchestration contract. For each arm, invoke the runner exactly once with `execute`, and keep its stdout as one JSON `execution-result.json` document.') [void]$builder.AppendLine() - [void]$builder.AppendLine('Do not read any `eval-metadata.json`, expected output, assertions, result grading, or paired output during Phase 1. Those files remain outside every run directory and are the grading key.') + [void]$builder.AppendLine('Do not read any `eval-metadata.json`, expected output, assertions, result grading, or paired output during Phase 1. The runner must never receive or inspect expected output, assertions, grading, paired output, benchmark data, or human feedback. Those files remain outside every run directory and are the grading key.') [void]$builder.AppendLine() [void]$builder.AppendLine('## Phase 2: bridge, grade, and report') [void]$builder.AppendLine() - [void]$builder.AppendLine('1. After every available arm has completed or failed, validate each `execution-result.json` and run the package bridge. The bridge checks prompt/run/profile hashes and artifact confinement, then writes the existing `eval-result/2` file while preserving unavailable telemetry as null/unavailable.') - [void]$builder.AppendLine((' `pwsh -NoProfile -File "' + $bridgePath + '" -Run "" -ExecutionResult "" -Result ""`')) - [void]$builder.AppendLine('2. Only now read each eval''s `eval-metadata.json` and reveal `expected_output` and `assertions` to the Grader. Follow `tools/skill-creator/agents/grader.md`; grade deterministically first, then use optional model judgement only where deterministic evidence cannot decide. Never infer tool or file behavior from model self-report without process evidence.') - [void]$builder.AppendLine('3. Write only `grading[].text`, `grading[].passed`, and `grading[].evidence` for grading. Do not alter raw execution results. Use null for genuinely unavailable judgement and leave missing arms visibly missing.') - [void]$builder.AppendLine(('4. Run the existing package report adapter now: `pwsh -NoProfile -File "' + $reportPath + '" -IterationDirectory "' + $IterationDirectory + '"`. It remains the bridge to Anthropic skill-creator''s grader-compatible aggregator and viewer; do not replace it with harness-specific reporting. The packaged compatibility tools remain `scripts/aggregate_benchmark.py` and `eval-viewer/generate_review.py`.')) + [void]$builder.AppendLine('1. After every available arm has completed or failed, invoke the deterministic package bridge below. It reads `manifest.json`, obtains each arm''s exact `run_manifest`, `execution_result`, and `result` paths. The bridge checks prompt/run/profile hashes and artifact confinement, validates the manifest paths, rejects unreferenced hyphen/underscore shadow results, and invokes the existing one-arm bridge with those exact paths. Do not manually construct a bridge command for an arm.') + [void]$builder.AppendLine((' `pwsh -NoProfile -File "' + $manifestBridgePath + '" -IterationDirectory "' + $IterationDirectory + '" -RequireComplete`')) + [void]$builder.AppendLine(' The bridge''s one-arm operation is conceptually `-Run runPath -ExecutionResult executionPath -Result resultPath`, where all three values are the exact strings read from `manifest.json`. Do not derive, normalize, rename, hyphenate, underscore, or otherwise reconstruct any of them.') + [void]$builder.AppendLine('2. Only if the package bridge succeeds, read each eval''s `eval-metadata.json` and reveal `expected_output` and `assertions` to the Grader. Follow `tools/skill-creator/agents/grader.md`; grade deterministically first, then use optional model judgement only where deterministic evidence cannot decide. Never infer tool or file behavior from model self-report without process evidence.') + [void]$builder.AppendLine('3. Write only `grading[].text`, `grading[].passed`, and `grading[].evidence` for grading. Do not alter raw execution results or replace the canonical result stubs. Use null for genuinely unavailable judgement and leave missing arms visibly missing.') + [void]$builder.AppendLine(('4. Run the existing package report adapter with its completion gate: `pwsh -NoProfile -File "' + $reportPath + '" -IterationDirectory "' + $IterationDirectory + '" -RequireComplete`. It remains the bridge to Anthropic skill-creator''s grader-compatible aggregator and viewer; do not replace it with harness-specific reporting. The packaged compatibility tools remain `scripts/aggregate_benchmark.py` and `eval-viewer/generate_review.py`.')) [void]$builder.AppendLine() [void]$builder.AppendLine('The completion artifacts are `report.html`, `skill-creator-report.html`, `benchmark.json`, and `benchmark.md` at the package root. Return their absolute paths, completed and missing arm counts, runner/model identity, and a concise evidence-backed summary. If the package cannot be written from the external environment, return one paste-ready block containing the completed result objects and report artifacts.') [void]$builder.AppendLine() @@ -1692,7 +1695,7 @@ function New-PackageReadme { [void]$builder.AppendLine() [void]$builder.AppendLine('1. Read `execution-profile.json`. If `runner` or `model` is missing, fail clearly instead of guessing. Resolve the selected package-local runner and run `describe`, then `preflight`, for every arm.') [void]$builder.AppendLine('2. Invoke `execute` exactly once for each `run.json`, preserving its one JSON `execution-result.json` unchanged. The runner must provide a fresh process/session, isolated home/config, isolated CWD, baseline skill exclusion, prompt fidelity, model/configuration lock, and complete response capture, or return `incompatible`. Hard filesystem confinement is not one of these mandatory controls: when the runner proves it the run reports strict isolation, and when it does not the run reports pragmatic isolation.') - [void]$builder.AppendLine('3. After all arms complete or fail, run `tools/eval-runners/bridge-execution-result.ps1` for each raw result. It writes the existing `eval-result/2` file and preserves explicit unavailable telemetry. Only then read the grading key, grade with `tools/skill-creator/agents/grader.md`, and run `tools/generate-eval-report.ps1`.') + [void]$builder.AppendLine('3. After all arms complete or fail, run `tools/eval-runners/bridge-manifest-results.ps1 -IterationDirectory -RequireComplete`. It reads the manifest-declared `run_manifest`, `execution_result`, and `result` paths for every arm and invokes the one-arm bridge with those exact paths. Only then read the grading key, grade with `tools/skill-creator/agents/grader.md`, and run `tools/generate-eval-report.ps1 -RequireComplete`.') [void]$builder.AppendLine() [void]$builder.AppendLine('`RUN-THIS.prompt.md` is the external Eval Orchestrator handoff. It selects the package-local runner from the profile, invokes the common protocol once per blind arm, bridges raw evidence into the existing result shape, reveals grading material only after execution, and invokes Anthropic skill-creator''s compatible aggregator and static viewer through the package adapter. It never executes an eval prompt in its own context.') [void]$builder.AppendLine() @@ -1846,46 +1849,66 @@ function Invoke-CollectMode { $errors = [System.Collections.Generic.List[string]]::new() $warnings = [System.Collections.Generic.List[string]]::new() $rows = [System.Collections.Generic.List[object]]::new() + $manifestRecords = @(Get-ManifestRunRecords -IterationDirectory $iterationDirectory -Manifest $manifest) + $runnerAware = $manifest.PSObject.Properties.Name -contains 'execution_profile' + if ($runnerAware) { + $packageBridgePath = Join-Path $iterationDirectory "$($manifest.runner_tools)/bridge-manifest-results.ps1" + if (-not (Test-Path -LiteralPath $packageBridgePath -PathType Leaf)) { + $packageBridgePath = Join-Path (Join-Path (Get-RepoRoot) 'scripts') 'eval-runners/bridge-manifest-results.ps1' + } + if (-not (Test-Path -LiteralPath $packageBridgePath -PathType Leaf)) { + $errors.Add("Manifest-driven bridge is missing at '$packageBridgePath'.") + } else { + $bridgeOutput = & pwsh -NoProfile -File $packageBridgePath -IterationDirectory $iterationDirectory 2>&1 + if ($LASTEXITCODE -ne 0) { + $errors.Add("Manifest-driven execution-result bridge failed: $([string]::Join(' ', @($bridgeOutput)))") + } else { + foreach ($line in @($bridgeOutput)) { + if (-not [string]::IsNullOrWhiteSpace([string]$line)) { + Write-Host $line + } + } + } + } + } foreach ($entry in @($manifest.evals)) { - $evalDirectory = Join-Path $iterationDirectory $entry.directory - $metadata = [System.IO.File]::ReadAllText((Join-Path $evalDirectory 'eval-metadata.json'), $utf8NoBom) | ConvertFrom-Json + $entryRecords = @($manifestRecords | Where-Object { [int]$_.EvalId -eq [int]$entry.eval_id }) + if ($entryRecords.Count -eq 0) { + $errors.Add("$($entry.eval_name) has no manifest-declared arm paths.") + continue + } + $evalDirectory = [string]$entryRecords[0].EvalDirectory + $metadata = [System.IO.File]::ReadAllText([string]$entryRecords[0].MetadataPath, $utf8NoBom) | ConvertFrom-Json $observed = @{} - $runnerAware = $manifest.PSObject.Properties.Name -contains 'execution_profile' - $bridgePath = if ($runnerAware) { Join-Path $iterationDirectory ($manifest.runner_tools + '/bridge-execution-result.ps1') } else { $null } - foreach ($configuration in @('with_skill', 'without_skill')) { - $fileName = if ($configuration -eq 'with_skill') { 'with-skill.result.json' } else { 'without-skill.result.json' } - $resultPath = Join-Path (Join-Path $evalDirectory 'results') $fileName - $runEntry = Get-JsonProperty -Object $entry.runs -Name $configuration -Default $null - $rawRelative = Get-JsonProperty -Object $runEntry -Name 'execution_result' -Default $null - $rawPath = if ([string]::IsNullOrWhiteSpace([string]$rawRelative)) { $null } else { Join-Path $iterationDirectory $rawRelative } - $runPath = Join-Path $iterationDirectory (Get-JsonProperty -Object $runEntry -Name 'run_manifest' -Default '') - if ($runnerAware -and $null -ne $rawPath -and (Test-Path -LiteralPath $rawPath -PathType Leaf)) { - $bridgeOutput = & pwsh -NoProfile -File $bridgePath -Run $runPath -ExecutionResult $rawPath -Result $resultPath 2>&1 - if ($LASTEXITCODE -ne 0) { - $errors.Add("$($entry.eval_name)/$configuration - execution-result bridge failed: $([string]::Join(' ', @($bridgeOutput)))") - continue - } + foreach ($configuration in @(Get-ManifestConfigurations -Manifest $manifest)) { + $runRecords = @($entryRecords | Where-Object { [string]$_.Configuration -eq $configuration }) + if ($runRecords.Count -ne 1) { + $errors.Add("$($entry.eval_name)/$configuration must have exactly one manifest-declared arm path set.") + continue } + $runRecord = $runRecords[0] + $resultPath = [string]$runRecord.ResultPath + $resultRelative = [string]$runRecord.ResultRelative if (-not (Test-Path -LiteralPath $resultPath)) { - $warnings.Add("$($entry.eval_name)/$configuration - no result file at results/$fileName.") + $warnings.Add("$($entry.eval_name)/$configuration - no result file at the manifest path '$resultRelative'.") continue } try { $result = [System.IO.File]::ReadAllText($resultPath, $utf8NoBom) | ConvertFrom-Json } catch { - $errors.Add("$($entry.eval_name)/$configuration - results/$fileName is not valid JSON: $($_.Exception.Message)") + $errors.Add("$($entry.eval_name)/$configuration - manifest result '$resultRelative' is not valid JSON: $($_.Exception.Message)") continue } if ([string]$result.configuration -ne $configuration) { - $errors.Add("$($entry.eval_name)/$configuration - results/$fileName declares configuration '$($result.configuration)'.") + $errors.Add("$($entry.eval_name)/$configuration - manifest result '$resultRelative' declares configuration '$($result.configuration)'.") continue } if ([int]$result.eval_id -ne [int]$metadata.eval_id) { - $errors.Add("$($entry.eval_name)/$configuration - results/$fileName declares eval_id $($result.eval_id) but the package says $($metadata.eval_id).") + $errors.Add("$($entry.eval_name)/$configuration - manifest result '$resultRelative' declares eval_id $($result.eval_id) but the package says $($metadata.eval_id).") continue } @@ -1960,6 +1983,17 @@ function Invoke-CollectMode { }) } + $completion = Test-ManifestResults ` + -IterationDirectory $iterationDirectory ` + -Manifest $manifest ` + -Records $manifestRecords ` + -RequireComplete + if (-not $completion.Complete) { + foreach ($completionError in @($completion.Errors)) { + $errors.Add("Completion gate: $completionError") + } + } + $builder = [System.Text.StringBuilder]::new() [void]$builder.AppendLine("# Eval comparison: $($manifest.skill_name) (iteration $($manifest.iteration))") [void]$builder.AppendLine() diff --git a/scripts/validate-skill-templates.ps1 b/scripts/validate-skill-templates.ps1 index 163ca4a..4961ac9 100644 --- a/scripts/validate-skill-templates.ps1 +++ b/scripts/validate-skill-templates.ps1 @@ -1452,8 +1452,17 @@ Add-ValidationResult -Results $results -Name 'Skill evaluation prepares portable 'never receive or inspect expected output, assertions, grading, paired output, benchmark data, or human feedback', 'Never fall back to the old generic isolated-worker behavior', 'execution-result.json', + '`runs..run_manifest`', + '`runs..execution_result`', + '`runs..result`', + '`runPath = exact run_manifest`', + '`executionPath = exact execution_result`', + '`resultPath = exact result`', + 'Do not derive, normalize, rename, hyphenate, underscore, or otherwise reconstruct any run, execution-result, or result path.', + 'bridge-manifest-results.ps1', + '-RequireComplete', 'The bridge checks prompt/run/profile hashes and artifact confinement', - 'Only now read each eval''s `eval-metadata.json`', + 'Only if the package bridge succeeds, read each eval''s `eval-metadata.json`', 'grading[].text', 'tools/skill-creator/agents/grader.md', 'scripts/aggregate_benchmark.py', @@ -1470,7 +1479,8 @@ Add-ValidationResult -Results $results -Name 'Skill evaluation prepares portable 'If you truly cannot, this package is not for you', 'Choose evaluation configuration', 'Codebelt Reference', - 'discover current models' + 'discover current models', + '' )) { if ($runner.Contains($forbidden)) { throw "RUN-THIS.prompt.md must not contain forbidden handoff text '$forbidden'." @@ -1502,6 +1512,9 @@ Add-ValidationResult -Results $results -Name 'Skill evaluation prepares portable throw "$($entry.eval_name)/$configuration manifest path '$mustExist' does not exist." } } + if (-not (Test-Path -LiteralPath (Join-Path $iterationDirectory $run.result) -PathType Leaf)) { + throw "$($entry.eval_name)/$configuration manifest result path '$($run.result)' does not exist." + } } $withRunDir = Join-Path $evalDirectory 'with_skill' @@ -1603,23 +1616,24 @@ Add-ValidationResult -Results $results -Name 'Skill evaluation prepares portable } foreach ($configuration in @('with_skill', 'without_skill')) { - $resultFile = if ($configuration -eq 'with_skill') { 'with-skill.result.json' } else { 'without-skill.result.json' } - $stubPath = Join-Path (Join-Path $evalDirectory 'results') $resultFile + $run = $entry.runs.$configuration + $resultLabel = [string]$run.result + $stubPath = Join-Path $iterationDirectory $run.result $stub = [System.IO.File]::ReadAllText($stubPath, $utf8NoBom) | ConvertFrom-Json if ([string]$stub.configuration -ne $configuration) { - throw "$($entry.eval_name) result stub $resultFile must declare configuration '$configuration'." + throw "$($entry.eval_name) result stub $resultLabel must declare configuration '$configuration'." } if (@($stub.grading).Count -ne @($metadata.assertions).Count) { - throw "$($entry.eval_name) result stub $resultFile must carry one grading entry per assertion." + throw "$($entry.eval_name) result stub $resultLabel must carry one grading entry per assertion." } foreach ($propertyName in @('transcript', 'shell_commands', 'files_read', 'files_written', 'exit_status', 'duration_seconds', 'total_tokens', 'tool_calls', 'turns', 'base_input_tokens', 'output_tokens', 'cache_read_tokens', 'cache_write_tokens', 'cache_write_1h_tokens', 'estimated_cost_usd', 'model_effort', 'isolation')) { if ($stub.PSObject.Properties.Name -notcontains $propertyName) { - throw "$($entry.eval_name) result stub $resultFile must expose optional field '$propertyName'." + throw "$($entry.eval_name) result stub $resultLabel must expose optional field '$propertyName'." } } foreach ($isolationField in @('fresh_context', 'isolated_home', 'isolated_cwd', 'filesystem_sandbox', 'candidate_skill_exposed', 'transcript_captured')) { if ($stub.isolation.PSObject.Properties.Name -notcontains $isolationField) { - throw "$($entry.eval_name) result stub $resultFile must expose isolation flag '$isolationField'." + throw "$($entry.eval_name) result stub $resultLabel must expose isolation flag '$isolationField'." } } } @@ -1649,7 +1663,7 @@ Add-ValidationResult -Results $results -Name 'Skill evaluation prepares portable throw 'execution-profile.json has an invalid schema or execution limit.' } $runnerTools = [System.Collections.Generic.List[string]]::new() - foreach ($runnerTool in @('runner-common.ps1', 'resolve-runner.ps1', 'bridge-execution-result.ps1', 'contracts/execution-profile.schema.json', 'contracts/execution-result.schema.json')) { $runnerTools.Add($runnerTool) } + foreach ($runnerTool in @('runner-common.ps1', 'resolve-runner.ps1', 'bridge-execution-result.ps1', 'bridge-manifest-results.ps1', 'manifest-paths.ps1', 'contracts/execution-profile.schema.json', 'contracts/execution-result.schema.json')) { $runnerTools.Add($runnerTool) } $runnerSourceRoot = Join-Path $repoRoot 'scripts/eval-runners' foreach ($runnerDirectory in Get-ChildItem -LiteralPath $runnerSourceRoot -Directory -Force | Sort-Object Name) { if (Test-Path -LiteralPath (Join-Path $runnerDirectory.FullName 'runner.ps1') -PathType Leaf) { @@ -1663,8 +1677,11 @@ Add-ValidationResult -Results $results -Name 'Skill evaluation prepares portable } $collectOutput = & pwsh -NoProfile -File $scriptPath -CollectResults $iterationDirectory 2>&1 - if ($LASTEXITCODE -ne 0) { - throw "prepare-skill-evals.ps1 -CollectResults failed on an unrun package: $($collectOutput -join [Environment]::NewLine)" + if ($LASTEXITCODE -eq 0) { + throw 'prepare-skill-evals.ps1 -CollectResults must fail closed on an unrun package.' + } + if (($collectOutput -join ' ') -notmatch 'Completion gate') { + throw "prepare-skill-evals.ps1 -CollectResults must report the completion gate for an unrun package: $($collectOutput -join [Environment]::NewLine)" } if (-not (Test-Path -LiteralPath (Join-Path $iterationDirectory 'comparison.md'))) { throw 'prepare-skill-evals.ps1 -CollectResults must write comparison.md.' @@ -1680,9 +1697,28 @@ Add-ValidationResult -Results $results -Name 'Skill evaluation prepares portable } $firstEntry = @($manifest.evals)[0] - $firstEvalDirectory = Join-Path $iterationDirectory $firstEntry.directory - foreach ($resultFile in @('with-skill.result.json', 'without-skill.result.json')) { - $resultPath = Join-Path (Join-Path $firstEvalDirectory 'results') $resultFile + $fakeRunnerPath = Join-Path $iterationDirectory 'tools/eval-runners/fake/runner.ps1' + $profilePath = Join-Path $iterationDirectory ([string]$manifest.execution_profile) + foreach ($entryToRun in @($manifest.evals)) { + foreach ($configuration in @('with_skill', 'without_skill')) { + $run = $entryToRun.runs.$configuration + $runPath = Join-Path $iterationDirectory ([string]$run.run_manifest) + $executionPath = Join-Path $iterationDirectory ([string]$run.execution_result) + $executionOutput = & pwsh -NoProfile -File $fakeRunnerPath execute -Run $runPath -Profile $profilePath 2>&1 + if ($LASTEXITCODE -ne 0) { + throw "The deterministic fake runner could not produce the manifest-declared execution result '$($run.execution_result)': $($executionOutput -join [Environment]::NewLine)" + } + $executionJson = [string]::Join([Environment]::NewLine, @($executionOutput)) | ConvertFrom-Json + [System.IO.File]::WriteAllText($executionPath, (($executionJson | ConvertTo-Json -Depth 100) + [Environment]::NewLine), $utf8NoBom) + } + } + $executionCollectOutput = & pwsh -NoProfile -File $scriptPath -CollectResults $iterationDirectory 2>&1 + if ($LASTEXITCODE -ne 0) { + throw "prepare-skill-evals.ps1 -CollectResults failed while bridging the complete deterministic fixture: $($executionCollectOutput -join [Environment]::NewLine)" + } + foreach ($configuration in @('with_skill', 'without_skill')) { + $run = $firstEntry.runs.$configuration + $resultPath = Join-Path $iterationDirectory ([string]$run.result) $result = [System.IO.File]::ReadAllText($resultPath, $utf8NoBom) | ConvertFrom-Json $result.model = 'validator-model' $result.harness = 'validator-harness' From 4f0a3416cf1dd17f2e8dd5cead4d292f34ba7b5e Mon Sep 17 00:00:00 2001 From: "aicia[bot]" Date: Sun, 23 Aug 2026 22:38:27 +0200 Subject: [PATCH 19/39] =?UTF-8?q?=F0=9F=92=AC=20clarify=20eval=20completio?= =?UTF-8?q?n=20gate=20validation?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Update repository guidance in AGENTS.md, CONTRIBUTING.md, and README.md to explicitly clarify that the eval completion gate must be satisfied before a package is presented as successfully completed. Incomplete or unrun evaluation packages must be flagged as incomplete, not misrepresented as passing. --- AGENTS.md | 4 ++-- CONTRIBUTING.md | 2 +- README.md | 2 +- 3 files changed, 4 insertions(+), 4 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 94efa77..564c57a 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -142,7 +142,7 @@ The candidate instructions are already inlined in the with_skill run's `prompt.m Use the same model, model version, configuration, tools, and limits for every worker. Disable persistent memory and cross-session recall. Independent runs may execute concurrently when the selected harness and the user's token budget allow it, but every run still gets a distinct context and no shared mutable workspace. -`RUN-THIS.prompt.md` requires a selected Eval Runner that can create isolated workers or sessions. A plain single-context client can still execute an individual self-contained prompt when the user opens it directly as the first message of a fresh session, but it cannot provide the paired comparison and report contract in that same context. A selected runner that cannot satisfy a required guarantee is `incompatible`; there is no generic fallback or runner substitution. Partial packages still grade and report what exists; missing arms remain visibly missing. +`RUN-THIS.prompt.md` requires a selected Eval Runner that can create isolated workers or sessions. A plain single-context client can still execute an individual self-contained prompt when the user opens it directly as the first message of a fresh session, but it cannot provide the paired comparison and report contract in that same context. A selected runner that cannot satisfy a required guarantee is `incompatible`; there is no generic fallback or runner substitution. Partial package state may be inspected and reported, but the completion gate must pass before it can be presented as a completed evaluation; missing or unrun arms remain visibly incomplete. An `output` is the model's own message in full, including questions, caveats, explanations, or a refusal. Where a run invoked a tool, that tool's stdout is evidence rather than a replacement for the response. Record the full worker transcript, duration, token usage, and tool-call count when the harness exposes them; omit unavailable metrics rather than estimating them. @@ -166,7 +166,7 @@ Validate and compare a collected iteration with: pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -CollectResults ``` -It checks that each result matches its eval and configuration, warns when an arm is missing, unrun, or ran on a different model, and writes `comparison.md`, the paired `report.html`, the exact upstream `skill-creator-report.html`, and the upstream `benchmark.json`/`benchmark.md`. The external evaluator may grade in its user-directed phase-two context; repository automation remains deterministic and never invokes a model. Use deterministic checks for mechanical assertions and evidence-backed human or evaluator judgement only where the assertion is genuinely qualitative. +It checks that each result matches its eval and configuration, may inspect and write `comparison.md`, the paired `report.html`, the exact upstream `skill-creator-report.html`, and the upstream `benchmark.json`/`benchmark.md` while warning when an arm is missing, unrun, or ran on a different model. It exits non-zero when the required completion gate is not satisfied, so an incomplete or unrun package is not a completed evaluation. The external evaluator may grade in its user-directed phase-two context; repository automation remains deterministic and never invokes a model. Use deterministic checks for mechanical assertions and evidence-backed human or evaluator judgement only where the assertion is genuinely qualitative. ### Workspace isolation diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 3d1dc3d..c9c926a 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -110,7 +110,7 @@ Record each external runner result in the matching `execution-result.json` first pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -CollectResults ``` -That writes `comparison.md`, the first-party side-by-side `report.html`, the exact upstream `skill-creator-report.html`, and the upstream `benchmark.json`/`benchmark.md`, while flagging missing arms, unrun configurations, and mixed models. The normal external Eval Orchestrator grades in the same handoff using deterministic checks for mechanical assertions and evidence-backed judgement where an assertion is genuinely qualitative. Repository automation remains deterministic and never invokes a model. Codex, OpenCode, and Cline are the conforming real runners; the deterministic fake runner is the CI conformance harness. Hard filesystem confinement is reported as strict versus pragmatic confidence and is not a universal Windows prerequisite. Freebuff remains planned/blocked until its official CLI provides a supported noninteractive machine-readable transport. Native skill activation, portability scoring, and additional runners are not part of v0.9.1. +It may write `comparison.md`, the first-party side-by-side `report.html`, the exact upstream `skill-creator-report.html`, and the upstream `benchmark.json`/`benchmark.md` while flagging missing arms, unrun configurations, and mixed models, but it exits non-zero when the required completion gate is not satisfied. Those diagnostic artifacts must not present an incomplete or unrun package as a successfully completed evaluation. The normal external Eval Orchestrator grades in the same handoff using deterministic checks for mechanical assertions and evidence-backed judgement where an assertion is genuinely qualitative. Repository automation remains deterministic and never invokes a model. Codex, OpenCode, and Cline are the conforming real runners; the deterministic fake runner is the CI conformance harness. Hard filesystem confinement is reported as strict versus pragmatic confidence and is not a universal Windows prerequisite. Freebuff remains planned/blocked until its official CLI provides a supported noninteractive machine-readable transport. Native skill activation, portability scoring, and additional runners are not part of v0.9.1. The eval package is a temp artifact. Do not commit it, its prompts, or its results unless the change explicitly calls for checked-in examples. diff --git a/README.md b/README.md index 7ee721e..cbb0d00 100644 --- a/README.md +++ b/README.md @@ -22,7 +22,7 @@ pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -Changed -Runner github- That resolves every skill the branch changed and prepares a package for each. `-Skill ` prepares one on demand, but execution selection must already be resolved; direct noninteractive use without `-Runner/-Model` or `-CodebeltReference` fails before a handoff is generated. `scripts/Get-HarnessModels.ps1 -Runner ` lists current model selectors for the selected harness. Use `-CodebeltReference` only when you want the script to verify the current Copilot catalog and select GitHub Copilot CLI + `claude-haiku-4.5`; if that model is absent, the script fails and prints the current choices rather than substituting another model. Packages land in the gitignored `.bot/-workspace/`, so a harness that refuses to work outside the repository folder can still reach them without anything entering the working tree. -Each eval becomes a directory holding the grading key (`eval-metadata.json` with the expected output, assertions, and fixture and skill hashes) and prefilled result stubs, plus two paired run directories. `with_skill/` is a self-contained run root: a `prompt.md` with the effective skill instructions inlined, a `run.json` contract naming only paths inside the run, a `repo/` working tree materialized from the fixtures as real files, an isolated empty `home/`, and a `skill//` copy of the exact candidate revision. `without_skill/` is the same run with a byte-identical `repo/`, no `skill/` directory, and no mention of the skill. The grading key and results sit outside both run directories, so workers are not intentionally given them. Neither prompt identifies itself as an eval or names its configuration. `RUN-THIS.prompt.md` makes the user-selected agent the Eval Orchestrator: it reads the profile, resolves the selected runner, validates `describe`, preflights each `run.json`, invokes one fresh runner process per arm, stores raw `execution-result.json`, bridges it into the existing result shape, reveals the grading key only after execution, grades, and invokes the package adapter. The adapter stages the results into Anthropic's upstream benchmark workspace, runs `aggregate_benchmark.py`, writes the exact upstream `skill-creator-report.html`, and writes a first-party `report.html` with paired outputs, expected outcomes, assertion evidence, telemetry, transcripts, and downloadable feedback, plus `benchmark.json` and `benchmark.md`. Missing telemetry is displayed as unavailable rather than estimated. A runner that cannot satisfy the mandatory experimental controls returns `incompatible`; lack of hard filesystem confinement downgrades the result to pragmatic isolation. `-CollectResults ` remains a deterministic fallback for transferred artifacts without reports. Packages land in gitignored `.bot/` storage by default and are not committed. +Each eval becomes a directory holding the grading key (`eval-metadata.json` with the expected output, assertions, and fixture and skill hashes) and prefilled result stubs, plus two paired run directories. `with_skill/` is a self-contained run root: a `prompt.md` with the effective skill instructions inlined, a `run.json` contract naming only paths inside the run, a `repo/` working tree materialized from the fixtures as real files, an isolated empty `home/`, and a `skill//` copy of the exact candidate revision. `without_skill/` is the same run with a byte-identical `repo/`, no `skill/` directory, and no mention of the skill. The grading key and results sit outside both run directories, so workers are not intentionally given them. Neither prompt identifies itself as an eval or names its configuration. `RUN-THIS.prompt.md` makes the user-selected agent the Eval Orchestrator: it reads the profile, resolves the selected runner, validates `describe`, preflights each `run.json`, invokes one fresh runner process per arm, stores raw `execution-result.json`, bridges it into the existing result shape, reveals the grading key only after execution, grades, and invokes the package adapter. The adapter stages the results into Anthropic's upstream benchmark workspace, runs `aggregate_benchmark.py`, writes the exact upstream `skill-creator-report.html`, and writes a first-party `report.html` with paired outputs, expected outcomes, assertion evidence, telemetry, transcripts, and downloadable feedback, plus `benchmark.json` and `benchmark.md`. Missing telemetry is displayed as unavailable rather than estimated. A runner that cannot satisfy the mandatory experimental controls returns `incompatible`; lack of hard filesystem confinement downgrades the result to pragmatic isolation. `-CollectResults ` remains a deterministic fallback for transferred artifacts without reports; it may inspect and report available package state, but exits non-zero when the required completion gate is not satisfied, so an incomplete or unrun package is not a successfully completed evaluation. Packages land in gitignored `.bot/` storage by default and are not committed. The four roles are intentionally separate: the Eval Runner is the harness-specific executor for one blind arm; the Eval Orchestrator coordinates the external handoff; the Grader assesses results only after execution; and the Human Reviewer remains the final evaluator. Behavioral evaluation is portable across supported runners. Native activation is harness-specific and out of scope for v0.9.1. Anthropic is the methodology and compatibility reference, not a required Claude runtime. From c5516b0ef72ec8ec5ceb174eb76e8f431bdb7581 Mon Sep 17 00:00:00 2001 From: "aicia[bot]" Date: Sun, 23 Aug 2026 22:38:34 +0200 Subject: [PATCH 20/39] =?UTF-8?q?=F0=9F=94=A8=20update=20eval=20package=20?= =?UTF-8?q?readme=20generation?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Update the generated README.md content in prepare-skill-evals.ps1 to reflect the clarified eval completion gate behavior. Packages generated by this script now include text clarifying that incomplete or unrun arms must be reported as such and that -CollectResults exits non-zero when the completion gate is not satisfied. --- scripts/prepare-skill-evals.ps1 | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/prepare-skill-evals.ps1 b/scripts/prepare-skill-evals.ps1 index 8f125a6..1c389d0 100644 --- a/scripts/prepare-skill-evals.ps1 +++ b/scripts/prepare-skill-evals.ps1 @@ -1699,7 +1699,7 @@ function New-PackageReadme { [void]$builder.AppendLine() [void]$builder.AppendLine('`RUN-THIS.prompt.md` is the external Eval Orchestrator handoff. It selects the package-local runner from the profile, invokes the common protocol once per blind arm, bridges raw evidence into the existing result shape, reveals grading material only after execution, and invokes Anthropic skill-creator''s compatible aggregator and static viewer through the package adapter. It never executes an eval prompt in its own context.') [void]$builder.AppendLine() - [void]$builder.AppendLine('A harness that cannot provide fresh, independent sessions with isolated working and config roots is incompatible with these evals. `-CollectResults` accepts a partial iteration and reports unfilled runs as missing.') + [void]$builder.AppendLine('A harness that cannot provide fresh, independent sessions with isolated working and config roots is incompatible with these evals. `-CollectResults` may inspect and report available package state, including missing or unrun arms, but it exits non-zero when the required completion gate is not satisfied. An incomplete or unrun package must not be presented as a successfully completed evaluation.') [void]$builder.AppendLine() [void]$builder.AppendLine('A with_skill run on one model compared against a baseline on another measures both the model and the skill. That is not a skill-effectiveness result, so do not report it as one. If you do mix models, say so explicitly and treat the comparison as directional only.') [void]$builder.AppendLine() From 24838e21557e06fbde4809ee89977b5a3440f4cd Mon Sep 17 00:00:00 2001 From: "aicia[bot]" Date: Mon, 24 Aug 2026 12:56:02 +0200 Subject: [PATCH 21/39] =?UTF-8?q?=F0=9F=91=B7=20add=20native=20worker=20or?= =?UTF-8?q?chestration=20to=20eval=20runner=20protocol?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Implements deterministic queue and state management for eval workers. Adds native delegation surface with full-capability and model-lock guarantees. Each eval arm executes in an isolated harness-native worker with working-directory isolation, result capture, and fresh session constraints. Supports concurrent worker coordination with configurable capacity slots and rejection handling without eval attempt increment. --- scripts/eval-runners/README.md | 64 ++++ scripts/eval-runners/cline/runner.ps1 | 91 ++++- scripts/eval-runners/codex/runner.ps1 | 75 +++- .../contracts/orchestration-plan.schema.json | 75 ++++ .../contracts/preflight-result.schema.json | 23 ++ .../contracts/runner-descriptor.schema.json | 27 ++ scripts/eval-runners/fake/runner.ps1 | 17 + .../eval-runners/github-copilot/runner.ps1 | 20 +- scripts/eval-runners/opencode/runner.ps1 | 20 +- scripts/eval-runners/orchestration.ps1 | 342 ++++++++++++++++++ scripts/eval-runners/runner-common.ps1 | 135 ++++++- .../eval-runners/tests/test-orchestration.ps1 | 207 +++++++++++ .../tests/test-runner-conformance.ps1 | 35 +- scripts/prepare-skill-evals.ps1 | 64 +++- 14 files changed, 1166 insertions(+), 29 deletions(-) create mode 100644 scripts/eval-runners/contracts/orchestration-plan.schema.json create mode 100644 scripts/eval-runners/orchestration.ps1 create mode 100644 scripts/eval-runners/tests/test-orchestration.ps1 diff --git a/scripts/eval-runners/README.md b/scripts/eval-runners/README.md index 2559817..507851b 100644 --- a/scripts/eval-runners/README.md +++ b/scripts/eval-runners/README.md @@ -1,5 +1,65 @@ # Eval Runner protocol +## Native worker orchestration + +The external handoff uses this topology: + +```text +Eval Orchestrator + | + +-- Eval Worker -> one eval arm + +-- Eval Worker -> one eval arm + +-- Eval Worker -> one eval arm + +-- ... +``` + +The Eval Orchestrator coordinates the manifest queue, native child creation, +terminal evidence, exact manifest destinations, and the phase boundary. It +does not execute an eval arm itself. One arm equals one fresh delegated +harness-native worker and one model-backed execution. The delegated worker +executes the prepared prompt directly; it must not call the runner's +model-spawning `execute` command or start another model session. + +`orchestration.ps1` is the deterministic queue/state helper copied into every +package. It creates one worker envelope per exact manifest arm, keeps unrelated +arms dependency-free, exposes at most the requested +`execution-profile.json.concurrency` active slots, and leaves a capacity +rejection pending without incrementing the eval attempt count. It contains no +harness-specific concurrency ceiling. `Assert-NativeWorkerDelegation` is the +fail-closed preflight gate: a conditional or unavailable native mechanism +cannot fall back to parent execution. + +The descriptor's `delegation` object records the native mechanism, worker role, +full-capability and model-lock guarantees, working-directory/result capture, +harness-authoritative capacity, and the invariant +`nested_model_execution = false`. The direct `execute` process surface remains +for compatibility and deterministic conformance; the external orchestrator +must use only `describe`, `preflight`, and the harness-native delegation +surface for the actual eval arms. It must reject a worker whose terminal +evidence does not prove the selected model, exact run working directory, +isolated home, and fresh session; a native surface that cannot prove those +controls is incompatible rather than a reason to fall back to the parent. + +Native delegation mechanisms: + +- GitHub Copilot: the CLI's native `task` tool with an explicit + full-capability `general-purpose` child agent for each arm. Fleet/subagent + lifecycle is harness-owned; Codebelt supplies the already-known one-arm + decomposition and observes child completion/model evidence. +- Codex: the installed CLI's native app-server child-session surface, + `thread/start` followed by `turn/start`, with the arm's `cwd`, selected + model, and ephemeral/fresh session settings. Do not wrap a native Codex + child in another `codex exec` invocation. +- OpenCode: the native Task tool with the full-capability built-in `General` + subagent. `Explore`/`Scout` read-only agents are not valid for a mutable + eval arm. +- Cline: a full-capability Cline SDK Agent Squad + `start_subagent(preset: "anvil")` child session backed by + `ClineCore.create`. The plugin's default `phantom` preset and Cline's + documented `use_subagents` research feature are read-only and are rejected + for mutable evals. The descriptor remains conditional until that + plugin/SDK mechanism is installed and preflight can prove it. + This directory contains the package-local implementation of the v0.9.1 Eval Runner protocol. It is copied into prepared packages so the external Eval Orchestrator can use the same runner implementation that was validated with the @@ -61,6 +121,10 @@ Cross-runner and cross-model numbers are never blended into one score; a paired `with_skill` versus `without_skill` comparison is only meaningful within one identical runner, model, and configuration stratum. +The following CLI details describe the compatibility `execute` transport only; +they are not a second model layer for native worker orchestration. The native +worker mechanisms above are authoritative for the external handoff. + GitHub Copilot uses `copilot -C --model --output-format json --allow-all-tools --no-ask-user --disable-builtin-mcps --no-color --log-level none --no-auto-update diff --git a/scripts/eval-runners/cline/runner.ps1 b/scripts/eval-runners/cline/runner.ps1 index 8fb1cdb..cdaef43 100644 --- a/scripts/eval-runners/cline/runner.ps1 +++ b/scripts/eval-runners/cline/runner.ps1 @@ -47,6 +47,23 @@ $descriptor = [ordered]@{ cost_telemetry = 'conditional' credential_child_filtering = 'conditional' native_skill_activation_evidence = 'unsupported' + native_worker_delegation = 'conditional' + delegated_worker_full_capability = 'conditional' + delegated_worker_model_lock = 'conditional' + delegated_worker_working_directory = 'conditional' + delegated_worker_result_capture = 'conditional' + delegated_worker_capacity_signal = 'conditional' + } + delegation = [ordered]@{ + mode = 'conditional' + mechanism = 'Cline SDK Agent Squad start_subagent(preset: anvil) child session backed by ClineCore.create; the read-only use_subagents research feature is not acceptable' + worker_role = 'agents-squad-child-agent' + full_capability = 'conditional' + model_lock = 'conditional' + working_directory = 'conditional' + result_capture = 'conditional' + capacity = 'harness_authoritative' + nested_model_execution = $false } supported_telemetry = @('transcript_event_capture', 'token_telemetry', 'cache_token_telemetry', 'tool_call_telemetry', 'command_evidence', 'file_evidence', 'cost_telemetry') configuration_profiles = @('isolated-default') @@ -197,10 +214,69 @@ function New-ClineCliArguments { return @($arguments) } +function Get-ClineAgentsSquadCapability { + $configuredPath = [Environment]::GetEnvironmentVariable('CLINE_AGENTS_SQUAD_PLUGIN') + if ([string]::IsNullOrWhiteSpace($configuredPath)) { + return [pscustomobject]@{ + Available = $false + Detail = 'No CLINE_AGENTS_SQUAD_PLUGIN path is configured; the read-only use_subagents feature cannot execute a mutable eval arm.' + } + } + try { + $resolvedPath = [System.IO.Path]::GetFullPath($configuredPath) + if (-not (Test-Path -LiteralPath $resolvedPath -PathType Container)) { + return [pscustomobject]@{ + Available = $false + Detail = "Configured Cline Agent Squad path '$resolvedPath' is not an existing directory." + } + } + $packageJson = Join-Path $resolvedPath 'package.json' + if (-not (Test-Path -LiteralPath $packageJson -PathType Leaf)) { + return [pscustomobject]@{ + Available = $false + Detail = "Configured Cline Agent Squad path '$resolvedPath' has no package.json marker." + } + } + $package = Read-RunnerJson -Path $packageJson + $pluginEntries = @((Get-JsonProperty -Object (Get-JsonProperty -Object $package -Name 'cline' -Default $null) -Name 'plugins' -Default @())) + $runtimeEntryProven = $false + foreach ($pluginEntry in $pluginEntries) { + $capabilities = @((Get-JsonProperty -Object $pluginEntry -Name 'capabilities' -Default @()) | ForEach-Object { [string]$_ }) + $entryPaths = @((Get-JsonProperty -Object $pluginEntry -Name 'paths' -Default @())) + if ($capabilities -contains 'hooks' -and $capabilities -contains 'tools' -and $entryPaths.Count -gt 0) { + foreach ($entryPath in $entryPaths) { + $candidateEntry = [System.IO.Path]::GetFullPath((Join-Path $resolvedPath ([string]$entryPath))) + if (Test-PathInside -BasePath $resolvedPath -CandidatePath $candidateEntry -and (Test-Path -LiteralPath $candidateEntry -PathType Leaf)) { + $runtimeEntryProven = $true + break + } + } + } + if ($runtimeEntryProven) { break } + } + if (-not $runtimeEntryProven) { + return [pscustomobject]@{ + Available = $false + Detail = "Configured Cline Agent Squad path '$resolvedPath' does not expose a cline plugin entry with hooks/tools and an existing entry file." + } + } + return [pscustomobject]@{ + Available = $true + Detail = "Cline Agent Squad plugin manifest and runtime entry are available at '$resolvedPath'; the worker must use start_subagent with the full-capability anvil preset, backed by ClineCore.create with explicit cwd, model, tools, and result polling." + } + } catch { + return [pscustomobject]@{ + Available = $false + Detail = "Cline Agent Squad plugin path could not be resolved: $($_.Exception.Message)" + } + } +} + function Get-ClineCapabilityMap { param( [Parameter(Mandatory = $true)][object]$Inputs, - [bool]$HardFilesystemConfinement = $false + [bool]$HardFilesystemConfinement = $false, + [bool]$NativeWorkerAvailable = $false ) $capabilities = [ordered]@{} @@ -209,6 +285,9 @@ function Get-ClineCapabilityMap { } $capabilities['filesystem_confinement'] = if ($HardFilesystemConfinement) { 'supported' } else { 'unsupported' } $capabilities['candidate_skill_exposure'] = if ($Inputs.Run.CandidateSkillExposed) { 'supported' } else { 'excluded' } + foreach ($name in @('native_worker_delegation', 'delegated_worker_full_capability', 'delegated_worker_model_lock', 'delegated_worker_working_directory', 'delegated_worker_result_capture', 'delegated_worker_capacity_signal')) { + $capabilities[$name] = if ($NativeWorkerAvailable) { 'supported' } else { 'unsupported' } + } return $capabilities } @@ -224,6 +303,7 @@ function Get-ClinePreflight { $commandInfo = Resolve-ExternalCommand -Name 'cline' $sandboxInfo = if ($platform -eq 'linux') { Resolve-ExternalCommand -Name 'bwrap' } elseif ($platform -eq 'macos') { Resolve-ExternalCommand -Name 'sandbox-exec' } else { $null } $versionObservation = $null + $agentsSquad = Get-ClineAgentsSquadCapability if ($profile.Runner -ne 'cline') { $reasons.Add("execution-profile.json selects '$($profile.Runner)' rather than cline.") @@ -242,6 +322,13 @@ function Get-ClinePreflight { if ($profile.ConfigurationProfile -ne 'isolated-default') { $reasons.Add("configuration_profile '$($profile.ConfigurationProfile)' is unsupported by cline.") } if ($profile.ToolProfile -ne 'default') { $reasons.Add("tool_profile '$($profile.ToolProfile)' is unsupported by cline.") } + if ($agentsSquad.Available) { + $checks.Add((New-PreflightCheck -Name 'native_worker_delegation' -Status passed -Detail $agentsSquad.Detail)) + } else { + $checks.Add((New-PreflightCheck -Name 'native_worker_delegation' -Status unavailable -Detail $agentsSquad.Detail)) + $warnings.Add('Cline native worker delegation is unavailable. The external orchestrator must not use the read-only use_subagents feature or the direct CLI execute transport as a fallback; install the full-capability Agent Squad plugin and set CLINE_AGENTS_SQUAD_PLUGIN.') + } + $environmentData = New-ClineEnvironment -Inputs $Inputs if ($null -eq $commandInfo) { $reasons.Add('The Cline CLI executable is not available on PATH.') @@ -309,7 +396,7 @@ function Get-ClinePreflight { $warnings.Add('Cline does not expose a supported child-tool environment filter in this CLI contract; the runner removes unrelated inherited variables but cannot independently prove that the selected provider credential is hidden from every Cline-launched tool.') $hardConfinement = $null -ne $sandboxInfo -and $platform -in @('linux', 'macos') - $capabilities = Get-ClineCapabilityMap -Inputs $Inputs -HardFilesystemConfinement $hardConfinement + $capabilities = Get-ClineCapabilityMap -Inputs $Inputs -HardFilesystemConfinement $hardConfinement -NativeWorkerAvailable $agentsSquad.Available $harnessVersion = if ($null -eq $versionObservation) { 'unavailable' } else { [string]$versionObservation.Version } $descriptorCopy = [ordered]@{} foreach ($key in $descriptor.Keys) { $descriptorCopy[$key] = $descriptor[$key] } diff --git a/scripts/eval-runners/codex/runner.ps1 b/scripts/eval-runners/codex/runner.ps1 index bf102e9..26393e3 100644 --- a/scripts/eval-runners/codex/runner.ps1 +++ b/scripts/eval-runners/codex/runner.ps1 @@ -46,6 +46,23 @@ $descriptor = [ordered]@{ cost_telemetry = 'conditional' credential_child_filtering = 'supported' native_skill_activation_evidence = 'unsupported' + native_worker_delegation = 'supported' + delegated_worker_full_capability = 'supported' + delegated_worker_model_lock = 'supported' + delegated_worker_working_directory = 'supported' + delegated_worker_result_capture = 'supported' + delegated_worker_capacity_signal = 'supported' + } + delegation = [ordered]@{ + mode = 'native_worker' + mechanism = 'Codex app-server native child session via thread/start and turn/start with per-worker cwd, model, and ephemeral context' + worker_role = 'native-codex-child-session' + full_capability = 'supported' + model_lock = 'supported' + working_directory = 'supported' + result_capture = 'supported' + capacity = 'harness_authoritative' + nested_model_execution = $false } supported_telemetry = @('transcript_event_capture', 'token_telemetry', 'cache_token_telemetry', 'tool_call_telemetry', 'command_evidence', 'file_evidence', 'cost_telemetry') configuration_profiles = @('isolated-default') @@ -116,6 +133,46 @@ function Get-CodexHelpResult { return Invoke-CodexCli -CommandInfo $CommandInfo -Arguments $Arguments -Inputs $Inputs -Environment $environment -TimeoutSeconds 30 } +function Get-CodexNativeWorkerProbe { + param( + [Parameter(Mandatory = $true)][object]$CommandInfo, + [Parameter(Mandatory = $true)][object]$Inputs + ) + + $environment = New-RunnerEnvironment -Run $Inputs.Run + $help = Invoke-CodexCli -CommandInfo $CommandInfo -Arguments @('app-server', '--help') -Inputs $Inputs -Environment $environment -TimeoutSeconds 30 + if ($help.TimedOut -or $help.ExitCode -ne 0) { + return [pscustomobject]@{ Available = $false; Detail = "codex app-server --help failed with exit status $($help.ExitCode)." } + } + $helpText = [string]::Join("`n", @($help.Stdout, $help.Stderr)) + if ($helpText -notmatch 'generate-json-schema') { + return [pscustomobject]@{ Available = $false; Detail = 'The installed Codex CLI does not advertise app-server schema generation.' } + } + $features = Invoke-CodexCli -CommandInfo $CommandInfo -Arguments @('features', 'list') -Inputs $Inputs -Environment $environment -TimeoutSeconds 30 + if ($features.TimedOut -or $features.ExitCode -ne 0 -or ([string]::Join("`n", @($features.Stdout, $features.Stderr)) -notmatch '(?im)multi_agent\s+stable\s+true')) { + return [pscustomobject]@{ Available = $false; Detail = 'The installed Codex CLI did not report the stable multi_agent feature required for native child workers.' } + } + + $schemaRelativeDirectory = Join-Path ([System.IO.Path]::GetRelativePath($Inputs.Run.WorkingDirectoryPath, $Inputs.Run.HomeDirectoryPath)) 'evidence/codex-app-server-schema' + $schemaDirectory = [System.IO.Path]::GetFullPath((Join-Path $Inputs.Run.WorkingDirectoryPath $schemaRelativeDirectory)) + New-Item -ItemType Directory -Path $schemaDirectory -Force | Out-Null + $schemaProcess = Invoke-CodexCli -CommandInfo $CommandInfo -Arguments @('app-server', 'generate-json-schema', "--out=$schemaRelativeDirectory") -Inputs $Inputs -Environment $environment -TimeoutSeconds 60 + if ($schemaProcess.TimedOut -or $schemaProcess.ExitCode -ne 0) { + return [pscustomobject]@{ Available = $false; Detail = "Codex app-server schema generation failed with exit status $($schemaProcess.ExitCode): $([string]::Join(' ', @($schemaProcess.Stdout, $schemaProcess.Stderr)))." } + } + $schemaFiles = @(Get-ChildItem -LiteralPath $schemaDirectory -Recurse -File -Filter '*.json' -ErrorAction SilentlyContinue) + $schemaText = [string]::Join("`n", @($schemaFiles | ForEach-Object { [System.IO.File]::ReadAllText($_.FullName, [System.Text.UTF8Encoding]::new($false)) })) + foreach ($needle in @('thread/start', 'turn/start', 'ThreadStartParams', 'TurnStartParams', '"cwd"', '"model"', '"ephemeral"')) { + if ($schemaText -notmatch [regex]::Escape($needle)) { + return [pscustomobject]@{ Available = $false; Detail = "Codex app-server schema did not prove native child-session field '$needle'." } + } + } + return [pscustomobject]@{ + Available = $true + Detail = 'Codex multi_agent is stable and app-server schema proves thread/start and turn/start with cwd, model, and ephemeral child-session controls.' + } +} + function Resolve-SandboxCommand { param([Parameter(Mandatory = $true)][string]$Name) @@ -159,7 +216,8 @@ function New-CodexCliArguments { function Get-CodexCapabilityMap { param( [Parameter(Mandatory = $true)][object]$Inputs, - [bool]$HardFilesystemConfinement = $false + [bool]$HardFilesystemConfinement = $false, + [bool]$NativeWorkerAvailable = $true ) $capabilities = [ordered]@{} @@ -168,6 +226,9 @@ function Get-CodexCapabilityMap { } $capabilities['filesystem_confinement'] = if ($HardFilesystemConfinement) { 'supported' } else { 'unsupported' } $capabilities['candidate_skill_exposure'] = if ($Inputs.Run.CandidateSkillExposed) { 'supported' } else { 'excluded' } + foreach ($name in @('native_worker_delegation', 'delegated_worker_full_capability', 'delegated_worker_model_lock', 'delegated_worker_working_directory', 'delegated_worker_result_capture', 'delegated_worker_capacity_signal')) { + $capabilities[$name] = if ($NativeWorkerAvailable) { 'supported' } else { 'unsupported' } + } return $capabilities } @@ -188,6 +249,7 @@ function Get-CodexPreflight { } $sandboxInfo = if ([string]::IsNullOrWhiteSpace([string]$sandboxName)) { $null } else { Resolve-SandboxCommand -Name $sandboxName } $versionObservation = $null + $nativeWorkerObservation = $null if ($profile.Runner -ne 'codex') { $reasons.Add("execution-profile.json selects '$($profile.Runner)' rather than codex.") @@ -248,6 +310,13 @@ function Get-CodexPreflight { $checks.Add((New-PreflightCheck -Name 'harness_contract' -Status passed -Detail 'Codex accepts the constructed noninteractive invocation: --ask-for-approval never, exec, --sandbox workspace-write, ephemeral JSON output, and isolated configuration controls.')) } } + $nativeWorkerObservation = Get-CodexNativeWorkerProbe -CommandInfo $commandInfo -Inputs $Inputs + if ($nativeWorkerObservation.Available) { + $checks.Add((New-PreflightCheck -Name 'native_worker_delegation' -Status passed -Detail $nativeWorkerObservation.Detail)) + } else { + $checks.Add((New-PreflightCheck -Name 'native_worker_delegation' -Status unavailable -Detail $nativeWorkerObservation.Detail)) + $reasons.Add($nativeWorkerObservation.Detail) + } } catch { $reasons.Add("Could not inspect Codex CLI capabilities: $($_.Exception.Message)") } @@ -278,13 +347,13 @@ function Get-CodexPreflight { $checks.Add((New-PreflightCheck -Name 'credential_boundary' -Status passed -Detail 'Only the selected provider API-key variable is passed to Codex; auth files are never copied into the worker HOME.')) $hardConfinement = $null -ne $sandboxInfo -and $platform -in @('linux', 'macos') - $capabilities = Get-CodexCapabilityMap -Inputs $Inputs -HardFilesystemConfinement $hardConfinement + $capabilities = Get-CodexCapabilityMap -Inputs $Inputs -HardFilesystemConfinement $hardConfinement -NativeWorkerAvailable ($null -ne $nativeWorkerObservation -and $nativeWorkerObservation.Available) $harnessVersion = if ($null -eq $versionObservation) { 'unavailable' } else { [string]$versionObservation.Version } $descriptorCopy = [ordered]@{} foreach ($key in $descriptor.Keys) { $descriptorCopy[$key] = $descriptor[$key] } $descriptorCopy.harness = [ordered]@{ name = 'OpenAI Codex CLI'; version = $harnessVersion } $mechanisms = [System.Collections.Generic.List[string]]::new() - foreach ($mechanism in @('--ask-for-approval never', 'codex exec --ephemeral', '--ignore-user-config', '--ignore-rules', '--sandbox workspace-write', 'shell_environment_policy.inherit=none', 'isolated CODEX_HOME', 'prompt on stdin', 'no session continuation')) { $mechanisms.Add($mechanism) } + foreach ($mechanism in @('native app-server thread/start + turn/start child session', '--ask-for-approval never', 'codex exec --ephemeral compatibility transport', '--ignore-user-config', '--ignore-rules', '--sandbox workspace-write', 'shell_environment_policy.inherit=none', 'isolated CODEX_HOME', 'prompt on stdin', 'no session continuation')) { $mechanisms.Add($mechanism) } if ($hardConfinement) { $mechanisms.Add("external $sandboxName filesystem sandbox") } else { $mechanisms.Add('pragmatic process/environment isolation without hard filesystem confinement') } return New-PreflightDocument -Descriptor $descriptorCopy -Profile $profile -Run $run -Compatible ($reasons.Count -eq 0) -Checks @($checks) -Mechanisms @($mechanisms) -ResolvedCapabilities $capabilities -Warnings @($warnings) -Reasons @($reasons) } diff --git a/scripts/eval-runners/contracts/orchestration-plan.schema.json b/scripts/eval-runners/contracts/orchestration-plan.schema.json new file mode 100644 index 0000000..643052d --- /dev/null +++ b/scripts/eval-runners/contracts/orchestration-plan.schema.json @@ -0,0 +1,75 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "https://codebelt.net/schemas/agentic/eval-orchestration-plan/1", + "title": "Codebelt Agentic Eval Native Worker Orchestration Plan", + "type": "object", + "additionalProperties": true, + "required": [ + "schema", + "protocol_version", + "runner", + "model", + "requested_concurrency", + "native_worker_required", + "parent_executes_arms", + "nested_model_execution", + "arms" + ], + "properties": { + "schema": { "const": "codebeltnet/agentic/eval-orchestration-plan/1" }, + "protocol_version": { "const": "codebeltnet/agentic/eval-runner-protocol/1" }, + "runner": { "type": "string", "minLength": 1 }, + "model": { "type": "string", "minLength": 1 }, + "requested_concurrency": { "type": "integer", "minimum": 1 }, + "native_worker_required": { "const": true }, + "parent_executes_arms": { "const": false }, + "nested_model_execution": { "const": false }, + "arms": { + "type": "array", + "items": { + "type": "object", + "required": ["worker_id", "eval_id", "eval_name", "configuration", "depends_on", "parent_paths", "worker"], + "properties": { + "worker_id": { "type": "string", "minLength": 1 }, + "eval_id": { "type": "integer", "minimum": 1 }, + "eval_name": { "type": "string", "minLength": 1 }, + "configuration": { "enum": ["with_skill", "without_skill"] }, + "depends_on": { "type": "array", "maxItems": 0 }, + "parent_paths": { + "type": "object", + "required": ["run_manifest", "execution_result", "result"], + "properties": { + "run_manifest": { "type": "string", "minLength": 1 }, + "execution_result": { "type": "string", "minLength": 1 }, + "result": { "type": "string", "minLength": 1 } + } + }, + "worker": { + "type": "object", + "required": [ + "worker_id", "eval_id", "eval_name", "configuration", "run_manifest", "run_manifest_path", + "model", "one_arm_only", "paired_arm_visible", "grading_material_visible", "parent_executes_arm", + "runner_execute_invocation", "nested_model_execution", "model_execution_count" + ], + "properties": { + "worker_id": { "type": "string", "minLength": 1 }, + "eval_id": { "type": "integer", "minimum": 1 }, + "eval_name": { "type": "string", "minLength": 1 }, + "configuration": { "enum": ["with_skill", "without_skill"] }, + "run_manifest": { "type": "string", "minLength": 1 }, + "run_manifest_path": { "type": "string", "minLength": 1 }, + "model": { "type": "string", "minLength": 1 }, + "one_arm_only": { "const": true }, + "paired_arm_visible": { "const": false }, + "grading_material_visible": { "const": false }, + "parent_executes_arm": { "const": false }, + "runner_execute_invocation": { "const": "forbidden" }, + "nested_model_execution": { "const": false }, + "model_execution_count": { "const": 1 } + } + } + } + } + } + } +} diff --git a/scripts/eval-runners/contracts/preflight-result.schema.json b/scripts/eval-runners/contracts/preflight-result.schema.json index b942e9c..07f8603 100644 --- a/scripts/eval-runners/contracts/preflight-result.schema.json +++ b/scripts/eval-runners/contracts/preflight-result.schema.json @@ -14,6 +14,7 @@ "requested", "checks", "resolved_capabilities", + "delegation", "isolation", "mechanisms", "warnings", @@ -43,6 +44,28 @@ }, "checks": { "type": "array" }, "resolved_capabilities": { "type": "object" }, + "delegation": { + "type": "object", + "additionalProperties": false, + "required": [ + "status", + "mode", + "mechanism", + "worker_role", + "nested_model_execution", + "required_controls", + "unproven_controls" + ], + "properties": { + "status": { "enum": ["supported", "conditional", "unsupported"] }, + "mode": { "enum": ["native_worker", "conditional", "unsupported"] }, + "mechanism": { "type": "string" }, + "worker_role": { "type": "string" }, + "nested_model_execution": { "type": "boolean" }, + "required_controls": { "type": "array", "items": { "type": "string" } }, + "unproven_controls": { "type": "array", "items": { "type": "string" } } + } + }, "isolation": { "type": "object", "required": ["level", "status", "hard_filesystem_confinement", "unproven_controls"], diff --git a/scripts/eval-runners/contracts/runner-descriptor.schema.json b/scripts/eval-runners/contracts/runner-descriptor.schema.json index e4ae3b4..8a2827f 100644 --- a/scripts/eval-runners/contracts/runner-descriptor.schema.json +++ b/scripts/eval-runners/contracts/runner-descriptor.schema.json @@ -12,6 +12,7 @@ "platforms", "harness", "capabilities", + "delegation", "supported_telemetry", "configuration_profiles", "tool_profiles" @@ -34,6 +35,32 @@ "type": "object", "additionalProperties": { "enum": ["supported", "conditional", "unsupported"] } }, + "delegation": { + "type": "object", + "additionalProperties": false, + "required": [ + "mode", + "mechanism", + "worker_role", + "full_capability", + "model_lock", + "working_directory", + "result_capture", + "capacity", + "nested_model_execution" + ], + "properties": { + "mode": { "enum": ["native_worker", "conditional", "unsupported"] }, + "mechanism": { "type": "string", "minLength": 1 }, + "worker_role": { "type": "string", "minLength": 1 }, + "full_capability": { "enum": ["supported", "conditional", "unsupported"] }, + "model_lock": { "enum": ["supported", "conditional", "unsupported"] }, + "working_directory": { "enum": ["supported", "conditional", "unsupported"] }, + "result_capture": { "enum": ["supported", "conditional", "unsupported"] }, + "capacity": { "enum": ["supported", "conditional", "unsupported", "harness_authoritative"] }, + "nested_model_execution": { "const": false } + } + }, "supported_telemetry": { "type": "array", "items": { "type": "string" } }, "configuration_profiles": { "type": "array", "items": { "type": "string" } }, "tool_profiles": { "type": "array", "items": { "type": "string" } } diff --git a/scripts/eval-runners/fake/runner.ps1 b/scripts/eval-runners/fake/runner.ps1 index 9453c56..697a359 100644 --- a/scripts/eval-runners/fake/runner.ps1 +++ b/scripts/eval-runners/fake/runner.ps1 @@ -49,6 +49,23 @@ $descriptor = [ordered]@{ file_evidence = 'supported' cost_telemetry = 'unsupported' native_skill_activation_evidence = 'unsupported' + native_worker_delegation = 'supported' + delegated_worker_full_capability = 'supported' + delegated_worker_model_lock = 'supported' + delegated_worker_working_directory = 'supported' + delegated_worker_result_capture = 'supported' + delegated_worker_capacity_signal = 'supported' + } + delegation = [ordered]@{ + mode = 'native_worker' + mechanism = 'deterministic fake native-worker fixture with one arm per worker' + worker_role = 'fake-eval-worker' + full_capability = 'supported' + model_lock = 'supported' + working_directory = 'supported' + result_capture = 'supported' + capacity = 'harness_authoritative' + nested_model_execution = $false } supported_telemetry = @('transcript_event_capture', 'tool_call_telemetry', 'command_evidence', 'file_evidence') configuration_profiles = @('isolated-default') diff --git a/scripts/eval-runners/github-copilot/runner.ps1 b/scripts/eval-runners/github-copilot/runner.ps1 index b5c21e0..9fd18a6 100644 --- a/scripts/eval-runners/github-copilot/runner.ps1 +++ b/scripts/eval-runners/github-copilot/runner.ps1 @@ -59,6 +59,23 @@ $descriptor = [ordered]@{ cost_telemetry = 'unsupported' credential_child_filtering = 'supported' native_skill_activation_evidence = 'unsupported' + native_worker_delegation = 'supported' + delegated_worker_full_capability = 'supported' + delegated_worker_model_lock = 'supported' + delegated_worker_working_directory = 'supported' + delegated_worker_result_capture = 'supported' + delegated_worker_capacity_signal = 'supported' + } + delegation = [ordered]@{ + mode = 'native_worker' + mechanism = 'Copilot CLI native task tool with an explicit full-capability general-purpose child agent; fleet/task lifecycle events observe completion' + worker_role = 'general-purpose' + full_capability = 'supported' + model_lock = 'supported' + working_directory = 'supported' + result_capture = 'supported' + capacity = 'harness_authoritative' + nested_model_execution = $false } supported_telemetry = @('transcript_event_capture', 'token_telemetry', 'cache_token_telemetry', 'tool_call_telemetry', 'command_evidence', 'file_evidence') configuration_profiles = @('isolated-default') @@ -384,6 +401,7 @@ function Get-CopilotPreflight { $checks.Add((New-PreflightCheck -Name 'run_paths' -Status passed -Detail "-C $($run.WorkingDirectoryPath); COPILOT_HOME under $($run.HomeDirectoryPath)")) $checks.Add((New-PreflightCheck -Name 'prompt_fidelity' -Status passed -Detail 'The prepared UTF-8 prompt bytes are supplied once through stdin; the execution fake proves the received bytes match the staged prompt.')) $checks.Add((New-PreflightCheck -Name 'credential_boundary' -Status passed -Detail 'Only supported authentication state is made available to Copilot; --secret-env-vars removes every listed token variable from shell and MCP child environments; no Copilot profile or credential file is copied.')) + $checks.Add((New-PreflightCheck -Name 'native_worker_delegation' -Status passed -Detail 'Use Copilot native task with a full-capability general-purpose child agent; the parent supplies one explicit arm per child and observes child completion/model evidence.')) if ($platform -eq 'macos') { $warnings.Add('macOS sandbox-exec is deprecated by Apple but is used only when present; a future runner revision may replace it with an equivalent supported mechanism.') } @@ -394,7 +412,7 @@ function Get-CopilotPreflight { foreach ($key in $descriptor.Keys) { $descriptorCopy[$key] = $descriptor[$key] } $descriptorCopy.harness = [ordered]@{ name = 'GitHub Copilot CLI'; version = $harnessVersion } $mechanisms = [System.Collections.Generic.List[string]]::new() - foreach ($mechanism in @('copilot --output-format json', 'prompt on stdin', '--allow-all-tools broad tool approval', 'path and URL verification preserved (no --allow-all-paths/--allow-all-urls)', '--no-ask-user', 'repository-owned custom instructions preserved', '--disable-builtin-mcps', '--secret-env-vars shell/MCP child filtering', 'isolated COPILOT_HOME and COPILOT_CACHE_HOME', 'isolated HOME/XDG roots', 'OS-keychain authentication delegated to Copilot', 'GitHub CLI fallback token resolved by the trusted runner when needed', 'no host GH_CONFIG_DIR exposed to the worker', 'no session continuation')) { $mechanisms.Add($mechanism) } + foreach ($mechanism in @('native task -> general-purpose full-capability child worker', 'Copilot fleet/task lifecycle observation', 'copilot --output-format json compatibility transport', 'prompt on stdin', '--allow-all-tools broad tool approval', 'path and URL verification preserved (no --allow-all-paths/--allow-all-urls)', '--no-ask-user', 'repository-owned custom instructions preserved', '--disable-builtin-mcps', '--secret-env-vars shell/MCP child filtering', 'isolated COPILOT_HOME and COPILOT_CACHE_HOME', 'isolated HOME/XDG roots', 'OS-keychain authentication delegated to Copilot', 'GitHub CLI fallback token resolved by the trusted runner when needed', 'no host GH_CONFIG_DIR exposed to the worker', 'no session continuation')) { $mechanisms.Add($mechanism) } if ($hardConfinement) { $mechanisms.Add("external $($sandboxInfo.Source) filesystem sandbox") } else { $mechanisms.Add('pragmatic process/environment isolation without hard filesystem confinement') } return New-PreflightDocument -Descriptor $descriptorCopy -Profile $profile -Run $run -Compatible ($reasons.Count -eq 0) -Checks @($checks) -Mechanisms @($mechanisms) -ResolvedCapabilities $capabilities -Warnings @($warnings) -Reasons @($reasons) } diff --git a/scripts/eval-runners/opencode/runner.ps1 b/scripts/eval-runners/opencode/runner.ps1 index 92adc75..ffba019 100644 --- a/scripts/eval-runners/opencode/runner.ps1 +++ b/scripts/eval-runners/opencode/runner.ps1 @@ -47,6 +47,23 @@ $descriptor = [ordered]@{ cost_telemetry = 'conditional' credential_child_filtering = 'conditional' native_skill_activation_evidence = 'unsupported' + native_worker_delegation = 'supported' + delegated_worker_full_capability = 'supported' + delegated_worker_model_lock = 'supported' + delegated_worker_working_directory = 'supported' + delegated_worker_result_capture = 'supported' + delegated_worker_capacity_signal = 'supported' + } + delegation = [ordered]@{ + mode = 'native_worker' + mechanism = 'OpenCode Task tool invoking the full-capability General subagent in a fresh child context' + worker_role = 'general' + full_capability = 'supported' + model_lock = 'supported' + working_directory = 'supported' + result_capture = 'supported' + capacity = 'harness_authoritative' + nested_model_execution = $false } supported_telemetry = @('transcript_event_capture', 'token_telemetry', 'cache_token_telemetry', 'tool_call_telemetry', 'command_evidence', 'file_evidence', 'cost_telemetry') configuration_profiles = @('isolated-default') @@ -260,6 +277,7 @@ function Get-OpenCodePreflight { $checks.Add((New-PreflightCheck -Name 'fresh_session' -Status passed -Detail 'The adapter starts one new opencode run process and supplies no resume, continue, or session id.')) $checks.Add((New-PreflightCheck -Name 'ambient_configuration' -Status passed -Detail 'The adapter isolates global/user configuration roots and deliberately preserves repository-owned project configuration; OPENCODE_DISABLE_PROJECT_CONFIG is not used.')) $checks.Add((New-PreflightCheck -Name 'prompt_fidelity' -Status passed -Detail 'The exact prompt bytes are sent on stdin as the first and only task input.')) + $checks.Add((New-PreflightCheck -Name 'native_worker_delegation' -Status passed -Detail 'Use OpenCode native Task with the full-capability General subagent; Explore and Scout read-only agents are not valid eval workers.')) $warnings.Add('OpenCode does not expose a supported child-tool environment filter in this CLI contract; the runner removes unrelated inherited variables but cannot independently prove that the selected provider credential is hidden from every OpenCode-launched tool.') $hardConfinement = $null -ne $sandboxInfo -and $platform -in @('linux', 'macos') @@ -272,7 +290,7 @@ function Get-OpenCodePreflight { foreach ($key in $descriptor.Keys) { $descriptorCopy[$key] = $descriptor[$key] } $descriptorCopy.harness = [ordered]@{ name = 'OpenCode CLI'; version = $harnessVersion } $mechanisms = [System.Collections.Generic.List[string]]::new() - foreach ($mechanism in @('opencode run --format json', '--auto', 'isolated OPENCODE_CONFIG_DIR', 'isolated OPENCODE_CONFIG', 'isolated HOME/XDG roots', 'repository-owned project configuration preserved', 'prompt on stdin', 'no session continuation')) { $mechanisms.Add($mechanism) } + foreach ($mechanism in @('native Task -> General full-capability child worker', 'opencode run --format json compatibility transport', '--auto', 'isolated OPENCODE_CONFIG_DIR', 'isolated OPENCODE_CONFIG', 'isolated HOME/XDG roots', 'repository-owned project configuration preserved', 'prompt on stdin', 'no session continuation')) { $mechanisms.Add($mechanism) } if ($hardConfinement) { $mechanisms.Add("external $($sandboxInfo.Source) filesystem sandbox") } else { $mechanisms.Add('pragmatic process/environment isolation without hard filesystem confinement') } return New-PreflightDocument -Descriptor $descriptorCopy -Profile $profile -Run $run -Compatible ($reasons.Count -eq 0) -Checks @($checks) -Mechanisms @($mechanisms) -ResolvedCapabilities $capabilities -Warnings @($warnings) -Reasons @($reasons) } diff --git a/scripts/eval-runners/orchestration.ps1 b/scripts/eval-runners/orchestration.ps1 new file mode 100644 index 0000000..93383e9 --- /dev/null +++ b/scripts/eval-runners/orchestration.ps1 @@ -0,0 +1,342 @@ +Set-StrictMode -Version Latest +$ErrorActionPreference = 'Stop' + +. (Join-Path $PSScriptRoot 'runner-common.ps1') +. (Join-Path $PSScriptRoot 'manifest-paths.ps1') + +function Get-OrchestrationProfileValue { + param( + [Parameter(Mandatory = $true)][object]$Profile, + [Parameter(Mandatory = $true)][string]$Name, + [object]$Default = $null + ) + + if ($Profile -is [System.Collections.IDictionary] -and $Profile.Contains($Name)) { + return $Profile[$Name] + } + if ($Profile.PSObject.Properties.Name -contains $Name) { + return $Profile.$Name + } + return $Default +} + +function New-EvalOrchestrationPlan { + <# + Deterministic parent-side planning only. This function never starts a + harness process, calls a model, reads grading material, or exposes a + paired arm to a worker envelope. + #> + param( + [Parameter(Mandatory = $true)][string]$IterationDirectory, + [Parameter(Mandatory = $true)][object]$Manifest, + [Parameter(Mandatory = $true)][object]$Profile + ) + + $records = @(Get-ManifestRunRecords -IterationDirectory $IterationDirectory -Manifest $Manifest) + $requestedConcurrency = [int](Get-OrchestrationProfileValue -Profile $Profile -Name 'concurrency' -Default 0) + if ($requestedConcurrency -lt 1) { + throw 'execution-profile.json concurrency must be at least 1 for native worker orchestration.' + } + + $runner = [string](Get-OrchestrationProfileValue -Profile $Profile -Name 'runner' -Default '') + $model = [string](Get-OrchestrationProfileValue -Profile $Profile -Name 'model' -Default '') + if ([string]::IsNullOrWhiteSpace($runner) -or [string]::IsNullOrWhiteSpace($model)) { + throw 'Native worker orchestration requires a selected runner and model.' + } + + $arms = [System.Collections.Generic.List[object]]::new() + $seenWorkers = [System.Collections.Generic.HashSet[string]]::new([System.StringComparer]::OrdinalIgnoreCase) + foreach ($record in $records) { + $workerId = ('arm-{0}-{1}' -f $record.EvalId, $record.Configuration) + if (-not $seenWorkers.Add($workerId)) { + throw "Manifest produced duplicate native worker id '$workerId'." + } + + # The parent retains the exact manifest-declared destinations. They are + # deliberately kept outside the worker envelope below. + $arms.Add([ordered]@{ + worker_id = $workerId + eval_id = $record.EvalId + eval_name = $record.EvalName + configuration = $record.Configuration + depends_on = @() + parent_paths = [ordered]@{ + run_manifest = $record.RunManifestRelative + execution_result = $record.ExecutionResultRelative + result = $record.ResultRelative + } + worker = [ordered]@{ + worker_id = $workerId + eval_id = $record.EvalId + eval_name = $record.EvalName + configuration = $record.Configuration + run_manifest = $record.RunManifestRelative + run_manifest_path = $record.RunManifestPath + model = $model + reasoning_effort = Get-OrchestrationProfileValue -Profile $Profile -Name 'reasoning_effort' + configuration_profile = Get-OrchestrationProfileValue -Profile $Profile -Name 'configuration_profile' + tool_profile = Get-OrchestrationProfileValue -Profile $Profile -Name 'tool_profile' + timeout_seconds = [int](Get-OrchestrationProfileValue -Profile $Profile -Name 'timeout_seconds' -Default 0) + one_arm_only = $true + paired_arm_visible = $false + grading_material_visible = $false + parent_executes_arm = $false + runner_execute_invocation = 'forbidden' + nested_model_execution = $false + model_execution_count = 1 + } + }) + } + + $schemas = Get-RunnerSchemaNames + return [ordered]@{ + schema = $schemas.OrchestrationPlan + protocol_version = $schemas.Protocol + runner = $runner + model = $model + requested_concurrency = $requestedConcurrency + native_worker_required = $true + parent_executes_arms = $false + nested_model_execution = $false + dispatch_policy = 'one fresh harness-native worker per arm; independent workers may run concurrently up to requested_concurrency' + capacity_policy = 'harness_authoritative; a rejected delegation that did not start remains queued and is not an eval attempt' + arms = $arms.ToArray() + } +} + +function Get-OrchestrationArmByWorkerId { + param( + [Parameter(Mandatory = $true)][object]$Plan, + [Parameter(Mandatory = $true)][string]$WorkerId + ) + + foreach ($arm in @($Plan.arms)) { + if ([string]$arm.worker_id -eq $WorkerId) { return $arm } + } + throw "Unknown orchestration worker '$WorkerId'." +} + +function New-OrchestrationState { + param([Parameter(Mandatory = $true)][object]$Plan) + + $pending = @($Plan.arms | ForEach-Object { [string]$_.worker_id }) + return [ordered]@{ + schema = 'codebeltnet/agentic/eval-orchestration-state/1' + plan_schema = [string]$Plan.schema + requested_concurrency = [int]$Plan.requested_concurrency + pending_worker_ids = @($pending) + active = [ordered]@{} + completed = [ordered]@{} + delegation_rejections = [ordered]@{} + eval_attempts = [ordered]@{} + max_observed_active = 0 + } +} + +function Get-OrchestrationActiveCount { + param([Parameter(Mandatory = $true)][object]$State) + + return @((Get-OrchestrationDictionary -Object $State -Name 'active').Keys).Count +} + +function Get-OrchestrationDictionary { + param( + [Parameter(Mandatory = $true)][object]$Object, + [Parameter(Mandatory = $true)][string]$Name + ) + + $value = Get-JsonProperty -Object $Object -Name $Name -Default $null + if ($null -eq $value -or -not ($value -is [System.Collections.IDictionary])) { + throw "Orchestration state field '$Name' must be a dictionary." + } + return $value +} + +function Get-NextWorkerDispatches { + <# + Returns dispatch envelopes without consuming pending work. An external + harness decides whether each native delegation request was accepted. That + is what lets a harness-owned capacity limit reject a request without + turning it into an eval attempt. + #> + param( + [Parameter(Mandatory = $true)][object]$Plan, + [Parameter(Mandatory = $true)][object]$State + ) + + $requested = [int]$Plan.requested_concurrency + $active = Get-OrchestrationActiveCount -State $State + $slots = [Math]::Max(0, $requested - $active) + if ($slots -eq 0) { return @() } + + $pending = @($State.pending_worker_ids) + $dispatches = [System.Collections.Generic.List[object]]::new() + foreach ($workerId in $pending | Select-Object -First $slots) { + $arm = Get-OrchestrationArmByWorkerId -Plan $Plan -WorkerId ([string]$workerId) + $dispatches.Add((New-WorkerDispatchEnvelope -Arm $arm)) + } + return @($dispatches) +} + +function Register-DelegationAccepted { + param( + [Parameter(Mandatory = $true)][object]$State, + [Parameter(Mandatory = $true)][string]$WorkerId, + [string]$WorkerSessionId = '' + ) + + $pending = [System.Collections.Generic.List[string]]::new() + foreach ($id in @($State.pending_worker_ids)) { [void]$pending.Add([string]$id) } + if (-not $pending.Contains($WorkerId)) { + throw "Worker '$WorkerId' cannot be accepted because it is not pending." + } + $active = Get-OrchestrationDictionary -Object $State -Name 'active' + $completed = Get-OrchestrationDictionary -Object $State -Name 'completed' + if ($active.Contains($WorkerId) -or $completed.Contains($WorkerId)) { + throw "Worker '$WorkerId' was already accepted or completed." + } + + [void]$pending.Remove($WorkerId) + $State.pending_worker_ids = @($pending) + $attempts = Get-OrchestrationDictionary -Object $State -Name 'eval_attempts' + $attempts[$WorkerId] = 1 + $active[$WorkerId] = [ordered]@{ + worker_id = $WorkerId + worker_session_id = if ([string]::IsNullOrWhiteSpace($WorkerSessionId)) { $null } else { $WorkerSessionId } + accepted_utc = [DateTime]::UtcNow.ToString('o') + attempt_count = 1 + } + $activeCount = Get-OrchestrationActiveCount -State $State + if ($activeCount -gt [int]$State.max_observed_active) { $State.max_observed_active = $activeCount } + return $true +} + +function Register-DelegationRejected { + param( + [Parameter(Mandatory = $true)][object]$State, + [Parameter(Mandatory = $true)][string]$WorkerId, + [Parameter(Mandatory = $true)][string]$Reason + ) + + $pending = @($State.pending_worker_ids) + if ($pending -notcontains $WorkerId) { + throw "Delegation rejection for '$WorkerId' is invalid because the arm is not pending." + } + $rejections = Get-OrchestrationDictionary -Object $State -Name 'delegation_rejections' + $count = if ($rejections.Contains($WorkerId)) { [int]$rejections[$WorkerId].count } else { 0 } + $rejections[$WorkerId] = [ordered]@{ + count = $count + 1 + last_reason = $Reason + last_rejected_utc = [DateTime]::UtcNow.ToString('o') + eval_attempt_started = $false + } + return $true +} + +function Register-WorkerTerminal { + param( + [Parameter(Mandatory = $true)][object]$Plan, + [Parameter(Mandatory = $true)][object]$State, + [Parameter(Mandatory = $true)][string]$WorkerId, + [Parameter(Mandatory = $true)][object]$ExecutionEvidence + ) + + $active = Get-OrchestrationDictionary -Object $State -Name 'active' + if (-not $active.Contains($WorkerId)) { + throw "Worker '$WorkerId' cannot become terminal because it is not active." + } + $status = [string](Get-JsonProperty -Object $ExecutionEvidence -Name 'status' -Default '') + if ($status -notin @('completed', 'failed', 'timed_out', 'cancelled', 'incompatible')) { + throw "Worker '$WorkerId' returned non-terminal status '$status'." + } + $arm = Get-OrchestrationArmByWorkerId -Plan $Plan -WorkerId $WorkerId + $activeWorker = $active[$WorkerId] + $runEvidence = Get-JsonProperty -Object $ExecutionEvidence -Name 'run' -Default $null + if ($null -ne $runEvidence) { + $evidenceEvalId = [int](Get-JsonProperty -Object $runEvidence -Name 'eval_id' -Default 0) + $evidenceConfiguration = [string](Get-JsonProperty -Object $runEvidence -Name 'configuration' -Default '') + if ($evidenceEvalId -ne [int]$arm.eval_id -or $evidenceConfiguration -ne [string]$arm.configuration) { + throw "Worker '$WorkerId' returned evidence for a different arm." + } + } + + $active.Remove($WorkerId) + $completed = Get-OrchestrationDictionary -Object $State -Name 'completed' + $completed[$WorkerId] = [ordered]@{ + worker_id = $WorkerId + eval_id = [int]$arm.eval_id + eval_name = [string]$arm.eval_name + configuration = [string]$arm.configuration + status = $status + terminal_utc = [DateTime]::UtcNow.ToString('o') + worker_session_id = Get-JsonProperty -Object $activeWorker -Name 'worker_session_id' -Default $null + } + return $true +} + +function New-WorkerDispatchEnvelope { + param([Parameter(Mandatory = $true)][object]$Arm) + + $worker = $Arm.worker + return [ordered]@{ + type = 'eval_worker_dispatch' + worker_id = [string]$worker.worker_id + arm = [ordered]@{ + eval_id = [int]$worker.eval_id + eval_name = [string]$worker.eval_name + configuration = [string]$worker.configuration + run_manifest = [string]$worker.run_manifest + run_manifest_path = [string]$worker.run_manifest_path + } + requested = [ordered]@{ + model = [string]$worker.model + reasoning_effort = $worker.reasoning_effort + configuration_profile = $worker.configuration_profile + tool_profile = $worker.tool_profile + timeout_seconds = [int]$worker.timeout_seconds + } + worker_contract = [ordered]@{ + one_arm_only = $true + paired_arm_visible = $false + grading_material_visible = $false + parent_executes_arm = $false + runner_execute_invocation = 'forbidden' + nested_model_execution = $false + model_execution_count = 1 + fresh_worker_required = $true + } + } +} + +function Assert-OrchestrationPlanContract { + param([Parameter(Mandatory = $true)][object]$Plan) + + if ([string]$Plan.schema -ne (Get-RunnerSchemaNames).OrchestrationPlan) { + throw 'Orchestration plan has an unsupported schema.' + } + if (-not [bool]$Plan.native_worker_required -or [bool]$Plan.parent_executes_arms -or [bool]$Plan.nested_model_execution) { + throw 'Orchestration plan must require native workers and forbid parent or nested model execution.' + } + $workerIds = [System.Collections.Generic.HashSet[string]]::new([System.StringComparer]::OrdinalIgnoreCase) + foreach ($arm in @($Plan.arms)) { + if (-not $workerIds.Add([string]$arm.worker_id)) { throw "Orchestration plan duplicates worker '$($arm.worker_id)'." } + if (@($arm.depends_on).Count -ne 0) { throw "Worker '$($arm.worker_id)' has an unrelated dependency." } + $worker = $arm.worker + foreach ($property in @('one_arm_only', 'paired_arm_visible', 'grading_material_visible', 'parent_executes_arm', 'nested_model_execution', 'model_execution_count')) { + if (-not (Test-JsonProperty -Object $worker -Name $property)) { throw "Worker '$($arm.worker_id)' is missing '$property'." } + } + if (-not [bool]$worker.one_arm_only -or [bool]$worker.paired_arm_visible -or [bool]$worker.grading_material_visible -or [bool]$worker.parent_executes_arm -or [bool]$worker.nested_model_execution -or [int]$worker.model_execution_count -ne 1) { + throw "Worker '$($arm.worker_id)' violates the one-arm/one-model contract." + } + if ([string]$worker.worker_id -ne [string]$arm.worker_id -or [int]$worker.eval_id -ne [int]$arm.eval_id -or [string]$worker.configuration -ne [string]$arm.configuration) { + throw "Worker '$($arm.worker_id)' does not identify exactly its manifest arm." + } + if ([string]$worker.runner_execute_invocation -ne 'forbidden') { throw "Worker '$($arm.worker_id)' may not invoke the direct runner execute transport." } + foreach ($forbiddenProperty in @('paired_arm', 'grading', 'expected_output', 'assertions', 'eval_metadata', 'execution_result', 'result')) { + if (Test-JsonProperty -Object $worker -Name $forbiddenProperty) { + throw "Worker '$($arm.worker_id)' exposes forbidden parent or grading field '$forbiddenProperty'." + } + } + } + return $true +} diff --git a/scripts/eval-runners/runner-common.ps1 b/scripts/eval-runners/runner-common.ps1 index 98d38ad..d191d6d 100644 --- a/scripts/eval-runners/runner-common.ps1 +++ b/scripts/eval-runners/runner-common.ps1 @@ -10,6 +10,7 @@ function Get-RunnerSchemaNames { Result = 'codebeltnet/agentic/eval-execution-result/1' PortableResult = 'codebeltnet/agentic/eval-result/2' Run = 'codebeltnet/agentic/eval-run/1' + OrchestrationPlan = 'codebeltnet/agentic/eval-orchestration-plan/1' } } @@ -279,6 +280,58 @@ function Get-IsolationCapabilityAssessment { } } +function Get-DelegationCapabilityAssessment { + param( + [Parameter(Mandatory = $true)][object]$Descriptor, + [System.Collections.IDictionary]$Capabilities + ) + + $required = @( + 'native_worker_delegation', + 'delegated_worker_full_capability', + 'delegated_worker_model_lock', + 'delegated_worker_working_directory', + 'delegated_worker_result_capture', + 'delegated_worker_capacity_signal' + ) + $unproven = [System.Collections.Generic.List[string]]::new() + foreach ($name in $required) { + $value = if ($null -ne $Capabilities -and $Capabilities.Contains($name)) { [string]$Capabilities[$name] } else { 'unavailable' } + if ($value -ne 'supported') { $unproven.Add($name) } + } + + $delegation = Get-JsonProperty -Object $Descriptor -Name 'delegation' -Default $null + $mode = [string](Get-JsonProperty -Object $delegation -Name 'mode' -Default 'unsupported') + $nestedModelExecution = [bool](Get-JsonProperty -Object $delegation -Name 'nested_model_execution' -Default $true) + $mechanism = [string](Get-JsonProperty -Object $delegation -Name 'mechanism' -Default '') + $workerRole = [string](Get-JsonProperty -Object $delegation -Name 'worker_role' -Default '') + $modeIsNative = $mode -eq 'native_worker' + if (-not $modeIsNative) { $unproven.Add('delegation.mode') } + if ($nestedModelExecution) { $unproven.Add('delegation.nested_model_execution') } + $delegationFields = [ordered]@{ + full_capability = 'delegated_worker_full_capability' + model_lock = 'delegated_worker_model_lock' + working_directory = 'delegated_worker_working_directory' + result_capture = 'delegated_worker_result_capture' + capacity = 'delegated_worker_capacity_signal' + } + foreach ($field in $delegationFields.Keys) { + $value = [string](Get-JsonProperty -Object $delegation -Name $field -Default 'unsupported') + $valid = if ($field -eq 'capacity') { $value -in @('supported', 'harness_authoritative') } else { $value -eq 'supported' } + if (-not $valid) { $unproven.Add("delegation.$field") } + } + + return [pscustomobject]@{ + MandatoryProven = $unproven.Count -eq 0 + Mode = $mode + Mechanism = $mechanism + WorkerRole = $workerRole + NestedModelExecution = $nestedModelExecution + Unproven = @($unproven) + Required = @($required) + } +} + function Resolve-RunContract { param([Parameter(Mandatory = $true)][string]$RunPath) @@ -422,7 +475,7 @@ function Assert-RunnerDescriptor { if ([string]$Descriptor.protocol_version -ne $schemas.Protocol) { throw "Runner descriptor protocol_version must be '$($schemas.Protocol)'." } - foreach ($field in @('name', 'version', 'platforms', 'harness', 'capabilities', 'configuration_profiles', 'tool_profiles')) { + foreach ($field in @('name', 'version', 'platforms', 'harness', 'capabilities', 'delegation', 'configuration_profiles', 'tool_profiles')) { if (-not (Test-JsonProperty -Object $Descriptor -Name $field)) { throw "Runner descriptor is missing '$field'." } @@ -458,6 +511,37 @@ function Assert-RunnerDescriptor { } } + $delegationRequired = @( + 'native_worker_delegation', + 'delegated_worker_full_capability', + 'delegated_worker_model_lock', + 'delegated_worker_working_directory', + 'delegated_worker_result_capture', + 'delegated_worker_capacity_signal' + ) + foreach ($name in $delegationRequired) { + if (-not (Test-JsonProperty -Object $Descriptor.capabilities -Name $name)) { + throw "Runner descriptor is missing required delegation capability '$name'." + } + } + $delegation = $Descriptor.delegation + foreach ($field in @('mode', 'mechanism', 'worker_role', 'full_capability', 'model_lock', 'working_directory', 'result_capture', 'capacity', 'nested_model_execution')) { + if (-not (Test-JsonProperty -Object $delegation -Name $field)) { + throw "Runner descriptor delegation is missing '$field'." + } + } + if ([string]$delegation.mode -notin @('native_worker', 'conditional', 'unsupported')) { + throw "Runner delegation mode '$($delegation.mode)' is unsupported." + } + foreach ($field in @('full_capability', 'model_lock', 'working_directory', 'result_capture', 'capacity')) { + if ([string]$delegation.$field -notin @('supported', 'conditional', 'unsupported', 'harness_authoritative')) { + throw "Runner delegation '$field' must be supported, conditional, unsupported, or harness_authoritative." + } + } + if ([bool]$delegation.nested_model_execution) { + throw 'Runner delegation must not describe nested model execution.' + } + return $true } @@ -487,6 +571,7 @@ function New-PreflightDocument { $schemas = Get-RunnerSchemaNames $capabilitiesForAssessment = if ($null -eq $ResolvedCapabilities) { [ordered]@{} } else { $ResolvedCapabilities } $assessment = Get-IsolationCapabilityAssessment -Capabilities $capabilitiesForAssessment + $delegationAssessment = Get-DelegationCapabilityAssessment -Descriptor $Descriptor -Capabilities $capabilitiesForAssessment $effectiveCompatible = $Compatible -and $assessment.MandatoryProven $unprovenControls = [string[]]$assessment.Unproven if (-not $effectiveCompatible) { $unprovenControls = [string[]](@($assessment.Unproven) + @('preflight')) } @@ -506,6 +591,15 @@ function New-PreflightDocument { } checks = @($Checks) resolved_capabilities = if ($null -eq $ResolvedCapabilities) { [ordered]@{} } else { $ResolvedCapabilities } + delegation = [ordered]@{ + status = if ($delegationAssessment.MandatoryProven) { 'supported' } elseif ($delegationAssessment.Mode -eq 'conditional') { 'conditional' } else { 'unsupported' } + mode = $delegationAssessment.Mode + mechanism = $delegationAssessment.Mechanism + worker_role = $delegationAssessment.WorkerRole + nested_model_execution = $delegationAssessment.NestedModelExecution + required_controls = @($delegationAssessment.Required) + unproven_controls = [string[]]$delegationAssessment.Unproven + } isolation = [ordered]@{ level = if ($effectiveCompatible) { $assessment.Level } else { 'unsupported' } status = if ($effectiveCompatible) { 'verified' } else { 'unverified' } @@ -518,6 +612,45 @@ function New-PreflightDocument { } } +function Assert-NativeWorkerDelegation { + param( + [Parameter(Mandatory = $true)][object]$Descriptor, + [Parameter(Mandatory = $true)][object]$Preflight + ) + + $delegation = Get-JsonProperty -Object $Descriptor -Name 'delegation' -Default $null + $mode = [string](Get-JsonProperty -Object $delegation -Name 'mode' -Default 'unsupported') + if ($mode -ne 'native_worker') { + throw "Runner '$($Descriptor.name)' cannot satisfy the mandatory native Eval Worker contract: delegation mode is '$mode'." + } + if ([bool](Get-JsonProperty -Object $delegation -Name 'nested_model_execution' -Default $true)) { + throw "Runner '$($Descriptor.name)' describes nested model execution; one eval arm must have exactly one model-backed worker." + } + if ([string](Get-JsonProperty -Object $Preflight -Name 'status' -Default 'incompatible') -ne 'compatible') { + throw "Runner '$($Descriptor.name)' preflight is incompatible; the orchestrator must not execute an arm in the parent context." + } + $delegationPreflight = Get-JsonProperty -Object $Preflight -Name 'delegation' -Default $null + if ([string](Get-JsonProperty -Object $delegationPreflight -Name 'status' -Default 'unsupported') -ne 'supported') { + $unproven = @((Get-JsonProperty -Object $delegationPreflight -Name 'unproven_controls' -Default @())) + throw "Runner '$($Descriptor.name)' native worker delegation is not proven during preflight: $([string]::Join(', ', $unproven))." + } + $capabilities = Get-JsonProperty -Object $Preflight -Name 'resolved_capabilities' -Default $null + $required = @( + 'native_worker_delegation', + 'delegated_worker_full_capability', + 'delegated_worker_model_lock', + 'delegated_worker_working_directory', + 'delegated_worker_result_capture', + 'delegated_worker_capacity_signal' + ) + foreach ($name in $required) { + if ([string](Get-JsonProperty -Object $capabilities -Name $name -Default 'unsupported') -ne 'supported') { + throw "Runner '$($Descriptor.name)' native worker capability '$name' is not supported; the orchestrator must not fall back to parent execution." + } + } + return $true +} + function New-UnavailableMetric { param([Parameter(Mandatory = $true)][string]$Reason) diff --git a/scripts/eval-runners/tests/test-orchestration.ps1 b/scripts/eval-runners/tests/test-orchestration.ps1 new file mode 100644 index 0000000..e483a6e --- /dev/null +++ b/scripts/eval-runners/tests/test-orchestration.ps1 @@ -0,0 +1,207 @@ +<# +.SYNOPSIS + Deterministic native-worker orchestration contract tests. + +.DESCRIPTION + Exercises only the manifest queue/state machinery. The fake capacity + harness below never starts a process or contacts a model. +#> +[CmdletBinding()] +param() + +$ErrorActionPreference = 'Stop' +Set-StrictMode -Version Latest + +$runnerRoot = (Resolve-Path (Join-Path $PSScriptRoot '..')).Path +. (Join-Path $runnerRoot 'runner-common.ps1') +. (Join-Path $runnerRoot 'orchestration.ps1') + +function Assert-True { + param([bool]$Condition, [string]$Message) + if (-not $Condition) { throw "ASSERT: $Message" } +} + +function Assert-Equal { + param([object]$Expected, [object]$Actual, [string]$Message) + if ([string]$Expected -ne [string]$Actual) { + throw "ASSERT: $Message (expected '$Expected', got '$Actual')" + } +} + +function Write-TestJson { + param([string]$Path, [object]$Value) + New-Item -ItemType Directory -Path (Split-Path -Parent $Path) -Force | Out-Null + [System.IO.File]::WriteAllText($Path, (($Value | ConvertTo-Json -Depth 100) + [Environment]::NewLine), [System.Text.UTF8Encoding]::new($false)) +} + +$testRoot = Join-Path ([System.IO.Path]::GetTempPath()) ('agentic-orchestration-' + [Guid]::NewGuid().ToString('N')) +try { + $iteration = Join-Path $testRoot 'iteration-1' + New-Item -ItemType Directory -Path $iteration -Force | Out-Null + + $manifestEvals = [System.Collections.Generic.List[object]]::new() + for ($evalId = 1; $evalId -le 8; $evalId++) { + $evalName = 'eval-{0:d2}' -f $evalId + $evalDirectory = Join-Path $iteration $evalName + New-Item -ItemType Directory -Path $evalDirectory -Force | Out-Null + Write-TestJson -Path (Join-Path $evalDirectory 'eval-metadata.json') -Value ([ordered]@{ + eval_id = $evalId + eval_name = $evalName + assertions = @('assertion') + }) + + $runs = [ordered]@{} + foreach ($configuration in @('with_skill', 'without_skill')) { + $runDirectory = Join-Path $evalDirectory $configuration + New-Item -ItemType Directory -Path $runDirectory -Force | Out-Null + Write-TestJson -Path (Join-Path $runDirectory 'run.json') -Value ([ordered]@{ + evalId = $evalId + evalName = $evalName + mode = $configuration + }) + $resultFileName = if ($configuration -eq 'with_skill') { 'with-skill.result.json' } else { 'without-skill.result.json' } + Write-TestJson -Path (Join-Path $evalDirectory (Join-Path 'results' $resultFileName)) -Value ([ordered]@{ + eval_id = $evalId + configuration = $configuration + execution_status = 'unrun' + grading = @() + }) + $executionFileName = if ($configuration -eq 'with_skill') { 'with-skill.execution-result.json' } else { 'without-skill.execution-result.json' } + $runs[$configuration] = [ordered]@{ + mode = $configuration + run_manifest = "$evalName/$configuration/run.json" + execution_result = "$evalName/results/$executionFileName" + result = "$evalName/results/$resultFileName" + } + } + $manifestEvals.Add([ordered]@{ + eval_id = $evalId + eval_name = $evalName + directory = $evalName + metadata = "$evalName/eval-metadata.json" + runs = $runs + }) + } + + $manifest = [ordered]@{ + schema = 'codebeltnet/agentic/eval-package/2' + configurations = @('with_skill', 'without_skill') + evals = @($manifestEvals) + } + $profile = [ordered]@{ + schema = (Get-RunnerSchemaNames).Profile + runner = 'fake' + model = 'fixture-model' + reasoning_effort = $null + configuration_profile = 'isolated-default' + tool_profile = 'default' + timeout_seconds = 60 + concurrency = 16 + } + + $plan = New-EvalOrchestrationPlan -IterationDirectory $iteration -Manifest $manifest -Profile $profile + [void](Assert-OrchestrationPlanContract -Plan $plan) + $serializedPlan = $plan | ConvertTo-Json -Depth 100 | ConvertFrom-Json + [void](Assert-OrchestrationPlanContract -Plan $serializedPlan) + Assert-Equal 16 @($plan.arms).Count '8 eval cases fan out to 16 independent arms' + Assert-Equal 16 $plan.requested_concurrency 'requested concurrency is preserved in the plan' + Assert-True ([bool]$plan.native_worker_required) 'native worker delegation is mandatory' + Assert-True (-not [bool]$plan.parent_executes_arms) 'the parent is forbidden from executing arms' + Assert-True (-not [bool]$plan.nested_model_execution) 'the plan forbids a nested model layer' + Assert-Equal 16 (@($plan.arms | ForEach-Object { [string]$_.worker_id } | Sort-Object -Unique).Count) 'every arm has a distinct worker identity' + Assert-Equal 16 (@($plan.arms | Where-Object { @($_.depends_on).Count -eq 0 }).Count) 'unrelated arms have no sequential dependencies' + + $initialState = New-OrchestrationState -Plan $plan + $initialDispatches = @(Get-NextWorkerDispatches -Plan $plan -State $initialState) + Assert-Equal 16 $initialDispatches.Count 'requested concurrency exposes all 16 pending arms' + foreach ($dispatch in $initialDispatches) { + Assert-True ([bool]$dispatch.worker_contract.one_arm_only) "$($dispatch.worker_id) receives one arm" + Assert-True (-not [bool]$dispatch.worker_contract.paired_arm_visible) "$($dispatch.worker_id) cannot see its paired arm" + Assert-True (-not [bool]$dispatch.worker_contract.grading_material_visible) "$($dispatch.worker_id) cannot see grading material" + Assert-True (-not [bool]$dispatch.worker_contract.parent_executes_arm) "$($dispatch.worker_id) cannot execute in the parent" + Assert-Equal 'forbidden' $dispatch.worker_contract.runner_execute_invocation "$($dispatch.worker_id) cannot invoke direct runner execute" + Assert-True (-not [bool]$dispatch.worker_contract.nested_model_execution) "$($dispatch.worker_id) has no nested model layer" + Assert-Equal 1 $dispatch.worker_contract.model_execution_count "$($dispatch.worker_id) has one model execution" + Assert-True ($dispatch.PSObject.Properties.Name -notcontains 'paired_arm') "$($dispatch.worker_id) has no paired-arm payload" + Assert-True ($dispatch.PSObject.Properties.Name -notcontains 'grading') "$($dispatch.worker_id) has no grading payload" + Assert-True ($dispatch.PSObject.Properties.Name -notcontains 'expected_output') "$($dispatch.worker_id) has no expected-output payload" + } + + # A fake harness accepts only four simultaneous native workers. This limit + # belongs to the fake harness, not to the portable plan or queue. + $capacityState = New-OrchestrationState -Plan $plan + $startedWorkers = [System.Collections.Generic.List[string]]::new() + $rejectedWorkers = [System.Collections.Generic.List[string]]::new() + $attemptsAtRejection = @{} + $maxObservedByHarness = 0 + $firstDispatches = @(Get-NextWorkerDispatches -Plan $plan -State $capacityState) + foreach ($dispatch in $firstDispatches) { + if ((Get-OrchestrationActiveCount -State $capacityState) -lt 4) { + [void](Register-DelegationAccepted -State $capacityState -WorkerId $dispatch.worker_id -WorkerSessionId ('session-' + $dispatch.worker_id)) + $startedWorkers.Add([string]$dispatch.worker_id) + } else { + [void](Register-DelegationRejected -State $capacityState -WorkerId $dispatch.worker_id -Reason 'fake harness capacity is four') + $rejectedWorkers.Add([string]$dispatch.worker_id) + $attemptsAtRejection[[string]$dispatch.worker_id] = $capacityState.eval_attempts.Contains([string]$dispatch.worker_id) + } + $maxObservedByHarness = [Math]::Max($maxObservedByHarness, (Get-OrchestrationActiveCount -State $capacityState)) + } + Assert-Equal 4 $startedWorkers.Count 'the fake harness starts four workers before rejecting capacity overflow' + Assert-True ($rejectedWorkers.Count -gt 0) 'capacity overflow creates queued rejections' + Assert-True (@($attemptsAtRejection.Values | Where-Object { $_ }).Count -eq 0) 'a rejected delegation is not an eval attempt' + Assert-Equal 16 $capacityState.requested_concurrency 'capacity handling does not lower portable requested concurrency' + + while (@($capacityState.completed.Keys).Count -lt @($plan.arms).Count) { + $dispatches = @(Get-NextWorkerDispatches -Plan $plan -State $capacityState) + foreach ($dispatch in $dispatches) { + if ((Get-OrchestrationActiveCount -State $capacityState) -lt 4) { + [void](Register-DelegationAccepted -State $capacityState -WorkerId $dispatch.worker_id -WorkerSessionId ('session-' + $dispatch.worker_id)) + if ($startedWorkers -notcontains [string]$dispatch.worker_id) { $startedWorkers.Add([string]$dispatch.worker_id) } + } else { + [void](Register-DelegationRejected -State $capacityState -WorkerId $dispatch.worker_id -Reason 'fake harness capacity is four') + } + $maxObservedByHarness = [Math]::Max($maxObservedByHarness, (Get-OrchestrationActiveCount -State $capacityState)) + } + + $activeIds = @($capacityState.active.Keys) + if ($activeIds.Count -gt 0) { + $workerId = [string]$activeIds[0] + $arm = Get-OrchestrationArmByWorkerId -Plan $plan -WorkerId $workerId + [void](Register-WorkerTerminal -Plan $plan -State $capacityState -WorkerId $workerId -ExecutionEvidence ([ordered]@{ + status = 'completed' + run = [ordered]@{ eval_id = [int]$arm.eval_id; configuration = [string]$arm.configuration } + })) + } elseif (@($capacityState.pending_worker_ids).Count -gt 0) { + throw 'capacity queue deadlocked with pending workers and no active worker.' + } + } + Assert-Equal 16 @($capacityState.completed.Keys).Count 'all 16 arms eventually become terminal' + Assert-Equal 16 $startedWorkers.Count 'all 16 arms start exactly once after capacity is released' + Assert-Equal 4 $capacityState.max_observed_active 'state records the fake harness maximum of four active workers' + Assert-Equal 4 $maxObservedByHarness 'the fake harness never exceeds four active workers' + Assert-Equal 0 @($capacityState.pending_worker_ids).Count 'no arm remains queued after capacity is released' + + $badDescriptor = [pscustomobject]@{ + name = 'fake-without-delegation' + delegation = [ordered]@{ mode = 'conditional'; nested_model_execution = $false } + } + $badPreflight = [ordered]@{ + status = 'compatible' + delegation = [ordered]@{ status = 'conditional'; unproven_controls = @('native_worker_delegation') } + resolved_capabilities = [ordered]@{} + } + $parentDispatches = 0 + $fallbackRejected = $false + try { + [void](Assert-NativeWorkerDelegation -Descriptor $badDescriptor -Preflight $badPreflight) + $parentDispatches++ + } catch { + $fallbackRejected = $true + } + Assert-True $fallbackRejected 'missing native delegation fails preflight' + Assert-Equal 0 $parentDispatches 'failed delegation preflight never invokes parent fallback' + + Write-Output 'Native worker orchestration: PASS' +} finally { + if (Test-Path -LiteralPath $testRoot) { Remove-Item -LiteralPath $testRoot -Recurse -Force } +} diff --git a/scripts/eval-runners/tests/test-runner-conformance.ps1 b/scripts/eval-runners/tests/test-runner-conformance.ps1 index 9596d76..56f50e1 100644 --- a/scripts/eval-runners/tests/test-runner-conformance.ps1 +++ b/scripts/eval-runners/tests/test-runner-conformance.ps1 @@ -61,7 +61,6 @@ try { [System.IO.File]::WriteAllText((Join-Path $with.Root 'repo\opencode.json'), '{"fixture_project_config":true}', [System.Text.UTF8Encoding]::new($false)) [System.IO.File]::WriteAllText((Join-Path $without.Root 'repo\opencode.json'), '{"fixture_project_config":true}', [System.Text.UTF8Encoding]::new($false)) $fakeCli = @' -[CmdletBinding()] param([Parameter(ValueFromRemainingArguments = $true)][string[]]$RemainingArguments) $harness = [System.IO.Path]::GetFileNameWithoutExtension($MyInvocation.MyCommand.Path) $logPath = Join-Path (Get-Location).Path ("{0}-fake-cli-log.jsonl" -f $harness) @@ -113,6 +112,25 @@ $record = [ordered]@{ ambient_copilot_instructions_visible = if ([string]::IsNullOrWhiteSpace($copilotHome)) { $false } else { Test-Path -LiteralPath (Join-Path $copilotHome 'copilot-instructions.md') -PathType Leaf } secret_env_vars_arg = @($arguments | Where-Object { $_ -like '--secret-env-vars=*' }) } +if ($harness -eq 'codex' -and $arguments -contains 'app-server' -and $arguments -contains 'generate-json-schema') { + $outArgument = @($arguments | Where-Object { $_ -like '--out=*' } | Select-Object -First 1) + if ($outArgument.Count -eq 0) { exit 2 } + $schemaDirectory = [IO.Path]::GetFullPath((Join-Path (Get-Location).Path ([string]$outArgument[0].Substring(6)))) + New-Item -ItemType Directory -Path $schemaDirectory -Force | Out-Null + [IO.File]::WriteAllText((Join-Path $schemaDirectory 'ClientRequest.json'), '{"thread/start":"ThreadStartParams","turn/start":"TurnStartParams","cwd":true,"model":true,"ephemeral":true}', [Text.UTF8Encoding]::new($false)) + [IO.File]::AppendAllText($logPath, (($record | ConvertTo-Json -Compress) + [Environment]::NewLine), [Text.UTF8Encoding]::new($false)) + exit 0 +} +if ($harness -eq 'codex' -and $arguments -contains 'app-server' -and $arguments -contains '--help') { + [IO.File]::AppendAllText($logPath, (($record | ConvertTo-Json -Compress) + [Environment]::NewLine), [Text.UTF8Encoding]::new($false)) + Write-Output 'generate-json-schema' + exit 0 +} +if ($harness -eq 'codex' -and $arguments -contains 'features' -and $arguments -contains 'list') { + [IO.File]::AppendAllText($logPath, (($record | ConvertTo-Json -Compress) + [Environment]::NewLine), [Text.UTF8Encoding]::new($false)) + Write-Output 'multi_agent stable true' + exit 0 +} if ($arguments -contains '--version') { $version = switch ($harness) { 'codex' { 'recorded-codex 9.1' } 'opencode' { 'recorded-opencode 9.2' } 'copilot' { 'GitHub Copilot CLI recorded-1.0.80' } default { 'recorded-cline 9.3' } } [IO.File]::AppendAllText($logPath, (($record | ConvertTo-Json -Compress) + [Environment]::NewLine), [Text.UTF8Encoding]::new($false)) @@ -294,6 +312,9 @@ exit 2 $runnerPath = Join-Path $runnerRoot "$runnerDir\runner.ps1" $description = Invoke-AdapterJson -RunnerPath $runnerPath -Command describe -RunPath $with.Path -ProfilePath $recordedProfiles[$runnerName] [void](Assert-RunnerDescriptor -Descriptor $description) + Assert-True ($description.PSObject.Properties.Name -contains 'delegation') "$runnerName descriptor declares native delegation" + Assert-True (-not [bool]$description.delegation.nested_model_execution) "$runnerName descriptor forbids nested model execution" + Assert-True (-not [string]::IsNullOrWhiteSpace([string]$description.delegation.mechanism)) "$runnerName descriptor records its native delegation mechanism" $expectedVersion = switch ($runnerName) { 'codex' { 'recorded-codex 9.1' } 'opencode' { 'recorded-opencode 9.2' } 'copilot' { 'GitHub Copilot CLI recorded-1.0.80' } default { 'recorded-cline 9.3' } } Assert-Equal $expectedVersion $description.harness.version "$runnerName exact describe version" $preflightWith = Invoke-AdapterJson -RunnerPath $runnerPath -Command preflight -RunPath $with.Path -ProfilePath $recordedProfiles[$runnerName] @@ -302,6 +323,12 @@ exit 2 Assert-Equal 'compatible' $preflightWithout.status "$runnerName without_skill pragmatic preflight" Assert-Equal $expectedVersion $preflightWith.harness.version "$runnerName exact preflight version" Assert-Equal 'pragmatic' $preflightWith.isolation.level "$runnerName pragmatic preflight level" + if ($runnerName -eq 'cline') { + Assert-True ($preflightWith.delegation.status -ne 'supported') 'Cline preflight does not claim unavailable Agent Squad delegation' + Assert-True (([string]::Join(' ', @($preflightWith.warnings))) -match 'use_subagents' -and ([string]::Join(' ', @($preflightWith.warnings))) -match 'Agent Squad') 'Cline preflight rejects read-only subagents as a mutable-arm fallback' + } else { + Assert-Equal 'supported' $preflightWith.delegation.status "$runnerName native delegation preflight" + } if ($runnerName -eq 'copilot') { Assert-True (@($preflightWith.checks | Where-Object { $_.name -eq 'authentication' -and $_.status -eq 'passed' }).Count -eq 1) 'Copilot preflight accepts explicit environment authentication' Assert-True (@($preflightWith.mechanisms | Where-Object { $_ -eq '--allow-all-tools broad tool approval' }).Count -eq 1) 'Copilot preflight describes --allow-all-tools as broad tool approval' @@ -836,6 +863,12 @@ try { Assert-True ($prepareText.Contains('bridge-manifest-results.ps1')) 'handoff preparation must use the deterministic package-level manifest bridge' Assert-True ($prepareText.Contains('runs..run_manifest') -and $prepareText.Contains('runs..execution_result') -and $prepareText.Contains('runs..result')) 'handoff preparation must require every exact manifest arm path' Assert-True ($prepareText.Contains('Do not derive, normalize, rename, hyphenate, underscore, or otherwise reconstruct any run, execution-result, or result path.')) 'handoff preparation must prohibit reconstructed paths' + Assert-True ($prepareText.Contains('DELEGATE EVERY eval arm to a fresh harness-native worker/subagent. The Eval Orchestrator MUST NOT execute an eval arm itself.')) 'handoff preparation must require delegated native workers and forbid parent execution' + Assert-True ($prepareText.Contains('One arm equals one delegated worker and one model-backed eval execution.')) 'handoff preparation must state the one-arm one-model invariant' + Assert-True ($prepareText.Contains('Require terminal evidence for the exact selected model, working directory, isolated home, and fresh session')) 'handoff preparation must require worker control evidence' + Assert-True ($prepareText.Contains('min(execution-profile.json.concurrency, remaining arms)')) 'handoff preparation must state requested concurrency fan-out' + Assert-True ($prepareText.Contains('rejected before the worker starts') -and $prepareText.Contains('record no eval attempt')) 'handoff preparation must queue capacity rejections without counting attempts' + Assert-True ($prepareText.Contains('orchestration.ps1')) 'handoff preparation must load the deterministic orchestration helper' Assert-True ($prepareText -notmatch '') 'handoff preparation must not expose an unconstrained result-file placeholder' Assert-True ($reportText -notmatch 'function Get-ResultPath') 'reporting must not contain a configuration-derived result path helper' Assert-True ($manifestBridgeText.Contains('Get-ManifestRunRecords') -and $manifestBridgeText.Contains('$record.ResultPath')) 'package-level bridge must resolve exact manifest records' diff --git a/scripts/prepare-skill-evals.ps1 b/scripts/prepare-skill-evals.ps1 index 1c389d0..31308fc 100644 --- a/scripts/prepare-skill-evals.ps1 +++ b/scripts/prepare-skill-evals.ps1 @@ -70,7 +70,7 @@ Per-arm runner timeout. Defaults to 900 seconds. .PARAMETER Concurrency - Requested external-orchestrator concurrency. Defaults to 1. It does not change paired-arm semantics. + Requested external-orchestrator concurrency. Defaults to 16. It does not change paired-arm semantics. .PARAMETER Changed Prepares a package for every repo-managed skill this branch changed, including uncommitted work. This is the form @@ -1593,9 +1593,11 @@ function Invoke-PrepareMode { Write-Host 'Point the harness at that path. Do not reproduce its contents in chat: a pasted copy' Write-Host 'loses the absolute paths it depends on, and the harness then cannot find the package.' Write-Host '' - Write-Host 'The runner-aware handoff uses execution-profile.json and the package-local Eval Runner protocol.' - Write-Host 'The selected external orchestrator must preflight and execute one runner process per blind arm,' - Write-Host 'then bridge, grade, and report the collected results.' + Write-Host 'The runner-aware handoff uses execution-profile.json, the package-local Eval Runner protocol,' + Write-Host 'and its deterministic native-worker orchestration plan.' + Write-Host 'The selected external orchestrator must delegate EVERY arm to a fresh harness-native worker.' + Write-Host 'The orchestrator coordinates; it must not execute an eval arm itself or invoke the direct execute transport.' + Write-Host 'Independent workers run concurrently up to execution-profile.json.concurrency; harness capacity remains authoritative.' Write-Host '' Write-Host 'This script prepared prompts only. It did not run them, and nothing here will.' Write-Host 'The selected evaluator should finish the package in one run. If it cannot write back to this package,' @@ -1613,6 +1615,7 @@ function New-RunnerPrompt { $builder = [System.Text.StringBuilder]::new() $profilePath = Join-Path $IterationDirectory 'execution-profile.json' $resolverPath = Join-Path $IterationDirectory "$evalRunnerToolRelativePath/resolve-runner.ps1" + $orchestrationPath = Join-Path $IterationDirectory "$evalRunnerToolRelativePath/orchestration.ps1" $manifestBridgePath = Join-Path $IterationDirectory "$evalRunnerToolRelativePath/bridge-manifest-results.ps1" $reportPath = Join-Path $IterationDirectory $reportToolRelativePath [void]$builder.AppendLine('# Execute, grade, and report this evaluation package') @@ -1627,29 +1630,37 @@ function New-RunnerPrompt { [void]$builder.AppendLine() [void]$builder.AppendLine('A human selected the external orchestrator and authorized this handoff. Repository preparation, validation, CI, hooks, and automatic completion gates remain model-free. Do not substitute a generic worker, another runner, or an improvised isolation scheme if the selected runner is unavailable or incompatible.') [void]$builder.AppendLine() - [void]$builder.AppendLine('## Phase 1: execute blind arms') + [void]$builder.AppendLine('## Phase 1: delegate blind arms') + [void]$builder.AppendLine() + [void]$builder.AppendLine('> DELEGATE EVERY eval arm to a fresh harness-native worker/subagent. The Eval Orchestrator MUST NOT execute an eval arm itself.') + [void]$builder.AppendLine() + [void]$builder.AppendLine('> One arm equals one delegated worker and one model-backed eval execution. The delegated worker is the eval worker; do not place another model-backed runner process inside it.') + [void]$builder.AppendLine() + [void]$builder.AppendLine('> Run independent workers concurrently up to `execution-profile.json.concurrency`. If the harness temporarily refuses another worker because its own concurrency limit is reached, keep that arm queued and dispatch it when capacity becomes available.') [void]$builder.AppendLine() [void]$builder.AppendLine('1. Read `manifest.json` and `execution-profile.json`. If `runner` or `model` is null, unavailable, or unsupported, fail clearly and list the supported package-local runner IDs; do not guess a default. The profile contains no credentials.') - [void]$builder.AppendLine('2. Resolve the selected package-local runner with the resolver. Ask it for `describe` and validate its protocol, descriptor, and capability declarations before running an arm. Do not invent harness-specific CLI commands.') - [void]$builder.AppendLine('3. For every eval arm, read the exact `run_manifest`, `execution_result`, and `result` fields from `manifest.json` at `runs..run_manifest`, `runs..execution_result`, and `runs..result`. Bind them without editing the strings: `runPath = exact run_manifest`, `executionPath = exact execution_result`, and `resultPath = exact result`.') - [void]$builder.AppendLine('4. Keep `with_skill` and `without_skill` in fresh independent processes/sessions. Use the same runner-native model selector, configuration, tools, and limits. The runner must send each `prompt.md` unchanged as the first task input and must enforce the run contract, including the baseline skill exclusion and the staged filesystem/workspace boundary.') - [void]$builder.AppendLine('5. Save the runner''s single normalized JSON response unchanged to that arm''s exact `executionPath`. Do not derive, normalize, rename, hyphenate, underscore, or otherwise reconstruct any run, execution-result, or result path. A runner execute operation writes only to the exact manifest-declared `execution_result` path. Preserve the complete final response, status, telemetry, evidence references, hashes, isolation mechanisms, warnings, and compatibility deviations. Do not retry for answer quality. A refusal is a result; timeout, harness failure, and incompatibility are results.') - [void]$builder.AppendLine('6. If the runner cannot satisfy a required guarantee, keep the normalized status `incompatible` and stop that arm. Never fall back to the old generic isolated-worker behavior and never substitute a different runner.') + [void]$builder.AppendLine(('2. Resolve the selected package-local runner with the resolver. Ask it for `describe` and `preflight`, validate its protocol, descriptor, and native-delegation capability declarations, and load the deterministic orchestration helper at ' + $orchestrationPath + '. Do not invent harness-specific CLI commands.')) + [void]$builder.AppendLine('3. Build the pending arm queue from `manifest.json` with the orchestration helper. For every arm, read the exact `run_manifest`, `execution_result`, and `result` fields from `runs..run_manifest`, `runs..execution_result`, and `runs..result`. Retain those exact manifest-declared strings without editing them: the parent owns those exact destinations; the worker receives only its own prepared arm contract. Do not derive, normalize, rename, hyphenate, underscore, or otherwise reconstruct any run, execution-result, or result path.') + [void]$builder.AppendLine('4. Before dispatching, require native worker delegation and all mandatory isolation controls. A conditional or unavailable delegation mechanism is an incompatible preflight. Do not continue by invoking a runner process in the parent, and do not silently serialize arms in the parent.') + [void]$builder.AppendLine('5. Dispatch each pending arm to one fresh harness-native full-capability worker. The worker must execute the prepared `prompt.md` as its first task from that arm''s staged run directory, with the selected model/configuration, exact working directory and isolated home. Require terminal evidence for the exact selected model, working directory, isolated home, and fresh session; if the native surface cannot lock or prove one of these, fail preflight rather than accepting a fallback. It must not receive its paired arm, `eval-metadata.json`, expected output, assertions, grading, benchmark/report data, or any result from another arm.') + [void]$builder.AppendLine('6. The delegated worker is the only model-backed execution for that arm. It must not invoke `runner.ps1 execute`, a second harness CLI, another model agent, or a nested session. Return the worker transcript/terminal evidence and normalized execution result to the parent; the parent writes it to the exact manifest-declared `execution_result` path without reconstructing any path.') + [void]$builder.AppendLine('7. Maintain up to `min(execution-profile.json.concurrency, remaining arms)` active delegated workers. If a delegation request is rejected before the worker starts because of harness capacity, leave that arm pending, record no eval attempt, and retry it after an active worker becomes terminal. Do not add a runner-specific ceiling or change the portable requested concurrency.') + [void]$builder.AppendLine('8. Preserve the complete terminal response, status, telemetry, evidence references, hashes, isolation mechanisms, warnings, and compatibility deviations. Do not retry for answer quality. A refusal is a result; timeout, harness failure, and incompatibility are results.') [void]$builder.AppendLine() [void]$builder.AppendLine('The package-local process surface is:') [void]$builder.AppendLine('```text') [void]$builder.AppendLine("pwsh -NoProfile -File `"$resolverPath`" ") [void]$builder.AppendLine('runner.ps1 describe') [void]$builder.AppendLine("runner.ps1 preflight -Run `"`" -Profile `"$profilePath`"") - [void]$builder.AppendLine("runner.ps1 execute -Run `"`" -Profile `"$profilePath`"") + [void]$builder.AppendLine("runner.ps1 execute -Run `"`" -Profile `"$profilePath`" # direct one-arm compatibility transport; forbidden to the parent orchestrator") [void]$builder.AppendLine('```') - [void]$builder.AppendLine('Use the resolver output to locate `runner.ps1`; `` is data from the profile, not a branch in this orchestration contract. For each arm, invoke the runner exactly once with `execute`, and keep its stdout as one JSON `execution-result.json` document.') + [void]$builder.AppendLine('Use the resolver output to locate `runner.ps1`; `` is data from the profile, not a branch in this orchestration contract. The parent may use `describe` and `preflight` only. The `execute` command remains part of the one-arm runner protocol for compatibility and conformance, but the native delegated worker path MUST NOT invoke it: doing so would create a second model-backed execution.') [void]$builder.AppendLine() - [void]$builder.AppendLine('Do not read any `eval-metadata.json`, expected output, assertions, result grading, or paired output during Phase 1. The runner must never receive or inspect expected output, assertions, grading, paired output, benchmark data, or human feedback. Those files remain outside every run directory and are the grading key.') + [void]$builder.AppendLine('Do not read any `eval-metadata.json`, expected output, assertions, result grading, benchmark/report data, or paired output during Phase 1. The orchestrator and worker must never expose those materials before all workers are terminal. They remain outside every run directory and are the grading key.') [void]$builder.AppendLine() [void]$builder.AppendLine('## Phase 2: bridge, grade, and report') [void]$builder.AppendLine() - [void]$builder.AppendLine('1. After every available arm has completed or failed, invoke the deterministic package bridge below. It reads `manifest.json`, obtains each arm''s exact `run_manifest`, `execution_result`, and `result` paths. The bridge checks prompt/run/profile hashes and artifact confinement, validates the manifest paths, rejects unreferenced hyphen/underscore shadow results, and invokes the existing one-arm bridge with those exact paths. Do not manually construct a bridge command for an arm.') + [void]$builder.AppendLine('1. Only after every available delegated worker is terminal, invoke the deterministic package bridge below. It reads `manifest.json`, obtains each arm''s exact `run_manifest`, `execution_result`, and `result` paths. The bridge checks prompt/run/profile hashes and artifact confinement, validates the manifest paths, rejects unreferenced hyphen/underscore shadow results, and invokes the existing one-arm bridge with those exact paths. Do not manually construct a bridge command for an arm.') [void]$builder.AppendLine((' `pwsh -NoProfile -File "' + $manifestBridgePath + '" -IterationDirectory "' + $IterationDirectory + '" -RequireComplete`')) [void]$builder.AppendLine(' The bridge''s one-arm operation is conceptually `-Run runPath -ExecutionResult executionPath -Result resultPath`, where all three values are the exact strings read from `manifest.json`. Do not derive, normalize, rename, hyphenate, underscore, or otherwise reconstruct any of them.') [void]$builder.AppendLine('2. Only if the package bridge succeeds, read each eval''s `eval-metadata.json` and reveal `expected_output` and `assertions` to the Grader. Follow `tools/skill-creator/agents/grader.md`; grade deterministically first, then use optional model judgement only where deterministic evidence cannot decide. Never infer tool or file behavior from model self-report without process evidence.') @@ -1673,7 +1684,7 @@ function New-PackageReadme { $builder = [System.Text.StringBuilder]::new() [void]$builder.AppendLine("# Eval package: $SkillName (iteration $IterationNumber)") [void]$builder.AppendLine() - [void]$builder.AppendLine('Prepared by `scripts/prepare-skill-evals.ps1` in `codebeltnet/agentic`. Nothing in this package was executed. `execution-profile.json` selects the user-chosen Eval Runner, runner-native model, and limits; the external Eval Orchestrator runs both configurations, grades them, and generates the report.') + [void]$builder.AppendLine('Prepared by `scripts/prepare-skill-evals.ps1` in `codebeltnet/agentic`. Nothing in this package was executed. `execution-profile.json` selects the user-chosen Eval Runner, runner-native model, and limits; the external Eval Orchestrator delegates every arm to a fresh harness-native Eval Worker, then grades and generates the report.') [void]$builder.AppendLine() [void]$builder.AppendLine('## What is here') [void]$builder.AppendLine() @@ -1682,7 +1693,19 @@ function New-PackageReadme { } [void]$builder.AppendLine() [void]$builder.AppendLine('Each eval directory holds the grading key (`eval-metadata.json`), result stubs under `results/`, and two isolated run directories: `with_skill/` and `without_skill/`. A run directory holds `prompt.md`, a `run.json` contract, a `repo/` working tree materialized from the fixtures, an isolated `home/`, and - for `with_skill` only - a `skill/` directory with the candidate skill. The grading key and results sit outside both run directories, so a worker that stays within its run directory is never handed them.') - [void]$builder.AppendLine('The package root also holds `execution-profile.json`, the package-local Eval Runner protocol under `tools/eval-runners/`, and raw `execution-result.json` paths beside the existing result stubs. `run.json` defines what one blind arm must execute; the profile defines with what runner/model/configuration; the selected runner defines how.') + [void]$builder.AppendLine('The package root also holds `execution-profile.json`, the package-local Eval Runner protocol and deterministic native-worker queue under `tools/eval-runners/`, and raw `execution-result.json` paths beside the existing result stubs. `run.json` defines what one blind arm must execute; the profile defines with what runner/model/configuration; the selected runner defines how its native worker is created.') + [void]$builder.AppendLine() + [void]$builder.AppendLine('## Orchestration topology') + [void]$builder.AppendLine() + [void]$builder.AppendLine('```text') + [void]$builder.AppendLine('Eval Orchestrator') + [void]$builder.AppendLine(' |') + [void]$builder.AppendLine(' +-- Eval Worker -> one eval arm') + [void]$builder.AppendLine(' +-- Eval Worker -> one eval arm') + [void]$builder.AppendLine(' +-- Eval Worker -> one eval arm') + [void]$builder.AppendLine(' +-- ...') + [void]$builder.AppendLine('```') + [void]$builder.AppendLine('The Eval Orchestrator coordinates and collects; it never executes an eval arm itself. One arm equals one delegated worker and one model-backed eval execution. Independent workers run concurrently up to `execution-profile.json.concurrency`; harness capacity is authoritative, so a rejected delegation stays queued and is not an attempt.') [void]$builder.AppendLine('The package also carries the exact Anthropic skill-creator assets used after execution under `tools/skill-creator`: `tools/skill-creator/agents/grader.md`, `tools/skill-creator/agents/comparator.md`, `tools/skill-creator/agents/analyzer.md`, `tools/skill-creator/references/schemas.md`, `tools/skill-creator/scripts/aggregate_benchmark.py`, and `tools/skill-creator/eval-viewer/generate_review.py` plus `tools/skill-creator/eval-viewer/viewer.html`.') [void]$builder.AppendLine() [void]$builder.AppendLine('## Isolation model') @@ -1693,11 +1716,12 @@ function New-PackageReadme { [void]$builder.AppendLine() [void]$builder.AppendLine('## How to run') [void]$builder.AppendLine() - [void]$builder.AppendLine('1. Read `execution-profile.json`. If `runner` or `model` is missing, fail clearly instead of guessing. Resolve the selected package-local runner and run `describe`, then `preflight`, for every arm.') - [void]$builder.AppendLine('2. Invoke `execute` exactly once for each `run.json`, preserving its one JSON `execution-result.json` unchanged. The runner must provide a fresh process/session, isolated home/config, isolated CWD, baseline skill exclusion, prompt fidelity, model/configuration lock, and complete response capture, or return `incompatible`. Hard filesystem confinement is not one of these mandatory controls: when the runner proves it the run reports strict isolation, and when it does not the run reports pragmatic isolation.') - [void]$builder.AppendLine('3. After all arms complete or fail, run `tools/eval-runners/bridge-manifest-results.ps1 -IterationDirectory -RequireComplete`. It reads the manifest-declared `run_manifest`, `execution_result`, and `result` paths for every arm and invokes the one-arm bridge with those exact paths. Only then read the grading key, grade with `tools/skill-creator/agents/grader.md`, and run `tools/generate-eval-report.ps1 -RequireComplete`.') + [void]$builder.AppendLine('1. Read `execution-profile.json`. If `runner` or `model` is missing, fail clearly instead of guessing. Resolve the selected package-local runner and run `describe`, then `preflight`, before any native worker is dispatched. Require the descriptor''s native delegation capability; do not use a parent sequential fallback.') + [void]$builder.AppendLine('2. Use the package-local orchestration helper to queue one worker per manifest arm. Delegate every arm to a fresh full-capability harness-native worker. Do not invoke the runner''s direct `execute` command from the parent or from the delegated worker, because it would add a second model execution. Each worker receives one arm only and no grading material or paired-arm data.') + [void]$builder.AppendLine('3. Maintain up to the requested concurrency. If the harness refuses a new worker because its own capacity is full, leave that arm queued and dispatch it when capacity is released; do not hardcode a runner-specific maximum and do not count the rejection as an attempt.') + [void]$builder.AppendLine('4. After all delegated workers complete or fail, run `tools/eval-runners/bridge-manifest-results.ps1 -IterationDirectory -RequireComplete`. It reads the manifest-declared `run_manifest`, `execution_result`, and `result` paths for every arm and invokes the one-arm bridge with those exact paths. Only then read the grading key, grade with `tools/skill-creator/agents/grader.md`, and run `tools/generate-eval-report.ps1 -RequireComplete`.') [void]$builder.AppendLine() - [void]$builder.AppendLine('`RUN-THIS.prompt.md` is the external Eval Orchestrator handoff. It selects the package-local runner from the profile, invokes the common protocol once per blind arm, bridges raw evidence into the existing result shape, reveals grading material only after execution, and invokes Anthropic skill-creator''s compatible aggregator and static viewer through the package adapter. It never executes an eval prompt in its own context.') + [void]$builder.AppendLine('`RUN-THIS.prompt.md` is the external Eval Orchestrator handoff. It selects the package-local runner from the profile, delegates one native Eval Worker per blind arm, queues capacity rejections, bridges raw evidence into the existing result shape, reveals grading material only after execution, and invokes Anthropic skill-creator''s compatible aggregator and static viewer through the package adapter. It never executes an eval prompt in its own context.') [void]$builder.AppendLine() [void]$builder.AppendLine('A harness that cannot provide fresh, independent sessions with isolated working and config roots is incompatible with these evals. `-CollectResults` may inspect and report available package state, including missing or unrun arms, but it exits non-zero when the required completion gate is not satisfied. An incomplete or unrun package must not be presented as a successfully completed evaluation.') [void]$builder.AppendLine() From 53aec32cea7837425b143233adb96d04dce2893a Mon Sep 17 00:00:00 2001 From: "aicia[bot]" Date: Mon, 24 Aug 2026 14:13:37 +0200 Subject: [PATCH 22/39] =?UTF-8?q?=F0=9F=91=B7=20refine=20native=20worker?= =?UTF-8?q?=20orchestration=20and=20terminal=20evidence=20validation?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Adds native worker terminal evidence requirements and validation logic. Enhances delegation capability assessment with status tracking (supported/conditional/unsupported). Implements terminal evidence collection in orchestration tests including session tracking, model observation, directory isolation verification, and prompt fidelity checks. Updates result bridging to validate native worker evidence when required. Expands runner descriptor validation and adds comprehensive terminal evidence audit trail. --- scripts/eval-runners/README.md | 54 +++-- .../eval-runners/bridge-execution-result.ps1 | 7 +- .../eval-runners/bridge-manifest-results.ps1 | 21 +- scripts/eval-runners/cline/runner.ps1 | 7 +- scripts/eval-runners/codex/runner.ps1 | 30 ++- .../contracts/execution-result.schema.json | 42 +++- .../contracts/preflight-result.schema.json | 6 +- scripts/eval-runners/fake/runner.ps1 | 30 +-- .../eval-runners/github-copilot/runner.ps1 | 26 +- scripts/eval-runners/opencode/runner.ps1 | 25 +- scripts/eval-runners/orchestration.ps1 | 50 +++- scripts/eval-runners/runner-common.ps1 | 227 +++++++++++++++++- .../eval-runners/tests/test-orchestration.ps1 | 184 +++++++++++++- .../tests/test-runner-conformance.ps1 | 14 +- scripts/prepare-skill-evals.ps1 | 29 ++- 15 files changed, 647 insertions(+), 105 deletions(-) diff --git a/scripts/eval-runners/README.md b/scripts/eval-runners/README.md index 507851b..4b8a2a2 100644 --- a/scripts/eval-runners/README.md +++ b/scripts/eval-runners/README.md @@ -26,39 +26,59 @@ arms dependency-free, exposes at most the requested `execution-profile.json.concurrency` active slots, and leaves a capacity rejection pending without incrementing the eval attempt count. It contains no harness-specific concurrency ceiling. `Assert-NativeWorkerDelegation` is the -fail-closed preflight gate: a conditional or unavailable native mechanism -cannot fall back to parent execution. +fail-closed handoff gate: an unavailable/unsupported native mechanism cannot +fall back to parent execution, while a conditional mechanism may dispatch only +when terminal evidence will be checked. + +The delegation contract has three distinct evidence levels: + +```text +descriptor advertised harness capability +preflight locally observable readiness +terminal evidence proof for the actual delegated Eval Worker +``` + +Descriptor fields describe a possible native mechanism; they do not prove an +individual child. Preflight may prove that the installed API, plugin, or CLI +surface is present, but child-specific model, cwd, HOME/config, fresh-session, +prompt, exclusion, and result facts remain `conditional` until terminal +evidence arrives. A worker is accepted only when `evidence.delegation` proves +the requested model, exact arm identity, exact run working directory, exact +isolated home/config boundary, prompt hash/fidelity, terminal capture, paired +arm/grading exclusion, fresh worker/session identity, and exactly one model +execution. Missing or mismatched evidence makes the arm `incompatible`; it is +never a reason to invoke the parent or the compatibility `runner.ps1 execute` +transport. The descriptor's `delegation` object records the native mechanism, worker role, -full-capability and model-lock guarantees, working-directory/result capture, -harness-authoritative capacity, and the invariant +advertised full-capability/model-lock/working-directory/result-capture +properties, harness-authoritative capacity, and the invariant `nested_model_execution = false`. The direct `execute` process surface remains for compatibility and deterministic conformance; the external orchestrator must use only `describe`, `preflight`, and the harness-native delegation -surface for the actual eval arms. It must reject a worker whose terminal -evidence does not prove the selected model, exact run working directory, -isolated home, and fresh session; a native surface that cannot prove those -controls is incompatible rather than a reason to fall back to the parent. +surface for the actual eval arms. Native delegation mechanisms: - GitHub Copilot: the CLI's native `task` tool with an explicit full-capability `general-purpose` child agent for each arm. Fleet/subagent lifecycle is harness-owned; Codebelt supplies the already-known one-arm - decomposition and observes child completion/model evidence. + decomposition and requires terminal child evidence. The direct CLI + `-C`/`--model`/HOME compatibility transport does not prove the child. - Codex: the installed CLI's native app-server child-session surface, `thread/start` followed by `turn/start`, with the arm's `cwd`, selected - model, and ephemeral/fresh session settings. Do not wrap a native Codex - child in another `codex exec` invocation. + model, and ephemeral/fresh session settings. The schema/feature probe is + preflight readiness only; terminal evidence must prove the actual child. + Do not wrap a native Codex child in another `codex exec` invocation. - OpenCode: the native Task tool with the full-capability built-in `General` - subagent. `Explore`/`Scout` read-only agents are not valid for a mutable - eval arm. + subagent. Task/General availability is preflight readiness only; + `Explore`/`Scout` read-only agents are not valid for a mutable eval arm. - Cline: a full-capability Cline SDK Agent Squad `start_subagent(preset: "anvil")` child session backed by `ClineCore.create`. The plugin's default `phantom` preset and Cline's documented `use_subagents` research feature are read-only and are rejected - for mutable evals. The descriptor remains conditional until that - plugin/SDK mechanism is installed and preflight can prove it. + for mutable evals. Plugin/SDK discovery is only preflight readiness; the + actual child remains conditional until terminal evidence proves its controls. This directory contains the package-local implementation of the v0.9.1 Eval Runner protocol. It is copied into prepared packages so the external Eval @@ -107,7 +127,9 @@ proven reports `strict` isolation when hard confinement is proven and boundary, prompt fidelity, result capture, or other mandatory control remains incompatible. -The fake runner is deterministic and is the conformance reference. GitHub +The fake runner is deterministic and is the conformance reference. It has no +harness-native delegation surface and its compatibility output is never proof +for a real harness. GitHub Copilot, Codex, OpenCode, and Cline are thin harness-specific adapters. Their native CLI flags, environment setup, event parsing, authentication injection, and isolation checks stay inside their own directories. Windows is supported in diff --git a/scripts/eval-runners/bridge-execution-result.ps1 b/scripts/eval-runners/bridge-execution-result.ps1 index 51dfc43..8323d0d 100644 --- a/scripts/eval-runners/bridge-execution-result.ps1 +++ b/scripts/eval-runners/bridge-execution-result.ps1 @@ -16,7 +16,9 @@ param( [string]$ExecutionResult, [Parameter(Mandatory = $true)] - [string]$Result + [string]$Result, + + [switch]$RequireNativeDelegation ) $ErrorActionPreference = 'Stop' @@ -137,6 +139,9 @@ try { if ([string]$raw.input.profile_sha256 -ne $profile.Hash) { throw 'execution-result input.profile_sha256 does not match execution-profile.json.' } + if ($RequireNativeDelegation -and [string]$raw.status -ne 'incompatible') { + [void](Assert-NativeWorkerTerminalEvidence -ExecutionEvidence $raw -Run $runData -RequestedModel ([string]$profile.Profile.Model)) + } if ([int]$raw.run.eval_id -ne $runData.EvalId -or [string]$raw.run.configuration -ne $runData.Mode) { throw 'execution-result run identity does not match run.json.' } diff --git a/scripts/eval-runners/bridge-manifest-results.ps1 b/scripts/eval-runners/bridge-manifest-results.ps1 index b096d78..80508fe 100644 --- a/scripts/eval-runners/bridge-manifest-results.ps1 +++ b/scripts/eval-runners/bridge-manifest-results.ps1 @@ -13,7 +13,9 @@ param( [Parameter(Mandatory = $true)] [string]$IterationDirectory, - [switch]$RequireComplete + [switch]$RequireComplete, + + [switch]$RequireNativeDelegation ) $ErrorActionPreference = 'Stop' @@ -56,7 +58,19 @@ try { $existingResult = Read-RunnerJson -Path $record.ResultPath $rawResult = Read-RunnerJson -Path $record.ExecutionResultPath $expectedExecutionFile = [System.IO.Path]::GetRelativePath($record.EvalDirectory, $record.ExecutionResultPath).Replace('\', '/') - $alreadyBridged = @('completed', 'failed', 'timed_out', 'cancelled', 'incompatible') -contains [string](Get-JsonProperty -Object $rawResult -Name 'status' -Default '') -and + $nativeEvidenceReady = -not $RequireNativeDelegation -or [string](Get-JsonProperty -Object $rawResult -Name 'status' -Default '') -eq 'incompatible' + if ($RequireNativeDelegation -and -not $nativeEvidenceReady) { + try { + $runData = Resolve-RunContract -RunPath $record.RunManifestPath + $profileData = Resolve-ExecutionProfile -ProfilePath (Join-Path $iterationPath 'execution-profile.json') + [void](Assert-NativeWorkerTerminalEvidence -ExecutionEvidence $rawResult -Run $runData -RequestedModel ([string]$profileData.Profile.Model)) + $nativeEvidenceReady = $true + } catch { + $nativeEvidenceReady = $false + } + } + $alreadyBridged = $nativeEvidenceReady -and + @('completed', 'failed', 'timed_out', 'cancelled', 'incompatible') -contains [string](Get-JsonProperty -Object $rawResult -Name 'status' -Default '') -and [string](Get-JsonProperty -Object $existingResult -Name 'execution_status' -Default '') -eq [string](Get-JsonProperty -Object $rawResult -Name 'status' -Default '') -and [string](Get-JsonProperty -Object $existingResult -Name 'execution_result_file' -Default '') -eq $expectedExecutionFile -and -not [string]::IsNullOrWhiteSpace([string](Get-JsonProperty -Object $existingResult -Name 'execution_run_id' -Default '')) @@ -72,7 +86,8 @@ try { $bridgeOutput = & pwsh -NoProfile -File $oneArmBridge ` -Run $record.RunManifestPath ` -ExecutionResult $record.ExecutionResultPath ` - -Result $record.ResultPath 2>&1 + -Result $record.ResultPath ` + -RequireNativeDelegation:$RequireNativeDelegation 2>&1 if ($LASTEXITCODE -ne 0) { throw "$($record.EvalName)/$($record.Configuration) bridge failed for manifest paths run='$($record.RunManifestRelative)', execution='$($record.ExecutionResultRelative)', result='$($record.ResultRelative)': $([string]::Join(' ', @($bridgeOutput)))" } diff --git a/scripts/eval-runners/cline/runner.ps1 b/scripts/eval-runners/cline/runner.ps1 index cdaef43..c516ac8 100644 --- a/scripts/eval-runners/cline/runner.ps1 +++ b/scripts/eval-runners/cline/runner.ps1 @@ -286,7 +286,9 @@ function Get-ClineCapabilityMap { $capabilities['filesystem_confinement'] = if ($HardFilesystemConfinement) { 'supported' } else { 'unsupported' } $capabilities['candidate_skill_exposure'] = if ($Inputs.Run.CandidateSkillExposed) { 'supported' } else { 'excluded' } foreach ($name in @('native_worker_delegation', 'delegated_worker_full_capability', 'delegated_worker_model_lock', 'delegated_worker_working_directory', 'delegated_worker_result_capture', 'delegated_worker_capacity_signal')) { - $capabilities[$name] = if ($NativeWorkerAvailable) { 'supported' } else { 'unsupported' } + # Agent Squad package discovery proves local readiness only. It does + # not prove what the child session actually used or returned. + $capabilities[$name] = if ($NativeWorkerAvailable) { 'conditional' } else { 'unsupported' } } return $capabilities } @@ -323,7 +325,8 @@ function Get-ClinePreflight { if ($profile.ToolProfile -ne 'default') { $reasons.Add("tool_profile '$($profile.ToolProfile)' is unsupported by cline.") } if ($agentsSquad.Available) { - $checks.Add((New-PreflightCheck -Name 'native_worker_delegation' -Status passed -Detail $agentsSquad.Detail)) + $checks.Add((New-PreflightCheck -Name 'native_worker_delegation' -Status passed -Detail ($agentsSquad.Detail + ' This proves plugin/API readiness only; the actual child remains conditional until terminal evidence.'))) + $warnings.Add('Cline Agent Squad native-worker controls remain conditional until terminal evidence proves the actual child model, cwd, HOME/config, fresh identity, prompt, exclusions, and terminal capture.') } else { $checks.Add((New-PreflightCheck -Name 'native_worker_delegation' -Status unavailable -Detail $agentsSquad.Detail)) $warnings.Add('Cline native worker delegation is unavailable. The external orchestrator must not use the read-only use_subagents feature or the direct CLI execute transport as a fallback; install the full-capability Agent Squad plugin and set CLINE_AGENTS_SQUAD_PLUGIN.') diff --git a/scripts/eval-runners/codex/runner.ps1 b/scripts/eval-runners/codex/runner.ps1 index 26393e3..da4a205 100644 --- a/scripts/eval-runners/codex/runner.ps1 +++ b/scripts/eval-runners/codex/runner.ps1 @@ -46,21 +46,24 @@ $descriptor = [ordered]@{ cost_telemetry = 'conditional' credential_child_filtering = 'supported' native_skill_activation_evidence = 'unsupported' - native_worker_delegation = 'supported' - delegated_worker_full_capability = 'supported' - delegated_worker_model_lock = 'supported' - delegated_worker_working_directory = 'supported' - delegated_worker_result_capture = 'supported' - delegated_worker_capacity_signal = 'supported' + # The app-server schema proves that a native child surface exists, not + # what the child actually resolved or inherited. Terminal evidence is + # required for every delegated-worker control. + native_worker_delegation = 'conditional' + delegated_worker_full_capability = 'conditional' + delegated_worker_model_lock = 'conditional' + delegated_worker_working_directory = 'conditional' + delegated_worker_result_capture = 'conditional' + delegated_worker_capacity_signal = 'conditional' } delegation = [ordered]@{ mode = 'native_worker' mechanism = 'Codex app-server native child session via thread/start and turn/start with per-worker cwd, model, and ephemeral context' worker_role = 'native-codex-child-session' - full_capability = 'supported' - model_lock = 'supported' - working_directory = 'supported' - result_capture = 'supported' + full_capability = 'conditional' + model_lock = 'conditional' + working_directory = 'conditional' + result_capture = 'conditional' capacity = 'harness_authoritative' nested_model_execution = $false } @@ -227,7 +230,9 @@ function Get-CodexCapabilityMap { $capabilities['filesystem_confinement'] = if ($HardFilesystemConfinement) { 'supported' } else { 'unsupported' } $capabilities['candidate_skill_exposure'] = if ($Inputs.Run.CandidateSkillExposed) { 'supported' } else { 'excluded' } foreach ($name in @('native_worker_delegation', 'delegated_worker_full_capability', 'delegated_worker_model_lock', 'delegated_worker_working_directory', 'delegated_worker_result_capture', 'delegated_worker_capacity_signal')) { - $capabilities[$name] = if ($NativeWorkerAvailable) { 'supported' } else { 'unsupported' } + # The app-server probe proves that the native surface is available; + # only the child terminal evidence can prove the selected controls. + $capabilities[$name] = if ($NativeWorkerAvailable) { 'conditional' } else { 'unsupported' } } return $capabilities } @@ -312,7 +317,8 @@ function Get-CodexPreflight { } $nativeWorkerObservation = Get-CodexNativeWorkerProbe -CommandInfo $commandInfo -Inputs $Inputs if ($nativeWorkerObservation.Available) { - $checks.Add((New-PreflightCheck -Name 'native_worker_delegation' -Status passed -Detail $nativeWorkerObservation.Detail)) + $checks.Add((New-PreflightCheck -Name 'native_worker_delegation' -Status passed -Detail ($nativeWorkerObservation.Detail + ' This proves API readiness only; the actual child remains conditional until terminal evidence.'))) + $warnings.Add('Codex app-server native-worker controls remain conditional until terminal evidence proves the actual child model, cwd, HOME/config, fresh identity, prompt, exclusions, and terminal capture.') } else { $checks.Add((New-PreflightCheck -Name 'native_worker_delegation' -Status unavailable -Detail $nativeWorkerObservation.Detail)) $reasons.Add($nativeWorkerObservation.Detail) diff --git a/scripts/eval-runners/contracts/execution-result.schema.json b/scripts/eval-runners/contracts/execution-result.schema.json index 161607e..9689ad8 100644 --- a/scripts/eval-runners/contracts/execution-result.schema.json +++ b/scripts/eval-runners/contracts/execution-result.schema.json @@ -115,7 +115,47 @@ } }, "telemetry": { "type": "object" }, - "evidence": { "type": "object" }, + "evidence": { + "type": "object", + "properties": { + "delegation": { + "type": "object", + "additionalProperties": true, + "required": [ + "mechanism", + "worker_session_id", + "observed_model", + "observed_working_directory", + "observed_home", + "fresh_worker", + "home_config_isolated", + "prompt_fidelity", + "prompt_sha256", + "terminal_result_capture", + "paired_arm_visible", + "grading_material_visible", + "nested_model_execution", + "model_execution_count" + ], + "properties": { + "mechanism": { "type": "string", "minLength": 1 }, + "worker_session_id": { "type": "string", "minLength": 1 }, + "observed_model": { "type": "string", "minLength": 1 }, + "observed_working_directory": { "type": "string", "minLength": 1 }, + "observed_home": { "type": "string", "minLength": 1 }, + "fresh_worker": { "const": true }, + "home_config_isolated": { "const": true }, + "prompt_fidelity": { "const": true }, + "prompt_sha256": { "type": "string", "pattern": "^[0-9a-fA-F]{64}$" }, + "terminal_result_capture": { "const": true }, + "paired_arm_visible": { "const": false }, + "grading_material_visible": { "const": false }, + "nested_model_execution": { "const": false }, + "model_execution_count": { "const": 1 } + } + } + } + }, "artifacts": { "type": "array" }, "warnings": { "type": "array", "items": { "type": "string" } }, "compatibility_deviations": { "type": "array", "items": { "type": "string" } }, diff --git a/scripts/eval-runners/contracts/preflight-result.schema.json b/scripts/eval-runners/contracts/preflight-result.schema.json index 07f8603..c4ae2ae 100644 --- a/scripts/eval-runners/contracts/preflight-result.schema.json +++ b/scripts/eval-runners/contracts/preflight-result.schema.json @@ -54,7 +54,8 @@ "worker_role", "nested_model_execution", "required_controls", - "unproven_controls" + "unproven_controls", + "terminal_evidence_required" ], "properties": { "status": { "enum": ["supported", "conditional", "unsupported"] }, @@ -63,7 +64,8 @@ "worker_role": { "type": "string" }, "nested_model_execution": { "type": "boolean" }, "required_controls": { "type": "array", "items": { "type": "string" } }, - "unproven_controls": { "type": "array", "items": { "type": "string" } } + "unproven_controls": { "type": "array", "items": { "type": "string" } }, + "terminal_evidence_required": { "type": "boolean" } } }, "isolation": { diff --git a/scripts/eval-runners/fake/runner.ps1 b/scripts/eval-runners/fake/runner.ps1 index 697a359..4eb4de2 100644 --- a/scripts/eval-runners/fake/runner.ps1 +++ b/scripts/eval-runners/fake/runner.ps1 @@ -49,22 +49,24 @@ $descriptor = [ordered]@{ file_evidence = 'supported' cost_telemetry = 'unsupported' native_skill_activation_evidence = 'unsupported' - native_worker_delegation = 'supported' - delegated_worker_full_capability = 'supported' - delegated_worker_model_lock = 'supported' - delegated_worker_working_directory = 'supported' - delegated_worker_result_capture = 'supported' - delegated_worker_capacity_signal = 'supported' + # This runner is a deterministic compatibility fixture. It has no + # harness-native worker surface and cannot prove a delegated child. + native_worker_delegation = 'unsupported' + delegated_worker_full_capability = 'unsupported' + delegated_worker_model_lock = 'unsupported' + delegated_worker_working_directory = 'unsupported' + delegated_worker_result_capture = 'unsupported' + delegated_worker_capacity_signal = 'unsupported' } delegation = [ordered]@{ - mode = 'native_worker' - mechanism = 'deterministic fake native-worker fixture with one arm per worker' - worker_role = 'fake-eval-worker' - full_capability = 'supported' - model_lock = 'supported' - working_directory = 'supported' - result_capture = 'supported' - capacity = 'harness_authoritative' + mode = 'unsupported' + mechanism = 'deterministic compatibility execute fixture; no harness-native worker surface' + worker_role = 'compatibility-fixture' + full_capability = 'unsupported' + model_lock = 'unsupported' + working_directory = 'unsupported' + result_capture = 'unsupported' + capacity = 'unsupported' nested_model_execution = $false } supported_telemetry = @('transcript_event_capture', 'tool_call_telemetry', 'command_evidence', 'file_evidence') diff --git a/scripts/eval-runners/github-copilot/runner.ps1 b/scripts/eval-runners/github-copilot/runner.ps1 index 9fd18a6..b1856a4 100644 --- a/scripts/eval-runners/github-copilot/runner.ps1 +++ b/scripts/eval-runners/github-copilot/runner.ps1 @@ -59,21 +59,24 @@ $descriptor = [ordered]@{ cost_telemetry = 'unsupported' credential_child_filtering = 'supported' native_skill_activation_evidence = 'unsupported' - native_worker_delegation = 'supported' - delegated_worker_full_capability = 'supported' - delegated_worker_model_lock = 'supported' - delegated_worker_working_directory = 'supported' - delegated_worker_result_capture = 'supported' - delegated_worker_capacity_signal = 'supported' + # These are harness capabilities advertised by the descriptor. The + # actual child model, cwd, home, session, and terminal result remain + # conditional until the delegated worker reports terminal evidence. + native_worker_delegation = 'conditional' + delegated_worker_full_capability = 'conditional' + delegated_worker_model_lock = 'conditional' + delegated_worker_working_directory = 'conditional' + delegated_worker_result_capture = 'conditional' + delegated_worker_capacity_signal = 'conditional' } delegation = [ordered]@{ mode = 'native_worker' mechanism = 'Copilot CLI native task tool with an explicit full-capability general-purpose child agent; fleet/task lifecycle events observe completion' worker_role = 'general-purpose' - full_capability = 'supported' - model_lock = 'supported' - working_directory = 'supported' - result_capture = 'supported' + full_capability = 'conditional' + model_lock = 'conditional' + working_directory = 'conditional' + result_capture = 'conditional' capacity = 'harness_authoritative' nested_model_execution = $false } @@ -401,7 +404,8 @@ function Get-CopilotPreflight { $checks.Add((New-PreflightCheck -Name 'run_paths' -Status passed -Detail "-C $($run.WorkingDirectoryPath); COPILOT_HOME under $($run.HomeDirectoryPath)")) $checks.Add((New-PreflightCheck -Name 'prompt_fidelity' -Status passed -Detail 'The prepared UTF-8 prompt bytes are supplied once through stdin; the execution fake proves the received bytes match the staged prompt.')) $checks.Add((New-PreflightCheck -Name 'credential_boundary' -Status passed -Detail 'Only supported authentication state is made available to Copilot; --secret-env-vars removes every listed token variable from shell and MCP child environments; no Copilot profile or credential file is copied.')) - $checks.Add((New-PreflightCheck -Name 'native_worker_delegation' -Status passed -Detail 'Use Copilot native task with a full-capability general-purpose child agent; the parent supplies one explicit arm per child and observes child completion/model evidence.')) + $checks.Add((New-PreflightCheck -Name 'native_worker_delegation' -Status unavailable -Detail 'Copilot task/general-purpose delegation is an advertised native mechanism, but this preflight cannot observe the child''s resolved model, cwd, HOME/config, fresh identity, prompt, exclusions, or terminal capture.')) + $warnings.Add('Copilot native-worker controls are conditional. The external orchestrator must require terminal delegation evidence; direct -C/--model/HOME compatibility-transport observations do not prove the native child.') if ($platform -eq 'macos') { $warnings.Add('macOS sandbox-exec is deprecated by Apple but is used only when present; a future runner revision may replace it with an equivalent supported mechanism.') } diff --git a/scripts/eval-runners/opencode/runner.ps1 b/scripts/eval-runners/opencode/runner.ps1 index ffba019..5540832 100644 --- a/scripts/eval-runners/opencode/runner.ps1 +++ b/scripts/eval-runners/opencode/runner.ps1 @@ -47,21 +47,23 @@ $descriptor = [ordered]@{ cost_telemetry = 'conditional' credential_child_filtering = 'conditional' native_skill_activation_evidence = 'unsupported' - native_worker_delegation = 'supported' - delegated_worker_full_capability = 'supported' - delegated_worker_model_lock = 'supported' - delegated_worker_working_directory = 'supported' - delegated_worker_result_capture = 'supported' - delegated_worker_capacity_signal = 'supported' + # Task/General availability is only an advertised native mechanism; + # the actual child must prove its controls at terminal time. + native_worker_delegation = 'conditional' + delegated_worker_full_capability = 'conditional' + delegated_worker_model_lock = 'conditional' + delegated_worker_working_directory = 'conditional' + delegated_worker_result_capture = 'conditional' + delegated_worker_capacity_signal = 'conditional' } delegation = [ordered]@{ mode = 'native_worker' mechanism = 'OpenCode Task tool invoking the full-capability General subagent in a fresh child context' worker_role = 'general' - full_capability = 'supported' - model_lock = 'supported' - working_directory = 'supported' - result_capture = 'supported' + full_capability = 'conditional' + model_lock = 'conditional' + working_directory = 'conditional' + result_capture = 'conditional' capacity = 'harness_authoritative' nested_model_execution = $false } @@ -277,7 +279,8 @@ function Get-OpenCodePreflight { $checks.Add((New-PreflightCheck -Name 'fresh_session' -Status passed -Detail 'The adapter starts one new opencode run process and supplies no resume, continue, or session id.')) $checks.Add((New-PreflightCheck -Name 'ambient_configuration' -Status passed -Detail 'The adapter isolates global/user configuration roots and deliberately preserves repository-owned project configuration; OPENCODE_DISABLE_PROJECT_CONFIG is not used.')) $checks.Add((New-PreflightCheck -Name 'prompt_fidelity' -Status passed -Detail 'The exact prompt bytes are sent on stdin as the first and only task input.')) - $checks.Add((New-PreflightCheck -Name 'native_worker_delegation' -Status passed -Detail 'Use OpenCode native Task with the full-capability General subagent; Explore and Scout read-only agents are not valid eval workers.')) + $checks.Add((New-PreflightCheck -Name 'native_worker_delegation' -Status unavailable -Detail 'OpenCode Task/General is an advertised native mechanism, but this preflight cannot observe the child''s resolved model, cwd, HOME/config, fresh identity, prompt, exclusions, or terminal capture. Explore and Scout remain invalid read-only workers.')) + $warnings.Add('OpenCode native-worker controls are conditional. The external orchestrator must require terminal delegation evidence; direct run --dir/--model/config observations do not prove the native child.') $warnings.Add('OpenCode does not expose a supported child-tool environment filter in this CLI contract; the runner removes unrelated inherited variables but cannot independently prove that the selected provider credential is hidden from every OpenCode-launched tool.') $hardConfinement = $null -ne $sandboxInfo -and $platform -in @('linux', 'macos') diff --git a/scripts/eval-runners/orchestration.ps1 b/scripts/eval-runners/orchestration.ps1 index 93383e9..5c74574 100644 --- a/scripts/eval-runners/orchestration.ps1 +++ b/scripts/eval-runners/orchestration.ps1 @@ -251,12 +251,42 @@ function Register-WorkerTerminal { } $arm = Get-OrchestrationArmByWorkerId -Plan $Plan -WorkerId $WorkerId $activeWorker = $active[$WorkerId] - $runEvidence = Get-JsonProperty -Object $ExecutionEvidence -Name 'run' -Default $null - if ($null -ne $runEvidence) { - $evidenceEvalId = [int](Get-JsonProperty -Object $runEvidence -Name 'eval_id' -Default 0) - $evidenceConfiguration = [string](Get-JsonProperty -Object $runEvidence -Name 'configuration' -Default '') - if ($evidenceEvalId -ne [int]$arm.eval_id -or $evidenceConfiguration -ne [string]$arm.configuration) { - throw "Worker '$WorkerId' returned evidence for a different arm." + $expectedSessionId = [string](Get-JsonProperty -Object $activeWorker -Name 'worker_session_id' -Default '') + $effectiveStatus = $status + $terminalEvidenceFailures = [System.Collections.Generic.List[string]]::new() + + # The plan stores the exact manifest-declared run path. Resolve only that + # arm here; do not infer a path from configuration or inspect grading data. + try { + $runData = Resolve-RunContract -RunPath ([string]$arm.worker.run_manifest_path) + $validation = Test-NativeWorkerTerminalEvidence -ExecutionEvidence $ExecutionEvidence -Run $runData -RequestedModel ([string]$arm.worker.model) -ExpectedWorkerSessionId $expectedSessionId + foreach ($failure in @($validation.Failures)) { $terminalEvidenceFailures.Add([string]$failure) } + $evidenceSessionId = [string](Get-JsonProperty -Object $validation.Delegation -Name 'worker_session_id' -Default '') + if ($terminalEvidenceFailures.Count -eq 0 -and -not [string]::IsNullOrWhiteSpace($evidenceSessionId)) { + $completedWorkers = Get-OrchestrationDictionary -Object $State -Name 'completed' + foreach ($completedWorker in @($completedWorkers.Values)) { + if ([string](Get-JsonProperty -Object $completedWorker -Name 'worker_session_id' -Default '') -eq $evidenceSessionId) { + $terminalEvidenceFailures.Add('fresh_worker') + break + } + } + } + } catch { + $terminalEvidenceFailures.Add('terminal_evidence_unresolvable') + $terminalEvidenceFailures.Add($_.Exception.Message) + } + + if ($terminalEvidenceFailures.Count -gt 0 -and $effectiveStatus -ne 'incompatible') { + $effectiveStatus = 'incompatible' + } + if ($terminalEvidenceFailures.Count -gt 0) { + # A worker that returned an answer without mandatory native evidence is + # an incompatible arm, not an invitation to retry through another path. + try { $ExecutionEvidence.status = 'incompatible' } catch { } + if ($ExecutionEvidence -is [System.Collections.IDictionary]) { + $ExecutionEvidence['native_worker_evidence_failures'] = @($terminalEvidenceFailures.ToArray()) + } elseif ($null -ne $ExecutionEvidence -and -not (Test-JsonProperty -Object $ExecutionEvidence -Name 'native_worker_evidence_failures')) { + Add-Member -InputObject $ExecutionEvidence -MemberType NoteProperty -Name native_worker_evidence_failures -Value @($terminalEvidenceFailures.ToArray()) -Force } } @@ -267,9 +297,13 @@ function Register-WorkerTerminal { eval_id = [int]$arm.eval_id eval_name = [string]$arm.eval_name configuration = [string]$arm.configuration - status = $status + status = $effectiveStatus terminal_utc = [DateTime]::UtcNow.ToString('o') - worker_session_id = Get-JsonProperty -Object $activeWorker -Name 'worker_session_id' -Default $null + worker_session_id = if ([string]::IsNullOrWhiteSpace($expectedSessionId)) { + Get-JsonProperty -Object (Get-JsonProperty -Object (Get-JsonProperty -Object $ExecutionEvidence -Name 'evidence' -Default $null) -Name 'delegation' -Default $null) -Name 'worker_session_id' -Default $null + } else { $expectedSessionId } + native_worker_evidence = if ($terminalEvidenceFailures.Count -eq 0) { 'verified' } else { 'incompatible' } + native_worker_evidence_failures = @($terminalEvidenceFailures.ToArray()) } return $true } diff --git a/scripts/eval-runners/runner-common.ps1 b/scripts/eval-runners/runner-common.ps1 index d191d6d..29aa20e 100644 --- a/scripts/eval-runners/runner-common.ps1 +++ b/scripts/eval-runners/runner-common.ps1 @@ -28,7 +28,7 @@ function Get-JsonProperty { return $Default } - if ($null -ne $Object -and $Object.PSObject.Properties.Name -contains $Name -and $null -ne $Object.$Name) { + if ($null -ne $Object -and @($Object.PSObject.Properties | ForEach-Object { [string]$_.Name }) -contains $Name -and $null -ne $Object.$Name) { return $Object.$Name } @@ -44,7 +44,7 @@ function Get-JsonPropertyNames { if ($Object -is [System.Collections.IDictionary]) { return @($Object.Keys | ForEach-Object { [string]$_ }) } - return @($Object.PSObject.Properties.Name) + return @($Object.PSObject.Properties | ForEach-Object { [string]$_.Name }) } function Test-JsonProperty { @@ -295,9 +295,14 @@ function Get-DelegationCapabilityAssessment { 'delegated_worker_capacity_signal' ) $unproven = [System.Collections.Generic.List[string]]::new() + $conditional = [System.Collections.Generic.List[string]]::new() + $unsupported = [System.Collections.Generic.List[string]]::new() foreach ($name in $required) { $value = if ($null -ne $Capabilities -and $Capabilities.Contains($name)) { [string]$Capabilities[$name] } else { 'unavailable' } - if ($value -ne 'supported') { $unproven.Add($name) } + if ($value -ne 'supported') { + $unproven.Add($name) + if ($value -eq 'conditional') { $conditional.Add($name) } else { $unsupported.Add($name) } + } } $delegation = Get-JsonProperty -Object $Descriptor -Name 'delegation' -Default $null @@ -306,8 +311,14 @@ function Get-DelegationCapabilityAssessment { $mechanism = [string](Get-JsonProperty -Object $delegation -Name 'mechanism' -Default '') $workerRole = [string](Get-JsonProperty -Object $delegation -Name 'worker_role' -Default '') $modeIsNative = $mode -eq 'native_worker' - if (-not $modeIsNative) { $unproven.Add('delegation.mode') } - if ($nestedModelExecution) { $unproven.Add('delegation.nested_model_execution') } + if (-not $modeIsNative) { + $unproven.Add('delegation.mode') + if ($mode -eq 'conditional') { $conditional.Add('delegation.mode') } else { $unsupported.Add('delegation.mode') } + } + if ($nestedModelExecution) { + $unproven.Add('delegation.nested_model_execution') + $unsupported.Add('delegation.nested_model_execution') + } $delegationFields = [ordered]@{ full_capability = 'delegated_worker_full_capability' model_lock = 'delegated_worker_model_lock' @@ -318,20 +329,201 @@ function Get-DelegationCapabilityAssessment { foreach ($field in $delegationFields.Keys) { $value = [string](Get-JsonProperty -Object $delegation -Name $field -Default 'unsupported') $valid = if ($field -eq 'capacity') { $value -in @('supported', 'harness_authoritative') } else { $value -eq 'supported' } - if (-not $valid) { $unproven.Add("delegation.$field") } + if (-not $valid) { + $unproven.Add("delegation.$field") + if ($value -eq 'conditional') { $conditional.Add("delegation.$field") } else { $unsupported.Add("delegation.$field") } + } + } + + $status = if ($unsupported.Count -gt 0) { + 'unsupported' + } elseif ($conditional.Count -gt 0) { + 'conditional' + } else { + 'supported' } return [pscustomobject]@{ - MandatoryProven = $unproven.Count -eq 0 + MandatoryProven = $status -eq 'supported' + Status = $status Mode = $mode Mechanism = $mechanism WorkerRole = $workerRole NestedModelExecution = $nestedModelExecution Unproven = @($unproven) + Conditional = @($conditional) + Unsupported = @($unsupported) Required = @($required) } } +function Get-NativeWorkerTerminalEvidenceRequirements { + return @( + 'mechanism', + 'worker_session_id', + 'observed_model', + 'observed_working_directory', + 'observed_home', + 'fresh_worker', + 'home_config_isolated', + 'prompt_fidelity', + 'prompt_sha256', + 'terminal_result_capture', + 'paired_arm_visible', + 'grading_material_visible', + 'nested_model_execution', + 'model_execution_count' + ) +} + +function ConvertTo-ComparablePath { + param([Parameter(Mandatory = $true)][AllowEmptyString()][string]$Path) + + if ([string]::IsNullOrWhiteSpace($Path)) { return $null } + try { + $full = [System.IO.Path]::GetFullPath($Path) + $root = [System.IO.Path]::GetPathRoot($full) + if (-not [string]::IsNullOrWhiteSpace($root) -and $full.Length -gt $root.Length) { + $full = $full.TrimEnd([char[]]@('\', '/')) + } + return $full + } catch { + return $null + } +} + +function Test-ExactObservedPath { + param( + [Parameter(Mandatory = $true)][string]$Expected, + [Parameter(Mandatory = $true)][AllowEmptyString()][string]$Observed + ) + + $expectedComparable = ConvertTo-ComparablePath -Path $Expected + $observedComparable = ConvertTo-ComparablePath -Path $Observed + if ($null -eq $expectedComparable -or $null -eq $observedComparable) { return $false } + $comparison = if ($IsWindows) { [System.StringComparison]::OrdinalIgnoreCase } else { [System.StringComparison]::Ordinal } + return [string]::Equals($expectedComparable, $observedComparable, $comparison) +} + +function Test-NativeWorkerTerminalEvidence { + <# + Descriptor fields describe what a harness advertises. This validator is + deliberately separate: it accepts only observations from the actual + delegated worker for this exact arm. A direct compatibility-run result + without evidence.delegation is therefore never native-worker evidence. + #> + param( + [Parameter(Mandatory = $true)][object]$ExecutionEvidence, + [Parameter(Mandatory = $true)][object]$Run, + [Parameter(Mandatory = $true)][string]$RequestedModel, + [string]$ExpectedWorkerSessionId = '' + ) + + $failures = [System.Collections.Generic.List[string]]::new() + $status = [string](Get-JsonProperty -Object $ExecutionEvidence -Name 'status' -Default '') + if ($status -notin @('completed', 'failed', 'timed_out', 'cancelled', 'incompatible')) { + $failures.Add('terminal_result_capture') + } + $runEvidence = Get-JsonProperty -Object $ExecutionEvidence -Name 'run' -Default $null + if ($null -eq $runEvidence) { + $failures.Add('arm_identity') + } else { + if ([int](Get-JsonProperty -Object $runEvidence -Name 'eval_id' -Default 0) -ne [int]$Run.EvalId -or + [string](Get-JsonProperty -Object $runEvidence -Name 'eval_name' -Default '') -ne [string]$Run.EvalName -or + [string](Get-JsonProperty -Object $runEvidence -Name 'configuration' -Default '') -ne [string]$Run.Mode) { + $failures.Add('arm_identity') + } + } + + $requestedEvidence = Get-JsonProperty -Object $ExecutionEvidence -Name 'requested' -Default $null + if ($null -eq $requestedEvidence -or [string](Get-JsonProperty -Object $requestedEvidence -Name 'model' -Default '') -ne $RequestedModel) { + $failures.Add('requested_model') + } + + $delegation = Get-JsonProperty -Object (Get-JsonProperty -Object $ExecutionEvidence -Name 'evidence' -Default $null) -Name 'delegation' -Default $null + if ($null -eq $delegation) { + $failures.Add('delegation_terminal_evidence') + return [pscustomobject]@{ Valid = $false; Failures = @($failures); Delegation = $null } + } + + foreach ($name in @(Get-NativeWorkerTerminalEvidenceRequirements)) { + if (-not (Test-JsonProperty -Object $delegation -Name $name)) { + $failures.Add($name) + } + } + + if ([string]::IsNullOrWhiteSpace([string](Get-JsonProperty -Object $delegation -Name 'mechanism' -Default ''))) { + $failures.Add('mechanism') + } + + $workerSessionId = [string](Get-JsonProperty -Object $delegation -Name 'worker_session_id' -Default '') + if ([string]::IsNullOrWhiteSpace($workerSessionId)) { + if ($failures -notcontains 'worker_session_id') { $failures.Add('worker_session_id') } + } elseif (-not [string]::IsNullOrWhiteSpace($ExpectedWorkerSessionId) -and $workerSessionId -ne $ExpectedWorkerSessionId) { + $failures.Add('worker_session_id') + } + + if ([string](Get-JsonProperty -Object $delegation -Name 'observed_model' -Default '') -ne $RequestedModel) { + $failures.Add('requested_model') + } + if (-not [bool](Get-JsonProperty -Object $delegation -Name 'fresh_worker' -Default $false)) { + $failures.Add('fresh_worker') + } + if (-not [bool](Get-JsonProperty -Object $delegation -Name 'home_config_isolated' -Default $false)) { + $failures.Add('isolated_home_config') + } + if (-not [bool](Get-JsonProperty -Object $delegation -Name 'prompt_fidelity' -Default $false) -or + [string](Get-JsonProperty -Object $delegation -Name 'prompt_sha256' -Default '') -ne [string]$Run.PromptHash) { + $failures.Add('prompt_fidelity') + } + if (-not [bool](Get-JsonProperty -Object $delegation -Name 'terminal_result_capture' -Default $false)) { + $failures.Add('terminal_result_capture') + } + if ([bool](Get-JsonProperty -Object $delegation -Name 'paired_arm_visible' -Default $true) -or + [bool](Get-JsonProperty -Object $delegation -Name 'grading_material_visible' -Default $true)) { + $failures.Add('paired_arm_and_grading_exclusion') + } + if ([bool](Get-JsonProperty -Object $delegation -Name 'nested_model_execution' -Default $true) -or + [int](Get-JsonProperty -Object $delegation -Name 'model_execution_count' -Default 0) -ne 1) { + $failures.Add('nested_model_execution') + } + if (-not (Test-ExactObservedPath -Expected ([string]$Run.WorkingDirectoryPath) -Observed ([string](Get-JsonProperty -Object $delegation -Name 'observed_working_directory' -Default '')))) { + $failures.Add('working_directory') + } + if (-not (Test-ExactObservedPath -Expected ([string]$Run.HomeDirectoryPath) -Observed ([string](Get-JsonProperty -Object $delegation -Name 'observed_home' -Default '')))) { + $failures.Add('isolated_home_config') + } + + $session = Get-JsonProperty -Object $ExecutionEvidence -Name 'session' -Default $null + if ($null -eq $session -or + -not [bool](Get-JsonProperty -Object $session -Name 'fresh' -Default $false) -or + [bool](Get-JsonProperty -Object $session -Name 'resumed' -Default $true) -or + [string](Get-JsonProperty -Object $session -Name 'id' -Default '') -ne $workerSessionId) { + $failures.Add('fresh_worker') + } + + return [pscustomobject]@{ + Valid = $failures.Count -eq 0 + Failures = @($failures | Select-Object -Unique) + Delegation = $delegation + } +} + +function Assert-NativeWorkerTerminalEvidence { + param( + [Parameter(Mandatory = $true)][object]$ExecutionEvidence, + [Parameter(Mandatory = $true)][object]$Run, + [Parameter(Mandatory = $true)][string]$RequestedModel, + [string]$ExpectedWorkerSessionId = '' + ) + + $validation = Test-NativeWorkerTerminalEvidence -ExecutionEvidence $ExecutionEvidence -Run $Run -RequestedModel $RequestedModel -ExpectedWorkerSessionId $ExpectedWorkerSessionId + if (-not $validation.Valid) { + throw "Native worker terminal evidence is incompatible: $([string]::Join(', ', @($validation.Failures)))." + } + return $true +} + function Resolve-RunContract { param([Parameter(Mandatory = $true)][string]$RunPath) @@ -572,6 +764,9 @@ function New-PreflightDocument { $capabilitiesForAssessment = if ($null -eq $ResolvedCapabilities) { [ordered]@{} } else { $ResolvedCapabilities } $assessment = Get-IsolationCapabilityAssessment -Capabilities $capabilitiesForAssessment $delegationAssessment = Get-DelegationCapabilityAssessment -Descriptor $Descriptor -Capabilities $capabilitiesForAssessment + # Isolation is the compatibility transport's local readiness gate. Native + # delegation has a separate gate: conditional controls may proceed to a + # delegated worker, while an unavailable/unsupported mechanism cannot. $effectiveCompatible = $Compatible -and $assessment.MandatoryProven $unprovenControls = [string[]]$assessment.Unproven if (-not $effectiveCompatible) { $unprovenControls = [string[]](@($assessment.Unproven) + @('preflight')) } @@ -592,13 +787,14 @@ function New-PreflightDocument { checks = @($Checks) resolved_capabilities = if ($null -eq $ResolvedCapabilities) { [ordered]@{} } else { $ResolvedCapabilities } delegation = [ordered]@{ - status = if ($delegationAssessment.MandatoryProven) { 'supported' } elseif ($delegationAssessment.Mode -eq 'conditional') { 'conditional' } else { 'unsupported' } + status = $delegationAssessment.Status mode = $delegationAssessment.Mode mechanism = $delegationAssessment.Mechanism worker_role = $delegationAssessment.WorkerRole nested_model_execution = $delegationAssessment.NestedModelExecution required_controls = @($delegationAssessment.Required) unproven_controls = [string[]]$delegationAssessment.Unproven + terminal_evidence_required = $delegationAssessment.Status -eq 'conditional' } isolation = [ordered]@{ level = if ($effectiveCompatible) { $assessment.Level } else { 'unsupported' } @@ -620,7 +816,7 @@ function Assert-NativeWorkerDelegation { $delegation = Get-JsonProperty -Object $Descriptor -Name 'delegation' -Default $null $mode = [string](Get-JsonProperty -Object $delegation -Name 'mode' -Default 'unsupported') - if ($mode -ne 'native_worker') { + if ($mode -notin @('native_worker', 'conditional')) { throw "Runner '$($Descriptor.name)' cannot satisfy the mandatory native Eval Worker contract: delegation mode is '$mode'." } if ([bool](Get-JsonProperty -Object $delegation -Name 'nested_model_execution' -Default $true)) { @@ -630,9 +826,13 @@ function Assert-NativeWorkerDelegation { throw "Runner '$($Descriptor.name)' preflight is incompatible; the orchestrator must not execute an arm in the parent context." } $delegationPreflight = Get-JsonProperty -Object $Preflight -Name 'delegation' -Default $null - if ([string](Get-JsonProperty -Object $delegationPreflight -Name 'status' -Default 'unsupported') -ne 'supported') { + $delegationStatus = [string](Get-JsonProperty -Object $delegationPreflight -Name 'status' -Default 'unsupported') + if ($delegationStatus -notin @('supported', 'conditional')) { $unproven = @((Get-JsonProperty -Object $delegationPreflight -Name 'unproven_controls' -Default @())) - throw "Runner '$($Descriptor.name)' native worker delegation is not proven during preflight: $([string]::Join(', ', $unproven))." + throw "Runner '$($Descriptor.name)' native worker delegation is unavailable during preflight: $([string]::Join(', ', $unproven)). No parent or compatibility-execute fallback is permitted." + } + if ($delegationStatus -eq 'conditional' -and -not [bool](Get-JsonProperty -Object $delegationPreflight -Name 'terminal_evidence_required' -Default $false)) { + throw "Runner '$($Descriptor.name)' reports conditional native worker controls without requiring terminal evidence." } $capabilities = Get-JsonProperty -Object $Preflight -Name 'resolved_capabilities' -Default $null $required = @( @@ -644,8 +844,9 @@ function Assert-NativeWorkerDelegation { 'delegated_worker_capacity_signal' ) foreach ($name in $required) { - if ([string](Get-JsonProperty -Object $capabilities -Name $name -Default 'unsupported') -ne 'supported') { - throw "Runner '$($Descriptor.name)' native worker capability '$name' is not supported; the orchestrator must not fall back to parent execution." + $value = [string](Get-JsonProperty -Object $capabilities -Name $name -Default 'unsupported') + if ($value -notin @('supported', 'conditional')) { + throw "Runner '$($Descriptor.name)' native worker capability '$name' is unavailable; the orchestrator must not fall back to parent or compatibility execution." } } return $true diff --git a/scripts/eval-runners/tests/test-orchestration.ps1 b/scripts/eval-runners/tests/test-orchestration.ps1 index e483a6e..0f5a8d7 100644 --- a/scripts/eval-runners/tests/test-orchestration.ps1 +++ b/scripts/eval-runners/tests/test-orchestration.ps1 @@ -34,6 +34,46 @@ function Write-TestJson { [System.IO.File]::WriteAllText($Path, (($Value | ConvertTo-Json -Depth 100) + [Environment]::NewLine), [System.Text.UTF8Encoding]::new($false)) } +function Copy-TestObject { + param([Parameter(Mandatory = $true)][object]$Value) + + return $Value | ConvertTo-Json -Depth 100 | ConvertFrom-Json +} + +function New-TestNativeTerminalEvidence { + param( + [Parameter(Mandatory = $true)][object]$Arm, + [Parameter(Mandatory = $true)][object]$RunData, + [Parameter(Mandatory = $true)][string]$WorkerSessionId + ) + + return [ordered]@{ + status = 'completed' + session = [ordered]@{ id = $WorkerSessionId; fresh = $true; resumed = $false } + run = [ordered]@{ eval_id = [int]$Arm.eval_id; eval_name = [string]$Arm.eval_name; configuration = [string]$Arm.configuration } + requested = [ordered]@{ model = [string]$Arm.worker.model } + input = [ordered]@{ prompt_sha256 = [string]$RunData.PromptHash } + evidence = [ordered]@{ + delegation = [ordered]@{ + mechanism = 'deterministic-fake-native-worker' + worker_session_id = $WorkerSessionId + observed_model = [string]$Arm.worker.model + observed_working_directory = [string]$RunData.WorkingDirectoryPath + observed_home = [string]$RunData.HomeDirectoryPath + fresh_worker = $true + home_config_isolated = $true + prompt_fidelity = $true + prompt_sha256 = [string]$RunData.PromptHash + terminal_result_capture = $true + paired_arm_visible = $false + grading_material_visible = $false + nested_model_execution = $false + model_execution_count = 1 + } + } + } +} + $testRoot = Join-Path ([System.IO.Path]::GetTempPath()) ('agentic-orchestration-' + [Guid]::NewGuid().ToString('N')) try { $iteration = Join-Path $testRoot 'iteration-1' @@ -54,10 +94,28 @@ try { foreach ($configuration in @('with_skill', 'without_skill')) { $runDirectory = Join-Path $evalDirectory $configuration New-Item -ItemType Directory -Path $runDirectory -Force | Out-Null + New-Item -ItemType Directory -Path (Join-Path $runDirectory 'repo') -Force | Out-Null + New-Item -ItemType Directory -Path (Join-Path $runDirectory 'home') -Force | Out-Null + [System.IO.File]::WriteAllText((Join-Path $runDirectory 'prompt.md'), "terminal test prompt for $evalName/$configuration", [System.Text.UTF8Encoding]::new($false)) + if ($configuration -eq 'with_skill') { + New-Item -ItemType Directory -Path (Join-Path $runDirectory 'skill') -Force | Out-Null + [System.IO.File]::WriteAllText((Join-Path (Join-Path $runDirectory 'skill') 'SKILL.md'), '# deterministic terminal test skill', [System.Text.UTF8Encoding]::new($false)) + } Write-TestJson -Path (Join-Path $runDirectory 'run.json') -Value ([ordered]@{ + schema = (Get-RunnerSchemaNames).Run evalId = $evalId evalName = $evalName mode = $configuration + promptFile = 'prompt.md' + workingDirectory = 'repo' + homeDirectory = 'home' + skillDirectory = if ($configuration -eq 'with_skill') { 'skill' } else { $null } + freshContextRequired = $true + filesystemIsolationRequired = $true + isolatedHomeRequired = $true + mustNotReadOutsideSandbox = $true + fixtureHash = ('a' * 64) + skillHash = if ($configuration -eq 'with_skill') { ('b' * 64) } else { $null } }) $resultFileName = if ($configuration -eq 'with_skill') { 'with-skill.result.json' } else { 'without-skill.result.json' } Write-TestJson -Path (Join-Path $evalDirectory (Join-Path 'results' $resultFileName)) -Value ([ordered]@{ @@ -167,9 +225,30 @@ try { if ($activeIds.Count -gt 0) { $workerId = [string]$activeIds[0] $arm = Get-OrchestrationArmByWorkerId -Plan $plan -WorkerId $workerId + $runData = Resolve-RunContract -RunPath ([string]$arm.worker.run_manifest_path) + $workerSessionId = [string]$capacityState.active[$workerId].worker_session_id [void](Register-WorkerTerminal -Plan $plan -State $capacityState -WorkerId $workerId -ExecutionEvidence ([ordered]@{ status = 'completed' - run = [ordered]@{ eval_id = [int]$arm.eval_id; configuration = [string]$arm.configuration } + session = [ordered]@{ id = $workerSessionId; fresh = $true; resumed = $false } + run = [ordered]@{ eval_id = [int]$arm.eval_id; eval_name = [string]$arm.eval_name; configuration = [string]$arm.configuration } + evidence = [ordered]@{ + delegation = [ordered]@{ + mechanism = 'deterministic-fake-native-worker' + worker_session_id = $workerSessionId + observed_model = [string]$arm.worker.model + observed_working_directory = [string]$runData.WorkingDirectoryPath + observed_home = [string]$runData.HomeDirectoryPath + fresh_worker = $true + home_config_isolated = $true + prompt_fidelity = $true + prompt_sha256 = [string]$runData.PromptHash + terminal_result_capture = $true + paired_arm_visible = $false + grading_material_visible = $false + nested_model_execution = $false + model_execution_count = 1 + } + } })) } elseif (@($capacityState.pending_worker_ids).Count -gt 0) { throw 'capacity queue deadlocked with pending workers and no active worker.' @@ -201,6 +280,109 @@ try { Assert-True $fallbackRejected 'missing native delegation fails preflight' Assert-Equal 0 $parentDispatches 'failed delegation preflight never invokes parent fallback' + # A native mechanism may be locally ready while worker-specific controls + # remain conditional. The gate must allow that handoff only when it will + # validate terminal evidence; it must never reuse compatibility execute. + $conditionalDescriptor = [pscustomobject]@{ + name = 'conditional-native' + delegation = [ordered]@{ mode = 'native_worker'; nested_model_execution = $false } + } + $conditionalCapabilities = [ordered]@{ + native_worker_delegation = 'conditional' + delegated_worker_full_capability = 'conditional' + delegated_worker_model_lock = 'conditional' + delegated_worker_working_directory = 'conditional' + delegated_worker_result_capture = 'conditional' + delegated_worker_capacity_signal = 'supported' + } + $conditionalPreflight = [ordered]@{ + status = 'compatible' + delegation = [ordered]@{ status = 'conditional'; unproven_controls = @('delegated_worker_model_lock'); terminal_evidence_required = $true } + resolved_capabilities = $conditionalCapabilities + } + Assert-True (Assert-NativeWorkerDelegation -Descriptor $conditionalDescriptor -Preflight $conditionalPreflight) 'conditional native preflight is accepted only for terminal validation' + $conditionalWithoutTerminalEvidence = [ordered]@{ + status = 'compatible' + delegation = [ordered]@{ status = 'conditional'; unproven_controls = @('delegated_worker_model_lock') } + resolved_capabilities = $conditionalCapabilities + } + $conditionalWithoutTerminalRejected = $false + try { [void](Assert-NativeWorkerDelegation -Descriptor $conditionalDescriptor -Preflight $conditionalWithoutTerminalEvidence) } catch { $conditionalWithoutTerminalRejected = $true } + Assert-True $conditionalWithoutTerminalRejected 'conditional native preflight without a terminal-evidence requirement is rejected' + + $terminalArm = $plan.arms[0] + $terminalRunData = Resolve-RunContract -RunPath ([string]$terminalArm.worker.run_manifest_path) + $validTerminalEvidence = New-TestNativeTerminalEvidence -Arm $terminalArm -RunData $terminalRunData -WorkerSessionId 'native-terminal-session' + Assert-True ((Test-NativeWorkerTerminalEvidence -ExecutionEvidence $validTerminalEvidence -Run $terminalRunData -RequestedModel ([string]$terminalArm.worker.model) -ExpectedWorkerSessionId 'native-terminal-session').Valid) 'valid terminal native-worker evidence is accepted' + Assert-True (Assert-NativeWorkerTerminalEvidence -ExecutionEvidence $validTerminalEvidence -Run $terminalRunData -RequestedModel ([string]$terminalArm.worker.model) -ExpectedWorkerSessionId 'native-terminal-session') 'valid terminal evidence passes the assert gate' + + function Invoke-TerminalEvidenceCase { + param( + [Parameter(Mandatory = $true)][string]$Name, + [Parameter(Mandatory = $true)][object]$Evidence, + [Parameter(Mandatory = $true)][string]$ExpectedFailure + ) + + $caseState = New-OrchestrationState -Plan ([pscustomobject]@{ + schema = $plan.schema + requested_concurrency = 1 + arms = @($terminalArm) + }) + [void](Register-DelegationAccepted -State $caseState -WorkerId ([string]$terminalArm.worker_id) -WorkerSessionId 'native-terminal-session') + [void](Register-WorkerTerminal -Plan ([pscustomobject]@{ arms = @($terminalArm) }) -State $caseState -WorkerId ([string]$terminalArm.worker_id) -ExecutionEvidence $Evidence) + Assert-Equal 'incompatible' $Evidence.status "$Name changes the arm to incompatible" + Assert-Equal 'incompatible' $caseState.completed[[string]$terminalArm.worker_id].status "$Name is terminally incompatible" + Assert-True (([string]::Join(',', @($caseState.completed[[string]$terminalArm.worker_id].native_worker_evidence_failures))) -match [regex]::Escape($ExpectedFailure)) "$Name records $ExpectedFailure" + } + + $modelMismatch = Copy-TestObject -Value $validTerminalEvidence + $modelMismatch.evidence.delegation.observed_model = 'different-model' + Invoke-TerminalEvidenceCase -Name 'model mismatch' -Evidence $modelMismatch -ExpectedFailure 'requested_model' + + $workingDirectoryMismatch = Copy-TestObject -Value $validTerminalEvidence + $workingDirectoryMismatch.evidence.delegation.observed_working_directory = (Join-Path $terminalRunData.RunRoot 'other-repo') + Invoke-TerminalEvidenceCase -Name 'working-directory mismatch' -Evidence $workingDirectoryMismatch -ExpectedFailure 'working_directory' + + $missingHomeProof = Copy-TestObject -Value $validTerminalEvidence + $missingHomeProof.evidence.delegation.home_config_isolated = $false + $missingHomeProof.evidence.delegation.observed_home = '' + Invoke-TerminalEvidenceCase -Name 'missing HOME/config proof' -Evidence $missingHomeProof -ExpectedFailure 'isolated_home_config' + + $missingDelegationEvidence = Copy-TestObject -Value $validTerminalEvidence + $missingDelegationEvidence.evidence = $null + Invoke-TerminalEvidenceCase -Name 'missing terminal delegation evidence' -Evidence $missingDelegationEvidence -ExpectedFailure 'delegation_terminal_evidence' + + $armMismatch = Copy-TestObject -Value $validTerminalEvidence + $armMismatch.run.eval_id = 999 + Invoke-TerminalEvidenceCase -Name 'arm identity mismatch' -Evidence $armMismatch -ExpectedFailure 'arm_identity' + + $nestedExecution = Copy-TestObject -Value $validTerminalEvidence + $nestedExecution.evidence.delegation.nested_model_execution = $true + $nestedExecution.evidence.delegation.model_execution_count = 2 + Invoke-TerminalEvidenceCase -Name 'nested model execution' -Evidence $nestedExecution -ExpectedFailure 'nested_model_execution' + + $freshWorkerMismatch = Copy-TestObject -Value $validTerminalEvidence + $freshWorkerMismatch.session.id = 'different-worker-session' + $freshWorkerMismatch.evidence.delegation.worker_session_id = 'different-worker-session' + Invoke-TerminalEvidenceCase -Name 'fresh worker/session mismatch' -Evidence $freshWorkerMismatch -ExpectedFailure 'worker_session_id' + + $duplicateSessionState = New-OrchestrationState -Plan ([pscustomobject]@{ + schema = $plan.schema + requested_concurrency = 1 + arms = @($terminalArm) + }) + $duplicateSessionState.completed['prior-worker'] = [ordered]@{ worker_id = 'prior-worker'; worker_session_id = 'native-terminal-session' } + [void](Register-DelegationAccepted -State $duplicateSessionState -WorkerId ([string]$terminalArm.worker_id) -WorkerSessionId 'native-terminal-session') + [void](Register-WorkerTerminal -Plan ([pscustomobject]@{ arms = @($terminalArm) }) -State $duplicateSessionState -WorkerId ([string]$terminalArm.worker_id) -ExecutionEvidence (Copy-TestObject -Value $validTerminalEvidence)) + Assert-Equal 'incompatible' $duplicateSessionState.completed[[string]$terminalArm.worker_id].status 'reused worker session makes the arm incompatible' + Assert-True (([string]::Join(',', @($duplicateSessionState.completed[[string]$terminalArm.worker_id].native_worker_evidence_failures))) -match 'fresh_worker') 'reused worker session records fresh-worker failure' + + # A compatibility-transport answer has no native delegation evidence and + # is rejected rather than retried through runner.ps1 execute or parent code. + $compatibilityTransportResult = Copy-TestObject -Value $validTerminalEvidence + $compatibilityTransportResult.evidence.PSObject.Properties.Remove('delegation') + Invoke-TerminalEvidenceCase -Name 'compatibility transport result' -Evidence $compatibilityTransportResult -ExpectedFailure 'delegation_terminal_evidence' + Write-Output 'Native worker orchestration: PASS' } finally { if (Test-Path -LiteralPath $testRoot) { Remove-Item -LiteralPath $testRoot -Recurse -Force } diff --git a/scripts/eval-runners/tests/test-runner-conformance.ps1 b/scripts/eval-runners/tests/test-runner-conformance.ps1 index 56f50e1..24dd7f1 100644 --- a/scripts/eval-runners/tests/test-runner-conformance.ps1 +++ b/scripts/eval-runners/tests/test-runner-conformance.ps1 @@ -51,6 +51,7 @@ $recordedOldGhToken = $env:GH_TOKEN $recordedOldGithubToken = $env:GITHUB_TOKEN $recordedOldCopilotHome = $env:COPILOT_HOME $recordedOldGhConfigDir = $env:GH_CONFIG_DIR +$recordedOldClineAgentsSquad = $env:CLINE_AGENTS_SQUAD_PLUGIN try { $fakeBin = Join-Path $recordedRoot 'bin' New-Item -ItemType Directory -Path $fakeBin -Force | Out-Null @@ -272,6 +273,7 @@ exit 2 $env:COPILOT_GITHUB_TOKEN = 'recorded-copilot-canary-not-logged' $env:GH_TOKEN = 'recorded-gh-canary-not-logged' $env:GITHUB_TOKEN = 'recorded-github-canary-not-logged' + $env:CLINE_AGENTS_SQUAD_PLUGIN = $null $recordedGhConfig = Join-Path $recordedRoot 'github-cli-auth' New-Item -ItemType Directory -Path $recordedGhConfig -Force | Out-Null [System.IO.File]::WriteAllText((Join-Path $recordedGhConfig 'auth-marker.txt'), 'fixture auth state without a credential value', [System.Text.UTF8Encoding]::new($false)) @@ -315,6 +317,8 @@ exit 2 Assert-True ($description.PSObject.Properties.Name -contains 'delegation') "$runnerName descriptor declares native delegation" Assert-True (-not [bool]$description.delegation.nested_model_execution) "$runnerName descriptor forbids nested model execution" Assert-True (-not [string]::IsNullOrWhiteSpace([string]$description.delegation.mechanism)) "$runnerName descriptor records its native delegation mechanism" + Assert-Equal 'conditional' $description.capabilities.native_worker_delegation "$runnerName descriptor does not present native delegation as terminal proof" + Assert-Equal 'conditional' $description.delegation.model_lock "$runnerName descriptor leaves child model resolution conditional" $expectedVersion = switch ($runnerName) { 'codex' { 'recorded-codex 9.1' } 'opencode' { 'recorded-opencode 9.2' } 'copilot' { 'GitHub Copilot CLI recorded-1.0.80' } default { 'recorded-cline 9.3' } } Assert-Equal $expectedVersion $description.harness.version "$runnerName exact describe version" $preflightWith = Invoke-AdapterJson -RunnerPath $runnerPath -Command preflight -RunPath $with.Path -ProfilePath $recordedProfiles[$runnerName] @@ -327,7 +331,8 @@ exit 2 Assert-True ($preflightWith.delegation.status -ne 'supported') 'Cline preflight does not claim unavailable Agent Squad delegation' Assert-True (([string]::Join(' ', @($preflightWith.warnings))) -match 'use_subagents' -and ([string]::Join(' ', @($preflightWith.warnings))) -match 'Agent Squad') 'Cline preflight rejects read-only subagents as a mutable-arm fallback' } else { - Assert-Equal 'supported' $preflightWith.delegation.status "$runnerName native delegation preflight" + Assert-Equal 'conditional' $preflightWith.delegation.status "$runnerName native delegation preflight requires terminal evidence" + Assert-True ([bool]$preflightWith.delegation.terminal_evidence_required) "$runnerName preflight requires terminal delegation evidence" } if ($runnerName -eq 'copilot') { Assert-True (@($preflightWith.checks | Where-Object { $_.name -eq 'authentication' -and $_.status -eq 'passed' }).Count -eq 1) 'Copilot preflight accepts explicit environment authentication' @@ -469,6 +474,7 @@ exit 2 Assert-True (($resultWithout | ConvertTo-Json -Depth 100) -notmatch 'recorded-canary|recorded-unrelated-canary|recorded-copilot-canary|recorded-gh-canary|recorded-github-canary|recorded-gh-fallback-token') 'Copilot baseline result evidence does not contain credential values' } } + $env:CLINE_AGENTS_SQUAD_PLUGIN = $recordedOldClineAgentsSquad $staleCli = $fakeCli.Replace("'opencode' { '--format --dir --model --auto --pure --continue --session' }", "'opencode' { '--format --dir --model --pure --continue --session' }") [System.IO.File]::WriteAllText((Join-Path $fakeBin 'opencode.ps1'), $staleCli, [System.Text.UTF8Encoding]::new($false)) $stalePreflight = Invoke-AdapterJson -RunnerPath (Join-Path $runnerRoot 'opencode\runner.ps1') -Command preflight -RunPath $with.Path -ProfilePath $recordedProfiles['opencode'] @@ -550,6 +556,7 @@ exit 2 $env:GITHUB_TOKEN = $recordedOldGithubToken $env:COPILOT_HOME = $recordedOldCopilotHome $env:GH_CONFIG_DIR = $recordedOldGhConfigDir + $env:CLINE_AGENTS_SQUAD_PLUGIN = $recordedOldClineAgentsSquad if (Test-Path -LiteralPath $recordedRoot) { Remove-Item -LiteralPath $recordedRoot -Recurse -Force } } } @@ -705,6 +712,7 @@ try { [void](Assert-RunnerDescriptor -Descriptor $descriptor) Assert-Equal 'fake' $descriptor.name 'descriptor identity' Assert-Equal (Get-RunnerSchemaNames).Protocol $descriptor.protocol_version 'descriptor protocol' + Assert-Equal 'unsupported' $descriptor.capabilities.native_worker_delegation 'deterministic fake does not advertise a native delegation surface' Assert-Throws { Assert-RunnerDescriptor -Descriptor ([pscustomobject]@{ schema = $descriptor.schema; protocol_version = 'changed'; name = 'fake' }) } 'changed protocol must fail descriptor validation' Assert-Throws { Resolve-ExecutionProfile -ProfilePath $legacyProviderProfilePath } 'execution profile rejects the removed provider field' @@ -865,7 +873,9 @@ try { Assert-True ($prepareText.Contains('Do not derive, normalize, rename, hyphenate, underscore, or otherwise reconstruct any run, execution-result, or result path.')) 'handoff preparation must prohibit reconstructed paths' Assert-True ($prepareText.Contains('DELEGATE EVERY eval arm to a fresh harness-native worker/subagent. The Eval Orchestrator MUST NOT execute an eval arm itself.')) 'handoff preparation must require delegated native workers and forbid parent execution' Assert-True ($prepareText.Contains('One arm equals one delegated worker and one model-backed eval execution.')) 'handoff preparation must state the one-arm one-model invariant' - Assert-True ($prepareText.Contains('Require terminal evidence for the exact selected model, working directory, isolated home, and fresh session')) 'handoff preparation must require worker control evidence' + Assert-True ($prepareText.Contains('Assert-NativeWorkerDelegation')) 'handoff preparation must invoke the native delegation gate' + Assert-True ($prepareText.Contains('Require terminal evidence for the exact selected model, exact arm identity, working directory, isolated HOME/config boundary, prompt fidelity, terminal result capture')) 'handoff preparation must require worker control evidence' + Assert-True ($prepareText.Contains('-RequireComplete -RequireNativeDelegation')) 'handoff preparation must revalidate native terminal evidence during the manifest bridge' Assert-True ($prepareText.Contains('min(execution-profile.json.concurrency, remaining arms)')) 'handoff preparation must state requested concurrency fan-out' Assert-True ($prepareText.Contains('rejected before the worker starts') -and $prepareText.Contains('record no eval attempt')) 'handoff preparation must queue capacity rejections without counting attempts' Assert-True ($prepareText.Contains('orchestration.ps1')) 'handoff preparation must load the deterministic orchestration helper' diff --git a/scripts/prepare-skill-evals.ps1 b/scripts/prepare-skill-evals.ps1 index 31308fc..fe89a8e 100644 --- a/scripts/prepare-skill-evals.ps1 +++ b/scripts/prepare-skill-evals.ps1 @@ -1639,10 +1639,10 @@ function New-RunnerPrompt { [void]$builder.AppendLine('> Run independent workers concurrently up to `execution-profile.json.concurrency`. If the harness temporarily refuses another worker because its own concurrency limit is reached, keep that arm queued and dispatch it when capacity becomes available.') [void]$builder.AppendLine() [void]$builder.AppendLine('1. Read `manifest.json` and `execution-profile.json`. If `runner` or `model` is null, unavailable, or unsupported, fail clearly and list the supported package-local runner IDs; do not guess a default. The profile contains no credentials.') - [void]$builder.AppendLine(('2. Resolve the selected package-local runner with the resolver. Ask it for `describe` and `preflight`, validate its protocol, descriptor, and native-delegation capability declarations, and load the deterministic orchestration helper at ' + $orchestrationPath + '. Do not invent harness-specific CLI commands.')) + [void]$builder.AppendLine(('2. Resolve the selected package-local runner with the resolver. Ask it for `describe` and `preflight`, validate its protocol, descriptor, and native-delegation capability declarations, invoke `Assert-NativeWorkerDelegation`, and load the deterministic orchestration helper at ' + $orchestrationPath + '. Do not invent harness-specific CLI commands.')) [void]$builder.AppendLine('3. Build the pending arm queue from `manifest.json` with the orchestration helper. For every arm, read the exact `run_manifest`, `execution_result`, and `result` fields from `runs..run_manifest`, `runs..execution_result`, and `runs..result`. Retain those exact manifest-declared strings without editing them: the parent owns those exact destinations; the worker receives only its own prepared arm contract. Do not derive, normalize, rename, hyphenate, underscore, or otherwise reconstruct any run, execution-result, or result path.') - [void]$builder.AppendLine('4. Before dispatching, require native worker delegation and all mandatory isolation controls. A conditional or unavailable delegation mechanism is an incompatible preflight. Do not continue by invoking a runner process in the parent, and do not silently serialize arms in the parent.') - [void]$builder.AppendLine('5. Dispatch each pending arm to one fresh harness-native full-capability worker. The worker must execute the prepared `prompt.md` as its first task from that arm''s staged run directory, with the selected model/configuration, exact working directory and isolated home. Require terminal evidence for the exact selected model, working directory, isolated home, and fresh session; if the native surface cannot lock or prove one of these, fail preflight rather than accepting a fallback. It must not receive its paired arm, `eval-metadata.json`, expected output, assertions, grading, benchmark/report data, or any result from another arm.') + [void]$builder.AppendLine('4. Before dispatching, require native worker delegation and all mandatory isolation controls. An unavailable or unsupported delegation mechanism is incompatible. A conditional mechanism is allowed only when the external orchestrator will require terminal evidence for the actual worker. Do not continue by invoking a runner process in the parent, and do not silently serialize arms in the parent.') + [void]$builder.AppendLine('5. Dispatch each pending arm to one fresh harness-native full-capability worker. The worker must execute the prepared `prompt.md` as its first task from that arm''s staged run directory, with the selected model/configuration, exact working directory and isolated home. Require terminal evidence for the exact selected model, exact arm identity, working directory, isolated HOME/config boundary, prompt fidelity, terminal result capture, paired-arm/grading exclusion, fresh session, and absence of nested model execution; if any required fact is missing or mismatched, mark the arm incompatible without retrying through a fallback. It must not receive its paired arm, `eval-metadata.json`, expected output, assertions, grading, benchmark/report data, or any result from another arm.') [void]$builder.AppendLine('6. The delegated worker is the only model-backed execution for that arm. It must not invoke `runner.ps1 execute`, a second harness CLI, another model agent, or a nested session. Return the worker transcript/terminal evidence and normalized execution result to the parent; the parent writes it to the exact manifest-declared `execution_result` path without reconstructing any path.') [void]$builder.AppendLine('7. Maintain up to `min(execution-profile.json.concurrency, remaining arms)` active delegated workers. If a delegation request is rejected before the worker starts because of harness capacity, leave that arm pending, record no eval attempt, and retry it after an active worker becomes terminal. Do not add a runner-specific ceiling or change the portable requested concurrency.') [void]$builder.AppendLine('8. Preserve the complete terminal response, status, telemetry, evidence references, hashes, isolation mechanisms, warnings, and compatibility deviations. Do not retry for answer quality. A refusal is a result; timeout, harness failure, and incompatibility are results.') @@ -1661,7 +1661,7 @@ function New-RunnerPrompt { [void]$builder.AppendLine('## Phase 2: bridge, grade, and report') [void]$builder.AppendLine() [void]$builder.AppendLine('1. Only after every available delegated worker is terminal, invoke the deterministic package bridge below. It reads `manifest.json`, obtains each arm''s exact `run_manifest`, `execution_result`, and `result` paths. The bridge checks prompt/run/profile hashes and artifact confinement, validates the manifest paths, rejects unreferenced hyphen/underscore shadow results, and invokes the existing one-arm bridge with those exact paths. Do not manually construct a bridge command for an arm.') - [void]$builder.AppendLine((' `pwsh -NoProfile -File "' + $manifestBridgePath + '" -IterationDirectory "' + $IterationDirectory + '" -RequireComplete`')) + [void]$builder.AppendLine((' `pwsh -NoProfile -File "' + $manifestBridgePath + '" -IterationDirectory "' + $IterationDirectory + '" -RequireComplete -RequireNativeDelegation`')) [void]$builder.AppendLine(' The bridge''s one-arm operation is conceptually `-Run runPath -ExecutionResult executionPath -Result resultPath`, where all three values are the exact strings read from `manifest.json`. Do not derive, normalize, rename, hyphenate, underscore, or otherwise reconstruct any of them.') [void]$builder.AppendLine('2. Only if the package bridge succeeds, read each eval''s `eval-metadata.json` and reveal `expected_output` and `assertions` to the Grader. Follow `tools/skill-creator/agents/grader.md`; grade deterministically first, then use optional model judgement only where deterministic evidence cannot decide. Never infer tool or file behavior from model self-report without process evidence.') [void]$builder.AppendLine('3. Write only `grading[].text`, `grading[].passed`, and `grading[].evidence` for grading. Do not alter raw execution results or replace the canonical result stubs. Use null for genuinely unavailable judgement and leave missing arms visibly missing.') @@ -1716,10 +1716,10 @@ function New-PackageReadme { [void]$builder.AppendLine() [void]$builder.AppendLine('## How to run') [void]$builder.AppendLine() - [void]$builder.AppendLine('1. Read `execution-profile.json`. If `runner` or `model` is missing, fail clearly instead of guessing. Resolve the selected package-local runner and run `describe`, then `preflight`, before any native worker is dispatched. Require the descriptor''s native delegation capability; do not use a parent sequential fallback.') + [void]$builder.AppendLine('1. Read `execution-profile.json`. If `runner` or `model` is missing, fail clearly instead of guessing. Resolve the selected package-local runner and run `describe`, then `preflight`, before any native worker is dispatched. Invoke `Assert-NativeWorkerDelegation`; conditional controls require terminal evidence, unsupported controls are incompatible, and no parent sequential fallback is allowed.') [void]$builder.AppendLine('2. Use the package-local orchestration helper to queue one worker per manifest arm. Delegate every arm to a fresh full-capability harness-native worker. Do not invoke the runner''s direct `execute` command from the parent or from the delegated worker, because it would add a second model execution. Each worker receives one arm only and no grading material or paired-arm data.') [void]$builder.AppendLine('3. Maintain up to the requested concurrency. If the harness refuses a new worker because its own capacity is full, leave that arm queued and dispatch it when capacity is released; do not hardcode a runner-specific maximum and do not count the rejection as an attempt.') - [void]$builder.AppendLine('4. After all delegated workers complete or fail, run `tools/eval-runners/bridge-manifest-results.ps1 -IterationDirectory -RequireComplete`. It reads the manifest-declared `run_manifest`, `execution_result`, and `result` paths for every arm and invokes the one-arm bridge with those exact paths. Only then read the grading key, grade with `tools/skill-creator/agents/grader.md`, and run `tools/generate-eval-report.ps1 -RequireComplete`.') + [void]$builder.AppendLine('4. After all delegated workers complete or fail, run `tools/eval-runners/bridge-manifest-results.ps1 -IterationDirectory -RequireComplete -RequireNativeDelegation`. It reads the manifest-declared `run_manifest`, `execution_result`, and `result` paths for every arm, validates native terminal evidence again, and invokes the one-arm bridge with those exact paths. Only then read the grading key, grade with `tools/skill-creator/agents/grader.md`, and run `tools/generate-eval-report.ps1 -RequireComplete`.') [void]$builder.AppendLine() [void]$builder.AppendLine('`RUN-THIS.prompt.md` is the external Eval Orchestrator handoff. It selects the package-local runner from the profile, delegates one native Eval Worker per blind arm, queues capacity rejections, bridges raw evidence into the existing result shape, reveals grading material only after execution, and invokes Anthropic skill-creator''s compatible aggregator and static viewer through the package adapter. It never executes an eval prompt in its own context.') [void]$builder.AppendLine() @@ -1751,7 +1751,7 @@ function New-PackageReadme { [void]$builder.AppendLine() [void]$builder.AppendLine('If the harness cannot write to the package machine, its final handoff should contain one paste-ready JSON array of completed result objects including grading, plus the report as a file artifact when supported. A prose-only recap is not sufficient.') [void]$builder.AppendLine() - [void]$builder.AppendLine('For transferred results without report artifacts, the repository-side fallback is `pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -CollectResults `, which validates the files and invokes the same packaged skill-creator aggregator and viewer.') + [void]$builder.AppendLine('For transferred results without report artifacts, the repository-side fallback is `pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -CollectResults `, which validates native terminal evidence for a real selected runner, the files, and invokes the same packaged skill-creator aggregator and viewer. The deterministic fake remains compatibility-only and is never accepted as native proof.') [void]$builder.AppendLine() [void]$builder.AppendLine('This workspace is temporary. Do not commit it to the repository unless someone explicitly asks for a checked-in example.') @@ -1883,7 +1883,20 @@ function Invoke-CollectMode { if (-not (Test-Path -LiteralPath $packageBridgePath -PathType Leaf)) { $errors.Add("Manifest-driven bridge is missing at '$packageBridgePath'.") } else { - $bridgeOutput = & pwsh -NoProfile -File $packageBridgePath -IterationDirectory $iterationDirectory 2>&1 + $requireNativeDelegation = $true + $profilePath = Join-Path $iterationDirectory ([string]$manifest.execution_profile) + if (Test-Path -LiteralPath $profilePath -PathType Leaf) { + $profileForCollection = [System.IO.File]::ReadAllText($profilePath, $utf8NoBom) | ConvertFrom-Json + if ([string]$profileForCollection.runner -eq 'fake') { + # The deterministic fake is a compatibility fixture, not a + # harness-native worker. Never treat its execute output as + # native delegation evidence. + $requireNativeDelegation = $false + } + } + $bridgeArguments = @('-NoProfile', '-File', $packageBridgePath, '-IterationDirectory', $iterationDirectory) + if ($requireNativeDelegation) { $bridgeArguments += '-RequireNativeDelegation' } + $bridgeOutput = & pwsh @bridgeArguments 2>&1 if ($LASTEXITCODE -ne 0) { $errors.Add("Manifest-driven execution-result bridge failed: $([string]::Join(' ', @($bridgeOutput)))") } else { From f0ce94be43b9d684b370e8072a3aea3cb6f1e2cc Mon Sep 17 00:00:00 2001 From: "aicia[bot]" Date: Mon, 24 Aug 2026 14:39:27 +0200 Subject: [PATCH 23/39] =?UTF-8?q?=E2=99=BB=EF=B8=8F=20refactor=20git-visua?= =?UTF-8?q?l-commits=20auto-approval=20guidance?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Update SKILL.md and evals.json to explicitly clarify that when the user provides 'yolo' or 'auto' in an explicit commit request, the skill should complete the workflow in the same turn after required checks pass, rather than awaiting additional approval or returning a pending plan. --- skills/git-visual-commits/SKILL.md | 6 +++++- skills/git-visual-commits/evals/evals.json | 4 +++- 2 files changed, 8 insertions(+), 2 deletions(-) diff --git a/skills/git-visual-commits/SKILL.md b/skills/git-visual-commits/SKILL.md index d8f4eb5..0700436 100644 --- a/skills/git-visual-commits/SKILL.md +++ b/skills/git-visual-commits/SKILL.md @@ -16,6 +16,7 @@ This skill drives the entire git commit workflow — reviewing changes, grouping - An explicit `git bot commit`, `git commit`, or `git our commit` phrase is an authoritative request to use this skill. Do not substitute a changelog, release-note, squash-summary, or generic commit workflow. - Interpret `Please do a git bot commit yolo` as `git bot commit` identity plus auto-approval for the full current worktree. `yolo` is not the commit message, and it does not request a changelog. +- That exact combination is also an instruction to complete the commit workflow in the current turn after the required checks pass. Treat the visible plan as status information, not as a request for another approval; do not end with a pending plan or ask whether to proceed. - Equivalent word order and punctuation, such as `git bot commit, yolo` or `yolo — do a git bot commit`, preserve the same routing when both the explicit commit command and modifier are present. - A competing skill may run only when the user also explicitly requests its distinct output, such as updating `CHANGELOG.md`, drafting release notes, or producing a squash summary. @@ -72,6 +73,7 @@ The validator enforces an emoji present in the bundled reference table, exactly - mixed-scope validation - deterministic subject validation - post-commit author verification +- In auto-approval mode, the user's `yolo` or `auto` is already the approval for this commit request. After the required checks pass, execute the commit command(s) in the same turn. Do not ask "Proceed?", "Should I commit?", or any equivalent confirmation question, and do not return a plan as if approval were still pending. Stop only for a concrete blocker such as a missing alias or failed validation, and report that blocker directly. If the user did **not** say `yolo` or `auto`, and session-level auto mode is not already enabled, do **not** run any commit command yet. You must stop after Step 4, present the plan, and wait for approval. @@ -403,7 +405,9 @@ If auto-approval is **not** active, Step 4 is a hard stop. Do not stage, do not Auto-committing: 🔧 build config → 🚚 rename auth to identity → ✅ identity tests → 💬 update changelog ``` -Even in auto-approval mode, surface the commit buckets explicitly before committing. Auto-approval removes the wait, not the planning step. +Even in auto-approval mode, surface the commit buckets explicitly before committing. Auto-approval removes the wait, not the planning step. + +The summary is status output, not a review request. Step 5 is mandatory in the same turn once its preconditions pass: never ask "Proceed with committing these groups?" (or an equivalent question), wait for a reply, or finish with a pending commit plan. If the user did not narrow scope, the plan you surface must account for the full worktree rather than an arbitrarily chosen subset. diff --git a/skills/git-visual-commits/evals/evals.json b/skills/git-visual-commits/evals/evals.json index 73d8b1e..6ab5043 100644 --- a/skills/git-visual-commits/evals/evals.json +++ b/skills/git-visual-commits/evals/evals.json @@ -273,11 +273,13 @@ { "id": 25, "prompt": "Please do a git bot commit yolo.", - "expected_output": "An unambiguous git-visual-commits invocation that uses bot identity and auto-approval for the full worktree without treating yolo as a message or routing to changelog work.", + "expected_output": "An unambiguous git-visual-commits invocation that uses bot identity and auto-approval for the full worktree, performs the required checks, and completes the commit workflow in the same turn without asking for another approval or treating yolo as a message or routing to changelog work.", "expectations": [ "Routes the request to git-visual-commits rather than git-keep-a-changelog, release notes, or a generic commit workflow", "Interprets git bot commit as an identity lock and executes the git bot commit command rather than regular git commit", "Interprets yolo as auto-approval for the commit workflow rather than as the commit subject or message", + "Treats yolo as explicit approval to complete the commit workflow in the same turn after required checks pass", + "Does not ask whether to proceed, wait for another approval, or return a pending commit plan after presenting the status summary", "Treats the full current worktree as scope because the user did not narrow it", "Does not replace bot identity with a human-authored commit plus a Co-authored-by trailer" ] From 0d54ec8fe49a486767f7344816c2c1815c67fed5 Mon Sep 17 00:00:00 2001 From: "aicia[bot]" Date: Mon, 24 Aug 2026 14:39:34 +0200 Subject: [PATCH 24/39] =?UTF-8?q?=E2=9C=85=20add=20assertions=20for=20git-?= =?UTF-8?q?visual-commits=20auto-approval?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add validation assertions to ensure the git-visual-commits SKILL.md contains required phrases about completing the commit workflow in the same turn and that the plan summary is status output rather than a review request. --- scripts/validate-skill-templates.ps1 | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/scripts/validate-skill-templates.ps1 b/scripts/validate-skill-templates.ps1 index 4961ac9..a7f62fc 100644 --- a/scripts/validate-skill-templates.ps1 +++ b/scripts/validate-skill-templates.ps1 @@ -2075,6 +2075,7 @@ Add-ValidationResult -Results $results -Name 'Git visual commits skill enforces Assert-Contains -Name 'git-visual-commits/SKILL.md' -Content $skill -Needle '### Invocation Routing Lock' Assert-Contains -Name 'git-visual-commits/SKILL.md' -Content $skill -Needle 'Interpret `Please do a git bot commit yolo` as `git bot commit` identity plus auto-approval for the full current worktree.' Assert-Contains -Name 'git-visual-commits/SKILL.md' -Content $skill -Needle '`yolo` is not the commit message, and it does not request a changelog.' + Assert-Contains -Name 'git-visual-commits/SKILL.md' -Content $skill -Needle 'complete the commit workflow in the current turn after the required checks pass' Assert-Contains -Name 'git-visual-commits/SKILL.md' -Content $skill -Needle '### Full-Skill Read and Subject Lock' Assert-Contains -Name 'git-visual-commits/SKILL.md' -Content $skill -Needle 'Before running any Git command or composing a subject, read this `SKILL.md` completely from the first line through EOF.' Assert-Contains -Name 'git-visual-commits/SKILL.md' -Content $skill -Needle 'If a tool truncates the file, continue from the first unread line until EOF before proceeding.' @@ -2095,6 +2096,7 @@ Add-ValidationResult -Results $results -Name 'Git visual commits skill enforces Assert-Contains -Name 'git-visual-commits/SKILL.md' -Content $skill -Needle '### Recovery Safety Rule' Assert-Contains -Name 'git-visual-commits/SKILL.md' -Content $skill -Needle 'Prefer non-destructive recovery first: targeted unstaging, precise re-staging, or `git stash`' Assert-Contains -Name 'git-visual-commits/SKILL.md' -Content $skill -Needle '`yolo` / `auto` skips user confirmation only.' + Assert-Contains -Name 'git-visual-commits/SKILL.md' -Content $skill -Needle 'the user''s `yolo` or `auto` is already the approval for this commit request' Assert-Contains -Name 'git-visual-commits/SKILL.md' -Content $skill -Needle 'If the user did **not** say `yolo` or `auto`, and session-level auto mode is not already enabled, do **not** run any commit command yet.' Assert-Contains -Name 'git-visual-commits/SKILL.md' -Content $skill -Needle '### Commit Language Lock' Assert-Contains -Name 'git-visual-commits/SKILL.md' -Content $skill -Needle 'Resolve that path from this skill''s own bundled `references/` directory or installed skill folder first.' @@ -2115,6 +2117,7 @@ Add-ValidationResult -Results $results -Name 'Git visual commits skill enforces Assert-NotContains -Name 'git-visual-commits/SKILL.md' -Content $skill -Needle '### Allowed Prefixes' Assert-NotContains -Name 'git-visual-commits/SKILL.md' -Content $skill -Needle '### Emoji Selection' Assert-Contains -Name 'git-visual-commits/SKILL.md' -Content $skill -Needle 'Even in auto-approval mode, surface the commit buckets explicitly before committing.' + Assert-Contains -Name 'git-visual-commits/SKILL.md' -Content $skill -Needle 'The summary is status output, not a review request.' Assert-Contains -Name 'git-visual-commits/SKILL.md' -Content $skill -Needle 'Do not pass literal `\n` escape sequences and assume the shell will rewrite them.' Assert-Contains -Name 'git-visual-commits/SKILL.md' -Content $skill -Needle 'Prefer grammatical sentence and paragraph breaks over column-based hard wrapping.' Assert-Contains -Name 'git-visual-commits/SKILL.md' -Content $skill -Needle 'Then always run `git log -1 --format="%an <%ae>"` and verify that the author matches the requested identity mode before reporting success.' @@ -2186,6 +2189,8 @@ Add-ValidationResult -Results $results -Name 'Git visual commits skill enforces Assert-Contains -Name 'git-visual-commits/evals/evals.json' -Content $evals -Needle 'Triggers the single-category context quality gate because more than one file is being placed in one category' Assert-Contains -Name 'git-visual-commits/evals/evals.json' -Content $evals -Needle 'Recognizes exactly one changed file as the explicit exception and skips the single-category context quality gate' Assert-Contains -Name 'git-visual-commits/evals/evals.json' -Content $evals -Needle 'Please do a git bot commit yolo.' + Assert-Contains -Name 'git-visual-commits/evals/evals.json' -Content $evals -Needle 'Treats yolo as explicit approval to complete the commit workflow in the same turn after required checks pass' + Assert-Contains -Name 'git-visual-commits/evals/evals.json' -Content $evals -Needle 'Does not ask whether to proceed, wait for another approval, or return a pending commit plan after presenting the status summary' Assert-Contains -Name 'git-visual-commits/evals/evals.json' -Content $evals -Needle 'Does not replace bot identity with a human-authored commit plus a Co-authored-by trailer' Assert-Contains -Name 'README.md' -Content $readme -Needle '**Single-category context gate**' Assert-Contains -Name 'README.md' -Content $readme -Needle 'Multi-file plans that initially collapse to one category also require a visible full-context quality gate' From 1c3bc2ad670cec3f5c9b0111dd9750f1d4abd6da Mon Sep 17 00:00:00 2001 From: "aicia[bot]" Date: Mon, 24 Aug 2026 14:39:41 +0200 Subject: [PATCH 25/39] =?UTF-8?q?=F0=9F=93=9D=20clarify=20auto-approval=20?= =?UTF-8?q?in=20agent=20operation=20safeguards?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Update Git Operations Safeguards section to explicitly state that yolo or auto on an explicit commit request counts as approval to complete the commit workflow in the same turn, and clarifies that such approval must be attached to the same explicit commit request rather than treated as a standalone approval modifier. --- AGENTS.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 564c57a..38b8094 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -195,12 +195,12 @@ Never set or override `git user.name`, `git user.email`, or `alias.bot` in the * ## Git Operations Safeguards -Agents must never automatically commit code changes or push to remote repositories. Both actions require explicit user approval: +Agents must never commit code changes or push to remote repositories without explicit user approval. A direct commit request that includes `yolo` or `auto` is explicit approval for the current commit request; it authorizes the agent to complete that commit workflow in the same turn after the required checks pass. -- **Commits**: Always request confirmation from the user before staging and committing code. Present a clear summary of changes and wait for user approval before executing the commit. +- **Commits**: Request confirmation from the user before staging and committing code unless the same explicit commit request includes `yolo` or `auto`. In that auto-approved case, present the plan as status information and continue directly to staging and committing; do not ask a second confirmation question or end with a pending plan. Required review, scope, identity, message-validation, and post-commit checks still apply. - **Remote Operations**: Do not push, pull, fetch, or interact with `origin` or any remote repository without explicit user instruction. These operations modify repository history and can cause data loss if performed unexpectedly. -**Why:** Automatic commits can pollute history with incomplete work, debugging code, or unintended changes. Unexpected remote operations can overwrite or lose commits on shared branches. Always require the user to explicitly approve these operations. +**Why:** Automatic commits can pollute history with incomplete work, debugging code, or unintended changes. Unexpected remote operations can overwrite or lose commits on shared branches. Never treat silence, urgency, or momentum as approval; `yolo` or `auto` counts as approval only when attached to the same explicit commit request. ### Commit Skill Routing From 3da4977b823184b676e71dd572c60a199a12cdc6 Mon Sep 17 00:00:00 2001 From: Eval Worker Date: Mon, 24 Aug 2026 15:20:36 +0200 Subject: [PATCH 26/39] =?UTF-8?q?=E2=99=BB=EF=B8=8F=20clarify=20native=20d?= =?UTF-8?q?elegated=20worker=20protocol=20in=20eval=20orchestrator=20guida?= =?UTF-8?q?nce?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- scripts/validate-skill-templates.ps1 | 11 ++++------- 1 file changed, 4 insertions(+), 7 deletions(-) diff --git a/scripts/validate-skill-templates.ps1 b/scripts/validate-skill-templates.ps1 index a7f62fc..1b0ab35 100644 --- a/scripts/validate-skill-templates.ps1 +++ b/scripts/validate-skill-templates.ps1 @@ -1448,16 +1448,13 @@ Add-ValidationResult -Results $results -Name 'Skill evaluation prepares portable 'START NOW. You are the external Eval Orchestrator', 'Do not execute evaluation prompts in the current agent context.', 'execution-profile.json` selects the runner/model/configuration', - 'invoke the runner exactly once with `execute`', - 'never receive or inspect expected output, assertions, grading, paired output, benchmark data, or human feedback', - 'Never fall back to the old generic isolated-worker behavior', - 'execution-result.json', + 'The `execute` command remains part of the one-arm runner protocol for compatibility and conformance, but the native delegated worker path MUST NOT invoke it', + 'Do not read any `eval-metadata.json`, expected output, assertions, result grading, benchmark/report data, or paired output during Phase 1.', + 'Do not substitute a generic worker, another runner, or an improvised isolation scheme if the selected runner is unavailable or incompatible.', '`runs..run_manifest`', '`runs..execution_result`', '`runs..result`', - '`runPath = exact run_manifest`', - '`executionPath = exact execution_result`', - '`resultPath = exact result`', + 'The bridge''s one-arm operation is conceptually `-Run runPath -ExecutionResult executionPath -Result resultPath`, where all three values are the exact strings read from `manifest.json`.', 'Do not derive, normalize, rename, hyphenate, underscore, or otherwise reconstruct any run, execution-result, or result path.', 'bridge-manifest-results.ps1', '-RequireComplete', From 05dc60ceb80337758ddaed45aabaa360c8453636 Mon Sep 17 00:00:00 2001 From: "aicia[bot]" Date: Mon, 24 Aug 2026 23:26:06 +0200 Subject: [PATCH 27/39] =?UTF-8?q?=F0=9F=93=9D=20clarify=20eval=20orchestra?= =?UTF-8?q?tion=20requirements=20and=20Codex=20model=20defaults?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Update AGENTS.md and CONTRIBUTING.md to document the parallel orchestration model and Codex model selection. Clarify that independent eval arms must execute concurrently when harness capacity permits, and establish gpt-5.6-luna with medium reasoning effort as the Codex default configuration. --- AGENTS.md | 4 ++-- CONTRIBUTING.md | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 38b8094..7eb4767 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -46,7 +46,7 @@ Where `skill-creator` says to spawn with-skill and baseline subagents in the sam `eval `, `evaluate `, `eval this skill`, `prepare evals for `, and `evaluate using the existing evals` are all requests for this workflow. Treat them as instructions to prepare the package, never to run it, and never as a request to write new eval cases unless the user asks for that too. -Resolve the execution configuration before running the package preparation script. In an interactive agent session, offer Codebelt Reference first (`GitHub Copilot CLI` + `claude-haiku-4.5`) and verify that model through `scripts/Get-HarnessModels.ps1`; if it is unavailable, show the current discovered Copilot models and ask for a replacement. For manual selection, ask for Harness, discover current models for that harness with `scripts/Get-HarnessModels.ps1`, then pass the resulting runner/model pair to the preparation script. Cline and OpenCode discovery is free-only; GitHub Copilot and Codex discovery lists all currently available models. Never guess stale model ids, silently switch harnesses, or generate an execution-ready package with a null runner or model. +Resolve the execution configuration before running the package preparation script. In an interactive agent session, offer Codebelt Reference first (`GitHub Copilot CLI` + `claude-haiku-4.5`) and verify that model through `scripts/Get-HarnessModels.ps1`; if it is unavailable, show the current discovered Copilot models and ask for a replacement. If the user selects Codex, default to `-Model gpt-5.6-luna` and medium reasoning (low is also acceptable when explicitly chosen); verify the model through `scripts/Get-HarnessModels.ps1` before preparation. For manual selection, ask for Harness, discover current models for that harness with `scripts/Get-HarnessModels.ps1`, then pass the resulting runner/model pair to the preparation script. Cline and OpenCode discovery is free-only; GitHub Copilot and Codex discovery lists all currently available models. Never guess stale model ids, silently switch harnesses, or generate an execution-ready package with a null runner or model. ``` pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -Skill dotnet-test -Runner github-copilot -Model claude-haiku-4.5 @@ -140,7 +140,7 @@ The selected executor has two ordered phases. Its current context may read `RUN- The candidate instructions are already inlined in the with_skill run's `prompt.md` and staged under its `skill//` directory; the orchestrator does not load or summarize them for the worker. The baseline run has no `skill/` directory and no candidate instructions, and the orchestrator must not expose the candidate skill through another route, including a globally installed copy. The generated prompt files and the baseline `run.json` also omit the skill name, eval identifiers, and configuration labels so workers receive an ordinary task rather than an announcement that they are under evaluation. -Use the same model, model version, configuration, tools, and limits for every worker. Disable persistent memory and cross-session recall. Independent runs may execute concurrently when the selected harness and the user's token budget allow it, but every run still gets a distinct context and no shared mutable workspace. +Use the same model, model version, configuration, tools, and limits for every worker. Disable persistent memory and cross-session recall. Independent runs must execute concurrently up to `min(execution-profile.json.concurrency, remaining arms)` when harness capacity permits, but every run still gets a distinct context and no shared mutable workspace. For an OpenCode profile, one-at-a-time Task dispatch with available capacity is non-compliant; only an explicit harness capacity rejection justifies a serial effective run, and that limit must be recorded in orchestration state. `RUN-THIS.prompt.md` requires a selected Eval Runner that can create isolated workers or sessions. A plain single-context client can still execute an individual self-contained prompt when the user opens it directly as the first message of a fresh session, but it cannot provide the paired comparison and report contract in that same context. A selected runner that cannot satisfy a required guarantee is `incompatible`; there is no generic fallback or runner substitution. Partial package state may be inspected and reported, but the completion gate must pass before it can be presented as a completed evaluation; missing or unrun arms remain visibly incomplete. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index c9c926a..e918c6b 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -102,7 +102,7 @@ Before running the script, choose a Harness + Model. Use `scripts/Get-HarnessMod Repository preparation, validation, CI, hooks, deterministic tests, and automatic completion gates never run those prompts or invoke a model. That boundary is the Priority 1 rule in `AGENTS.md`, and preparing a prompt is not permission to execute one. A human-selected external Eval Orchestrator handed a specific package may invoke the selected package-local Eval Runner; this explicit handoff boundary does not weaken the repository prohibition or authorize CI/live evals. -Run both configurations on the same model, same version, and same configuration. A with-skill run on one model against a baseline on another measures the model as much as the skill and is not a skill-effectiveness result. +Run both configurations on the same model, same version, and same configuration. Independent arms must be dispatched concurrently up to `execution-profile.json.concurrency` when the harness permits it. For OpenCode, deliberate one-at-a-time Task dispatch is not compliant; serialize only after an explicit harness capacity rejection and preserve that evidence. A with-skill run on one model against a baseline on another measures the model as much as the skill and is not a skill-effectiveness result. Record each external runner result in the matching `execution-result.json` first. It must carry the protocol/schema, opaque run and fresh session ids, status, complete final response or explicit unavailability, runner/harness identity, requested and resolved model selection, timestamps and duration, exit/failure state, prompt/run/profile hashes, resolved isolation mechanisms, warnings, and artifact references. Include token, cache, cost, tool, command, file, and transcript evidence only when the harness exposes it; unavailable values remain explicitly unavailable and are never estimated. The deterministic bridge then writes the existing `results/*.result.json` shape, after which grading may add `grading[].passed` and evidence. Assertions about tool, shell, or file behavior are only gradeable from a run that captured that evidence. If the results were transferred without those report artifacts, validate and compare with: From 3341cb65b3fbf770f9656d9c08fdff01f7a8898e Mon Sep 17 00:00:00 2001 From: "aicia[bot]" Date: Mon, 24 Aug 2026 23:26:17 +0200 Subject: [PATCH 28/39] =?UTF-8?q?=E2=99=BB=EF=B8=8F=20implement=20parallel?= =?UTF-8?q?=20eval=20orchestration=20across=20runners?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Enhance eval orchestration to support concurrent worker dispatch. Update orchestration.ps1 with parallel execution coordination: track parallel_dispatch_required, minimum_parallel_workers, and capacity limits. Expand Codex runner (331 lines) with full implementation for gpt-5.6-luna and reasoning effort configuration. Update bridge-manifest-results.ps1 for new orchestration-plan schema fields. Add comprehensive runner conformance tests (133 lines) validating isolation, model selection, and parallel dispatch behavior. Update prepare-skill-evals.ps1 and validate-skill-templates.ps1 to support Codex preparation and validation. Update eval-runners/README.md documentation. --- scripts/eval-runners/README.md | 6 +- .../eval-runners/bridge-manifest-results.ps1 | 21 +- scripts/eval-runners/codex/runner.ps1 | 331 ++++++++++++++++-- .../contracts/orchestration-plan.schema.json | 4 + scripts/eval-runners/opencode/runner.ps1 | 8 +- scripts/eval-runners/orchestration.ps1 | 57 ++- .../eval-runners/tests/test-orchestration.ps1 | 20 +- .../tests/test-runner-conformance.ps1 | 133 ++++++- scripts/prepare-skill-evals.ps1 | 22 +- scripts/validate-skill-templates.ps1 | 10 + 10 files changed, 562 insertions(+), 50 deletions(-) diff --git a/scripts/eval-runners/README.md b/scripts/eval-runners/README.md index 4b8a2a2..5f8a2c2 100644 --- a/scripts/eval-runners/README.md +++ b/scripts/eval-runners/README.md @@ -24,7 +24,11 @@ model-spawning `execute` command or start another model session. package. It creates one worker envelope per exact manifest arm, keeps unrelated arms dependency-free, exposes at most the requested `execution-profile.json.concurrency` active slots, and leaves a capacity -rejection pending without incrementing the eval attempt count. It contains no +rejection pending without incrementing the eval attempt count. Independent arms +must use at least two active slots when the requested concurrency and harness +capacity permit it. `Assert-OrchestrationConcurrency` rejects a serial run that +has no explicit capacity-limit evidence; `bridge-manifest-results.ps1 +-RequireParallelDispatch` applies that gate before completion. It contains no harness-specific concurrency ceiling. `Assert-NativeWorkerDelegation` is the fail-closed handoff gate: an unavailable/unsupported native mechanism cannot fall back to parent execution, while a conditional mechanism may dispatch only diff --git a/scripts/eval-runners/bridge-manifest-results.ps1 b/scripts/eval-runners/bridge-manifest-results.ps1 index 80508fe..250965d 100644 --- a/scripts/eval-runners/bridge-manifest-results.ps1 +++ b/scripts/eval-runners/bridge-manifest-results.ps1 @@ -15,13 +15,18 @@ param( [switch]$RequireComplete, - [switch]$RequireNativeDelegation + [switch]$RequireNativeDelegation, + + [switch]$RequireParallelDispatch, + + [string]$OrchestrationStatePath = 'orchestration-state.json' ) $ErrorActionPreference = 'Stop' Set-StrictMode -Version Latest . (Join-Path $PSScriptRoot 'manifest-paths.ps1') +. (Join-Path $PSScriptRoot 'orchestration.ps1') try { $iterationPath = (Resolve-Path -LiteralPath $IterationDirectory -ErrorAction Stop).Path @@ -32,6 +37,19 @@ try { $manifest = Read-RunnerJson -Path $manifestPath $records = @(Get-ManifestRunRecords -IterationDirectory $iterationPath -Manifest $manifest) + $parallelDispatch = $null + if ($RequireParallelDispatch) { + Assert-SafeRelativePath -RelativePath $OrchestrationStatePath -FieldName 'orchestration state path' + $statePath = Join-Path $iterationPath ($OrchestrationStatePath -replace '/', [System.IO.Path]::DirectorySeparatorChar) + if (-not (Test-Path -LiteralPath $statePath -PathType Leaf)) { + throw "Parallel native-worker orchestration requires '$OrchestrationStatePath' at the iteration root." + } + + $profileData = Resolve-ExecutionProfile -ProfilePath (Join-Path $iterationPath 'execution-profile.json') + $plan = New-EvalOrchestrationPlan -IterationDirectory $iterationPath -Manifest $manifest -Profile $profileData.Profile + $state = Read-RunnerJson -Path $statePath + $parallelDispatch = Assert-OrchestrationConcurrency -Plan $plan -State $state + } $shadows = @(Get-ManifestShadowResultFiles -Records $records) if ($shadows.Count -gt 0) { $messages = @($shadows | ForEach-Object { @@ -111,6 +129,7 @@ try { terminal_execution_results = $validation.TerminalExecutionResults missing_execution_results = @($missing) complete = $validation.Complete + parallel_dispatch = $parallelDispatch warnings = @($validation.Warnings) }) -AsOutput } catch { diff --git a/scripts/eval-runners/codex/runner.ps1 b/scripts/eval-runners/codex/runner.ps1 index da4a205..465eddc 100644 --- a/scripts/eval-runners/codex/runner.ps1 +++ b/scripts/eval-runners/codex/runner.ps1 @@ -105,7 +105,7 @@ function Get-CodexAuthSource { } $authPath = Join-Path $codexHome 'auth.json' if (Test-Path -LiteralPath $authPath -PathType Leaf) { - return [pscustomobject]@{ Kind = 'file_unsupported'; Name = 'auth.json'; Path = (Resolve-Path -LiteralPath $authPath).Path } + return [pscustomobject]@{ Kind = 'subscription_file'; Name = 'auth.json'; Path = (Resolve-Path -LiteralPath $authPath).Path } } return [pscustomobject]@{ Kind = 'missing'; Name = $null; Path = $null } @@ -125,6 +125,263 @@ function Invoke-CodexCli { return Invoke-RunnerProcess -FileName $CommandInfo.FileName -ArgumentList $allArguments -WorkingDirectory $Inputs.Run.WorkingDirectoryPath -Environment $Environment -InputBytes $InputBytes -TimeoutSeconds $TimeoutSeconds } +function Invoke-CodexAppServer { + param( + [Parameter(Mandatory = $true)][object]$CommandInfo, + [Parameter(Mandatory = $true)][object]$Inputs, + [Parameter(Mandatory = $true)][object]$Auth, + [int]$TimeoutSeconds = 900 + ) + + if ($Auth.Kind -ne 'subscription_file') { + throw 'Codex app-server subscription transport requires auth.json authentication.' + } + + $start = [DateTime]::UtcNow + $deadline = $start.AddSeconds($TimeoutSeconds) + $psi = [System.Diagnostics.ProcessStartInfo]::new() + $psi.FileName = $CommandInfo.FileName + $psi.WorkingDirectory = $Inputs.Run.WorkingDirectoryPath + $psi.UseShellExecute = $false + $psi.CreateNoWindow = $true + $psi.RedirectStandardInput = $true + $psi.RedirectStandardOutput = $true + $psi.RedirectStandardError = $true + foreach ($argument in @($CommandInfo.Prefix) + @('app-server', '--stdio', '-c', 'shell_environment_policy.inherit=none')) { [void]$psi.ArgumentList.Add([string]$argument) } + $parentEnvironment = New-RunnerEnvironment -Run $Inputs.Run -Additional @{ CODEX_HOME = (Split-Path -Parent $Auth.Path) } + $psi.Environment.Clear() + foreach ($name in @($parentEnvironment.Keys)) { $psi.Environment[$name] = [string]$parentEnvironment[$name] } + + $process = [System.Diagnostics.Process]::new() + $process.StartInfo = $psi + $writer = $null + $reader = $null + $stderrTask = $null + $events = [System.Collections.Generic.List[string]]::new() + $normalized = [System.Collections.Generic.List[string]]::new() + $threadId = $null + $turnId = $null + $finalText = $null + $latestUsage = $null + $timedOut = $false + $transportFailure = $null + $turnCompleted = $false + $stderr = '' + $actualExitCode = $null + $processStarted = $false + + try { + if (-not $process.Start()) { throw 'Could not start Codex app-server.' } + $processStarted = $true + $writer = $process.StandardInput + $reader = $process.StandardOutput + $stderrTask = $process.StandardError.ReadToEndAsync() + + $writeMessage = { + param([Parameter(Mandatory = $true)][object]$Value) + $writer.WriteLine(($Value | ConvertTo-Json -Depth 50 -Compress)) + $writer.Flush() + } + $readMessage = { + $remaining = $deadline - [DateTime]::UtcNow + if ($remaining.TotalMilliseconds -le 0) { throw [TimeoutException]::new('Codex app-server timed out.') } + $readTask = $reader.ReadLineAsync() + $waitMilliseconds = [int][Math]::Min([int]::MaxValue, [Math]::Ceiling($remaining.TotalMilliseconds)) + if (-not $readTask.Wait($waitMilliseconds)) { throw [TimeoutException]::new('Codex app-server timed out.') } + $line = $readTask.GetAwaiter().GetResult() + if ($null -eq $line) { throw [EndOfStreamException]::new('Codex app-server closed stdout before turn completion.') } + $events.Add($line) + try { return ($line | ConvertFrom-Json -Depth 50) } catch { throw [FormatException]::new("Codex app-server emitted malformed JSON: $($_.Exception.Message)") } + } + $waitForResponse = { + param([Parameter(Mandatory = $true)][int]$ExpectedId, [Parameter(Mandatory = $true)][string]$Operation) + while ($true) { + $message = & $readMessage + $messageId = Get-JsonProperty -Object $message -Name 'id' -Default $null + $method = [string](Get-JsonProperty -Object $message -Name 'method' -Default '') + if (-not [string]::IsNullOrWhiteSpace($method) -and $null -ne $messageId) { + throw "Codex app-server requested unsupported interactive method '$method'." + } + if ($null -eq $messageId -or [int]$messageId -ne $ExpectedId) { continue } + $error = Get-JsonProperty -Object $message -Name 'error' -Default $null + if ($null -ne $error) { + $errorMessage = [string](Get-JsonProperty -Object $error -Name 'message' -Default ($error | ConvertTo-Json -Depth 20 -Compress)) + throw "Codex app-server $Operation failed: $errorMessage" + } + return $message + } + } + + $initializeRequest = 1 + & $writeMessage ([ordered]@{ + jsonrpc = '2.0' + id = $initializeRequest + method = 'initialize' + params = [ordered]@{ + clientInfo = [ordered]@{ name = 'codebelt-agentic-eval-runner'; title = 'Codebelt Eval Runner'; version = '0.9.1' } + capabilities = [ordered]@{ experimentalApi = $true } + } + }) + $null = & $waitForResponse $initializeRequest 'initialize' + & $writeMessage ([ordered]@{ jsonrpc = '2.0'; method = 'initialized' }) + + $threadRequest = 2 + & $writeMessage ([ordered]@{ + jsonrpc = '2.0' + id = $threadRequest + method = 'thread/start' + params = [ordered]@{ + model = $Inputs.Profile.Model + cwd = $Inputs.Run.WorkingDirectoryPath + approvalPolicy = 'never' + # thread/start can persist project trust when it begins in a + # writable sandbox. Keep the ephemeral thread read-only and + # apply the intended workspace-write policy to the turn only. + sandbox = 'readOnly' + ephemeral = $true + } + }) + $threadResponse = & $waitForResponse $threadRequest 'thread/start' + $threadId = [string]$threadResponse.result.thread.id + if ([string]::IsNullOrWhiteSpace($threadId)) { throw 'Codex app-server thread/start returned no thread id.' } + + $turnRequest = 3 + & $writeMessage ([ordered]@{ + jsonrpc = '2.0' + id = $turnRequest + method = 'turn/start' + params = [ordered]@{ + threadId = $threadId + input = @([ordered]@{ type = 'text'; text = [System.Text.Encoding]::UTF8.GetString($Inputs.Run.PromptBytes) }) + cwd = $Inputs.Run.WorkingDirectoryPath + model = $Inputs.Profile.Model + effort = $Inputs.Profile.ReasoningEffort + approvalPolicy = 'never' + sandboxPolicy = [ordered]@{ + type = 'workspaceWrite' + writableRoots = @($Inputs.Run.WorkingDirectoryPath) + networkAccess = $true + } + } + }) + $turnResponse = & $waitForResponse $turnRequest 'turn/start' + $turnId = [string]$turnResponse.result.turn.id + if ([string]::IsNullOrWhiteSpace($turnId)) { throw 'Codex app-server turn/start returned no turn id.' } + + while (-not $turnCompleted) { + $message = & $readMessage + $messageId = Get-JsonProperty -Object $message -Name 'id' -Default $null + $method = [string](Get-JsonProperty -Object $message -Name 'method' -Default '') + if (-not [string]::IsNullOrWhiteSpace($method) -and $null -ne $messageId) { + throw "Codex app-server requested unsupported interactive method '$method'." + } + switch ($method) { + 'thread/started' { + $normalized.Add(([ordered]@{ type = 'thread.started'; thread_id = $message.params.thread.id } | ConvertTo-Json -Compress)) + } + 'item/completed' { + $item = $message.params.item + $itemType = [string]$item.type + $normalizedType = switch ($itemType) { + 'agentMessage' { 'agent_message' } + 'commandExecution' { 'command_execution' } + 'fileChange' { 'file_change' } + 'mcpToolCall' { 'mcp_tool_call' } + default { $itemType } + } + $normalizedItem = [ordered]@{ type = $normalizedType; id = Get-JsonProperty -Object $item -Name 'id' -Default $null } + if ($itemType -eq 'agentMessage') { + $normalizedItem.text = [string]$item.text + $finalText = [string]$item.text + } elseif ($itemType -eq 'commandExecution') { + $normalizedItem.command = Get-JsonProperty -Object $item -Name 'command' -Default $null + $normalizedItem.exit_code = Get-JsonProperty -Object $item -Name 'exitCode' -Default $null + $normalizedItem.aggregated_output = Get-JsonProperty -Object $item -Name 'aggregatedOutput' -Default $null + } elseif ($itemType -eq 'fileChange') { + $normalizedItem.changes = Get-JsonProperty -Object $item -Name 'changes' -Default @() + } else { + $normalizedItem.raw = $item + } + $normalized.Add(([ordered]@{ type = 'item.completed'; item = $normalizedItem } | ConvertTo-Json -Depth 40 -Compress)) + } + 'thread/tokenUsage/updated' { + $latestUsage = $message.params.tokenUsage.last + } + 'turn/completed' { + $turn = $message.params.turn + if ([string]::IsNullOrWhiteSpace($finalText)) { + $turnItems = @($turn.items) + for ($itemIndex = $turnItems.Count - 1; $itemIndex -ge 0; $itemIndex--) { + if ([string]$turnItems[$itemIndex].type -eq 'agentMessage' -and -not [string]::IsNullOrWhiteSpace([string]$turnItems[$itemIndex].text)) { + $finalText = [string]$turnItems[$itemIndex].text + break + } + } + } + if ([string]$turn.status -eq 'failed') { + $errorMessage = [string](Get-JsonProperty -Object $turn.error -Name 'message' -Default 'Codex turn failed.') + $normalized.Add(([ordered]@{ type = 'turn.failed'; error = $errorMessage } | ConvertTo-Json -Compress)) + } elseif ([string]$turn.status -eq 'interrupted') { + $normalized.Add(([ordered]@{ type = 'turn.failed'; error = 'Codex turn was interrupted.' } | ConvertTo-Json -Compress)) + } else { + $usage = $null + if ($null -ne $latestUsage) { + $usage = [ordered]@{ + input_tokens = $latestUsage.inputTokens + cached_input_tokens = $latestUsage.cachedInputTokens + output_tokens = $latestUsage.outputTokens + reasoning_output_tokens = $latestUsage.reasoningOutputTokens + } + } + $normalized.Add(([ordered]@{ type = 'turn.completed'; usage = $usage } | ConvertTo-Json -Depth 20 -Compress)) + } + $turnCompleted = $true + } + 'error' { + $errorMessage = [string](Get-JsonProperty -Object $message.params -Name 'message' -Default 'Codex app-server emitted an error.') + throw $errorMessage + } + } + } + } catch [TimeoutException] { + $timedOut = $true + } catch { + $transportFailure = $_.Exception.Message + $normalized.Add(([ordered]@{ type = 'error'; message = $transportFailure } | ConvertTo-Json -Compress)) + } finally { + if ($null -ne $writer) { try { $writer.Close() } catch { } } + if ($processStarted) { + try { + if (-not $process.HasExited -and -not $process.WaitForExit(2000)) { + $process.Kill($true) + $process.WaitForExit() + } + if ($process.HasExited) { $actualExitCode = $process.ExitCode } + } catch { } + } + if ($null -ne $stderrTask) { + try { $stderr = $stderrTask.GetAwaiter().GetResult() } catch { $stderr = $_.Exception.Message } + } + $process.Dispose() + } + + $finish = [DateTime]::UtcNow + $exitCode = if ($timedOut) { $null } elseif ($turnCompleted -and $null -eq $transportFailure) { 0 } elseif ($null -ne $actualExitCode -and $actualExitCode -ne 0) { $actualExitCode } else { 1 } + return [pscustomobject]@{ + Stdout = [string]::Join("`n", $normalized) + RawStdout = [string]::Join("`n", $events) + Stderr = $stderr + ExitCode = $exitCode + TimedOut = $timedOut + StartedUtc = $start + FinishedUtc = $finish + DurationSeconds = [Math]::Round(($finish - $start).TotalSeconds, 3) + FinalText = $finalText + ThreadId = $threadId + TurnId = $turnId + } +} + function Get-CodexHelpResult { param( [Parameter(Mandatory = $true)][object]$CommandInfo, @@ -330,14 +587,17 @@ function Get-CodexPreflight { $auth = Get-CodexAuthSource if ($auth.Kind -eq 'missing') { - $reasons.Add('No narrow Codex provider API-key environment variable is available.') - } elseif ($auth.Kind -eq 'file_unsupported') { - $reasons.Add('Codex auth.json cannot be copied into the worker HOME: the evaluated agent could read that credential file. Set the provider API-key environment variable instead.') + $reasons.Add('Neither a narrow Codex provider API-key environment variable nor subscription auth.json is available.') + } elseif ($auth.Kind -eq 'subscription_file') { + $checks.Add((New-PreflightCheck -Name 'authentication' -Status passed -Detail 'Codex app-server can read the existing subscription auth.json through its parent-only CODEX_HOME. The file is not copied into the staged worker home.')) } else { $checks.Add((New-PreflightCheck -Name 'authentication' -Status passed -Detail "Authentication is available through the narrow $($auth.Name) environment variable; the child shell policy is set to inherit=none.")) } - if ($null -eq $sandboxName) { + if ($auth.Kind -eq 'subscription_file') { + $checks.Add((New-PreflightCheck -Name 'filesystem_confinement' -Status unavailable -Detail 'The subscription app-server parent must read the existing Codex credential home and is not wrapped by the external run-only sandbox. Codex workspace-write remains enabled for the turn.')) + $warnings.Add('Subscription execution uses pragmatic isolation. The adapter does not claim that an external filesystem sandbox protects the app-server transport.') + } elseif ($null -eq $sandboxName) { $checks.Add((New-PreflightCheck -Name 'filesystem_confinement' -Status not_applicable -Detail "Platform '$platform' has no configured external hard-confinement mechanism; pragmatic isolation remains available.")) $warnings.Add("Platform '$platform' has no external hard filesystem confinement in this adapter; execution will report pragmatic isolation.") } elseif ($null -eq $sandboxInfo) { @@ -347,19 +607,29 @@ function Get-CodexPreflight { $checks.Add((New-PreflightCheck -Name 'filesystem_confinement' -Status passed -Detail "External $sandboxName confines Codex to the staged run and required system runtime paths; Codex sandbox=workspace-write remains enabled inside it.")) } - $checks.Add((New-PreflightCheck -Name 'fresh_session' -Status passed -Detail 'The adapter uses --ephemeral and never supplies a resume, continue, or session identifier.')) - $checks.Add((New-PreflightCheck -Name 'ambient_configuration' -Status passed -Detail 'The adapter uses an isolated CODEX_HOME plus --ignore-user-config and --ignore-rules; unrelated inherited environment variables are removed.')) - $checks.Add((New-PreflightCheck -Name 'run_paths' -Status passed -Detail "--cd $($run.WorkingDirectoryPath); CODEX_HOME under $($run.HomeDirectoryPath)")) - $checks.Add((New-PreflightCheck -Name 'credential_boundary' -Status passed -Detail 'Only the selected provider API-key variable is passed to Codex; auth files are never copied into the worker HOME.')) + $checks.Add((New-PreflightCheck -Name 'fresh_session' -Status passed -Detail 'The selected transport starts an ephemeral thread and never supplies a resume, continue, or existing session identifier.')) + if ($auth.Kind -eq 'subscription_file') { + $checks.Add((New-PreflightCheck -Name 'ambient_configuration' -Status passed -Detail 'The app-server parent receives a filtered environment plus the existing CODEX_HOME needed for subscription auth. Child shell inheritance is disabled with shell_environment_policy.inherit=none.')) + $checks.Add((New-PreflightCheck -Name 'run_paths' -Status passed -Detail "thread/start and turn/start set cwd to $($run.WorkingDirectoryPath); HOME and USERPROFILE remain staged under $($run.HomeDirectoryPath).")) + $checks.Add((New-PreflightCheck -Name 'credential_boundary' -Status passed -Detail 'The adapter does not copy auth.json into the run or deliberately forward CODEX_HOME to child shell tools. This does not claim hard filesystem confinement where none is available.')) + } else { + $checks.Add((New-PreflightCheck -Name 'ambient_configuration' -Status passed -Detail 'The compatibility transport uses an isolated CODEX_HOME plus --ignore-user-config and --ignore-rules; unrelated inherited environment variables are removed.')) + $checks.Add((New-PreflightCheck -Name 'run_paths' -Status passed -Detail "--cd $($run.WorkingDirectoryPath); CODEX_HOME under $($run.HomeDirectoryPath)")) + $checks.Add((New-PreflightCheck -Name 'credential_boundary' -Status passed -Detail 'Only the selected provider API-key variable is passed to Codex; auth files are not copied into the worker HOME.')) + } - $hardConfinement = $null -ne $sandboxInfo -and $platform -in @('linux', 'macos') + $hardConfinement = $auth.Kind -eq 'environment' -and $null -ne $sandboxInfo -and $platform -in @('linux', 'macos') $capabilities = Get-CodexCapabilityMap -Inputs $Inputs -HardFilesystemConfinement $hardConfinement -NativeWorkerAvailable ($null -ne $nativeWorkerObservation -and $nativeWorkerObservation.Available) $harnessVersion = if ($null -eq $versionObservation) { 'unavailable' } else { [string]$versionObservation.Version } $descriptorCopy = [ordered]@{} foreach ($key in $descriptor.Keys) { $descriptorCopy[$key] = $descriptor[$key] } $descriptorCopy.harness = [ordered]@{ name = 'OpenAI Codex CLI'; version = $harnessVersion } $mechanisms = [System.Collections.Generic.List[string]]::new() - foreach ($mechanism in @('native app-server thread/start + turn/start child session', '--ask-for-approval never', 'codex exec --ephemeral compatibility transport', '--ignore-user-config', '--ignore-rules', '--sandbox workspace-write', 'shell_environment_policy.inherit=none', 'isolated CODEX_HOME', 'prompt on stdin', 'no session continuation')) { $mechanisms.Add($mechanism) } + if ($auth.Kind -eq 'subscription_file') { + foreach ($mechanism in @('native app-server initialize + thread/start + turn/start', 'parent-only subscription CODEX_HOME', 'ephemeral thread', 'approvalPolicy=never', 'sandboxPolicy=workspaceWrite', 'shell_environment_policy.inherit=none', 'filtered parent process environment', 'prompt in turn/start input', 'no session continuation')) { $mechanisms.Add($mechanism) } + } else { + foreach ($mechanism in @('--ask-for-approval never', 'codex exec --ephemeral compatibility transport', '--ignore-user-config', '--ignore-rules', '--sandbox workspace-write', 'shell_environment_policy.inherit=none', 'isolated CODEX_HOME', 'prompt on stdin', 'no session continuation')) { $mechanisms.Add($mechanism) } + } if ($hardConfinement) { $mechanisms.Add("external $sandboxName filesystem sandbox") } else { $mechanisms.Add('pragmatic process/environment isolation without hard filesystem confinement') } return New-PreflightDocument -Descriptor $descriptorCopy -Profile $profile -Run $run -Compatible ($reasons.Count -eq 0) -Checks @($checks) -Mechanisms @($mechanisms) -ResolvedCapabilities $capabilities -Warnings @($warnings) -Reasons @($reasons) } @@ -373,9 +643,7 @@ function New-CodexEnvironment { $codexHome = Join-Path $Inputs.Run.HomeDirectoryPath '.codex' New-Item -ItemType Directory -Path $codexHome -Force | Out-Null $environment = New-RunnerEnvironment -Run $Inputs.Run -AuthenticationVariables @(Get-ProviderAuthenticationVariables -Provider 'openai') -Additional @{ CODEX_HOME = $codexHome } - if ($Auth.Kind -ne 'environment') { - throw 'Codex execution requires a provider environment credential; file credentials are not safe to expose in the worker HOME.' - } + if ($Auth.Kind -eq 'environment') { return $environment } return $environment } @@ -493,16 +761,18 @@ function Invoke-CodexExecute { $commandInfo = Resolve-ExternalCommand -Name 'codex' $auth = Get-CodexAuthSource - $environment = New-CodexEnvironment -Inputs $Inputs -Auth $auth + $environment = if ($auth.Kind -eq 'environment') { New-CodexEnvironment -Inputs $Inputs -Auth $auth } else { $null } $lastResponsePath = 'evidence/codex-final.txt' New-Item -ItemType Directory -Path (Join-Path $Inputs.Run.RunRoot 'evidence') -Force | Out-Null $platform = Get-PlatformName $sandboxInfo = if ($platform -eq 'linux') { Resolve-SandboxCommand -Name 'bwrap' } elseif ($platform -eq 'macos') { Resolve-SandboxCommand -Name 'sandbox-exec' } else { $null } - $hardFilesystem = $null -ne $sandboxInfo -and $platform -in @('linux', 'macos') + $hardFilesystem = $auth.Kind -eq 'environment' -and $null -ne $sandboxInfo -and $platform -in @('linux', 'macos') $visiblePlatform = if ($hardFilesystem) { $platform } elseif ($platform -eq 'linux') { 'unknown' } else { $platform } $arguments = New-CodexCliArguments -Inputs $Inputs -LastResponsePath (Join-Path $Inputs.Run.RunRoot ($lastResponsePath -replace '/', [System.IO.Path]::DirectorySeparatorChar)) -VisiblePlatform $visiblePlatform - if ($platform -eq 'linux' -and $hardFilesystem) { + if ($auth.Kind -eq 'subscription_file') { + $process = Invoke-CodexAppServer -CommandInfo $commandInfo -Inputs $Inputs -Auth $auth -TimeoutSeconds $Inputs.Profile.TimeoutSeconds + } elseif ($platform -eq 'linux' -and $hardFilesystem) { $sandboxArguments = Get-LinuxCodexSandboxArguments -Inputs $Inputs -CommandInfo $commandInfo -Environment $environment $process = Invoke-RunnerProcess -FileName $sandboxInfo.FileName -ArgumentList (@($sandboxArguments) + @($arguments)) -WorkingDirectory $Inputs.Run.WorkingDirectoryPath -Environment $environment -InputBytes $Inputs.Run.PromptBytes -TimeoutSeconds $Inputs.Profile.TimeoutSeconds } elseif ($platform -eq 'macos' -and $hardFilesystem) { @@ -517,12 +787,19 @@ function Invoke-CodexExecute { $artifacts = [System.Collections.Generic.List[object]]::new() $artifacts.Add($stdoutArtifact) $artifacts.Add($stderrArtifact) + $transcriptArtifactPath = 'evidence/codex-events.jsonl' + if ($auth.Kind -eq 'subscription_file') { + $rawStdoutArtifact = Write-CodexCapture -RunData $Inputs -RelativePath 'evidence/codex-app-server-events.jsonl' -Text $process.RawStdout + $artifacts.Add($rawStdoutArtifact) + $transcriptArtifactPath = 'evidence/codex-app-server-events.jsonl' + } $parsed = ConvertFrom-JsonLines -Text $process.Stdout $warnings = [System.Collections.Generic.List[string]]::new() foreach ($parseError in @($parsed.Errors)) { $warnings.Add("Codex event parse error: $parseError") } - $finalText = $null - $threadId = $null + $finalText = if ($auth.Kind -eq 'subscription_file') { $process.FinalText } else { $null } + $threadId = if ($auth.Kind -eq 'subscription_file') { $process.ThreadId } else { $null } + $turnId = if ($auth.Kind -eq 'subscription_file') { $process.TurnId } else { $null } $turnFailure = $null $usage = $null $toolCalls = 0 @@ -549,7 +826,7 @@ function Invoke-CodexExecute { if ($itemType -eq 'command_execution') { $commands.Add([ordered]@{ type = $itemType; command = Get-JsonProperty -Object $item -Name 'command'; exit_code = Get-JsonProperty -Object $item -Name 'exit_code' }) } else { - $files.Add([ordered]@{ type = $itemType; item = $itemType }) + $files.Add([ordered]@{ type = $itemType; item = $item }) } } } @@ -581,7 +858,8 @@ function Invoke-CodexExecute { } elseif ($process.ExitCode -ne 0 -or $null -ne $turnFailure) { $status = 'failed' $reason = 'codex_failure' - $failure = New-ExecutionFailure -Code 'codex_failure' -Message ([string]$turnFailure) + $failureMessage = if ($null -ne $turnFailure) { [string]$turnFailure } elseif (-not [string]::IsNullOrWhiteSpace($process.Stderr)) { $process.Stderr.Trim() } else { "Codex exited with status $($process.ExitCode)." } + $failure = New-ExecutionFailure -Code 'codex_failure' -Message $failureMessage } elseif ([string]::IsNullOrWhiteSpace($finalText)) { $warnings.Add('Codex exited successfully without a final agent message.') $reason = 'codex_did_not_return_final_response' @@ -598,7 +876,7 @@ function Invoke-CodexExecute { if ($usageValue.Count -eq 0) { New-UnavailableMetric -Reason 'codex_usage_event_had_no_supported_buckets' } else { New-AvailableMetric -Value $usageValue } } $telemetry = [ordered]@{ - transcript = New-AvailableMetric -Value ([ordered]@{ artifact = 'evidence/codex-events.jsonl'; complete = $true }) + transcript = New-AvailableMetric -Value ([ordered]@{ artifact = $transcriptArtifactPath; complete = $true }) tokens = $tokenMetric tool_calls = New-AvailableMetric -Value $toolCalls cost = New-UnavailableMetric -Reason 'codex_runner_does_not_estimate_cost' @@ -607,7 +885,11 @@ function Invoke-CodexExecute { $sessionResultId = if ([string]::IsNullOrWhiteSpace($threadId)) { $sessionId } else { $threadId } $capabilities = Get-CodexCapabilityMap -Inputs $Inputs -HardFilesystemConfinement $hardFilesystem $mechanisms = [System.Collections.Generic.List[string]]::new() - foreach ($mechanism in @('--ask-for-approval never', 'codex exec --ephemeral', '--ignore-user-config', '--ignore-rules', '--sandbox workspace-write', 'shell_environment_policy.inherit=none', 'isolated CODEX_HOME', 'prompt on stdin', 'no session continuation')) { $mechanisms.Add($mechanism) } + if ($auth.Kind -eq 'subscription_file') { + foreach ($mechanism in @('native app-server initialize + thread/start + turn/start', 'parent-only subscription CODEX_HOME', 'ephemeral thread', 'approvalPolicy=never', 'sandboxPolicy=workspaceWrite', 'shell_environment_policy.inherit=none', 'filtered parent process environment', 'prompt in turn/start input', 'no session continuation')) { $mechanisms.Add($mechanism) } + } else { + foreach ($mechanism in @('--ask-for-approval never', 'codex exec --ephemeral', '--ignore-user-config', '--ignore-rules', '--sandbox workspace-write', 'shell_environment_policy.inherit=none', 'isolated CODEX_HOME', 'prompt on stdin', 'no session continuation')) { $mechanisms.Add($mechanism) } + } if ($hardFilesystem) { $mechanisms.Add("external $($sandboxInfo.Source) filesystem sandbox") } else { $mechanisms.Add('pragmatic process/environment isolation without hard filesystem confinement') } if (-not $hardFilesystem) { $warnings.Add('Hard filesystem confinement was unavailable; the completed arm is reported as pragmatic isolation.') } $sandboxEvidence = if (-not $hardFilesystem) { 'unavailable' } elseif ($platform -eq 'linux') { 'bwrap' } else { 'sandbox-exec' } @@ -618,7 +900,8 @@ function Invoke-CodexExecute { child_tool_visibility = 'codex_shell_environment_policy_inherit_none' value_observed = $false } - return New-ExecutionResult -Descriptor $executionDescriptor -Profile $Inputs.Profile -Run $Inputs.Run -Status $status -FinalResponse $finalText -FinalResponseReason $reason -StartedUtc $process.StartedUtc.ToString('o') -FinishedUtc $finished.ToString('o') -DurationSeconds $process.DurationSeconds -ExitStatus $exitStatus -Failure $failure -SessionId $sessionResultId -IsolationCapabilities $capabilities -IsolationMechanisms @($mechanisms) -ResolvedConfiguration ([ordered]@{ status = 'accepted_request'; reason = 'Codex accepted the requested model and configuration but did not expose concrete backend resolution.'; observations = [ordered]@{ model = $Inputs.Profile.Model; reasoning_effort = $Inputs.Profile.ReasoningEffort } }) -Telemetry $telemetry -Artifacts @($artifacts) -Warnings @($warnings) -Evidence ([ordered]@{ thread_id = $threadId; event_counts = $eventCounts; commands = @($commands); files = @($files); prompt_first_input = $true; resume = $false; stdout_exit_code = $process.ExitCode; sandbox = $sandboxEvidence; output_last_message_argument = (Get-SandboxVisiblePath -HostPath (Join-Path $Inputs.Run.RunRoot ($lastResponsePath -replace '/', [System.IO.Path]::DirectorySeparatorChar)) -RunRoot $Inputs.Run.RunRoot -Platform $visiblePlatform); credential = $credentialEvidence }) -AttemptCount 1 + $outputLastMessageArgument = if ($auth.Kind -eq 'subscription_file') { $null } else { Get-SandboxVisiblePath -HostPath (Join-Path $Inputs.Run.RunRoot ($lastResponsePath -replace '/', [System.IO.Path]::DirectorySeparatorChar)) -RunRoot $Inputs.Run.RunRoot -Platform $visiblePlatform } + return New-ExecutionResult -Descriptor $executionDescriptor -Profile $Inputs.Profile -Run $Inputs.Run -Status $status -FinalResponse $finalText -FinalResponseReason $reason -StartedUtc $process.StartedUtc.ToString('o') -FinishedUtc $finished.ToString('o') -DurationSeconds $process.DurationSeconds -ExitStatus $exitStatus -Failure $failure -SessionId $sessionResultId -IsolationCapabilities $capabilities -IsolationMechanisms @($mechanisms) -ResolvedConfiguration ([ordered]@{ status = 'accepted_request'; reason = 'Codex accepted the requested model and configuration but did not expose concrete backend resolution.'; observations = [ordered]@{ model = $Inputs.Profile.Model; reasoning_effort = $Inputs.Profile.ReasoningEffort } }) -Telemetry $telemetry -Artifacts @($artifacts) -Warnings @($warnings) -Evidence ([ordered]@{ thread_id = $threadId; turn_id = $turnId; event_counts = $eventCounts; commands = @($commands); files = @($files); prompt_first_input = $true; resume = $false; stdout_exit_code = $process.ExitCode; sandbox = $sandboxEvidence; output_last_message_argument = $outputLastMessageArgument; credential = $credentialEvidence }) -AttemptCount 1 } try { diff --git a/scripts/eval-runners/contracts/orchestration-plan.schema.json b/scripts/eval-runners/contracts/orchestration-plan.schema.json index 643052d..4004ee7 100644 --- a/scripts/eval-runners/contracts/orchestration-plan.schema.json +++ b/scripts/eval-runners/contracts/orchestration-plan.schema.json @@ -10,6 +10,8 @@ "runner", "model", "requested_concurrency", + "parallel_dispatch_required", + "minimum_parallel_workers", "native_worker_required", "parent_executes_arms", "nested_model_execution", @@ -21,6 +23,8 @@ "runner": { "type": "string", "minLength": 1 }, "model": { "type": "string", "minLength": 1 }, "requested_concurrency": { "type": "integer", "minimum": 1 }, + "parallel_dispatch_required": { "type": "boolean" }, + "minimum_parallel_workers": { "type": "integer", "minimum": 1 }, "native_worker_required": { "const": true }, "parent_executes_arms": { "const": false }, "nested_model_execution": { "const": false }, diff --git a/scripts/eval-runners/opencode/runner.ps1 b/scripts/eval-runners/opencode/runner.ps1 index 5540832..93580b5 100644 --- a/scripts/eval-runners/opencode/runner.ps1 +++ b/scripts/eval-runners/opencode/runner.ps1 @@ -213,6 +213,12 @@ function Get-OpenCodePreflight { } else { $checks.Add((New-PreflightCheck -Name 'model' -Status passed -Detail $profile.Model)) } + if ([int]$profile.Concurrency -lt 2) { + $checks.Add((New-PreflightCheck -Name 'parallel_dispatch' -Status failed -Detail 'OpenCode native-worker evaluations require at least two concurrent worker slots; a serial execution profile is not supported.')) + $reasons.Add('OpenCode native-worker evaluations require execution-profile.json concurrency >= 2. Sequential dispatch is incompatible unless the external harness reports a capacity limit during orchestration.') + } else { + $checks.Add((New-PreflightCheck -Name 'parallel_dispatch' -Status passed -Detail "OpenCode native-worker evaluations require bounded concurrent dispatch; requested slots=$($profile.Concurrency).")) + } if ($profile.ConfigurationProfile -ne 'isolated-default') { $reasons.Add("configuration_profile '$($profile.ConfigurationProfile)' is unsupported by opencode.") } @@ -293,7 +299,7 @@ function Get-OpenCodePreflight { foreach ($key in $descriptor.Keys) { $descriptorCopy[$key] = $descriptor[$key] } $descriptorCopy.harness = [ordered]@{ name = 'OpenCode CLI'; version = $harnessVersion } $mechanisms = [System.Collections.Generic.List[string]]::new() - foreach ($mechanism in @('native Task -> General full-capability child worker', 'opencode run --format json compatibility transport', '--auto', 'isolated OPENCODE_CONFIG_DIR', 'isolated OPENCODE_CONFIG', 'isolated HOME/XDG roots', 'repository-owned project configuration preserved', 'prompt on stdin', 'no session continuation')) { $mechanisms.Add($mechanism) } + foreach ($mechanism in @('native Task -> General full-capability child worker', 'bounded concurrent native-worker dispatch required', 'opencode run --format json compatibility transport', '--auto', 'isolated OPENCODE_CONFIG_DIR', 'isolated OPENCODE_CONFIG', 'isolated HOME/XDG roots', 'repository-owned project configuration preserved', 'prompt on stdin', 'no session continuation')) { $mechanisms.Add($mechanism) } if ($hardConfinement) { $mechanisms.Add("external $($sandboxInfo.Source) filesystem sandbox") } else { $mechanisms.Add('pragmatic process/environment isolation without hard filesystem confinement') } return New-PreflightDocument -Descriptor $descriptorCopy -Profile $profile -Run $run -Compatible ($reasons.Count -eq 0) -Checks @($checks) -Mechanisms @($mechanisms) -ResolvedCapabilities $capabilities -Warnings @($warnings) -Reasons @($reasons) } diff --git a/scripts/eval-runners/orchestration.ps1 b/scripts/eval-runners/orchestration.ps1 index 5c74574..9e87450 100644 --- a/scripts/eval-runners/orchestration.ps1 +++ b/scripts/eval-runners/orchestration.ps1 @@ -89,16 +89,19 @@ function New-EvalOrchestrationPlan { } $schemas = Get-RunnerSchemaNames + $parallelDispatchRequired = @($arms).Count -gt 1 -and $requestedConcurrency -gt 1 return [ordered]@{ schema = $schemas.OrchestrationPlan protocol_version = $schemas.Protocol runner = $runner model = $model requested_concurrency = $requestedConcurrency + parallel_dispatch_required = $parallelDispatchRequired + minimum_parallel_workers = if ($parallelDispatchRequired) { 2 } else { 1 } native_worker_required = $true parent_executes_arms = $false nested_model_execution = $false - dispatch_policy = 'one fresh harness-native worker per arm; independent workers may run concurrently up to requested_concurrency' + dispatch_policy = 'one fresh harness-native worker per arm; independent workers must run concurrently up to requested_concurrency when capacity permits' capacity_policy = 'harness_authoritative; a rejected delegation that did not start remains queued and is not an eval attempt' arms = $arms.ToArray() } @@ -124,10 +127,13 @@ function New-OrchestrationState { schema = 'codebeltnet/agentic/eval-orchestration-state/1' plan_schema = [string]$Plan.schema requested_concurrency = [int]$Plan.requested_concurrency + parallel_dispatch_required = [bool](Get-JsonProperty -Object $Plan -Name 'parallel_dispatch_required' -Default $false) + minimum_parallel_workers = [int](Get-JsonProperty -Object $Plan -Name 'minimum_parallel_workers' -Default 1) pending_worker_ids = @($pending) active = [ordered]@{} completed = [ordered]@{} delegation_rejections = [ordered]@{} + capacity_limit_reported = $false eval_attempts = [ordered]@{} max_observed_active = 0 } @@ -215,7 +221,8 @@ function Register-DelegationRejected { param( [Parameter(Mandatory = $true)][object]$State, [Parameter(Mandatory = $true)][string]$WorkerId, - [Parameter(Mandatory = $true)][string]$Reason + [Parameter(Mandatory = $true)][string]$Reason, + [switch]$CapacityLimited ) $pending = @($State.pending_worker_ids) @@ -228,11 +235,46 @@ function Register-DelegationRejected { count = $count + 1 last_reason = $Reason last_rejected_utc = [DateTime]::UtcNow.ToString('o') + capacity_limited = [bool]$CapacityLimited eval_attempt_started = $false } + if ($CapacityLimited) { + $State.capacity_limit_reported = $true + } return $true } +function Assert-OrchestrationConcurrency { + <# + The queue exposes concurrent slots; this completion gate proves that + the external orchestrator used them. A serial run is valid only when the + harness explicitly rejected additional workers for its own capacity. + #> + param( + [Parameter(Mandatory = $true)][object]$Plan, + [Parameter(Mandatory = $true)][object]$State + ) + + $requestedConcurrency = [int](Get-JsonProperty -Object $Plan -Name 'requested_concurrency' -Default 0) + $armCount = @((Get-JsonProperty -Object $Plan -Name 'arms' -Default @())).Count + $defaultRequired = if ($armCount -gt 1 -and $requestedConcurrency -gt 1) { 2 } else { 1 } + $required = [int](Get-JsonProperty -Object $Plan -Name 'minimum_parallel_workers' -Default $defaultRequired) + $maxObserved = [int](Get-JsonProperty -Object $State -Name 'max_observed_active' -Default 0) + $capacityReported = [bool](Get-JsonProperty -Object $State -Name 'capacity_limit_reported' -Default $false) + + if ($required -gt 1 -and $maxObserved -lt $required -and -not $capacityReported) { + throw "Native worker orchestration was serial: max_observed_active=$maxObserved, required_parallel_workers=$required, requested_concurrency=$requestedConcurrency. A serial dispatch is incompatible unless the harness records an explicit capacity rejection." + } + + return [ordered]@{ + status = if ($maxObserved -ge $required) { 'verified' } else { 'capacity_limited' } + requested_concurrency = $requestedConcurrency + required_parallel_workers = $required + max_observed_active = $maxObserved + capacity_limit_reported = $capacityReported + } +} + function Register-WorkerTerminal { param( [Parameter(Mandatory = $true)][object]$Plan, @@ -351,6 +393,17 @@ function Assert-OrchestrationPlanContract { if (-not [bool]$Plan.native_worker_required -or [bool]$Plan.parent_executes_arms -or [bool]$Plan.nested_model_execution) { throw 'Orchestration plan must require native workers and forbid parent or nested model execution.' } + $armCount = @($Plan.arms).Count + $requestedConcurrency = [int]$Plan.requested_concurrency + $expectedParallel = $armCount -gt 1 -and $requestedConcurrency -gt 1 + if ([bool](Get-JsonProperty -Object $Plan -Name 'parallel_dispatch_required' -Default $false) -ne $expectedParallel) { + throw 'Orchestration plan parallel_dispatch_required does not match its independent arm count and requested concurrency.' + } + $minimumParallelWorkers = [int](Get-JsonProperty -Object $Plan -Name 'minimum_parallel_workers' -Default 1) + $expectedMinimumParallelWorkers = if ($expectedParallel) { 2 } else { 1 } + if ($minimumParallelWorkers -ne $expectedMinimumParallelWorkers) { + throw 'Orchestration plan minimum_parallel_workers must require two workers whenever independent concurrent dispatch is requested.' + } $workerIds = [System.Collections.Generic.HashSet[string]]::new([System.StringComparer]::OrdinalIgnoreCase) foreach ($arm in @($Plan.arms)) { if (-not $workerIds.Add([string]$arm.worker_id)) { throw "Orchestration plan duplicates worker '$($arm.worker_id)'." } diff --git a/scripts/eval-runners/tests/test-orchestration.ps1 b/scripts/eval-runners/tests/test-orchestration.ps1 index 0f5a8d7..066f660 100644 --- a/scripts/eval-runners/tests/test-orchestration.ps1 +++ b/scripts/eval-runners/tests/test-orchestration.ps1 @@ -163,6 +163,8 @@ try { [void](Assert-OrchestrationPlanContract -Plan $serializedPlan) Assert-Equal 16 @($plan.arms).Count '8 eval cases fan out to 16 independent arms' Assert-Equal 16 $plan.requested_concurrency 'requested concurrency is preserved in the plan' + Assert-True ([bool]$plan.parallel_dispatch_required) 'independent arms require concurrent dispatch' + Assert-Equal 2 $plan.minimum_parallel_workers 'independent arms require at least two active workers' Assert-True ([bool]$plan.native_worker_required) 'native worker delegation is mandatory' Assert-True (-not [bool]$plan.parent_executes_arms) 'the parent is forbidden from executing arms' Assert-True (-not [bool]$plan.nested_model_execution) 'the plan forbids a nested model layer' @@ -198,7 +200,7 @@ try { [void](Register-DelegationAccepted -State $capacityState -WorkerId $dispatch.worker_id -WorkerSessionId ('session-' + $dispatch.worker_id)) $startedWorkers.Add([string]$dispatch.worker_id) } else { - [void](Register-DelegationRejected -State $capacityState -WorkerId $dispatch.worker_id -Reason 'fake harness capacity is four') + [void](Register-DelegationRejected -State $capacityState -WorkerId $dispatch.worker_id -Reason 'fake harness capacity is four' -CapacityLimited) $rejectedWorkers.Add([string]$dispatch.worker_id) $attemptsAtRejection[[string]$dispatch.worker_id] = $capacityState.eval_attempts.Contains([string]$dispatch.worker_id) } @@ -216,7 +218,7 @@ try { [void](Register-DelegationAccepted -State $capacityState -WorkerId $dispatch.worker_id -WorkerSessionId ('session-' + $dispatch.worker_id)) if ($startedWorkers -notcontains [string]$dispatch.worker_id) { $startedWorkers.Add([string]$dispatch.worker_id) } } else { - [void](Register-DelegationRejected -State $capacityState -WorkerId $dispatch.worker_id -Reason 'fake harness capacity is four') + [void](Register-DelegationRejected -State $capacityState -WorkerId $dispatch.worker_id -Reason 'fake harness capacity is four' -CapacityLimited) } $maxObservedByHarness = [Math]::Max($maxObservedByHarness, (Get-OrchestrationActiveCount -State $capacityState)) } @@ -259,6 +261,20 @@ try { Assert-Equal 4 $capacityState.max_observed_active 'state records the fake harness maximum of four active workers' Assert-Equal 4 $maxObservedByHarness 'the fake harness never exceeds four active workers' Assert-Equal 0 @($capacityState.pending_worker_ids).Count 'no arm remains queued after capacity is released' + [void](Assert-OrchestrationConcurrency -Plan $plan -State $capacityState) + + $serialState = New-OrchestrationState -Plan $plan + $serialState.max_observed_active = 1 + $serialState.pending_worker_ids = @() + $serialState.active = [ordered]@{} + $serialState.completed = [ordered]@{} + $serialRejected = $false + try { [void](Assert-OrchestrationConcurrency -Plan $plan -State $serialState) } catch { $serialRejected = $true } + Assert-True $serialRejected 'serial dispatch without capacity evidence fails the concurrency gate' + + $capacityStateJson = $capacityState | ConvertTo-Json -Depth 100 | ConvertFrom-Json + [void](Assert-OrchestrationConcurrency -Plan $plan -State $capacityStateJson) + Assert-True ([bool]$capacityStateJson.capacity_limit_reported) 'serialized fallback is permitted only with persisted capacity evidence' $badDescriptor = [pscustomobject]@{ name = 'fake-without-delegation' diff --git a/scripts/eval-runners/tests/test-runner-conformance.ps1 b/scripts/eval-runners/tests/test-runner-conformance.ps1 index 24dd7f1..655a095 100644 --- a/scripts/eval-runners/tests/test-runner-conformance.ps1 +++ b/scripts/eval-runners/tests/test-runner-conformance.ps1 @@ -132,6 +132,59 @@ if ($harness -eq 'codex' -and $arguments -contains 'features' -and $arguments -c Write-Output 'multi_agent stable true' exit 0 } +if ($harness -eq 'codex' -and $arguments -contains 'app-server') { + function Read-AppServerMessage { + $line = [Console]::In.ReadLine() + if ($null -eq $line) { throw 'recorded app-server reached EOF before the expected request' } + return ($line | ConvertFrom-Json -Depth 50) + } + function Write-AppServerMessage { + param([Parameter(Mandatory = $true)][object]$Value) + [Console]::Out.WriteLine(($Value | ConvertTo-Json -Depth 50 -Compress)) + [Console]::Out.Flush() + } + + $initialize = Read-AppServerMessage + Write-AppServerMessage ([ordered]@{ jsonrpc = '2.0'; id = $initialize.id; result = [ordered]@{ serverInfo = [ordered]@{ name = 'recorded-codex'; version = '9.1' } } }) + $initialized = Read-AppServerMessage + $threadStart = Read-AppServerMessage + Write-AppServerMessage ([ordered]@{ jsonrpc = '2.0'; id = $threadStart.id; result = [ordered]@{ thread = [ordered]@{ id = 'recorded-subscription-thread'; ephemeral = $true; path = $null } } }) + Write-AppServerMessage ([ordered]@{ jsonrpc = '2.0'; method = 'thread/started'; params = [ordered]@{ thread = [ordered]@{ id = 'recorded-subscription-thread' } } }) + $turnStart = Read-AppServerMessage + Write-AppServerMessage ([ordered]@{ jsonrpc = '2.0'; id = $turnStart.id; result = [ordered]@{ turn = [ordered]@{ id = 'recorded-subscription-turn'; status = 'inProgress'; items = @() } } }) + + $promptText = [string]$turnStart.params.input[0].text + $promptBytes = [Text.Encoding]::UTF8.GetBytes($promptText) + $expectedPromptPath = Join-Path (Split-Path -Parent (Get-Location).Path) 'prompt.md' + $expectedPromptBytes = [IO.File]::ReadAllBytes($expectedPromptPath) + $record.stdin_received = $promptBytes.Length -gt 0 + $record.stdin_delivery_count = if ($promptBytes.Length -gt 0) { 1 } else { 0 } + $record.stdin_byte_length = $promptBytes.Length + $record.stdin_sha256 = [Convert]::ToHexString(([Security.Cryptography.SHA256]::HashData($promptBytes))).ToLowerInvariant() + $record.stdin_expected_sha256 = [Convert]::ToHexString(([Security.Cryptography.SHA256]::HashData($expectedPromptBytes))).ToLowerInvariant() + $record.stdin_exact = $record.stdin_sha256 -eq $record.stdin_expected_sha256 + $record.stdin_utf8_round_trip = $record.stdin_exact + $record.worker_provider_visible = $false + $record.worker_copilot_token_visible = $false + $record.worker_gh_token_visible = $false + $record.worker_github_token_visible = $false + $record.worker_auth_file_visible = $false + $record.worker_global_secret_visible = $false + $record.worker_project_disable_visible = $false + $record.parent_codex_home = [Environment]::GetEnvironmentVariable('CODEX_HOME') + $record.parent_auth_file_visible = Test-Path -LiteralPath (Join-Path $record.parent_codex_home 'auth.json') -PathType Leaf + $record.rpc_methods = @($initialize.method, $initialized.method, $threadStart.method, $turnStart.method) + $record.thread_params = $threadStart.params + $record.turn_params = $turnStart.params + [IO.File]::AppendAllText($logPath, (($record | ConvertTo-Json -Depth 50 -Compress) + [Environment]::NewLine), [Text.UTF8Encoding]::new($false)) + + Write-AppServerMessage ([ordered]@{ jsonrpc = '2.0'; method = 'item/completed'; params = [ordered]@{ threadId = 'recorded-subscription-thread'; turnId = 'recorded-subscription-turn'; completedAtMs = 1; item = [ordered]@{ type = 'commandExecution'; id = 'command-1'; command = 'recorded command'; commandActions = @(); cwd = (Get-Location).Path; status = 'completed'; exitCode = 0; aggregatedOutput = 'recorded output' } } }) + Write-AppServerMessage ([ordered]@{ jsonrpc = '2.0'; method = 'item/completed'; params = [ordered]@{ threadId = 'recorded-subscription-thread'; turnId = 'recorded-subscription-turn'; completedAtMs = 2; item = [ordered]@{ type = 'fileChange'; id = 'file-1'; status = 'completed'; changes = @([ordered]@{ path = 'recorded.txt'; kind = [ordered]@{ type = 'add' } }) } } }) + Write-AppServerMessage ([ordered]@{ jsonrpc = '2.0'; method = 'item/completed'; params = [ordered]@{ threadId = 'recorded-subscription-thread'; turnId = 'recorded-subscription-turn'; completedAtMs = 3; item = [ordered]@{ type = 'agentMessage'; id = 'message-1'; text = 'recorded subscription response' } } }) + Write-AppServerMessage ([ordered]@{ jsonrpc = '2.0'; method = 'thread/tokenUsage/updated'; params = [ordered]@{ threadId = 'recorded-subscription-thread'; turnId = 'recorded-subscription-turn'; tokenUsage = [ordered]@{ total = [ordered]@{ inputTokens = 2; cachedInputTokens = 1; outputTokens = 3; reasoningOutputTokens = 1; totalTokens = 6 }; last = [ordered]@{ inputTokens = 2; cachedInputTokens = 1; outputTokens = 3; reasoningOutputTokens = 1; totalTokens = 6 } } } }) + Write-AppServerMessage ([ordered]@{ jsonrpc = '2.0'; method = 'turn/completed'; params = [ordered]@{ threadId = 'recorded-subscription-thread'; turn = [ordered]@{ id = 'recorded-subscription-turn'; status = 'completed'; items = @() } } }) + exit 0 +} if ($arguments -contains '--version') { $version = switch ($harness) { 'codex' { 'recorded-codex 9.1' } 'opencode' { 'recorded-opencode 9.2' } 'copilot' { 'GitHub Copilot CLI recorded-1.0.80' } default { 'recorded-cline 9.3' } } [IO.File]::AppendAllText($logPath, (($record | ConvertTo-Json -Compress) + [Environment]::NewLine), [Text.UTF8Encoding]::new($false)) @@ -300,7 +353,7 @@ exit 2 configuration_profile = 'isolated-default' tool_profile = 'default' timeout_seconds = 30 - concurrency = 1 + concurrency = if ($runnerName -eq 'opencode') { 2 } else { 1 } }) $recordedProfiles[$runnerName] = $profilePath } @@ -339,6 +392,10 @@ exit 2 Assert-True (@($preflightWith.mechanisms | Where-Object { $_ -eq '--allow-all-tools broad tool approval' }).Count -eq 1) 'Copilot preflight describes --allow-all-tools as broad tool approval' Assert-True (@($preflightWith.mechanisms | Where-Object { $_ -eq 'path and URL verification preserved (no --allow-all-paths/--allow-all-urls)' }).Count -eq 1) 'Copilot preflight records preserved path and URL verification' } + if ($runnerName -eq 'opencode') { + Assert-True (@($preflightWith.checks | Where-Object { $_.name -eq 'parallel_dispatch' -and $_.status -eq 'passed' }).Count -eq 1) 'OpenCode preflight requires bounded concurrent dispatch' + Assert-True (@($preflightWith.mechanisms | Where-Object { $_ -eq 'bounded concurrent native-worker dispatch required' }).Count -eq 1) 'OpenCode preflight records the concurrency requirement' + } if ($runnerName -in @('opencode', 'cline')) { Assert-True (@($preflightWith.warnings | Where-Object { $_ -match 'child-tool environment filter' }).Count -gt 0) "$runnerName reports the child credential-filter limitation" } @@ -474,6 +531,13 @@ exit 2 Assert-True (($resultWithout | ConvertTo-Json -Depth 100) -notmatch 'recorded-canary|recorded-unrelated-canary|recorded-copilot-canary|recorded-gh-canary|recorded-github-canary|recorded-gh-fallback-token') 'Copilot baseline result evidence does not contain credential values' } } + $serialOpenCodeProfile = Join-Path $recordedRoot 'opencode-serial-profile.json' + $serialOpenCodeData = Read-RunnerJson -Path $recordedProfiles['opencode'] + $serialOpenCodeData.concurrency = 1 + Write-TestJson -Path $serialOpenCodeProfile -Value $serialOpenCodeData + $serialOpenCodePreflight = Invoke-AdapterJson -RunnerPath (Join-Path $runnerRoot 'opencode\runner.ps1') -Command preflight -RunPath $with.Path -ProfilePath $serialOpenCodeProfile + Assert-Equal 'incompatible' $serialOpenCodePreflight.status 'OpenCode rejects a serial execution profile' + Assert-True (@($serialOpenCodePreflight.reasons | Where-Object { $_ -match 'concurrency >= 2|Sequential dispatch' }).Count -gt 0) 'OpenCode serial-profile failure explains the concurrency requirement' $env:CLINE_AGENTS_SQUAD_PLUGIN = $recordedOldClineAgentsSquad $staleCli = $fakeCli.Replace("'opencode' { '--format --dir --model --auto --pure --continue --session' }", "'opencode' { '--format --dir --model --pure --continue --session' }") [System.IO.File]::WriteAllText((Join-Path $fakeBin 'opencode.ps1'), $staleCli, [System.Text.UTF8Encoding]::new($false)) @@ -487,8 +551,35 @@ exit 2 $env:OPENAI_API_KEY = $null $env:CODEX_HOME = $fileAuthHome $fileAuthPreflight = Invoke-AdapterJson -RunnerPath (Join-Path $runnerRoot 'codex\runner.ps1') -Command preflight -RunPath $with.Path -ProfilePath $recordedProfiles['codex'] - Assert-Equal 'incompatible' $fileAuthPreflight.status 'Codex file-only authentication is fail-closed' - Assert-True (@($fileAuthPreflight.reasons | Where-Object { $_ -match 'auth\.json' }).Count -gt 0) 'Codex file-auth limitation is explicit' + Assert-Equal 'compatible' $fileAuthPreflight.status 'Codex subscription auth is accepted through app-server' + Assert-True (@($fileAuthPreflight.checks | Where-Object { $_.name -eq 'authentication' -and $_.status -eq 'passed' }).Count -eq 1) 'Codex subscription authentication is explicit in preflight' + $fileAuthResult = Invoke-AdapterJson -RunnerPath (Join-Path $runnerRoot 'codex\runner.ps1') -Command execute -RunPath $with.Path -ProfilePath $recordedProfiles['codex'] + Assert-Equal 'completed' $fileAuthResult.status 'Codex subscription app-server execution completes' + Assert-Equal 'recorded subscription response' $fileAuthResult.final_response.text 'Codex app-server captures the final agent message' + Assert-Equal 'recorded-subscription-thread' $fileAuthResult.session.id 'Codex app-server preserves thread identity' + Assert-Equal 'recorded-subscription-turn' $fileAuthResult.evidence.turn_id 'Codex app-server preserves turn identity' + Assert-Equal 'available' $fileAuthResult.telemetry.tokens.status 'Codex app-server maps token usage notifications' + Assert-Equal 2 ([int]$fileAuthResult.telemetry.tokens.value.input_tokens) 'Codex app-server maps input token usage' + Assert-Equal 3 ([int]$fileAuthResult.telemetry.tokens.value.output_tokens) 'Codex app-server maps output token usage' + Assert-Equal 2 ([int]$fileAuthResult.telemetry.tool_calls.value) 'Codex app-server counts command and file-change evidence' + Assert-Equal 1 @($fileAuthResult.evidence.commands).Count 'Codex app-server preserves command evidence' + Assert-Equal 1 @($fileAuthResult.evidence.files).Count 'Codex app-server preserves file-change evidence' + $subscriptionLogPath = Join-Path $with.Root 'repo\codex-fake-cli-log.jsonl' + $subscriptionRecord = Get-Content -LiteralPath $subscriptionLogPath | ForEach-Object { $_ | ConvertFrom-Json } | Where-Object { $_.PSObject.Properties.Name -contains 'rpc_methods' } | Select-Object -Last 1 + Assert-Equal 'initialize,initialized,thread/start,turn/start' ([string]::Join(',', @($subscriptionRecord.rpc_methods))) 'Codex app-server follows the required handshake order' + Assert-Equal 'gpt-5.6-luna' $subscriptionRecord.thread_params.model 'Codex app-server thread receives the requested model' + Assert-True ([bool]$subscriptionRecord.thread_params.ephemeral) 'Codex app-server thread is ephemeral' + Assert-Equal 'readOnly' $subscriptionRecord.thread_params.sandbox 'Codex app-server thread avoids persisting writable project trust' + Assert-Equal 'gpt-5.6-luna' $subscriptionRecord.turn_params.model 'Codex app-server turn receives the requested model' + Assert-Equal 'medium' $subscriptionRecord.turn_params.effort 'Codex app-server turn receives the requested reasoning effort' + Assert-Equal (Join-Path $with.Root 'repo') $subscriptionRecord.turn_params.cwd 'Codex app-server turn receives the staged working directory' + Assert-Equal 'never' $subscriptionRecord.turn_params.approvalPolicy 'Codex app-server turn rejects interactive approvals' + Assert-Equal 'workspaceWrite' $subscriptionRecord.turn_params.sandboxPolicy.type 'Codex app-server turn receives workspace-write sandbox policy' + Assert-Equal $fileAuthHome $subscriptionRecord.parent_codex_home 'Codex app-server parent receives the subscription CODEX_HOME' + Assert-True ([bool]$subscriptionRecord.parent_auth_file_visible) 'Codex app-server parent can read the subscription auth file' + Assert-True (-not [bool]$subscriptionRecord.unrelated_present) 'Codex app-server parent excludes unrelated inherited environment variables' + Assert-True (-not [bool]$subscriptionRecord.worker_auth_file_visible) 'Codex app-server worker fixture does not receive auth.json' + Assert-True (@($subscriptionRecord.args) -contains 'shell_environment_policy.inherit=none') 'Codex app-server disables child shell environment inheritance' $env:OPENAI_API_KEY = 'recorded-canary-not-logged' $env:CODEX_HOME = $recordedOldCodexHome # GitHub Copilot authentication: explicit env, OS-keychain, GitHub CLI, and @@ -875,8 +966,9 @@ try { Assert-True ($prepareText.Contains('One arm equals one delegated worker and one model-backed eval execution.')) 'handoff preparation must state the one-arm one-model invariant' Assert-True ($prepareText.Contains('Assert-NativeWorkerDelegation')) 'handoff preparation must invoke the native delegation gate' Assert-True ($prepareText.Contains('Require terminal evidence for the exact selected model, exact arm identity, working directory, isolated HOME/config boundary, prompt fidelity, terminal result capture')) 'handoff preparation must require worker control evidence' - Assert-True ($prepareText.Contains('-RequireComplete -RequireNativeDelegation')) 'handoff preparation must revalidate native terminal evidence during the manifest bridge' + Assert-True ($prepareText.Contains('-RequireComplete -RequireNativeDelegation -RequireParallelDispatch')) 'handoff preparation must revalidate native terminal and parallel-dispatch evidence during the manifest bridge' Assert-True ($prepareText.Contains('min(execution-profile.json.concurrency, remaining arms)')) 'handoff preparation must state requested concurrency fan-out' + Assert-True ($prepareText.Contains('Assert-OrchestrationConcurrency') -and $prepareText.Contains('orchestration-state.json')) 'handoff preparation must persist and validate orchestration concurrency state' Assert-True ($prepareText.Contains('rejected before the worker starts') -and $prepareText.Contains('record no eval attempt')) 'handoff preparation must queue capacity rejections without counting attempts' Assert-True ($prepareText.Contains('orchestration.ps1')) 'handoff preparation must load the deterministic orchestration helper' Assert-True ($prepareText -notmatch '') 'handoff preparation must not expose an unconstrained result-file placeholder' @@ -967,9 +1059,12 @@ try { }) } Write-TestJson -Path (Join-Path $manifestPackage 'manifest.json') -Value $manifest - Copy-Item -LiteralPath $profilePath -Destination (Join-Path $manifestPackage 'execution-profile.json') -Force - $manifestWithExecutionResult = Invoke-Fake -FakePath $fakePath -Command execute -Run $manifestWith.Path -Profile $profilePath - $manifestWithoutExecutionResult = Invoke-Fake -FakePath $fakePath -Command execute -Run $manifestWithout.Path -Profile $profilePath + $parallelProfilePath = Join-Path $manifestPackage 'execution-profile.json' + $parallelProfile = Read-RunnerJson -Path $profilePath + $parallelProfile.concurrency = 2 + Write-TestJson -Path $parallelProfilePath -Value $parallelProfile + $manifestWithExecutionResult = Invoke-Fake -FakePath $fakePath -Command execute -Run $manifestWith.Path -Profile $parallelProfilePath + $manifestWithoutExecutionResult = Invoke-Fake -FakePath $fakePath -Command execute -Run $manifestWithout.Path -Profile $parallelProfilePath Write-TestJson -Path $manifestWithExecution -Value $manifestWithExecutionResult Write-TestJson -Path $manifestWithoutExecution -Value $manifestWithoutExecutionResult @@ -995,7 +1090,27 @@ try { Assert-Equal 'unrun' $canonicalBeforeBridge.execution_status 'shadow result is never selected as the canonical result' Remove-Item -LiteralPath $shadowPath -Force - $manifestBridgeOutput = & pwsh -NoProfile -File $manifestBridgePath -IterationDirectory $manifestPackage -RequireComplete 2>&1 + $manifestStatePath = Join-Path $manifestPackage 'orchestration-state.json' + $manifestState = [ordered]@{ + schema = 'codebeltnet/agentic/eval-orchestration-state/1' + requested_concurrency = 2 + parallel_dispatch_required = $true + minimum_parallel_workers = 2 + capacity_limit_reported = $false + max_observed_active = 1 + pending_worker_ids = @() + active = [ordered]@{} + completed = [ordered]@{} + delegation_rejections = [ordered]@{} + eval_attempts = [ordered]@{ 'arm-1-with_skill' = 1; 'arm-1-without_skill' = 1 } + } + Write-TestJson -Path $manifestStatePath -Value $manifestState + $serialBridgeOutput = & pwsh -NoProfile -File $manifestBridgePath -IterationDirectory $manifestPackage -RequireComplete -RequireParallelDispatch 2>&1 + Assert-True ($LASTEXITCODE -ne 0) 'manifest bridge rejects serial orchestration without capacity evidence' + Assert-True (([string]::Join(' ', @($serialBridgeOutput))) -match 'serial|parallel') 'serial bridge rejection explains the concurrency requirement' + $manifestState.max_observed_active = 2 + Write-TestJson -Path $manifestStatePath -Value $manifestState + $manifestBridgeOutput = & pwsh -NoProfile -File $manifestBridgePath -IterationDirectory $manifestPackage -RequireComplete -RequireParallelDispatch 2>&1 if ($LASTEXITCODE -ne 0) { throw "manifest path bridge failed: $([string]::Join(' ', @($manifestBridgeOutput)))" } $canonicalWith = Get-Content -LiteralPath $manifestWithResult -Raw | ConvertFrom-Json Assert-Equal 'completed' $canonicalWith.execution_status 'manifest bridge populates the canonical hyphen result' @@ -1009,7 +1124,7 @@ try { $canonicalWith.grading[0].passed = $true $canonicalWith.grading[0].evidence = 'graded after the first bridge' Write-TestJson -Path $manifestWithResult -Value $canonicalWith - $repeatBridgeOutput = & pwsh -NoProfile -File $manifestBridgePath -IterationDirectory $manifestPackage -RequireComplete 2>&1 + $repeatBridgeOutput = & pwsh -NoProfile -File $manifestBridgePath -IterationDirectory $manifestPackage -RequireComplete -RequireParallelDispatch 2>&1 if ($LASTEXITCODE -ne 0) { throw "repeat manifest path bridge failed: $([string]::Join(' ', @($repeatBridgeOutput)))" } $canonicalAfterRepeat = Get-Content -LiteralPath $manifestWithResult -Raw | ConvertFrom-Json Assert-True ([bool]$canonicalAfterRepeat.grading[0].passed) 'repeat manifest bridge preserves completed grading' diff --git a/scripts/prepare-skill-evals.ps1 b/scripts/prepare-skill-evals.ps1 index fe89a8e..577fa1f 100644 --- a/scripts/prepare-skill-evals.ps1 +++ b/scripts/prepare-skill-evals.ps1 @@ -58,7 +58,8 @@ Optional deterministic catalog JSON used by the model discovery helper. Intended for tests and offline validation. .PARAMETER ReasoningEffort - Optional runner-supported reasoning/effort setting written to execution-profile.json. + Optional runner-supported reasoning/effort setting written to execution-profile.json. Codex defaults to medium + when this is omitted. .PARAMETER ConfigurationProfile Runner configuration profile. Defaults to isolated-default. @@ -711,7 +712,7 @@ function New-ExecutionProfile { schema = $executionProfileSchema runner = $ExecutionSelection.Runner model = $ExecutionSelection.Model - reasoning_effort = if ([string]::IsNullOrWhiteSpace($ReasoningEffort)) { $null } else { $ReasoningEffort } + reasoning_effort = if (-not [string]::IsNullOrWhiteSpace($ReasoningEffort)) { $ReasoningEffort } elseif ($ExecutionSelection.Runner -eq 'codex') { 'medium' } else { $null } configuration_profile = $ConfigurationProfile tool_profile = $ToolProfile timeout_seconds = $TimeoutSeconds @@ -1644,8 +1645,9 @@ function New-RunnerPrompt { [void]$builder.AppendLine('4. Before dispatching, require native worker delegation and all mandatory isolation controls. An unavailable or unsupported delegation mechanism is incompatible. A conditional mechanism is allowed only when the external orchestrator will require terminal evidence for the actual worker. Do not continue by invoking a runner process in the parent, and do not silently serialize arms in the parent.') [void]$builder.AppendLine('5. Dispatch each pending arm to one fresh harness-native full-capability worker. The worker must execute the prepared `prompt.md` as its first task from that arm''s staged run directory, with the selected model/configuration, exact working directory and isolated home. Require terminal evidence for the exact selected model, exact arm identity, working directory, isolated HOME/config boundary, prompt fidelity, terminal result capture, paired-arm/grading exclusion, fresh session, and absence of nested model execution; if any required fact is missing or mismatched, mark the arm incompatible without retrying through a fallback. It must not receive its paired arm, `eval-metadata.json`, expected output, assertions, grading, benchmark/report data, or any result from another arm.') [void]$builder.AppendLine('6. The delegated worker is the only model-backed execution for that arm. It must not invoke `runner.ps1 execute`, a second harness CLI, another model agent, or a nested session. Return the worker transcript/terminal evidence and normalized execution result to the parent; the parent writes it to the exact manifest-declared `execution_result` path without reconstructing any path.') - [void]$builder.AppendLine('7. Maintain up to `min(execution-profile.json.concurrency, remaining arms)` active delegated workers. If a delegation request is rejected before the worker starts because of harness capacity, leave that arm pending, record no eval attempt, and retry it after an active worker becomes terminal. Do not add a runner-specific ceiling or change the portable requested concurrency.') - [void]$builder.AppendLine('8. Preserve the complete terminal response, status, telemetry, evidence references, hashes, isolation mechanisms, warnings, and compatibility deviations. Do not retry for answer quality. A refusal is a result; timeout, harness failure, and incompatibility are results.') + [void]$builder.AppendLine('7. Maintain up to `min(execution-profile.json.concurrency, remaining arms)` active delegated workers. If a delegation request is rejected before the worker starts because of harness capacity, leave that arm pending, record no eval attempt, mark the rejection as capacity-limited in orchestration state, and retry it after an active worker becomes terminal. Do not add a runner-specific ceiling or change the portable requested concurrency.') + [void]$builder.AppendLine('8. Persist the orchestration state at `orchestration-state.json` as workers are accepted, capacity-rejected, and terminal. Before bridging, invoke `Assert-OrchestrationConcurrency`. If independent arms requested concurrency > 1 but `max_observed_active` is 1 without explicit capacity-limit evidence, fail the handoff; deliberate serial pacing is not a completed evaluation.') + [void]$builder.AppendLine('9. Preserve the complete terminal response, status, telemetry, evidence references, hashes, isolation mechanisms, warnings, and compatibility deviations. Do not retry for answer quality. A refusal is a result; timeout, harness failure, and incompatibility are results.') [void]$builder.AppendLine() [void]$builder.AppendLine('The package-local process surface is:') [void]$builder.AppendLine('```text') @@ -1660,8 +1662,8 @@ function New-RunnerPrompt { [void]$builder.AppendLine() [void]$builder.AppendLine('## Phase 2: bridge, grade, and report') [void]$builder.AppendLine() - [void]$builder.AppendLine('1. Only after every available delegated worker is terminal, invoke the deterministic package bridge below. It reads `manifest.json`, obtains each arm''s exact `run_manifest`, `execution_result`, and `result` paths. The bridge checks prompt/run/profile hashes and artifact confinement, validates the manifest paths, rejects unreferenced hyphen/underscore shadow results, and invokes the existing one-arm bridge with those exact paths. Do not manually construct a bridge command for an arm.') - [void]$builder.AppendLine((' `pwsh -NoProfile -File "' + $manifestBridgePath + '" -IterationDirectory "' + $IterationDirectory + '" -RequireComplete -RequireNativeDelegation`')) + [void]$builder.AppendLine('1. Only after every available delegated worker is terminal and `Assert-OrchestrationConcurrency` passes, invoke the deterministic package bridge below. It reads `manifest.json`, obtains each arm''s exact `run_manifest`, `execution_result`, and `result` paths. The bridge checks prompt/run/profile hashes and artifact confinement, validates the manifest paths, rejects unreferenced hyphen/underscore shadow results, verifies the persisted parallel-dispatch state, and invokes the existing one-arm bridge with those exact paths. Do not manually construct a bridge command for an arm.') + [void]$builder.AppendLine((' `pwsh -NoProfile -File "' + $manifestBridgePath + '" -IterationDirectory "' + $IterationDirectory + '" -RequireComplete -RequireNativeDelegation -RequireParallelDispatch`')) [void]$builder.AppendLine(' The bridge''s one-arm operation is conceptually `-Run runPath -ExecutionResult executionPath -Result resultPath`, where all three values are the exact strings read from `manifest.json`. Do not derive, normalize, rename, hyphenate, underscore, or otherwise reconstruct any of them.') [void]$builder.AppendLine('2. Only if the package bridge succeeds, read each eval''s `eval-metadata.json` and reveal `expected_output` and `assertions` to the Grader. Follow `tools/skill-creator/agents/grader.md`; grade deterministically first, then use optional model judgement only where deterministic evidence cannot decide. Never infer tool or file behavior from model self-report without process evidence.') [void]$builder.AppendLine('3. Write only `grading[].text`, `grading[].passed`, and `grading[].evidence` for grading. Do not alter raw execution results or replace the canonical result stubs. Use null for genuinely unavailable judgement and leave missing arms visibly missing.') @@ -1705,7 +1707,7 @@ function New-PackageReadme { [void]$builder.AppendLine(' +-- Eval Worker -> one eval arm') [void]$builder.AppendLine(' +-- ...') [void]$builder.AppendLine('```') - [void]$builder.AppendLine('The Eval Orchestrator coordinates and collects; it never executes an eval arm itself. One arm equals one delegated worker and one model-backed eval execution. Independent workers run concurrently up to `execution-profile.json.concurrency`; harness capacity is authoritative, so a rejected delegation stays queued and is not an attempt.') + [void]$builder.AppendLine('The Eval Orchestrator coordinates and collects; it never executes an eval arm itself. One arm equals one delegated worker and one model-backed eval execution. Independent workers must run concurrently up to `execution-profile.json.concurrency` when capacity permits; harness capacity is authoritative, so a rejected delegation stays queued and is not an attempt. A serial result without persisted capacity-limit evidence fails the completion gate.') [void]$builder.AppendLine('The package also carries the exact Anthropic skill-creator assets used after execution under `tools/skill-creator`: `tools/skill-creator/agents/grader.md`, `tools/skill-creator/agents/comparator.md`, `tools/skill-creator/agents/analyzer.md`, `tools/skill-creator/references/schemas.md`, `tools/skill-creator/scripts/aggregate_benchmark.py`, and `tools/skill-creator/eval-viewer/generate_review.py` plus `tools/skill-creator/eval-viewer/viewer.html`.') [void]$builder.AppendLine() [void]$builder.AppendLine('## Isolation model') @@ -1716,10 +1718,10 @@ function New-PackageReadme { [void]$builder.AppendLine() [void]$builder.AppendLine('## How to run') [void]$builder.AppendLine() - [void]$builder.AppendLine('1. Read `execution-profile.json`. If `runner` or `model` is missing, fail clearly instead of guessing. Resolve the selected package-local runner and run `describe`, then `preflight`, before any native worker is dispatched. Invoke `Assert-NativeWorkerDelegation`; conditional controls require terminal evidence, unsupported controls are incompatible, and no parent sequential fallback is allowed.') + [void]$builder.AppendLine('1. Read `execution-profile.json`. If `runner` or `model` is missing, fail clearly instead of guessing. Resolve the selected package-local runner and run `describe`, then `preflight`, before any native worker is dispatched. Invoke `Assert-NativeWorkerDelegation`; conditional controls require terminal evidence, unsupported controls are incompatible, and no parent sequential fallback is allowed. For OpenCode, a profile with concurrency below 2 is incompatible.') [void]$builder.AppendLine('2. Use the package-local orchestration helper to queue one worker per manifest arm. Delegate every arm to a fresh full-capability harness-native worker. Do not invoke the runner''s direct `execute` command from the parent or from the delegated worker, because it would add a second model execution. Each worker receives one arm only and no grading material or paired-arm data.') - [void]$builder.AppendLine('3. Maintain up to the requested concurrency. If the harness refuses a new worker because its own capacity is full, leave that arm queued and dispatch it when capacity is released; do not hardcode a runner-specific maximum and do not count the rejection as an attempt.') - [void]$builder.AppendLine('4. After all delegated workers complete or fail, run `tools/eval-runners/bridge-manifest-results.ps1 -IterationDirectory -RequireComplete -RequireNativeDelegation`. It reads the manifest-declared `run_manifest`, `execution_result`, and `result` paths for every arm, validates native terminal evidence again, and invokes the one-arm bridge with those exact paths. Only then read the grading key, grade with `tools/skill-creator/agents/grader.md`, and run `tools/generate-eval-report.ps1 -RequireComplete`.') + [void]$builder.AppendLine('3. Maintain up to the requested concurrency. If the harness refuses a new worker because its own capacity is full, leave that arm queued, mark the rejection as capacity-limited in `orchestration-state.json`, and dispatch it when capacity is released; do not hardcode a runner-specific maximum and do not count the rejection as an attempt.') + [void]$builder.AppendLine('4. After all delegated workers complete or fail, invoke `Assert-OrchestrationConcurrency`, persist `orchestration-state.json`, and run `tools/eval-runners/bridge-manifest-results.ps1 -IterationDirectory -RequireComplete -RequireNativeDelegation -RequireParallelDispatch`. It reads the manifest-declared `run_manifest`, `execution_result`, and `result` paths for every arm, validates native terminal evidence and parallel-dispatch evidence again, and invokes the one-arm bridge with those exact paths. Only then read the grading key, grade with `tools/skill-creator/agents/grader.md`, and run `tools/generate-eval-report.ps1 -RequireComplete`.') [void]$builder.AppendLine() [void]$builder.AppendLine('`RUN-THIS.prompt.md` is the external Eval Orchestrator handoff. It selects the package-local runner from the profile, delegates one native Eval Worker per blind arm, queues capacity rejections, bridges raw evidence into the existing result shape, reveals grading material only after execution, and invokes Anthropic skill-creator''s compatible aggregator and static viewer through the package adapter. It never executes an eval prompt in its own context.') [void]$builder.AppendLine() diff --git a/scripts/validate-skill-templates.ps1 b/scripts/validate-skill-templates.ps1 index 1b0ab35..4d6ffd9 100644 --- a/scripts/validate-skill-templates.ps1 +++ b/scripts/validate-skill-templates.ps1 @@ -1388,6 +1388,16 @@ Add-ValidationResult -Results $results -Name 'Skill evaluation prepares portable throw 'Codebelt Reference preparation must write github-copilot + claude-haiku-4.5 atomically.' } + $codexPackageRoot = Join-Path $packageRoot 'codex-package' + $codexPrepareOutput = & pwsh -NoProfile -File $scriptPath -Skill 'dotnet-strong-name-signing' -Eval 1 -OutputRoot $codexPackageRoot -Runner 'codex' -Model 'gpt-5.6-luna' 2>&1 + if ($LASTEXITCODE -ne 0) { + throw "prepare-skill-evals.ps1 failed for the Codex default fixture: $($codexPrepareOutput -join [Environment]::NewLine)" + } + $codexProfile = [System.IO.File]::ReadAllText((Join-Path $codexPackageRoot 'iteration-1\execution-profile.json'), $utf8NoBom) | ConvertFrom-Json + if ([string]$codexProfile.runner -ne 'codex' -or [string]$codexProfile.model -ne 'gpt-5.6-luna' -or [string]$codexProfile.reasoning_effort -ne 'medium') { + throw 'Codex preparation must preserve gpt-5.6-luna and default omitted reasoning effort to medium.' + } + $missingSelectionRoot = Join-Path ([System.IO.Path]::GetTempPath()) ('agentic-eval-missing-selection-' + [Guid]::NewGuid().ToString('N')) $missingSelectionOutput = & pwsh -NoProfile -File $scriptPath -Skill 'dotnet-strong-name-signing' -OutputRoot $missingSelectionRoot 2>&1 if ($LASTEXITCODE -eq 0) { From e07fc2860fd508ede53f6800035773753c89ed24 Mon Sep 17 00:00:00 2001 From: "aicia[bot]" Date: Tue, 25 Aug 2026 01:09:13 +0200 Subject: [PATCH 29/39] =?UTF-8?q?=E2=99=BB=EF=B8=8F=20harden=20parallel=20?= =?UTF-8?q?evaluation=20orchestration?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Improve runner descriptor handling, native worker delegation validation, grading reset logic on model/run ID mismatches, and terminal artifact capture validation. Add execution-run ID tracking to prevent stale grading state. Improve prepare-skill-evals prompt generation with better orchestration requirements for OpenCode native Task dispatch. Add conformance tests for orchestration state management and runner compliance. --- .../eval-runners/bridge-execution-result.ps1 | 116 ++++++++++++++++-- .../eval-runners/bridge-manifest-results.ps1 | 34 +---- .../contracts/execution-result.schema.json | 24 ++++ scripts/eval-runners/orchestration.ps1 | 16 +-- scripts/eval-runners/runner-common.ps1 | 54 +++++++- .../eval-runners/tests/test-orchestration.ps1 | 24 ++++ .../tests/test-runner-conformance.ps1 | 35 ++++++ scripts/generate-eval-report.ps1 | 8 +- scripts/prepare-skill-evals.ps1 | 64 +++++++--- scripts/validate-skill-templates.ps1 | 20 +-- 10 files changed, 314 insertions(+), 81 deletions(-) diff --git a/scripts/eval-runners/bridge-execution-result.ps1 b/scripts/eval-runners/bridge-execution-result.ps1 index 8323d0d..ca98ded 100644 --- a/scripts/eval-runners/bridge-execution-result.ps1 +++ b/scripts/eval-runners/bridge-execution-result.ps1 @@ -95,7 +95,10 @@ function Get-ResultRelativeArtifactPath { } function Get-ExistingGrading { - param([Parameter(Mandatory = $true)][string]$ResultPath) + param( + [Parameter(Mandatory = $true)][string]$ResultPath, + [string]$ExecutionRunId = '' + ) if (-not (Test-Path -LiteralPath $ResultPath -PathType Leaf)) { throw "Manifest-declared result stub '$ResultPath' does not exist; the bridge will not create a new grading-less result file." @@ -105,7 +108,63 @@ function Get-ExistingGrading { if (-not (Test-JsonProperty -Object $existing -Name 'grading')) { throw "Manifest-declared result stub '$ResultPath' is missing its grading array." } - return @(Get-JsonProperty -Object $existing -Name 'grading' -Default @()) + $grading = @(Get-JsonProperty -Object $existing -Name 'grading' -Default @()) + $existingRunId = [string](Get-JsonProperty -Object $existing -Name 'execution_run_id' -Default '') + if (-not [string]::IsNullOrWhiteSpace($ExecutionRunId) -and + -not [string]::IsNullOrWhiteSpace($existingRunId) -and + $existingRunId -ne $ExecutionRunId) { + return @($grading | ForEach-Object { + $reset = $_ | ConvertTo-Json -Depth 100 | ConvertFrom-Json + $reset.passed = $null + $reset.evidence = '' + $reset + }) + } + return $grading +} + +function Get-PackageRunnerDescriptor { + param([Parameter(Mandatory = $true)][string]$RunnerName) + + if ($RunnerName -notmatch '^[a-z0-9-]+$') { + throw "execution-profile.json runner '$RunnerName' is not a valid package runner name." + } + $runnerPath = Join-Path (Join-Path $PSScriptRoot $RunnerName) 'runner.ps1' + if (-not (Test-Path -LiteralPath $runnerPath -PathType Leaf)) { + throw "Package-local runner '$RunnerName' is missing its runner.ps1 descriptor." + } + + $descriptorOutput = & pwsh -NoProfile -File $runnerPath describe 2>&1 + if ($LASTEXITCODE -ne 0) { + throw "Package-local runner '$RunnerName' descriptor failed: $([string]::Join(' ', @($descriptorOutput)))" + } + try { + $descriptor = [string]::Join([Environment]::NewLine, @($descriptorOutput)) | ConvertFrom-Json + [void](Assert-RunnerDescriptor -Descriptor $descriptor) + } catch { + throw "Package-local runner '$RunnerName' returned an invalid descriptor: $($_.Exception.Message)" + } + if ([string]$descriptor.name -ne $RunnerName) { + throw "Package-local runner descriptor name '$($descriptor.name)' does not match selected runner '$RunnerName'." + } + return $descriptor +} + +function Assert-NativeTerminalCaptureArtifact { + param([Parameter(Mandatory = $true)][object]$ExecutionResult) + + $transcriptMetric = Get-JsonProperty -Object $ExecutionResult.telemetry -Name 'transcript' -Default $null + $transcriptStatus = [string](Get-JsonProperty -Object $transcriptMetric -Name 'status' -Default '') + $transcriptArtifact = [string](Get-JsonProperty -Object (Get-JsonProperty -Object $transcriptMetric -Name 'value' -Default $null) -Name 'artifact' -Default '') + if ($transcriptStatus -ne 'available' -or [string]::IsNullOrWhiteSpace($transcriptArtifact)) { + throw 'Native worker execution must provide an available terminal transcript artifact.' + } + $matchingArtifacts = @($ExecutionResult.artifacts | Where-Object { + [string](Get-JsonProperty -Object $_ -Name 'path' -Default '') -eq $transcriptArtifact + }) + if ($matchingArtifacts.Count -ne 1) { + throw "Native worker transcript artifact '$transcriptArtifact' is not recorded exactly once in execution-result.json." + } } try { @@ -115,6 +174,7 @@ try { $evalDirectory = Split-Path -Parent $runDirectory $iterationDirectory = Split-Path -Parent $evalDirectory $executionPath = (Resolve-Path -LiteralPath $ExecutionResult -ErrorAction Stop).Path + $executionResultHash = Get-Sha256HexFromFile -Path $executionPath $resultPath = [System.IO.Path]::GetFullPath($Result, (Get-Location).Path) if (-not (Test-PathInside -BasePath $iterationDirectory -CandidatePath $executionPath)) { throw 'execution-result.json must remain inside the prepared iteration package.' @@ -139,8 +199,9 @@ try { if ([string]$raw.input.profile_sha256 -ne $profile.Hash) { throw 'execution-result input.profile_sha256 does not match execution-profile.json.' } - if ($RequireNativeDelegation -and [string]$raw.status -ne 'incompatible') { - [void](Assert-NativeWorkerTerminalEvidence -ExecutionEvidence $raw -Run $runData -RequestedModel ([string]$profile.Profile.Model)) + $rawRunnerName = [string](Get-JsonProperty -Object $raw.runner -Name 'name' -Default '') + if ($rawRunnerName -ne [string]$profile.Profile.runner) { + throw "execution-result runner '$rawRunnerName' does not match selected runner '$($profile.Profile.runner)'." } if ([int]$raw.run.eval_id -ne $runData.EvalId -or [string]$raw.run.configuration -ne $runData.Mode) { throw 'execution-result run identity does not match run.json.' @@ -160,6 +221,20 @@ try { $artifactPaths.Add((Get-ResultRelativeArtifactPath -EvalDirectory $evalDirectory -FullPath $full)) } + if ($RequireNativeDelegation) { + if ([string]$raw.status -eq 'incompatible') { + throw 'An incompatible native-worker arm is diagnostic only and cannot be bridged into a gradeable canonical result.' + } + $runnerDescriptor = Get-PackageRunnerDescriptor -RunnerName ([string]$profile.Profile.runner) + [void](Assert-NativeWorkerTerminalEvidence ` + -ExecutionEvidence $raw ` + -Run $runData ` + -RequestedModel ([string]$profile.Profile.Model) ` + -ExpectedRunner ([string]$profile.Profile.runner) ` + -ExpectedMechanism ([string]$runnerDescriptor.delegation.mechanism)) + Assert-NativeTerminalCaptureArtifact -ExecutionResult $raw + } + $finalStatus = [string]$raw.final_response.status $output = if ($finalStatus -eq 'available') { [string]$raw.final_response.text } else { '' } $transcript = '' @@ -183,7 +258,7 @@ try { if ($finalStatus -eq 'unavailable') { $notes.Add("final_response_unavailable=$($raw.final_response.reason)") } foreach ($warning in $warnings) { if (-not [string]::IsNullOrWhiteSpace([string]$warning)) { $notes.Add([string]$warning) } } - $existingGrading = @(Get-ExistingGrading -ResultPath $resultPath) + $existingGrading = @(Get-ExistingGrading -ResultPath $resultPath -ExecutionRunId ([string]$raw.run_id)) $caps = Get-JsonProperty -Object $raw.isolation -Name 'capabilities' -Default ([ordered]@{}) $requestedModel = [string](Get-JsonProperty -Object $raw.requested -Name 'model' -Default '') $resolvedModelValue = Get-JsonProperty -Object $raw.resolved -Name 'model' -Default $null @@ -237,11 +312,32 @@ try { candidate_skill_exposed = Get-CapabilityBoolean (Get-JsonProperty -Object $caps -Name 'candidate_skill_exposure' -Default $null) transcript_captured = if ($null -ne $transcriptMetricObject) { [string](Get-JsonProperty -Object $transcriptMetricObject -Name 'status' -Default '') -eq 'available' } else { $null } } - execution_status = [string]$raw.status - execution_run_id = [string]$raw.run_id - execution_result_file = [System.IO.Path]::GetRelativePath($evalDirectory, $executionPath).Replace('\', '/') - grading = @($existingGrading) - notes = [string]::Join("`n", @($notes)) + execution_status = [string]$raw.status + execution_run_id = [string]$raw.run_id + execution_result_file = [System.IO.Path]::GetRelativePath($evalDirectory, $executionPath).Replace('\', '/') + execution_result_sha256 = $executionResultHash + grading = @($existingGrading) + notes = [string]::Join("`n", @($notes)) + } + + # Re-bridging the same immutable raw result must not erase grading or + # telemetry that an external evaluator has already attached. A changed + # raw hash takes the fresh-result path above and resets stale grading. + $existingCanonical = Read-RunnerJson -Path $resultPath + $existingRawHash = [string](Get-JsonProperty -Object $existingCanonical -Name 'execution_result_sha256' -Default '') + if ($existingRawHash -eq $executionResultHash) { + foreach ($field in @( + 'model', 'requested_model', 'resolved_model', 'configuration_resolution_status', + 'configuration_resolution_reason', 'harness', 'executed_utc', 'output', 'output_files', + 'transcript', 'shell_commands', 'files_read', 'files_written', 'stdout', 'stderr', + 'exit_status', 'duration_seconds', 'total_tokens', 'tool_calls', 'turns', + 'base_input_tokens', 'output_tokens', 'cache_read_tokens', 'cache_write_tokens', + 'cache_write_1h_tokens', 'estimated_cost_usd', 'model_effort', 'isolation', 'grading', 'notes' + )) { + if (Test-JsonProperty -Object $existingCanonical -Name $field) { + $portableResult[$field] = Get-JsonProperty -Object $existingCanonical -Name $field + } + } } Write-BridgeJson -Path $resultPath -Value $portableResult Write-RunnerJson -Value ([ordered]@{ schema = 'codebeltnet/agentic/eval-result-bridge/1'; result = [System.IO.Path]::GetRelativePath($iterationDirectory, $resultPath).Replace('\', '/'); execution_status = $raw.status }) -AsOutput diff --git a/scripts/eval-runners/bridge-manifest-results.ps1 b/scripts/eval-runners/bridge-manifest-results.ps1 index 250965d..6f4deb8 100644 --- a/scripts/eval-runners/bridge-manifest-results.ps1 +++ b/scripts/eval-runners/bridge-manifest-results.ps1 @@ -71,36 +71,10 @@ try { continue } - $alreadyBridged = $false - try { - $existingResult = Read-RunnerJson -Path $record.ResultPath - $rawResult = Read-RunnerJson -Path $record.ExecutionResultPath - $expectedExecutionFile = [System.IO.Path]::GetRelativePath($record.EvalDirectory, $record.ExecutionResultPath).Replace('\', '/') - $nativeEvidenceReady = -not $RequireNativeDelegation -or [string](Get-JsonProperty -Object $rawResult -Name 'status' -Default '') -eq 'incompatible' - if ($RequireNativeDelegation -and -not $nativeEvidenceReady) { - try { - $runData = Resolve-RunContract -RunPath $record.RunManifestPath - $profileData = Resolve-ExecutionProfile -ProfilePath (Join-Path $iterationPath 'execution-profile.json') - [void](Assert-NativeWorkerTerminalEvidence -ExecutionEvidence $rawResult -Run $runData -RequestedModel ([string]$profileData.Profile.Model)) - $nativeEvidenceReady = $true - } catch { - $nativeEvidenceReady = $false - } - } - $alreadyBridged = $nativeEvidenceReady -and - @('completed', 'failed', 'timed_out', 'cancelled', 'incompatible') -contains [string](Get-JsonProperty -Object $rawResult -Name 'status' -Default '') -and - [string](Get-JsonProperty -Object $existingResult -Name 'execution_status' -Default '') -eq [string](Get-JsonProperty -Object $rawResult -Name 'status' -Default '') -and - [string](Get-JsonProperty -Object $existingResult -Name 'execution_result_file' -Default '') -eq $expectedExecutionFile -and - -not [string]::IsNullOrWhiteSpace([string](Get-JsonProperty -Object $existingResult -Name 'execution_run_id' -Default '')) - } catch { - $alreadyBridged = $false - } - - if ($alreadyBridged) { - $bridged.Add("$($record.EvalName)/$($record.Configuration)") - continue - } - + # Always invoke the one-arm bridge. The raw result is the authoritative + # terminal evidence; a status/path match alone cannot prove that the + # canonical result reflects the current raw file. The one-arm bridge + # preserves existing grading while revalidating hashes and provenance. $bridgeOutput = & pwsh -NoProfile -File $oneArmBridge ` -Run $record.RunManifestPath ` -ExecutionResult $record.ExecutionResultPath ` diff --git a/scripts/eval-runners/contracts/execution-result.schema.json b/scripts/eval-runners/contracts/execution-result.schema.json index 9689ad8..832e0d5 100644 --- a/scripts/eval-runners/contracts/execution-result.schema.json +++ b/scripts/eval-runners/contracts/execution-result.schema.json @@ -52,6 +52,22 @@ "reason": { "type": "string" } } }, + "runner": { + "type": "object", + "required": ["name", "version"], + "properties": { + "name": { "type": "string", "minLength": 1 }, + "version": { "type": "string", "minLength": 1 } + } + }, + "harness": { + "type": "object", + "required": ["name", "version"], + "properties": { + "name": { "type": "string", "minLength": 1 }, + "version": { "type": "string", "minLength": 1 } + } + }, "requested": { "type": "object", "additionalProperties": false, @@ -97,6 +113,14 @@ } }, "duration_seconds": { "type": "number", "minimum": 0 }, + "exit": { + "type": "object", + "required": ["status"], + "properties": { + "status": { "type": ["integer", "null"] }, + "failure": {} + } + }, "input": { "type": "object", "required": ["prompt_sha256", "run_json_sha256", "profile_sha256"] diff --git a/scripts/eval-runners/orchestration.ps1 b/scripts/eval-runners/orchestration.ps1 index 9e87450..7726820 100644 --- a/scripts/eval-runners/orchestration.ps1 +++ b/scripts/eval-runners/orchestration.ps1 @@ -191,15 +191,15 @@ function Register-DelegationAccepted { [string]$WorkerSessionId = '' ) - $pending = [System.Collections.Generic.List[string]]::new() - foreach ($id in @($State.pending_worker_ids)) { [void]$pending.Add([string]$id) } - if (-not $pending.Contains($WorkerId)) { - throw "Worker '$WorkerId' cannot be accepted because it is not pending." - } $active = Get-OrchestrationDictionary -Object $State -Name 'active' $completed = Get-OrchestrationDictionary -Object $State -Name 'completed' if ($active.Contains($WorkerId) -or $completed.Contains($WorkerId)) { - throw "Worker '$WorkerId' was already accepted or completed." + throw "Worker '$WorkerId' was already accepted or completed; delegation acceptance is exactly-once and must not be retried." + } + $pending = [System.Collections.Generic.List[string]]::new() + foreach ($id in @($State.pending_worker_ids)) { [void]$pending.Add([string]$id) } + if (-not $pending.Contains($WorkerId)) { + throw "Worker '$WorkerId' cannot be accepted because it is not pending; preserve the orchestration state and do not register a second attempt." } [void]$pending.Remove($WorkerId) @@ -285,7 +285,9 @@ function Register-WorkerTerminal { $active = Get-OrchestrationDictionary -Object $State -Name 'active' if (-not $active.Contains($WorkerId)) { - throw "Worker '$WorkerId' cannot become terminal because it is not active." + $completed = Get-OrchestrationDictionary -Object $State -Name 'completed' + $stateDescription = if ($completed.Contains($WorkerId)) { 'already terminal' } else { 'not accepted' } + throw "Worker '$WorkerId' cannot become terminal because it is $stateDescription; terminal registration is exactly-once and must not be retried." } $status = [string](Get-JsonProperty -Object $ExecutionEvidence -Name 'status' -Default '') if ($status -notin @('completed', 'failed', 'timed_out', 'cancelled', 'incompatible')) { diff --git a/scripts/eval-runners/runner-common.ps1 b/scripts/eval-runners/runner-common.ps1 index 29aa20e..b635136 100644 --- a/scripts/eval-runners/runner-common.ps1 +++ b/scripts/eval-runners/runner-common.ps1 @@ -416,7 +416,9 @@ function Test-NativeWorkerTerminalEvidence { [Parameter(Mandatory = $true)][object]$ExecutionEvidence, [Parameter(Mandatory = $true)][object]$Run, [Parameter(Mandatory = $true)][string]$RequestedModel, - [string]$ExpectedWorkerSessionId = '' + [string]$ExpectedWorkerSessionId = '', + [string]$ExpectedRunner = '', + [string]$ExpectedMechanism = '' ) $failures = [System.Collections.Generic.List[string]]::new() @@ -440,6 +442,13 @@ function Test-NativeWorkerTerminalEvidence { $failures.Add('requested_model') } + if (-not [string]::IsNullOrWhiteSpace($ExpectedRunner)) { + $runnerEvidence = Get-JsonProperty -Object $ExecutionEvidence -Name 'runner' -Default $null + if ($null -eq $runnerEvidence -or [string](Get-JsonProperty -Object $runnerEvidence -Name 'name' -Default '') -ne $ExpectedRunner) { + $failures.Add('runner_identity') + } + } + $delegation = Get-JsonProperty -Object (Get-JsonProperty -Object $ExecutionEvidence -Name 'evidence' -Default $null) -Name 'delegation' -Default $null if ($null -eq $delegation) { $failures.Add('delegation_terminal_evidence') @@ -455,6 +464,10 @@ function Test-NativeWorkerTerminalEvidence { if ([string]::IsNullOrWhiteSpace([string](Get-JsonProperty -Object $delegation -Name 'mechanism' -Default ''))) { $failures.Add('mechanism') } + if (-not [string]::IsNullOrWhiteSpace($ExpectedMechanism) -and + [string](Get-JsonProperty -Object $delegation -Name 'mechanism' -Default '') -ne $ExpectedMechanism) { + $failures.Add('native_mechanism') + } $workerSessionId = [string](Get-JsonProperty -Object $delegation -Name 'worker_session_id' -Default '') if ([string]::IsNullOrWhiteSpace($workerSessionId)) { @@ -514,10 +527,18 @@ function Assert-NativeWorkerTerminalEvidence { [Parameter(Mandatory = $true)][object]$ExecutionEvidence, [Parameter(Mandatory = $true)][object]$Run, [Parameter(Mandatory = $true)][string]$RequestedModel, - [string]$ExpectedWorkerSessionId = '' + [string]$ExpectedWorkerSessionId = '', + [string]$ExpectedRunner = '', + [string]$ExpectedMechanism = '' ) - $validation = Test-NativeWorkerTerminalEvidence -ExecutionEvidence $ExecutionEvidence -Run $Run -RequestedModel $RequestedModel -ExpectedWorkerSessionId $ExpectedWorkerSessionId + $validation = Test-NativeWorkerTerminalEvidence ` + -ExecutionEvidence $ExecutionEvidence ` + -Run $Run ` + -RequestedModel $RequestedModel ` + -ExpectedWorkerSessionId $ExpectedWorkerSessionId ` + -ExpectedRunner $ExpectedRunner ` + -ExpectedMechanism $ExpectedMechanism if (-not $validation.Valid) { throw "Native worker terminal evidence is incompatible: $([string]::Join(', ', @($validation.Failures)))." } @@ -1026,6 +1047,19 @@ function Assert-ExecutionResult { throw "execution-result.json is missing '$field'." } } + foreach ($identityName in @('runner', 'harness')) { + $identity = Get-JsonProperty -Object $Result -Name $identityName -Default $null + foreach ($identityField in @('name', 'version')) { + if ($null -eq $identity -or -not (Test-JsonProperty -Object $identity -Name $identityField) -or + [string]::IsNullOrWhiteSpace([string](Get-JsonProperty -Object $identity -Name $identityField -Default ''))) { + throw "execution-result.json $identityName.$identityField must be non-empty." + } + } + } + $exitObject = Get-JsonProperty -Object $Result -Name 'exit' -Default $null + if ($null -eq $exitObject -or -not (Test-JsonProperty -Object $exitObject -Name 'status')) { + throw 'execution-result.json exit.status must be present and numeric or null.' + } if ([string]::IsNullOrWhiteSpace([string]$Result.run_id)) { throw 'execution-result.json run_id must be non-empty.' } @@ -1037,6 +1071,20 @@ function Assert-ExecutionResult { if ([int](Get-JsonProperty -Object $Result -Name 'attempt_count' -Default 0) -ne 1) { throw 'execution-result.json attempt_count must be exactly 1; quality retries are not allowed.' } + $exitStatus = Get-JsonProperty -Object $exitObject -Name 'status' -Default $null + if ($null -ne $exitStatus) { + $numericExitStatus = $exitStatus -is [byte] -or + $exitStatus -is [sbyte] -or + $exitStatus -is [int16] -or + $exitStatus -is [uint16] -or + $exitStatus -is [int32] -or + $exitStatus -is [uint32] -or + $exitStatus -is [int64] -or + $exitStatus -is [uint64] + if (-not $numericExitStatus) { + throw 'execution-result.json exit.status must be a JSON number or null; textual lifecycle labels are not valid exit codes.' + } + } $isolationStatus = [string](Get-JsonProperty -Object $Result.isolation -Name 'status' -Default '') $isolationLevel = [string](Get-JsonProperty -Object $Result.isolation -Name 'level' -Default '') $hardFilesystem = [bool](Get-JsonProperty -Object $Result.isolation -Name 'hard_filesystem_confinement' -Default $false) diff --git a/scripts/eval-runners/tests/test-orchestration.ps1 b/scripts/eval-runners/tests/test-orchestration.ps1 index 066f660..b972817 100644 --- a/scripts/eval-runners/tests/test-orchestration.ps1 +++ b/scripts/eval-runners/tests/test-orchestration.ps1 @@ -332,6 +332,30 @@ try { Assert-True ((Test-NativeWorkerTerminalEvidence -ExecutionEvidence $validTerminalEvidence -Run $terminalRunData -RequestedModel ([string]$terminalArm.worker.model) -ExpectedWorkerSessionId 'native-terminal-session').Valid) 'valid terminal native-worker evidence is accepted' Assert-True (Assert-NativeWorkerTerminalEvidence -ExecutionEvidence $validTerminalEvidence -Run $terminalRunData -RequestedModel ([string]$terminalArm.worker.model) -ExpectedWorkerSessionId 'native-terminal-session') 'valid terminal evidence passes the assert gate' + $exactOnceState = New-OrchestrationState -Plan ([pscustomobject]@{ + schema = $plan.schema + requested_concurrency = 1 + arms = @($terminalArm) + }) + [void](Register-DelegationAccepted -State $exactOnceState -WorkerId ([string]$terminalArm.worker_id) -WorkerSessionId 'native-terminal-session') + $duplicateAcceptanceRejected = $false + try { + [void](Register-DelegationAccepted -State $exactOnceState -WorkerId ([string]$terminalArm.worker_id) -WorkerSessionId 'duplicate-session') + } catch { + $duplicateAcceptanceRejected = $_.Exception.Message -match 'exactly-once' + } + Assert-True $duplicateAcceptanceRejected 'duplicate worker acceptance is rejected with an exactly-once diagnostic' + Assert-Equal 1 $exactOnceState.active[[string]$terminalArm.worker_id].attempt_count 'duplicate acceptance does not create another attempt' + [void](Register-WorkerTerminal -Plan ([pscustomobject]@{ arms = @($terminalArm) }) -State $exactOnceState -WorkerId ([string]$terminalArm.worker_id) -ExecutionEvidence (Copy-TestObject -Value $validTerminalEvidence)) + $duplicateTerminalRejected = $false + try { + [void](Register-WorkerTerminal -Plan ([pscustomobject]@{ arms = @($terminalArm) }) -State $exactOnceState -WorkerId ([string]$terminalArm.worker_id) -ExecutionEvidence (Copy-TestObject -Value $validTerminalEvidence)) + } catch { + $duplicateTerminalRejected = $_.Exception.Message -match 'already terminal|exactly-once' + } + Assert-True $duplicateTerminalRejected 'duplicate terminal registration is rejected after the worker is terminal' + Assert-Equal 1 @($exactOnceState.completed.Keys).Count 'duplicate terminal registration does not add another completion' + function Invoke-TerminalEvidenceCase { param( [Parameter(Mandatory = $true)][string]$Name, diff --git a/scripts/eval-runners/tests/test-runner-conformance.ps1 b/scripts/eval-runners/tests/test-runner-conformance.ps1 index 655a095..32251e9 100644 --- a/scripts/eval-runners/tests/test-runner-conformance.ps1 +++ b/scripts/eval-runners/tests/test-runner-conformance.ps1 @@ -956,6 +956,7 @@ try { $reportText = [System.IO.File]::ReadAllText((Join-Path $repoRoot 'scripts\generate-eval-report.ps1'), [System.Text.UTF8Encoding]::new($false)) $bridgeText = [System.IO.File]::ReadAllText((Join-Path $runnerRoot 'bridge-execution-result.ps1'), [System.Text.UTF8Encoding]::new($false)) $manifestBridgeText = [System.IO.File]::ReadAllText((Join-Path $runnerRoot 'bridge-manifest-results.ps1'), [System.Text.UTF8Encoding]::new($false)) + $commonText = [System.IO.File]::ReadAllText((Join-Path $runnerRoot 'runner-common.ps1'), [System.Text.UTF8Encoding]::new($false)) Assert-True ($prepareText -notmatch '(?i)codex\s+exec|opencode\s+run|cline\s+--|copilot\s+-p|copilot\s+--prompt|Profile\.Provider') 'portable preparation must not contain harness-specific CLI invocations or provider-field branches' Assert-True ($reportText -notmatch '(?i)codex\s+exec|opencode\s+run|cline\s+--|copilot\s+-p|copilot\s+--prompt|Profile\.Provider') 'reporting must not contain harness-specific or provider-field branches' Assert-True ($bridgeText -notmatch '(?i)codex\s+exec|opencode\s+run|cline\s+--|copilot\s+-p|copilot\s+--prompt|Profile\.Provider') 'the raw-to-portable bridge must remain runner-neutral' @@ -968,8 +969,15 @@ try { Assert-True ($prepareText.Contains('Require terminal evidence for the exact selected model, exact arm identity, working directory, isolated HOME/config boundary, prompt fidelity, terminal result capture')) 'handoff preparation must require worker control evidence' Assert-True ($prepareText.Contains('-RequireComplete -RequireNativeDelegation -RequireParallelDispatch')) 'handoff preparation must revalidate native terminal and parallel-dispatch evidence during the manifest bridge' Assert-True ($prepareText.Contains('min(execution-profile.json.concurrency, remaining arms)')) 'handoff preparation must state requested concurrency fan-out' + Assert-True ($prepareText.Contains('DISPATCH IS AN ACTION, NOT A CONFIRMATION STEP.')) 'handoff preparation must forbid confirmation pauses before native dispatch' + Assert-True ($prepareText.Contains('sibling `Task` tool calls') -and $prepareText.Contains('same assistant turn before any result is awaited')) 'OpenCode handoff preparation must require same-turn sibling Task dispatch' + Assert-True ($prepareText.Contains('Want me to re-dispatch...')) 'OpenCode handoff preparation must reject deferred re-dispatch questions' Assert-True ($prepareText.Contains('Assert-OrchestrationConcurrency') -and $prepareText.Contains('orchestration-state.json')) 'handoff preparation must persist and validate orchestration concurrency state' Assert-True ($prepareText.Contains('rejected before the worker starts') -and $prepareText.Contains('record no eval attempt')) 'handoff preparation must queue capacity rejections without counting attempts' + Assert-True ($prepareText.Contains('Register each worker acceptance and terminal result exactly once') -and $prepareText.Contains('incompatibility is diagnostic-only') -and $prepareText.Contains('Do not grade incompatible arms')) 'handoff preparation must make duplicate registration and incompatible-arm handling fail closed' + Assert-True ($prepareText.Contains('Skipping report generation because the completion gate failed') -and $prepareText.Contains('Diagnostic comparison (incomplete)')) 'incomplete collection must remain diagnostic and skip report generation' + Assert-True ($bridgeText.Contains('Get-PackageRunnerDescriptor') -and $bridgeText.Contains('Assert-NativeTerminalCaptureArtifact') -and $bridgeText.Contains('ExpectedMechanism')) 'native bridge must require runner-produced terminal evidence' + Assert-True ($commonText.Contains('exit.status must be a JSON number or null')) 'execution results must reject textual exit statuses' Assert-True ($prepareText.Contains('orchestration.ps1')) 'handoff preparation must load the deterministic orchestration helper' Assert-True ($prepareText -notmatch '') 'handoff preparation must not expose an unconstrained result-file placeholder' Assert-True ($reportText -notmatch 'function Get-ResultPath') 'reporting must not contain a configuration-derived result path helper' @@ -1073,6 +1081,13 @@ try { Assert-True (-not $preBridgeValidation.Success) 'terminal execution plus an unrun canonical result fails validation before bridging' Assert-True (@($preBridgeValidation.Errors | Where-Object { $_ -match 'remains unrun' }).Count -gt 0) 'pre-bridge validation reports the canonical unrun result' + $incompatibleManifestResult = Invoke-Fake -FakePath $fakePath -Command execute -Run $manifestWithout.Path -Profile $parallelProfilePath -Scenario incompatible + Write-TestJson -Path $manifestWithoutExecution -Value $incompatibleManifestResult + $incompatibleValidation = Test-ManifestResults -IterationDirectory $manifestPackage -Manifest $manifestObject -RequireComplete + Assert-True (-not $incompatibleValidation.Complete) 'incompatible execution evidence fails the completion gate' + Assert-True (@($incompatibleValidation.Errors | Where-Object { $_ -match 'diagnostic only' }).Count -gt 0) 'incompatible completion rejection explains that the arm is diagnostic only' + Write-TestJson -Path $manifestWithoutExecution -Value $manifestWithoutExecutionResult + $shadowPath = Join-Path $manifestEval 'results\with_skill.result.json' Write-TestJson -Path $shadowPath -Value ([ordered]@{ schema = (Get-RunnerSchemaNames).PortableResult @@ -1130,6 +1145,26 @@ try { Assert-True ([bool]$canonicalAfterRepeat.grading[0].passed) 'repeat manifest bridge preserves completed grading' Assert-Equal 'graded after the first bridge' $canonicalAfterRepeat.grading[0].evidence 'repeat manifest bridge preserves grading evidence' + $staleReplacement = Get-Content -LiteralPath $manifestWithExecution -Raw | ConvertFrom-Json + $staleReplacement.run_id = 'replacement-terminal-result' + $staleReplacement.final_response.text = 'replacement terminal output' + Write-TestJson -Path $manifestWithExecution -Value $staleReplacement + $replacementBridgeOutput = & pwsh -NoProfile -File $manifestBridgePath -IterationDirectory $manifestPackage -RequireComplete -RequireParallelDispatch 2>&1 + if ($LASTEXITCODE -ne 0) { throw "replacement manifest path bridge failed: $([string]::Join(' ', @($replacementBridgeOutput)))" } + $canonicalAfterReplacement = Get-Content -LiteralPath $manifestWithResult -Raw | ConvertFrom-Json + Assert-Equal 'replacement-terminal-result' $canonicalAfterReplacement.execution_run_id 'manifest bridge revalidates and replaces stale canonical execution evidence' + Assert-Equal 'replacement terminal output' $canonicalAfterReplacement.output 'manifest bridge does not skip a changed raw terminal result' + Assert-True ($null -eq $canonicalAfterReplacement.grading[0].passed) 'raw replacement clears grading tied to a prior execution run' + Assert-True ([string]::IsNullOrWhiteSpace([string]$canonicalAfterReplacement.grading[0].evidence)) 'raw replacement clears stale grading evidence' + + $invalidExitPath = Join-Path $manifestEval 'results\invalid-exit.execution-result.json' + $invalidExitResult = $bridgeResult | ConvertTo-Json -Depth 100 | ConvertFrom-Json + $invalidExitResult.exit.status = 'completed' + Write-TestJson -Path $invalidExitPath -Value $invalidExitResult + $invalidExitOutput = & pwsh -NoProfile -File $bridgePath -Run $with.Path -ExecutionResult $invalidExitPath -Result $resultPath 2>&1 + Assert-True ($LASTEXITCODE -ne 0) 'one-arm bridge rejects a textual exit status' + Assert-True (([string]::Join(' ', @($invalidExitOutput))) -match 'JSON number or null') 'textual exit rejection explains the numeric contract' + $acceptedBridgeResult = $bridgeResult | ConvertTo-Json -Depth 100 | ConvertFrom-Json $acceptedBridgeResult.resolved.status = 'accepted_request' $acceptedBridgeResult.resolved.model = $null diff --git a/scripts/generate-eval-report.ps1 b/scripts/generate-eval-report.ps1 index 2e3bd7f..5deed72 100644 --- a/scripts/generate-eval-report.ps1 +++ b/scripts/generate-eval-report.ps1 @@ -28,8 +28,8 @@ default so a prepared package remains self-contained after preparation. .PARAMETER RequireComplete - Fail before reporting success unless every manifest-declared arm has a terminal execution result and a validated - canonical result bridged from that exact path. + Retained for command-line compatibility. Report generation always requires every manifest-declared arm to have a + terminal execution result and a validated canonical result bridged from that exact path. #> [CmdletBinding()] param( @@ -675,14 +675,14 @@ function New-UpstreamWorkspace { $iterationPath = (Resolve-Path -LiteralPath $IterationDirectory).Path $manifest = Read-JsonFile -Path (Join-Path $iterationPath 'manifest.json') $manifestRecords = @(Get-ManifestRunRecords -IterationDirectory $iterationPath -Manifest $manifest) -$validation = Test-ManifestResults -IterationDirectory $iterationPath -Manifest $manifest -Records $manifestRecords +$validation = Test-ManifestResults -IterationDirectory $iterationPath -Manifest $manifest -Records $manifestRecords -RequireComplete if (-not $validation.Success) { throw ([string]::Join([Environment]::NewLine, @($validation.Errors))) } foreach ($warning in @($validation.Warnings)) { Write-Host "[WARN] $warning" } -if ($RequireComplete -and -not $validation.Complete) { +if (-not $validation.Complete) { throw "Evaluation completion gate failed: expected $($validation.ExpectedArmCount) bridged terminal arms, found $($validation.BridgedResults)." } $skillCreatorPathResolved = Resolve-SkillCreatorPath -RequestedPath $SkillCreatorPath diff --git a/scripts/prepare-skill-evals.ps1 b/scripts/prepare-skill-evals.ps1 index 577fa1f..df91108 100644 --- a/scripts/prepare-skill-evals.ps1 +++ b/scripts/prepare-skill-evals.ps1 @@ -58,7 +58,7 @@ Optional deterministic catalog JSON used by the model discovery helper. Intended for tests and offline validation. .PARAMETER ReasoningEffort - Optional runner-supported reasoning/effort setting written to execution-profile.json. Codex defaults to medium + Optional runner-supported reasoning/effort setting written to execution-profile.json. Codex defaults to low when this is omitted. .PARAMETER ConfigurationProfile @@ -712,7 +712,7 @@ function New-ExecutionProfile { schema = $executionProfileSchema runner = $ExecutionSelection.Runner model = $ExecutionSelection.Model - reasoning_effort = if (-not [string]::IsNullOrWhiteSpace($ReasoningEffort)) { $ReasoningEffort } elseif ($ExecutionSelection.Runner -eq 'codex') { 'medium' } else { $null } + reasoning_effort = if (-not [string]::IsNullOrWhiteSpace($ReasoningEffort)) { $ReasoningEffort } elseif ($ExecutionSelection.Runner -eq 'codex') { 'low' } else { $null } configuration_profile = $ConfigurationProfile tool_profile = $ToolProfile timeout_seconds = $TimeoutSeconds @@ -1566,7 +1566,7 @@ function Invoke-PrepareMode { Write-Utf8File -Path (Join-Path $iterationDirectory 'README.md') -Content (New-PackageReadme -SkillName $Skill -IterationNumber $iterationNumber -IterationDirectory $iterationDirectory -ManifestEvals @($manifestEvals)) $runnerPath = Join-Path $iterationDirectory 'RUN-THIS.prompt.md' - Write-Utf8File -Path $runnerPath -Content (New-RunnerPrompt -IterationDirectory $iterationDirectory -IterationNumber $iterationNumber -ManifestEvals @($manifestEvals)) + Write-Utf8File -Path $runnerPath -Content (New-RunnerPrompt -IterationDirectory $iterationDirectory -IterationNumber $iterationNumber -ManifestEvals @($manifestEvals) -ExecutionSelection $executionSelection -RequestedConcurrency $Concurrency) Write-Host 'Evaluation package prepared.' Write-Host '' @@ -1610,7 +1610,9 @@ function New-RunnerPrompt { param( [string]$IterationDirectory, [int]$IterationNumber, - [object[]]$ManifestEvals + [object[]]$ManifestEvals, + [Parameter(Mandatory = $true)][object]$ExecutionSelection, + [Parameter(Mandatory = $true)][int]$RequestedConcurrency ) $builder = [System.Text.StringBuilder]::new() @@ -1619,6 +1621,12 @@ function New-RunnerPrompt { $orchestrationPath = Join-Path $IterationDirectory "$evalRunnerToolRelativePath/orchestration.ps1" $manifestBridgePath = Join-Path $IterationDirectory "$evalRunnerToolRelativePath/bridge-manifest-results.ps1" $reportPath = Join-Path $IterationDirectory $reportToolRelativePath + $armCount = @($ManifestEvals).Count * 2 + $firstDispatchCount = [Math]::Min($RequestedConcurrency, $armCount) + $armLabels = foreach ($manifestEval in @($ManifestEvals)) { + "arm-$($manifestEval.eval_id)-with_skill" + "arm-$($manifestEval.eval_id)-without_skill" + } [void]$builder.AppendLine('# Execute, grade, and report this evaluation package') [void]$builder.AppendLine() [void]$builder.AppendLine('START NOW. You are the external Eval Orchestrator for this user-directed handoff. Complete execution, deterministic grading, optional judgement, aggregation, and reporting in this run. Do not execute evaluation prompts in the current agent context.') @@ -1644,10 +1652,19 @@ function New-RunnerPrompt { [void]$builder.AppendLine('3. Build the pending arm queue from `manifest.json` with the orchestration helper. For every arm, read the exact `run_manifest`, `execution_result`, and `result` fields from `runs..run_manifest`, `runs..execution_result`, and `runs..result`. Retain those exact manifest-declared strings without editing them: the parent owns those exact destinations; the worker receives only its own prepared arm contract. Do not derive, normalize, rename, hyphenate, underscore, or otherwise reconstruct any run, execution-result, or result path.') [void]$builder.AppendLine('4. Before dispatching, require native worker delegation and all mandatory isolation controls. An unavailable or unsupported delegation mechanism is incompatible. A conditional mechanism is allowed only when the external orchestrator will require terminal evidence for the actual worker. Do not continue by invoking a runner process in the parent, and do not silently serialize arms in the parent.') [void]$builder.AppendLine('5. Dispatch each pending arm to one fresh harness-native full-capability worker. The worker must execute the prepared `prompt.md` as its first task from that arm''s staged run directory, with the selected model/configuration, exact working directory and isolated home. Require terminal evidence for the exact selected model, exact arm identity, working directory, isolated HOME/config boundary, prompt fidelity, terminal result capture, paired-arm/grading exclusion, fresh session, and absence of nested model execution; if any required fact is missing or mismatched, mark the arm incompatible without retrying through a fallback. It must not receive its paired arm, `eval-metadata.json`, expected output, assertions, grading, benchmark/report data, or any result from another arm.') - [void]$builder.AppendLine('6. The delegated worker is the only model-backed execution for that arm. It must not invoke `runner.ps1 execute`, a second harness CLI, another model agent, or a nested session. Return the worker transcript/terminal evidence and normalized execution result to the parent; the parent writes it to the exact manifest-declared `execution_result` path without reconstructing any path.') - [void]$builder.AppendLine('7. Maintain up to `min(execution-profile.json.concurrency, remaining arms)` active delegated workers. If a delegation request is rejected before the worker starts because of harness capacity, leave that arm pending, record no eval attempt, mark the rejection as capacity-limited in orchestration state, and retry it after an active worker becomes terminal. Do not add a runner-specific ceiling or change the portable requested concurrency.') + [void]$builder.AppendLine('6. The delegated worker is the only model-backed execution for that arm. It must not invoke `runner.ps1 execute`, a second harness CLI, another model agent, or a nested session. Return the runner-produced terminal `execution-result.json` unchanged, including its exact runner identity, the descriptor''s exact native delegation mechanism, and a hashed transcript/event artifact. A parent-created summary or normalized result is not genuine runner evidence and is incompatible. The parent writes that raw result to the exact manifest-declared `execution_result` path without reconstructing any path.') + [void]$builder.AppendLine('7. Register each worker acceptance and terminal result exactly once. If orchestration reports that an arm is already accepted or terminal, do not retry or re-register it; preserve the state and resolve the arm from its existing terminal record. Maintain up to `min(execution-profile.json.concurrency, remaining arms)` active delegated workers. If a delegation request is rejected before the worker starts because of harness capacity, leave that arm pending, record no eval attempt, mark the rejection as capacity-limited in orchestration state, and retry it after an active worker becomes terminal. Do not add a runner-specific ceiling or change the portable requested concurrency.') + [void]$builder.AppendLine('7a. DISPATCH IS AN ACTION, NOT A CONFIRMATION STEP. After preflight, emit the full first batch of native worker calls immediately. Do not send a prose status message, ask the user whether to start or re-dispatch, wait for confirmation, or wait for the first worker result before emitting the remaining calls. If the client cannot emit the batch in one assistant turn, mark the selected runner incompatible and stop; never replace the batch with deliberate serial dispatch.') + if ([string]::Equals([string]$ExecutionSelection.Runner, 'opencode', [System.StringComparison]::OrdinalIgnoreCase)) { + $armLabelText = [string]::Join(', ', @($armLabels)) + $deferredDispatchExample = 'Want me to re-dispatch...' + [void]$builder.AppendLine() + [void]$builder.AppendLine('OpenCode NATIVE TASK DISPATCH (MANDATORY): This package selected OpenCode. Use the native `Task` tool with the full-capability built-in `General` worker for every arm; do not use a prose plan as a substitute.') + [void]$builder.AppendLine("Immediately after preflight, in the first execution assistant turn, emit exactly $firstDispatchCount sibling `Task` tool calls for the first batch of $armCount arms (requested concurrency=$RequestedConcurrency). The arms in this package are: $armLabelText. Each call must create a fresh worker for one arm only and pass only that arm's prompt/run contract. Do not wait between calls. The UI/API may show the sibling calls as separate tool messages, but all calls must be emitted from this same assistant turn before any result is awaited.") + [void]$builder.AppendLine("For this OpenCode package, a response such as [$deferredDispatchExample], any equivalent confirmation question, or one Task call followed by a wait is non-compliant. Begin the sibling Task calls now. If OpenCode/the client cannot issue multiple sibling Task calls in one assistant turn, record the runner as incompatible before executing any arm; do not run the $armCount arms sequentially and do not claim a completed evaluation.") + } [void]$builder.AppendLine('8. Persist the orchestration state at `orchestration-state.json` as workers are accepted, capacity-rejected, and terminal. Before bridging, invoke `Assert-OrchestrationConcurrency`. If independent arms requested concurrency > 1 but `max_observed_active` is 1 without explicit capacity-limit evidence, fail the handoff; deliberate serial pacing is not a completed evaluation.') - [void]$builder.AppendLine('9. Preserve the complete terminal response, status, telemetry, evidence references, hashes, isolation mechanisms, warnings, and compatibility deviations. Do not retry for answer quality. A refusal is a result; timeout, harness failure, and incompatibility are results.') + [void]$builder.AppendLine('9. Preserve the complete terminal response, status, telemetry, evidence references, hashes, isolation mechanisms, warnings, and compatibility deviations. Do not retry for answer quality. A refusal is a result; timeout and harness failure are terminal results; incompatibility is diagnostic-only and cannot be graded or benchmarked.') [void]$builder.AppendLine() [void]$builder.AppendLine('The package-local process surface is:') [void]$builder.AppendLine('```text') @@ -1666,7 +1683,7 @@ function New-RunnerPrompt { [void]$builder.AppendLine((' `pwsh -NoProfile -File "' + $manifestBridgePath + '" -IterationDirectory "' + $IterationDirectory + '" -RequireComplete -RequireNativeDelegation -RequireParallelDispatch`')) [void]$builder.AppendLine(' The bridge''s one-arm operation is conceptually `-Run runPath -ExecutionResult executionPath -Result resultPath`, where all three values are the exact strings read from `manifest.json`. Do not derive, normalize, rename, hyphenate, underscore, or otherwise reconstruct any of them.') [void]$builder.AppendLine('2. Only if the package bridge succeeds, read each eval''s `eval-metadata.json` and reveal `expected_output` and `assertions` to the Grader. Follow `tools/skill-creator/agents/grader.md`; grade deterministically first, then use optional model judgement only where deterministic evidence cannot decide. Never infer tool or file behavior from model self-report without process evidence.') - [void]$builder.AppendLine('3. Write only `grading[].text`, `grading[].passed`, and `grading[].evidence` for grading. Do not alter raw execution results or replace the canonical result stubs. Use null for genuinely unavailable judgement and leave missing arms visibly missing.') + [void]$builder.AppendLine('3. Write only `grading[].text`, `grading[].passed`, and `grading[].evidence` for grading. Do not grade incompatible arms, do not alter raw execution results, and do not replace the canonical result stubs. Use null for genuinely unavailable judgement and leave missing or incompatible arms visibly ungradeable.') [void]$builder.AppendLine(('4. Run the existing package report adapter with its completion gate: `pwsh -NoProfile -File "' + $reportPath + '" -IterationDirectory "' + $IterationDirectory + '" -RequireComplete`. It remains the bridge to Anthropic skill-creator''s grader-compatible aggregator and viewer; do not replace it with harness-specific reporting. The packaged compatibility tools remain `scripts/aggregate_benchmark.py` and `eval-viewer/generate_review.py`.')) [void]$builder.AppendLine() [void]$builder.AppendLine('The completion artifacts are `report.html`, `skill-creator-report.html`, `benchmark.json`, and `benchmark.md` at the package root. Return their absolute paths, completed and missing arm counts, runner/model identity, and a concise evidence-backed summary. If the package cannot be written from the external environment, return one paste-ready block containing the completed result objects and report artifacts.') @@ -1969,7 +1986,12 @@ function Invoke-CollectMode { # A transferred or partial result may arrive without grading. Fall back to the assertion count from the # package so the row still shows how much is left to check. $grading = @(Get-JsonProperty -Object $result -Name 'grading' -Default @()) - $graded = @($grading | Where-Object { $null -ne (Get-JsonProperty -Object $_ -Name 'passed') }) + if ($executionStatus -eq 'incompatible') { + $warnings.Add("$($entry.eval_name)/$configuration - execution is incompatible; its output is diagnostic only and cannot contribute grading evidence.") + $graded = @() + } else { + $graded = @($grading | Where-Object { $null -ne (Get-JsonProperty -Object $_ -Name 'passed') }) + } $passed = @($graded | Where-Object { [bool]$_.passed }).Count $total = if ($grading.Count -gt 0) { $grading.Count } else { @($metadata.assertions).Count } if ($graded.Count -eq 0) { @@ -2034,9 +2056,13 @@ function Invoke-CollectMode { } $builder = [System.Text.StringBuilder]::new() - [void]$builder.AppendLine("# Eval comparison: $($manifest.skill_name) (iteration $($manifest.iteration))") + $comparisonKind = if ($errors.Count -gt 0) { 'Diagnostic comparison (incomplete)' } else { 'Eval comparison' } + [void]$builder.AppendLine("# $comparisonKind`: $($manifest.skill_name) (iteration $($manifest.iteration))") [void]$builder.AppendLine() [void]$builder.AppendLine('This repository-side comparison validates recorded grading and never invokes a model. Grading may have been performed by the user-directed external evaluator before this report was generated.') + if ($errors.Count -gt 0) { + [void]$builder.AppendLine('This package did not pass the completion gate. The rows below are diagnostic evidence only and must not be treated as a valid paired benchmark.') + } [void]$builder.AppendLine() [void]$builder.AppendLine('| Eval | Assertions | with_skill model | with_skill graded | without_skill model | without_skill graded |') [void]$builder.AppendLine('| --- | --- | --- | --- | --- | --- |') @@ -2097,14 +2123,18 @@ function Invoke-CollectMode { Write-Host '' Write-Host "Wrote $comparisonPath" - $reportScript = Join-Path (Join-Path (Get-RepoRoot) 'scripts') 'generate-eval-report.ps1' - $reportOutput = & pwsh -NoProfile -File $reportScript -IterationDirectory $iterationDirectory 2>&1 - if ($LASTEXITCODE -ne 0) { - $errors.Add("Report generation failed: $($reportOutput -join [Environment]::NewLine)") - } else { - foreach ($line in @($reportOutput)) { - Write-Host $line + if ($errors.Count -eq 0) { + $reportScript = Join-Path (Join-Path (Get-RepoRoot) 'scripts') 'generate-eval-report.ps1' + $reportOutput = & pwsh -NoProfile -File $reportScript -IterationDirectory $iterationDirectory 2>&1 + if ($LASTEXITCODE -ne 0) { + $errors.Add("Report generation failed: $($reportOutput -join [Environment]::NewLine)") + } else { + foreach ($line in @($reportOutput)) { + Write-Host $line + } } + } else { + Write-Host 'Skipping report generation because the completion gate failed; comparison.md is diagnostic only.' } if ($errors.Count -gt 0) { diff --git a/scripts/validate-skill-templates.ps1 b/scripts/validate-skill-templates.ps1 index 4d6ffd9..264df2a 100644 --- a/scripts/validate-skill-templates.ps1 +++ b/scripts/validate-skill-templates.ps1 @@ -1394,8 +1394,8 @@ Add-ValidationResult -Results $results -Name 'Skill evaluation prepares portable throw "prepare-skill-evals.ps1 failed for the Codex default fixture: $($codexPrepareOutput -join [Environment]::NewLine)" } $codexProfile = [System.IO.File]::ReadAllText((Join-Path $codexPackageRoot 'iteration-1\execution-profile.json'), $utf8NoBom) | ConvertFrom-Json - if ([string]$codexProfile.runner -ne 'codex' -or [string]$codexProfile.model -ne 'gpt-5.6-luna' -or [string]$codexProfile.reasoning_effort -ne 'medium') { - throw 'Codex preparation must preserve gpt-5.6-luna and default omitted reasoning effort to medium.' + if ([string]$codexProfile.runner -ne 'codex' -or [string]$codexProfile.model -ne 'gpt-5.6-luna' -or [string]$codexProfile.reasoning_effort -ne 'low') { + throw 'Codex preparation must preserve gpt-5.6-luna and default omitted reasoning effort to low.' } $missingSelectionRoot = Join-Path ([System.IO.Path]::GetTempPath()) ('agentic-eval-missing-selection-' + [Guid]::NewGuid().ToString('N')) @@ -1693,19 +1693,19 @@ Add-ValidationResult -Results $results -Name 'Skill evaluation prepares portable if (-not (Test-Path -LiteralPath (Join-Path $iterationDirectory 'comparison.md'))) { throw 'prepare-skill-evals.ps1 -CollectResults must write comparison.md.' } - if (-not (Test-Path -LiteralPath (Join-Path $iterationDirectory 'report.html'))) { - throw 'prepare-skill-evals.ps1 -CollectResults must write report.html.' - } - if (-not (Test-Path -LiteralPath (Join-Path $iterationDirectory 'benchmark.json'))) { - throw 'prepare-skill-evals.ps1 -CollectResults must write benchmark.json.' - } - if (-not (Test-Path -LiteralPath (Join-Path $iterationDirectory 'benchmark.md'))) { - throw 'prepare-skill-evals.ps1 -CollectResults must write benchmark.md.' + foreach ($invalidCompletionArtifact in @('report.html', 'skill-creator-report.html', 'benchmark.json', 'benchmark.md')) { + if (Test-Path -LiteralPath (Join-Path $iterationDirectory $invalidCompletionArtifact)) { + throw "prepare-skill-evals.ps1 -CollectResults must not write '$invalidCompletionArtifact' for an incomplete diagnostic package." + } } $firstEntry = @($manifest.evals)[0] $fakeRunnerPath = Join-Path $iterationDirectory 'tools/eval-runners/fake/runner.ps1' $profilePath = Join-Path $iterationDirectory ([string]$manifest.execution_profile) + $deterministicProfile = [System.IO.File]::ReadAllText($profilePath, $utf8NoBom) | ConvertFrom-Json + $deterministicProfile.runner = 'fake' + $deterministicProfile.model = 'fixture-model' + [System.IO.File]::WriteAllText($profilePath, (($deterministicProfile | ConvertTo-Json -Depth 100) + [Environment]::NewLine), $utf8NoBom) foreach ($entryToRun in @($manifest.evals)) { foreach ($configuration in @('with_skill', 'without_skill')) { $run = $entryToRun.runs.$configuration From 88a481107eb1ea22b6926b38e2f15d5d2800ac05 Mon Sep 17 00:00:00 2001 From: "aicia[bot]" Date: Tue, 25 Aug 2026 01:09:20 +0200 Subject: [PATCH 30/39] =?UTF-8?q?=F0=9F=93=9D=20clarify=20evaluation=20orc?= =?UTF-8?q?hestration=20requirements=20for=20parallel=20execution?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Update AGENTS.md orchestration guidance to emphasize native delegation, runner identity preservation, and OpenCode-specific Task dispatch requirements for parallel arms. Adjust Codex default reasoning effort to low. Clarify that incompatible arms cannot contribute to grading evidence. Expand eval-runners README with implementation notes. --- AGENTS.md | 6 ++++-- CONTRIBUTING.md | 6 +++--- scripts/eval-runners/README.md | 14 +++++++++++++- 3 files changed, 20 insertions(+), 6 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 7eb4767..137f5dd 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -46,7 +46,7 @@ Where `skill-creator` says to spawn with-skill and baseline subagents in the sam `eval `, `evaluate `, `eval this skill`, `prepare evals for `, and `evaluate using the existing evals` are all requests for this workflow. Treat them as instructions to prepare the package, never to run it, and never as a request to write new eval cases unless the user asks for that too. -Resolve the execution configuration before running the package preparation script. In an interactive agent session, offer Codebelt Reference first (`GitHub Copilot CLI` + `claude-haiku-4.5`) and verify that model through `scripts/Get-HarnessModels.ps1`; if it is unavailable, show the current discovered Copilot models and ask for a replacement. If the user selects Codex, default to `-Model gpt-5.6-luna` and medium reasoning (low is also acceptable when explicitly chosen); verify the model through `scripts/Get-HarnessModels.ps1` before preparation. For manual selection, ask for Harness, discover current models for that harness with `scripts/Get-HarnessModels.ps1`, then pass the resulting runner/model pair to the preparation script. Cline and OpenCode discovery is free-only; GitHub Copilot and Codex discovery lists all currently available models. Never guess stale model ids, silently switch harnesses, or generate an execution-ready package with a null runner or model. +Resolve the execution configuration before running the package preparation script. In an interactive agent session, offer Codebelt Reference first (`GitHub Copilot CLI` + `claude-haiku-4.5`) and verify that model through `scripts/Get-HarnessModels.ps1`; if it is unavailable, show the current discovered Copilot models and ask for a replacement. If the user selects Codex, default to `-Model gpt-5.6-luna` and low reasoning; verify the model through `scripts/Get-HarnessModels.ps1` before preparation. For manual selection, ask for Harness, discover current models for that harness with `scripts/Get-HarnessModels.ps1`, then pass the resulting runner/model pair to the preparation script. Cline and OpenCode discovery is free-only; GitHub Copilot and Codex discovery lists all currently available models. Never guess stale model ids, silently switch harnesses, or generate an execution-ready package with a null runner or model. ``` pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -Skill dotnet-test -Runner github-copilot -Model claude-haiku-4.5 @@ -136,12 +136,14 @@ Four things still hold while you execute: An agent that prepared a package in this session does not get to turn around and execute it. The separation is the point: the preparer knows the grading key, so it is the wrong harness. This is the only role-based disqualification. -The selected executor has two ordered phases. Its current context may read `RUN-THIS.prompt.md`, `manifest.json`, `execution-profile.json`, and the runner protocol files, but it must not execute an eval prompt itself. In phase one, it resolves the selected runner, validates `describe`, preflights each `run.json`, and invokes `execute` exactly once for every case's fresh `with_skill` and `without_skill` arm. The runner launches each native harness session from its own run directory with `repo/` as the working directory, `home/` as the isolated profile, and the required isolation controls; hard filesystem confinement, when a runner proves it, raises the reported isolation from pragmatic to strict but is not itself a prerequisite. The runner receives only `run.json` and `execution-profile.json`; workers never see the runner, manifest, grading key, sibling results, or orchestration commentary, because all of those live outside the run directory. Never reuse a worker or session between runs. In phase two, after all available execution results are complete or failed, the executor validates and freezes the raw results, bridges them into `eval-result/2`, reads the grading key, follows the packaged `skill-creator` grader guidance, writes the grading evidence, invokes the package adapter so Anthropic's aggregator and eval viewer produce the report, and returns the report path and comparison. It does not ask the user whether to start either phase. +The selected executor has two ordered phases. Its current context may read `RUN-THIS.prompt.md`, `manifest.json`, `execution-profile.json`, and the runner protocol files, but it must not execute an eval prompt itself. In phase one, it resolves the selected runner, validates `describe`, preflights each `run.json`, and dispatches every arm through the selected runner's declared harness-native worker mechanism. The parent and delegated worker must not invoke the compatibility `execute` transport or substitute a generic subagent. The runner launches each native harness session from its own run directory with `repo/` as the working directory, `home/` as the isolated profile, and the required isolation controls; hard filesystem confinement, when a runner proves it, raises the reported isolation from pragmatic to strict but is not itself a prerequisite. The runner-produced terminal result must retain the selected runner identity, exact native mechanism, and hashed transcript/event artifact; a parent-created summary is incompatible. The runner receives only `run.json` and `execution-profile.json`; workers never see the runner, manifest, grading key, sibling results, or orchestration commentary, because all of those live outside the run directory. Never reuse a worker or session between runs. In phase two, after all available execution results are complete or failed, the executor validates and freezes the raw results, bridges them into `eval-result/2`, reads the grading key, follows the packaged `skill-creator` grader guidance, writes the grading evidence, invokes the package adapter so Anthropic's aggregator and eval viewer produce the report, and returns the report path and comparison. It does not ask the user whether to start either phase. The candidate instructions are already inlined in the with_skill run's `prompt.md` and staged under its `skill//` directory; the orchestrator does not load or summarize them for the worker. The baseline run has no `skill/` directory and no candidate instructions, and the orchestrator must not expose the candidate skill through another route, including a globally installed copy. The generated prompt files and the baseline `run.json` also omit the skill name, eval identifiers, and configuration labels so workers receive an ordinary task rather than an announcement that they are under evaluation. Use the same model, model version, configuration, tools, and limits for every worker. Disable persistent memory and cross-session recall. Independent runs must execute concurrently up to `min(execution-profile.json.concurrency, remaining arms)` when harness capacity permits, but every run still gets a distinct context and no shared mutable workspace. For an OpenCode profile, one-at-a-time Task dispatch with available capacity is non-compliant; only an explicit harness capacity rejection justifies a serial effective run, and that limit must be recorded in orchestration state. +For OpenCode, the external orchestrator must perform that concurrency as one assistant turn: after preflight, emit one sibling native `Task` call per pending arm up to the available slots, then await the results. It must not ask whether to start or re-dispatch, send a prose status message between calls, or wait for the first Task result before emitting the remaining calls. If the client cannot emit multiple sibling Task calls in one turn, the selected runner is incompatible; do not silently fall back to serial execution. + `RUN-THIS.prompt.md` requires a selected Eval Runner that can create isolated workers or sessions. A plain single-context client can still execute an individual self-contained prompt when the user opens it directly as the first message of a fresh session, but it cannot provide the paired comparison and report contract in that same context. A selected runner that cannot satisfy a required guarantee is `incompatible`; there is no generic fallback or runner substitution. Partial package state may be inspected and reported, but the completion gate must pass before it can be presented as a completed evaluation; missing or unrun arms remain visibly incomplete. An `output` is the model's own message in full, including questions, caveats, explanations, or a refusal. Where a run invoked a tool, that tool's stdout is evidence rather than a replacement for the response. Record the full worker transcript, duration, token usage, and tool-call count when the harness exposes them; omit unavailable metrics rather than estimating them. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index e918c6b..3e28008 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -98,11 +98,11 @@ Run it after the last skill edit and before `scripts/sync-skill-install.ps1`, wh pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -Skill -Runner -Model ``` -Before running the script, choose a Harness + Model. Use `scripts/Get-HarnessModels.ps1 -Runner ` to list current selectors; Cline and OpenCode are free-only, while GitHub Copilot and Codex list all currently available models. The Codebelt Reference shortcut is GitHub Copilot CLI + `claude-haiku-4.5`; `-CodebeltReference` verifies that the model still exists and fails instead of silently substituting a different model. The script writes `.bot/-workspace/iteration-/` with one directory per eval. Each holds the grading key `eval-metadata.json` and result stubs under `results/` at the eval-case level, plus two paired run directories, `with_skill/` and `without_skill/`. A run directory is the worker's run root: `prompt.md`, a `run.json` contract, a `repo/` working tree materialized from the fixtures, an isolated `home/`, and - for `with_skill` only - a `skill//` copy of the candidate. The grading key and results sit outside both run directories. At the root it writes `manifest.json`, `execution-profile.json`, the package-local Eval Runner protocol, the package report adapter, the exact Anthropic skill-creator grader/aggregator/viewer assets, and `RUN-THIS.prompt.md`, the one prompt you hand to the external Eval Orchestrator. That orchestrator resolves and preflights the selected runner, invokes its common `describe`/`preflight`/`execute` surface once per fresh blind arm, stores raw `execution-result.json`, bridges the results, grades only after execution, and runs the adapter, which invokes `aggregate_benchmark.py` and `eval-viewer/generate_review.py --static`. It never runs an eval prompt in the coordinator context, never chooses runner/model policy, and never reuses a worker. Both worker prompts carry the same task, materialized repository, and response contract; only the operating instructions and the presence of `skill/` differ, and neither prompt identifies itself as an eval. `.gitignore` covers `.bot/*`, so nothing there reaches git. The script refuses an `-OutputRoot` inside the repository but outside `.bot/`; pass an explicit temp path when the harness does not need repository-local storage. +Before running the script, choose a Harness + Model. Use `scripts/Get-HarnessModels.ps1 -Runner ` to list current selectors; Cline and OpenCode are free-only, while GitHub Copilot and Codex list all currently available models. The Codebelt Reference shortcut is GitHub Copilot CLI + `claude-haiku-4.5`; `-CodebeltReference` verifies that the model still exists and fails instead of silently substituting a different model. The script writes `.bot/-workspace/iteration-/` with one directory per eval. Each holds the grading key `eval-metadata.json` and result stubs under `results/` at the eval-case level, plus two paired run directories, `with_skill/` and `without_skill/`. A run directory is the worker's run root: `prompt.md`, a `run.json` contract, a `repo/` working tree materialized from the fixtures, an isolated `home/`, and - for `with_skill` only - a `skill//` copy of the candidate. The grading key and results sit outside both run directories. At the root it writes `manifest.json`, `execution-profile.json`, the package-local Eval Runner protocol, the package report adapter, the exact Anthropic skill-creator grader/aggregator/viewer assets, and `RUN-THIS.prompt.md`, the one prompt you hand to the external Eval Orchestrator. That orchestrator resolves and preflights the selected runner, dispatches each arm through its declared harness-native worker mechanism (never the compatibility `execute` transport), stores genuine runner-produced `execution-result.json` evidence, bridges the results, grades only after execution, and runs the adapter, which invokes `aggregate_benchmark.py` and `eval-viewer/generate_review.py --static`. It never runs an eval prompt in the coordinator context, never chooses runner/model policy, and never reuses a worker. Both worker prompts carry the same task, materialized repository, and response contract; only the operating instructions and the presence of `skill/` differ, and neither prompt identifies itself as an eval. `.gitignore` covers `.bot/*`, so nothing there reaches git. The script refuses an `-OutputRoot` inside the repository but outside `.bot/`; pass an explicit temp path when the harness does not need repository-local storage. Repository preparation, validation, CI, hooks, deterministic tests, and automatic completion gates never run those prompts or invoke a model. That boundary is the Priority 1 rule in `AGENTS.md`, and preparing a prompt is not permission to execute one. A human-selected external Eval Orchestrator handed a specific package may invoke the selected package-local Eval Runner; this explicit handoff boundary does not weaken the repository prohibition or authorize CI/live evals. -Run both configurations on the same model, same version, and same configuration. Independent arms must be dispatched concurrently up to `execution-profile.json.concurrency` when the harness permits it. For OpenCode, deliberate one-at-a-time Task dispatch is not compliant; serialize only after an explicit harness capacity rejection and preserve that evidence. A with-skill run on one model against a baseline on another measures the model as much as the skill and is not a skill-effectiveness result. +Run both configurations on the same model, same version, and same configuration. Independent arms must be dispatched concurrently up to `execution-profile.json.concurrency` when the harness permits it. For OpenCode, emit the sibling native `Task` calls for the first batch in one assistant turn, without asking for confirmation or waiting between calls; deliberate one-at-a-time Task dispatch is not compliant. Serialize only after an explicit harness capacity rejection and preserve that evidence. If the client cannot issue multiple sibling Task calls in one turn, the runner is incompatible. A with-skill run on one model against a baseline on another measures the model as much as the skill and is not a skill-effectiveness result. Record each external runner result in the matching `execution-result.json` first. It must carry the protocol/schema, opaque run and fresh session ids, status, complete final response or explicit unavailability, runner/harness identity, requested and resolved model selection, timestamps and duration, exit/failure state, prompt/run/profile hashes, resolved isolation mechanisms, warnings, and artifact references. Include token, cache, cost, tool, command, file, and transcript evidence only when the harness exposes it; unavailable values remain explicitly unavailable and are never estimated. The deterministic bridge then writes the existing `results/*.result.json` shape, after which grading may add `grading[].passed` and evidence. Assertions about tool, shell, or file behavior are only gradeable from a run that captured that evidence. If the results were transferred without those report artifacts, validate and compare with: @@ -110,7 +110,7 @@ Record each external runner result in the matching `execution-result.json` first pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -CollectResults ``` -It may write `comparison.md`, the first-party side-by-side `report.html`, the exact upstream `skill-creator-report.html`, and the upstream `benchmark.json`/`benchmark.md` while flagging missing arms, unrun configurations, and mixed models, but it exits non-zero when the required completion gate is not satisfied. Those diagnostic artifacts must not present an incomplete or unrun package as a successfully completed evaluation. The normal external Eval Orchestrator grades in the same handoff using deterministic checks for mechanical assertions and evidence-backed judgement where an assertion is genuinely qualitative. Repository automation remains deterministic and never invokes a model. Codex, OpenCode, and Cline are the conforming real runners; the deterministic fake runner is the CI conformance harness. Hard filesystem confinement is reported as strict versus pragmatic confidence and is not a universal Windows prerequisite. Freebuff remains planned/blocked until its official CLI provides a supported noninteractive machine-readable transport. Native skill activation, portability scoring, and additional runners are not part of v0.9.1. +It may write a diagnostic `comparison.md` while flagging missing arms, unrun configurations, incompatible evidence, and mixed models, but it exits non-zero and does not write benchmark/report artifacts until the required completion gate is satisfied. Those diagnostic artifacts must not present an incomplete or unrun package as a successfully completed evaluation. The normal external Eval Orchestrator grades in the same handoff using deterministic checks for mechanical assertions and evidence-backed judgement where an assertion is genuinely qualitative. Repository automation remains deterministic and never invokes a model. Codex, OpenCode, and Cline are the conforming real runners; the deterministic fake runner is the CI conformance harness. Hard filesystem confinement is reported as strict versus pragmatic confidence and is not a universal Windows prerequisite. Freebuff remains planned/blocked until its official CLI provides a supported noninteractive machine-readable transport. Native skill activation, portability scoring, and additional runners are not part of v0.9.1. The eval package is a temp artifact. Do not commit it, its prompts, or its results unless the change explicitly calls for checked-in examples. diff --git a/scripts/eval-runners/README.md b/scripts/eval-runners/README.md index 5f8a2c2..e569c86 100644 --- a/scripts/eval-runners/README.md +++ b/scripts/eval-runners/README.md @@ -54,6 +54,13 @@ execution. Missing or mismatched evidence makes the arm `incompatible`; it is never a reason to invoke the parent or the compatibility `runner.ps1 execute` transport. +The terminal result must come from the selected package runner. The parent may +persist that result at the exact manifest-declared path, but it must not replace +it with a worker summary or synthesize a normalized result. Native bridging +also checks the result's runner identity, the descriptor's exact delegation +mechanism, and a hashed transcript/event artifact. An `incompatible` arm is +diagnostic-only: it is never gradeable and fails the completion/benchmark gate. + The descriptor's `delegation` object records the native mechanism, worker role, advertised full-capability/model-lock/working-directory/result-capture properties, harness-authoritative capacity, and the invariant @@ -77,6 +84,10 @@ Native delegation mechanisms: - OpenCode: the native Task tool with the full-capability built-in `General` subagent. Task/General availability is preflight readiness only; `Explore`/`Scout` read-only agents are not valid for a mutable eval arm. + When more than one arm is pending, the external orchestrator must emit the + sibling Task calls for the first batch in one assistant turn and must not ask + for confirmation or wait between calls. A client that cannot do that is + incompatible; available-capacity serial dispatch is not a fallback. - Cline: a full-capability Cline SDK Agent Squad `start_subagent(preset: "anvil")` child session backed by `ClineCore.create`. The plugin's default `phantom` preset and Cline's @@ -106,7 +117,8 @@ runner may pass it through unchanged or split it internally when its native CLI requires separate provider/model arguments. The profile contains no credentials, secrets, or portable provider field. `execution-result.json` normalizes one blind execution and keeps grading -separate from raw evidence. +separate from raw evidence. Its `exit.status` is a numeric process exit code or +`null`, never a textual lifecycle label such as `completed`. Every runner exposes the same process surface: From fb261d270f9bdfb8d9f3cf2fbffbdf2b6e9fc6d1 Mon Sep 17 00:00:00 2001 From: Eval Worker Date: Tue, 25 Aug 2026 01:15:34 +0200 Subject: [PATCH 31/39] =?UTF-8?q?=E2=99=BB=EF=B8=8F=20harden=20manifest=20?= =?UTF-8?q?result=20validation=20with=20sha256=20and=20incompatible=20chec?= =?UTF-8?q?ks?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- scripts/eval-runners/manifest-paths.ps1 | 18 ++++++++++++++++-- 1 file changed, 16 insertions(+), 2 deletions(-) diff --git a/scripts/eval-runners/manifest-paths.ps1 b/scripts/eval-runners/manifest-paths.ps1 index 1c98816..92c8d62 100644 --- a/scripts/eval-runners/manifest-paths.ps1 +++ b/scripts/eval-runners/manifest-paths.ps1 @@ -275,6 +275,9 @@ function Test-ManifestResults { $errors.Add("$($record.EvalName)/$($record.Configuration) execution result has non-terminal status '$rawStatus'.") } else { $terminalExecutionResults++ + if ($RequireComplete -and $rawStatus -eq 'incompatible') { + $errors.Add("$($record.EvalName)/$($record.Configuration) is incompatible; incompatible execution evidence is diagnostic only and cannot be graded or benchmarked.") + } } } elseif ($RequireComplete) { $errors.Add("$($record.EvalName)/$($record.Configuration) is missing its manifest-declared execution result '$($record.ExecutionResultRelative)'.") @@ -317,7 +320,7 @@ function Test-ManifestResults { $errors.Add("$($record.EvalName)/$($record.Configuration) canonical execution_status '$canonicalStatus' does not match raw status '$rawStatus'.") } - foreach ($field in @('model', 'harness', 'execution_status', 'execution_run_id', 'execution_result_file')) { + foreach ($field in @('model', 'harness', 'execution_status', 'execution_run_id', 'execution_result_file', 'execution_result_sha256')) { if (-not (Test-JsonProperty -Object $canonical -Name $field) -or [string]::IsNullOrWhiteSpace([string](Get-JsonProperty -Object $canonical -Name $field -Default ''))) { $errors.Add("$($record.EvalName)/$($record.Configuration) canonical result is missing populated '$field'.") } @@ -327,7 +330,18 @@ function Test-ManifestResults { if ($actualExecutionFile -ne $expectedExecutionFile) { $errors.Add("$($record.EvalName)/$($record.Configuration) canonical execution_result_file '$actualExecutionFile' does not match the manifest execution_result path '$($record.ExecutionResultRelative)'.") } - if ($canonicalStatus -eq $rawStatus -and $grading.Count -eq $assertionCount) { + $expectedExecutionHash = Get-Sha256HexFromFile -Path $record.ExecutionResultPath + $actualExecutionHash = [string](Get-JsonProperty -Object $canonical -Name 'execution_result_sha256' -Default '') + if ($actualExecutionHash -ne $expectedExecutionHash) { + $errors.Add("$($record.EvalName)/$($record.Configuration) canonical execution_result_sha256 does not match the current manifest execution result.") + } + if ($rawStatus -eq 'incompatible' -and $RequireComplete) { + $gradedEntries = @($grading | Where-Object { $null -ne (Get-JsonProperty -Object $_ -Name 'passed' -Default $null) }) + if ($gradedEntries.Count -gt 0) { + $errors.Add("$($record.EvalName)/$($record.Configuration) has grading for an incompatible execution; diagnostic arms must not contribute grading evidence.") + } + } + if ($canonicalStatus -eq $rawStatus -and $grading.Count -eq $assertionCount -and $rawStatus -ne 'incompatible') { $bridgedResults++ } } elseif (-not $rawExists -and $RequireComplete -and $canonicalStatus -ne 'unrun') { From 55a5201f45cccf737524c95e07e17d592b110584 Mon Sep 17 00:00:00 2001 From: "aicia[bot]" Date: Tue, 25 Aug 2026 02:01:22 +0200 Subject: [PATCH 32/39] =?UTF-8?q?=F0=9F=94=A5=20remove=20cline=20eval=20ru?= =?UTF-8?q?nner=20support?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Cline is no longer a supported eval runner platform. Remove the adapter, its test fixtures, and related model discovery code. --- scripts/Get-HarnessModels.ps1 | 68 +- scripts/eval-runners/README.md | 56 +- scripts/eval-runners/cline/runner.ps1 | 642 ------------------ .../tests/fixtures/cline-events.jsonl | 11 - 4 files changed, 41 insertions(+), 736 deletions(-) delete mode 100644 scripts/eval-runners/cline/runner.ps1 delete mode 100644 scripts/eval-runners/tests/fixtures/cline-events.jsonl diff --git a/scripts/Get-HarnessModels.ps1 b/scripts/Get-HarnessModels.ps1 index 93a5f7a..ede25c8 100644 --- a/scripts/Get-HarnessModels.ps1 +++ b/scripts/Get-HarnessModels.ps1 @@ -4,11 +4,11 @@ .DESCRIPTION Discovers runner-native model selectors without executing model requests. GitHub Copilot and Codex return every - model the harness exposes. Cline and OpenCode return only models whose current catalog metadata proves free + model the harness exposes. OpenCode returns only models whose current catalog metadata proves free availability. Discovery failures are local to the selected harness and never fall back to stale hardcoded catalogs. .PARAMETER Runner - Internal Eval Runner id: github-copilot, codex, opencode, or cline. + Internal Eval Runner id: github-copilot, codex, or opencode. .PARAMETER CatalogPath Optional deterministic catalog fixture used by tests. When supplied, no harness command is invoked. @@ -23,7 +23,7 @@ [CmdletBinding()] param( [Parameter(Mandatory = $true)] - [ValidateSet('github-copilot', 'codex', 'opencode', 'cline')] + [ValidateSet('github-copilot', 'codex', 'opencode')] [string]$Runner, [string]$CatalogPath, @@ -48,7 +48,6 @@ function Get-HarnessDisplayName { 'github-copilot' { return 'GitHub Copilot CLI' } 'codex' { return 'Codex CLI' } 'opencode' { return 'OpenCode' } - 'cline' { return 'Cline' } default { return $RunnerName } } } @@ -56,7 +55,7 @@ function Get-HarnessDisplayName { function Get-PolicyName { param([Parameter(Mandatory = $true)][string]$RunnerName) - if ($RunnerName -in @('cline', 'opencode')) { + if ($RunnerName -eq 'opencode') { return 'free' } return 'all' @@ -190,7 +189,7 @@ function ConvertTo-ModelChoice { } $provider = Get-FirstPropertyValue -Object $Model -Names @('providerID', 'providerId', 'provider') - if ($RunnerName -in @('cline', 'opencode') -and $id -notmatch '/' -and -not [string]::IsNullOrWhiteSpace($provider)) { + if ($RunnerName -eq 'opencode' -and $id -notmatch '/' -and -not [string]::IsNullOrWhiteSpace($provider)) { $id = "$provider/$id" } @@ -335,29 +334,6 @@ function ConvertFrom-OpenCodeTextCatalog { return @($models) } -function Resolve-ClineModelsModulePath { - $command = Resolve-ExternalCommand -Name 'cline' - if ($null -eq $command) { - throw 'Cline CLI executable is not available on PATH.' - } - - $source = [string]$command.Source - $directory = Split-Path -Parent $source - $candidates = @( - (Join-Path $directory 'node_modules/cline/node_modules/@cline/llms/dist/models.js'), - (Join-Path $directory '../lib/node_modules/cline/node_modules/@cline/llms/dist/models.js'), - (Join-Path $directory '../node_modules/cline/node_modules/@cline/llms/dist/models.js') - ) - foreach ($candidate in $candidates) { - $full = [System.IO.Path]::GetFullPath($candidate) - if (Test-Path -LiteralPath $full -PathType Leaf) { - return $full - } - } - - throw 'Cline model registry module was not found under the installed CLI package.' -} - function Resolve-CopilotSdkPath { $command = Resolve-ExternalCommand -Name 'copilot' if ($null -eq $command) { @@ -410,39 +386,6 @@ function Get-OpenCodeModels { return ConvertFrom-OpenCodeTextCatalog -Text $result.Stdout } -function Get-ClineModels { - $modulePath = Resolve-ClineModelsModulePath - $node = Resolve-ExternalCommand -Name 'node' - if ($null -eq $node) { - throw 'Node.js is required to read the Cline model registry.' - } - - $script = @' -import { pathToFileURL } from "node:url"; -const modulePath = process.argv[1]; -const mod = await import(pathToFileURL(modulePath).href); -const providerIds = await mod.getProviderIds(); -const models = []; -for (const providerId of providerIds) { - const providerModels = await mod.getModelsForProvider(providerId); - for (const [key, model] of Object.entries(providerModels || {})) { - models.push({ - id: model.id || key, - providerID: providerId, - name: model.name || model.id || key, - pricing: model.pricing, - operation: model.operation, - capabilities: model.capabilities - }); - } -} -console.log(JSON.stringify({ models })); -'@ - $result = Invoke-JsonCommand -CommandInfo $node -Arguments @('--input-type=module', '-e', $script, $modulePath) -TimeoutSeconds 60 - $catalog = $result.Stdout | ConvertFrom-Json - return ConvertTo-ModelChoices -Catalog $catalog -RunnerName 'cline' -Source '@cline/llms model registry' -} - function Get-CopilotModels { $sdkPath = Resolve-CopilotSdkPath $node = Resolve-ExternalCommand -Name 'node' @@ -472,7 +415,6 @@ try { 'github-copilot' { Get-CopilotModels } 'codex' { Get-CodexModels } 'opencode' { Get-OpenCodeModels } - 'cline' { Get-ClineModels } } } $models = @(Select-ModelsByPolicy -Models @($rawModels) -RunnerName $Runner) diff --git a/scripts/eval-runners/README.md b/scripts/eval-runners/README.md index e569c86..aae32ef 100644 --- a/scripts/eval-runners/README.md +++ b/scripts/eval-runners/README.md @@ -54,12 +54,24 @@ execution. Missing or mismatched evidence makes the arm `incompatible`; it is never a reason to invoke the parent or the compatibility `runner.ps1 execute` transport. -The terminal result must come from the selected package runner. The parent may -persist that result at the exact manifest-declared path, but it must not replace -it with a worker summary or synthesize a normalized result. Native bridging -also checks the result's runner identity, the descriptor's exact delegation -mechanism, and a hashed transcript/event artifact. An `incompatible` arm is -diagnostic-only: it is never gradeable and fails the completion/benchmark gate. +The harness-native transport returns a terminal envelope with schema +`codebeltnet/agentic/eval-native-worker-result/1`. The envelope declares +`capture.source = harness_native_transport`, `capture.terminal = true`, and +`capture.worker_authored = false`; the model worker's answer is data inside the +envelope, never its author. If the selected harness exposes only assistant text +or a worker-authored summary, the arm is incompatible. After the worker is +terminal, the orchestrator writes only that captured envelope to a package-local +temporary path and invokes `record-native-result.ps1`. That deterministic +package-runner helper derives the exact run/profile identity, timestamps, +requested configuration, runner/harness identity, and `eval-execution-result/1` +shape, then validates the native evidence before writing the manifest-declared +raw result. The parent may persist only that helper-produced result; it must not +replace it with a worker summary, hand-write `execution-result.json`, or +synthesize a normalized result. +Native bridging also checks the result's runner identity, the descriptor's exact +delegation mechanism, and a hashed transcript/event artifact. An `incompatible` +arm is diagnostic-only: it is never gradeable and fails the completion/benchmark +gate. The descriptor's `delegation` object records the native mechanism, worker role, advertised full-capability/model-lock/working-directory/result-capture @@ -88,22 +100,15 @@ Native delegation mechanisms: sibling Task calls for the first batch in one assistant turn and must not ask for confirmation or wait between calls. A client that cannot do that is incompatible; available-capacity serial dispatch is not a fallback. -- Cline: a full-capability Cline SDK Agent Squad - `start_subagent(preset: "anvil")` child session backed by - `ClineCore.create`. The plugin's default `phantom` preset and Cline's - documented `use_subagents` research feature are read-only and are rejected - for mutable evals. Plugin/SDK discovery is only preflight readiness; the - actual child remains conditional until terminal evidence proves its controls. - This directory contains the package-local implementation of the v0.9.1 Eval Runner protocol. It is copied into prepared packages so the external Eval Orchestrator can use the same runner implementation that was validated with the package. It is not a model executor used by repository automation. -The boundary has three documents: +The boundary has a native terminal envelope plus the runner-owned raw result: ```text -run.json + execution-profile.json -> runner -> execution-result.json +run.json + execution-profile.json -> native worker envelope -> record-native-result.ps1 -> execution-result.json ``` `run.json` is the existing portable one-arm contract. It owns the prompt, @@ -128,6 +133,17 @@ runner.ps1 preflight -Run -Profile runner.ps1 execute -Run -Profile ``` +The native handoff additionally uses: + +```text +record-native-result.ps1 -Runner -Run -Profile -NativeResult -Output +``` + +`record-native-result.ps1` is deterministic and never starts a harness or a +model. The direct `execute` command remains the compatibility/conformance +transport; native delegation must not invoke it because that would create a +second model execution. + The commands emit one JSON document. `describe` and `preflight` do not consume model tokens. `execute` runs exactly one arm, never resumes a session, never grades or retries for answer quality, and returns a normalized result even for @@ -146,7 +162,7 @@ incompatible. The fake runner is deterministic and is the conformance reference. It has no harness-native delegation surface and its compatibility output is never proof for a real harness. GitHub -Copilot, Codex, OpenCode, and Cline are thin harness-specific adapters. Their +Copilot, Codex, and OpenCode are thin harness-specific adapters. Their native CLI flags, environment setup, event parsing, authentication injection, and isolation checks stay inside their own directories. Windows is supported in pragmatic mode when the native CLI satisfies the mandatory controls. @@ -188,14 +204,14 @@ workspace-write`; it does not combine explicit sandbox selection with `--approve-for-me`. OpenCode uses `run --format json --auto --model ` with isolated global/config roots and preserves repository-owned project configuration; it does not depend on -`OPENCODE_DISABLE_PROJECT_CONFIG` or use `--pure`. Cline expects a -`provider/model` selector in the profile and derives its native `--provider` -and `--model` arguments inside the adapter. Each captures an exact observable +`OPENCODE_DISABLE_PROJECT_CONFIG` or use `--pure`. + +Each captures an exact observable CLI version and passes only documented environment credentials when the selected runner supports them. None copies a global skill directory, memory store, plugin set, or normal agent profile into a run. -Model discovery lives in `scripts/Get-HarnessModels.ps1`. It uses the current local harness catalog where available: Copilot through the installed CLI SDK help-visible model list, Codex through `codex debug models`, OpenCode through `opencode models opencode --verbose`, and Cline through the installed `@cline/llms` registry. Cline and OpenCode discovery returns only models with current metadata proving free availability; zero free models is a clear local failure, not a fallback to paid models. +Model discovery lives in `scripts/Get-HarnessModels.ps1`. It uses the current local harness catalog where available: Copilot through the installed CLI SDK help-visible model list, Codex through `codex debug models`, OpenCode through `opencode models opencode --verbose`. OpenCode discovery returns only models with current metadata proving free availability; zero free models is a clear local failure, not a fallback to paid models. Freebuff is currently documented as planned/blocked. Its supported CLI remains TUI-oriented and does not provide the required one-prompt, noninteractive, diff --git a/scripts/eval-runners/cline/runner.ps1 b/scripts/eval-runners/cline/runner.ps1 deleted file mode 100644 index c516ac8..0000000 --- a/scripts/eval-runners/cline/runner.ps1 +++ /dev/null @@ -1,642 +0,0 @@ -<#! -.SYNOPSIS - Cline Eval Runner adapter. - -.DESCRIPTION - The adapter uses Cline's supported headless JSON/NDJSON surface. It starts - one fresh process per arm, supplies the prompt on stdin, uses run-local - data/config and hooks directories, and never supplies a session id. -#> -[CmdletBinding()] -param( - [Parameter(Mandatory = $true, Position = 0)] - [ValidateSet('describe', 'preflight', 'execute')] - [string]$Command, - - [string]$Run, - [string]$Profile -) - -$ErrorActionPreference = 'Stop' -Set-StrictMode -Version Latest -. (Join-Path $PSScriptRoot '..\runner-common.ps1') - -$descriptor = [ordered]@{ - schema = (Get-RunnerSchemaNames).Descriptor - protocol_version = (Get-RunnerSchemaNames).Protocol - name = 'cline' - version = '0.9.1' - platforms = @('windows', 'linux', 'macos') - harness = [ordered]@{ name = 'Cline CLI'; version = 'unavailable' } - capabilities = [ordered]@{ - fresh_context = 'supported' - isolated_home_config = 'supported' - isolated_working_directory = 'supported' - filesystem_confinement = 'conditional' - ambient_candidate_skill_exclusion = 'supported' - candidate_skill_exposure = 'supported' - prompt_fidelity = 'supported' - model_configuration_lock = 'supported' - response_capture = 'supported' - transcript_event_capture = 'supported' - token_telemetry = 'conditional' - cache_token_telemetry = 'conditional' - tool_call_telemetry = 'conditional' - command_evidence = 'conditional' - file_evidence = 'conditional' - cost_telemetry = 'conditional' - credential_child_filtering = 'conditional' - native_skill_activation_evidence = 'unsupported' - native_worker_delegation = 'conditional' - delegated_worker_full_capability = 'conditional' - delegated_worker_model_lock = 'conditional' - delegated_worker_working_directory = 'conditional' - delegated_worker_result_capture = 'conditional' - delegated_worker_capacity_signal = 'conditional' - } - delegation = [ordered]@{ - mode = 'conditional' - mechanism = 'Cline SDK Agent Squad start_subagent(preset: anvil) child session backed by ClineCore.create; the read-only use_subagents research feature is not acceptable' - worker_role = 'agents-squad-child-agent' - full_capability = 'conditional' - model_lock = 'conditional' - working_directory = 'conditional' - result_capture = 'conditional' - capacity = 'harness_authoritative' - nested_model_execution = $false - } - supported_telemetry = @('transcript_event_capture', 'token_telemetry', 'cache_token_telemetry', 'tool_call_telemetry', 'command_evidence', 'file_evidence', 'cost_telemetry') - configuration_profiles = @('isolated-default') - tool_profiles = @('default') -} - -function Write-ProtocolError { - param([string]$Message) - - [Console]::Error.WriteLine($Message) - exit 2 -} - -function Resolve-ClineInputs { - if ([string]::IsNullOrWhiteSpace($Run) -or [string]::IsNullOrWhiteSpace($Profile)) { - throw 'preflight and execute require -Run and -Profile.' - } - return [pscustomobject]@{ - Run = Resolve-RunContract -RunPath $Run - Profile = Resolve-ExecutionProfile -ProfilePath $Profile - } -} - -function Invoke-ClineCli { - param( - [Parameter(Mandatory = $true)][object]$CommandInfo, - [Parameter(Mandatory = $true)][string[]]$Arguments, - [Parameter(Mandatory = $true)][object]$Inputs, - [System.Collections.IDictionary]$Environment, - [byte[]]$InputBytes = @(), - [int]$TimeoutSeconds = 60 - ) - - return Invoke-RunnerProcess -FileName $CommandInfo.FileName -ArgumentList (@($CommandInfo.Prefix) + @($Arguments)) -WorkingDirectory $Inputs.Run.WorkingDirectoryPath -Environment $Environment -InputBytes $InputBytes -TimeoutSeconds $TimeoutSeconds -} - -function Get-ClineDescriptor { - $copy = [ordered]@{} - foreach ($key in $descriptor.Keys) { $copy[$key] = $descriptor[$key] } - $commandInfo = Resolve-ExternalCommand -Name 'cline' - $version = 'unavailable' - if ($null -ne $commandInfo) { - $observation = Get-ExternalCommandVersion -CommandInfo $commandInfo - $version = [string]$observation.Version - } - $copy.harness = [ordered]@{ name = 'Cline CLI'; version = $version } - return $copy -} - -function Resolve-ClineModelSelector { - param([string]$Model) - - if ([string]::IsNullOrWhiteSpace($Model)) { - return [pscustomobject]@{ Provider = $null; Model = $null; Valid = $false } - } - - $parts = $Model.Split([char[]]@('/'), 2, [System.StringSplitOptions]::None) - if ($parts.Count -lt 2 -or [string]::IsNullOrWhiteSpace($parts[0]) -or [string]::IsNullOrWhiteSpace($parts[1])) { - return [pscustomobject]@{ Provider = $null; Model = $Model; Valid = $false } - } - - return [pscustomobject]@{ Provider = $parts[0]; Model = $parts[1]; Valid = $true } -} - -function New-ClineEnvironment { - param([Parameter(Mandatory = $true)][object]$Inputs) - - $clineRoot = Join-Path $Inputs.Run.HomeDirectoryPath '.cline' - $dataDirectory = Join-Path $clineRoot 'data' - $settingsDirectory = Join-Path $dataDirectory 'settings' - $sandboxDataDirectory = Join-Path $clineRoot 'sandbox-data' - $teamDataDirectory = Join-Path $dataDirectory 'teams' - $hooksDirectory = Join-Path $clineRoot 'hooks' - foreach ($directory in @($clineRoot, $dataDirectory, $settingsDirectory, $sandboxDataDirectory, $teamDataDirectory, $hooksDirectory)) { - New-Item -ItemType Directory -Path $directory -Force | Out-Null - } - $selector = Resolve-ClineModelSelector -Model ([string]$Inputs.Profile.Model) - $authVariables = @(if ($selector.Valid) { Get-ProviderAuthenticationVariables -Provider $selector.Provider }) - return [pscustomobject]@{ - Environment = New-RunnerEnvironment -Run $Inputs.Run -AuthenticationVariables $authVariables -Additional @{ - CLINE_DATA_DIR = $dataDirectory - CLINE_SANDBOX_DATA_DIR = $sandboxDataDirectory - CLINE_HOOKS_DIR = $hooksDirectory - CLINE_SESSION_BACKEND_MODE = 'local' - } - Root = $clineRoot - DataDirectory = $dataDirectory - SettingsDirectory = $settingsDirectory - SandboxDataDirectory = $sandboxDataDirectory - TeamDataDirectory = $teamDataDirectory - HooksDirectory = $hooksDirectory - ConfigPath = $clineRoot - } -} - -function Get-ClineInsideEnvironment { - param( - [Parameter(Mandatory = $true)][object]$Inputs, - [Parameter(Mandatory = $true)][object]$EnvironmentData - ) - - $inside = [ordered]@{ - HOME = '/run/home' - USERPROFILE = '/run/home' - XDG_CONFIG_HOME = '/run/home/.config' - XDG_DATA_HOME = '/run/home/.local/share' - XDG_CACHE_HOME = '/run/home/.cache' - TEMP = '/run/home/tmp' - TMP = '/run/home/tmp' - CLINE_DATA_DIR = '/run/home/.cline/data' - CLINE_SANDBOX_DATA_DIR = '/run/home/.cline/sandbox-data' - CLINE_HOOKS_DIR = '/run/home/.cline/hooks' - CLINE_SESSION_BACKEND_MODE = 'local' - PATH = '/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin' - CI = '1' - NO_COLOR = '1' - } - $selector = Resolve-ClineModelSelector -Model ([string]$Inputs.Profile.Model) - $authVariables = @(if ($selector.Valid) { Get-ProviderAuthenticationVariables -Provider $selector.Provider }) - foreach ($name in $authVariables) { - if ($EnvironmentData.Environment.Contains($name) -and -not [string]::IsNullOrWhiteSpace([string]$EnvironmentData.Environment[$name])) { - $inside[$name] = [string]$EnvironmentData.Environment[$name] - } - } - return $inside -} - -function New-ClineCliArguments { - param( - [Parameter(Mandatory = $true)][object]$Inputs, - [Parameter(Mandatory = $true)][object]$EnvironmentData, - [ValidateSet('windows', 'linux', 'macos', 'unknown')][string]$VisiblePlatform = (Get-PlatformName) - ) - - $workingDirectory = Get-SandboxVisiblePath -HostPath $Inputs.Run.WorkingDirectoryPath -RunRoot $Inputs.Run.RunRoot -Platform $VisiblePlatform - $configPath = Get-SandboxVisiblePath -HostPath $EnvironmentData.ConfigPath -RunRoot $Inputs.Run.RunRoot -Platform $VisiblePlatform - $dataRoot = Get-SandboxVisiblePath -HostPath $EnvironmentData.DataDirectory -RunRoot $Inputs.Run.RunRoot -Platform $VisiblePlatform - $hooksDirectory = Get-SandboxVisiblePath -HostPath $EnvironmentData.HooksDirectory -RunRoot $Inputs.Run.RunRoot -Platform $VisiblePlatform - $selector = Resolve-ClineModelSelector -Model ([string]$Inputs.Profile.Model) - $arguments = [System.Collections.Generic.List[string]]::new() - foreach ($argument in @('--json', '--auto-approve', 'true', '--cwd', $workingDirectory, '--config', $configPath, '--data-dir', $dataRoot, '--hooks-dir', $hooksDirectory, '--provider', $selector.Provider, '--model', $selector.Model, '--retries', '0', '--timeout', [string]$Inputs.Profile.TimeoutSeconds)) { - $arguments.Add([string]$argument) - } - if (-not [string]::IsNullOrWhiteSpace([string]$Inputs.Profile.ReasoningEffort)) { - $arguments.Add('--thinking') - $arguments.Add([string]$Inputs.Profile.ReasoningEffort) - } - return @($arguments) -} - -function Get-ClineAgentsSquadCapability { - $configuredPath = [Environment]::GetEnvironmentVariable('CLINE_AGENTS_SQUAD_PLUGIN') - if ([string]::IsNullOrWhiteSpace($configuredPath)) { - return [pscustomobject]@{ - Available = $false - Detail = 'No CLINE_AGENTS_SQUAD_PLUGIN path is configured; the read-only use_subagents feature cannot execute a mutable eval arm.' - } - } - try { - $resolvedPath = [System.IO.Path]::GetFullPath($configuredPath) - if (-not (Test-Path -LiteralPath $resolvedPath -PathType Container)) { - return [pscustomobject]@{ - Available = $false - Detail = "Configured Cline Agent Squad path '$resolvedPath' is not an existing directory." - } - } - $packageJson = Join-Path $resolvedPath 'package.json' - if (-not (Test-Path -LiteralPath $packageJson -PathType Leaf)) { - return [pscustomobject]@{ - Available = $false - Detail = "Configured Cline Agent Squad path '$resolvedPath' has no package.json marker." - } - } - $package = Read-RunnerJson -Path $packageJson - $pluginEntries = @((Get-JsonProperty -Object (Get-JsonProperty -Object $package -Name 'cline' -Default $null) -Name 'plugins' -Default @())) - $runtimeEntryProven = $false - foreach ($pluginEntry in $pluginEntries) { - $capabilities = @((Get-JsonProperty -Object $pluginEntry -Name 'capabilities' -Default @()) | ForEach-Object { [string]$_ }) - $entryPaths = @((Get-JsonProperty -Object $pluginEntry -Name 'paths' -Default @())) - if ($capabilities -contains 'hooks' -and $capabilities -contains 'tools' -and $entryPaths.Count -gt 0) { - foreach ($entryPath in $entryPaths) { - $candidateEntry = [System.IO.Path]::GetFullPath((Join-Path $resolvedPath ([string]$entryPath))) - if (Test-PathInside -BasePath $resolvedPath -CandidatePath $candidateEntry -and (Test-Path -LiteralPath $candidateEntry -PathType Leaf)) { - $runtimeEntryProven = $true - break - } - } - } - if ($runtimeEntryProven) { break } - } - if (-not $runtimeEntryProven) { - return [pscustomobject]@{ - Available = $false - Detail = "Configured Cline Agent Squad path '$resolvedPath' does not expose a cline plugin entry with hooks/tools and an existing entry file." - } - } - return [pscustomobject]@{ - Available = $true - Detail = "Cline Agent Squad plugin manifest and runtime entry are available at '$resolvedPath'; the worker must use start_subagent with the full-capability anvil preset, backed by ClineCore.create with explicit cwd, model, tools, and result polling." - } - } catch { - return [pscustomobject]@{ - Available = $false - Detail = "Cline Agent Squad plugin path could not be resolved: $($_.Exception.Message)" - } - } -} - -function Get-ClineCapabilityMap { - param( - [Parameter(Mandatory = $true)][object]$Inputs, - [bool]$HardFilesystemConfinement = $false, - [bool]$NativeWorkerAvailable = $false - ) - - $capabilities = [ordered]@{} - foreach ($capabilityName in @(Get-JsonPropertyNames -Object $descriptor.capabilities)) { - $capabilities[$capabilityName] = [string](Get-JsonProperty -Object $descriptor.capabilities -Name $capabilityName) - } - $capabilities['filesystem_confinement'] = if ($HardFilesystemConfinement) { 'supported' } else { 'unsupported' } - $capabilities['candidate_skill_exposure'] = if ($Inputs.Run.CandidateSkillExposed) { 'supported' } else { 'excluded' } - foreach ($name in @('native_worker_delegation', 'delegated_worker_full_capability', 'delegated_worker_model_lock', 'delegated_worker_working_directory', 'delegated_worker_result_capture', 'delegated_worker_capacity_signal')) { - # Agent Squad package discovery proves local readiness only. It does - # not prove what the child session actually used or returned. - $capabilities[$name] = if ($NativeWorkerAvailable) { 'conditional' } else { 'unsupported' } - } - return $capabilities -} - -function Get-ClinePreflight { - param([Parameter(Mandatory = $true)][object]$Inputs) - - $checks = [System.Collections.Generic.List[object]]::new() - $reasons = [System.Collections.Generic.List[string]]::new() - $warnings = [System.Collections.Generic.List[string]]::new() - $profile = $Inputs.Profile - $run = $Inputs.Run - $platform = Get-PlatformName - $commandInfo = Resolve-ExternalCommand -Name 'cline' - $sandboxInfo = if ($platform -eq 'linux') { Resolve-ExternalCommand -Name 'bwrap' } elseif ($platform -eq 'macos') { Resolve-ExternalCommand -Name 'sandbox-exec' } else { $null } - $versionObservation = $null - $agentsSquad = Get-ClineAgentsSquadCapability - - if ($profile.Runner -ne 'cline') { - $reasons.Add("execution-profile.json selects '$($profile.Runner)' rather than cline.") - } else { - $checks.Add((New-PreflightCheck -Name 'runner_selection' -Status passed -Detail 'The selected runner is cline.')) - } - $selector = Resolve-ClineModelSelector -Model ([string]$profile.Model) - if ([string]::IsNullOrWhiteSpace($profile.Model)) { - $reasons.Add('Cline requires a model in execution-profile.json.') - } elseif (-not $selector.Valid) { - $reasons.Add("Cline requires a runner-native model selector in 'provider/model' form; received '$($profile.Model)'.") - } else { - $checks.Add((New-PreflightCheck -Name 'model' -Status passed -Detail $profile.Model)) - $checks.Add((New-PreflightCheck -Name 'native_model_selector' -Status passed -Detail "--provider $($selector.Provider); --model $($selector.Model)")) - } - if ($profile.ConfigurationProfile -ne 'isolated-default') { $reasons.Add("configuration_profile '$($profile.ConfigurationProfile)' is unsupported by cline.") } - if ($profile.ToolProfile -ne 'default') { $reasons.Add("tool_profile '$($profile.ToolProfile)' is unsupported by cline.") } - - if ($agentsSquad.Available) { - $checks.Add((New-PreflightCheck -Name 'native_worker_delegation' -Status passed -Detail ($agentsSquad.Detail + ' This proves plugin/API readiness only; the actual child remains conditional until terminal evidence.'))) - $warnings.Add('Cline Agent Squad native-worker controls remain conditional until terminal evidence proves the actual child model, cwd, HOME/config, fresh identity, prompt, exclusions, and terminal capture.') - } else { - $checks.Add((New-PreflightCheck -Name 'native_worker_delegation' -Status unavailable -Detail $agentsSquad.Detail)) - $warnings.Add('Cline native worker delegation is unavailable. The external orchestrator must not use the read-only use_subagents feature or the direct CLI execute transport as a fallback; install the full-capability Agent Squad plugin and set CLINE_AGENTS_SQUAD_PLUGIN.') - } - - $environmentData = New-ClineEnvironment -Inputs $Inputs - if ($null -eq $commandInfo) { - $reasons.Add('The Cline CLI executable is not available on PATH.') - } else { - $checks.Add((New-PreflightCheck -Name 'harness_executable' -Status passed -Detail $commandInfo.Source)) - try { - $versionObservation = Get-ExternalCommandVersion -CommandInfo $commandInfo -WorkingDirectory $run.WorkingDirectoryPath -Environment $environmentData.Environment -TimeoutSeconds 30 - if (-not $versionObservation.Available) { - $reasons.Add('The Cline CLI did not expose an exact observable version through --version.') - $checks.Add((New-PreflightCheck -Name 'harness_version' -Status unavailable -Detail 'cline --version did not return a usable version string.')) - } else { - $checks.Add((New-PreflightCheck -Name 'harness_version' -Status passed -Detail ([string]$versionObservation.Version))) - } - $help = Invoke-ClineCli -CommandInfo $commandInfo -Arguments @('--retries', '0', '--help') -Inputs $Inputs -Environment $environmentData.Environment -TimeoutSeconds 30 - if ($help.TimedOut -or $help.ExitCode -ne 0) { - $reasons.Add("Cline --retries 0 --help failed with exit status $($help.ExitCode).") - } else { - $helpText = [string]::Join("`n", @($help.Stdout, $help.Stderr)) - foreach ($flag in @('--json', '--auto-approve', '--cwd', '--config', '--data-dir', '--hooks-dir', '--provider', '--model', '--thinking', '--timeout', '--retries')) { - if ($helpText -notmatch [regex]::Escape($flag)) { $reasons.Add("The installed Cline CLI does not advertise required flag '$flag'.") } - } - $visiblePlatform = if ($platform -eq 'linux' -and $null -ne $sandboxInfo) { 'linux' } else { $platform } - $constructed = New-ClineCliArguments -Inputs $Inputs -EnvironmentData $environmentData -VisiblePlatform $visiblePlatform - foreach ($forbidden in @('--id', '--continue', '--session', '--yolo', '--zen', '--tui')) { - if (@($constructed) -contains $forbidden) { $reasons.Add("The constructed Cline invocation must not use resume or interactive option '$forbidden'.") } - } - $retryIndex = [Array]::IndexOf([string[]]$constructed, '--retries') - if ($retryIndex -lt 0 -or $constructed[$retryIndex + 1] -ne '0') { $reasons.Add('The constructed Cline invocation must set --retries 0.') } - if ($reasons.Count -eq 0) { - $checks.Add((New-PreflightCheck -Name 'harness_contract' -Status passed -Detail 'Cline advertises JSON/NDJSON output, isolated directories, provider/model selection, timeout, thinking, auto-approval, and zero internal retries.')) - } - } - } catch { - $reasons.Add("Could not inspect Cline CLI capabilities: $($_.Exception.Message)") - } - } - - $authVariables = @(if ($selector.Valid) { Get-ProviderAuthenticationVariables -Provider $selector.Provider }) - $authVariable = $null - foreach ($name in $authVariables) { - if (-not [string]::IsNullOrWhiteSpace([Environment]::GetEnvironmentVariable($name))) { $authVariable = $name; break } - } - if ($authVariables.Count -eq 0) { - $checks.Add((New-PreflightCheck -Name 'authentication' -Status not_applicable -Detail 'No runner-known provider API-key environment variable is required for this Cline model selector.')) - } elseif ([string]::IsNullOrWhiteSpace($authVariable)) { - $reasons.Add("No narrow provider authentication environment variable is available for model provider '$($selector.Provider)'; ambient Cline auth profiles are not copied into an eval run.") - } else { - $checks.Add((New-PreflightCheck -Name 'authentication' -Status passed -Detail "Provider credential will be passed only as $authVariable.")) - } - - if ($platform -notin @('linux', 'macos')) { - $checks.Add((New-PreflightCheck -Name 'filesystem_confinement' -Status not_applicable -Detail "Platform '$platform' has no configured external hard-confinement mechanism; pragmatic isolation remains available.")) - $warnings.Add("Platform '$platform' has no external hard filesystem confinement in this adapter; execution will report pragmatic isolation.") - } elseif ($null -eq $sandboxInfo) { - $sandboxName = if ($platform -eq 'linux') { 'bwrap' } else { 'sandbox-exec' } - $checks.Add((New-PreflightCheck -Name 'filesystem_confinement' -Status unavailable -Detail "External '$sandboxName' is unavailable; pragmatic isolation remains available.")) - $warnings.Add("External '$sandboxName' was unavailable; execution will report pragmatic isolation.") - } else { - $checks.Add((New-PreflightCheck -Name 'filesystem_confinement' -Status passed -Detail "External $($sandboxInfo.Source) confines Cline to the staged run and run-local data/config roots.")) - } - $checks.Add((New-PreflightCheck -Name 'fresh_session' -Status passed -Detail 'The adapter starts one new Cline process, supplies no --id, and never reuses a session.')) - $checks.Add((New-PreflightCheck -Name 'retry_semantics' -Status passed -Detail '--retries 0 disables Cline consecutive operational retries; the runner still starts exactly one semantic process with attempt_count=1.')) - $checks.Add((New-PreflightCheck -Name 'ambient_configuration' -Status passed -Detail 'HOME, Cline data/config, hooks, sessions, and plugin roots are run-local and empty; no ambient user profile is copied.')) - $checks.Add((New-PreflightCheck -Name 'prompt_fidelity' -Status passed -Detail 'The exact prompt bytes are sent on stdin as the first and only task input.')) - $warnings.Add('Cline does not expose a supported child-tool environment filter in this CLI contract; the runner removes unrelated inherited variables but cannot independently prove that the selected provider credential is hidden from every Cline-launched tool.') - - $hardConfinement = $null -ne $sandboxInfo -and $platform -in @('linux', 'macos') - $capabilities = Get-ClineCapabilityMap -Inputs $Inputs -HardFilesystemConfinement $hardConfinement -NativeWorkerAvailable $agentsSquad.Available - $harnessVersion = if ($null -eq $versionObservation) { 'unavailable' } else { [string]$versionObservation.Version } - $descriptorCopy = [ordered]@{} - foreach ($key in $descriptor.Keys) { $descriptorCopy[$key] = $descriptor[$key] } - $descriptorCopy.harness = [ordered]@{ name = 'Cline CLI'; version = $harnessVersion } - $mechanisms = [System.Collections.Generic.List[string]]::new() - foreach ($mechanism in @('cline --json', '--auto-approve true', '--retries 0', 'no --id session resume', 'run-local HOME', 'run-local Cline data/config/hooks directories', 'prompt on stdin')) { $mechanisms.Add($mechanism) } - if ($hardConfinement) { $mechanisms.Add("external $($sandboxInfo.Source) filesystem sandbox") } else { $mechanisms.Add('pragmatic process/environment isolation without hard filesystem confinement') } - return New-PreflightDocument -Descriptor $descriptorCopy -Profile $profile -Run $run -Compatible ($reasons.Count -eq 0) -Checks @($checks) -Mechanisms @($mechanisms) -ResolvedCapabilities $capabilities -Warnings @($warnings) -Reasons @($reasons) -} - -function Write-ClineCapture { - param( - [Parameter(Mandatory = $true)][object]$RunData, - [Parameter(Mandatory = $true)][string]$RelativePath, - [Parameter(Mandatory = $true)][AllowEmptyString()][string]$Text - ) - - $path = Join-Path $RunData.Run.RunRoot ($RelativePath -replace '/', [System.IO.Path]::DirectorySeparatorChar) - New-Item -ItemType Directory -Path (Split-Path -Parent $path) -Force | Out-Null - [System.IO.File]::WriteAllText($path, $Text, [System.Text.UTF8Encoding]::new($false)) - return New-ArtifactReference -Run $RunData.Run -Path $RelativePath -Scope run -MediaType (Get-MediaType -Path $RelativePath) -} - -function Invoke-ClineExecute { - param([Parameter(Mandatory = $true)][object]$Inputs) - - $preflight = Get-ClinePreflight -Inputs $Inputs - $started = [DateTime]::UtcNow - $sessionId = [Guid]::NewGuid().ToString('D') - $executionDescriptor = [ordered]@{} - foreach ($key in $descriptor.Keys) { $executionDescriptor[$key] = $descriptor[$key] } - $executionDescriptor.harness = $preflight.harness - if ($preflight.status -ne 'compatible') { - $finished = [DateTime]::UtcNow - return New-ExecutionResult -Descriptor $executionDescriptor -Profile $Inputs.Profile -Run $Inputs.Run -Status incompatible -FinalResponseReason 'preflight_incompatible' -StartedUtc $started.ToString('o') -FinishedUtc $finished.ToString('o') -DurationSeconds ($finished - $started).TotalSeconds -Failure (New-ExecutionFailure -Code 'incompatible' -Message ([string]::Join('; ', @($preflight.reasons)))) -SessionId $sessionId -IsolationCapabilities ([ordered]@{}) -IsolationMechanisms @('preflight-only') -Evidence ([ordered]@{ preflight = $preflight; resume = $false; session_id_supplied = $false }) -AttemptCount 1 - } - - $commandInfo = Resolve-ExternalCommand -Name 'cline' - $environmentData = New-ClineEnvironment -Inputs $Inputs - $platform = Get-PlatformName - $sandboxInfo = if ($platform -eq 'linux') { Resolve-ExternalCommand -Name 'bwrap' } elseif ($platform -eq 'macos') { Resolve-ExternalCommand -Name 'sandbox-exec' } else { $null } - $hardFilesystem = $null -ne $sandboxInfo -and $platform -in @('linux', 'macos') - $visiblePlatform = if ($hardFilesystem) { $platform } elseif ($platform -eq 'linux') { 'unknown' } else { $platform } - $arguments = New-ClineCliArguments -Inputs $Inputs -EnvironmentData $environmentData -VisiblePlatform $visiblePlatform - if ($platform -eq 'linux' -and $hardFilesystem) { - $insideEnvironment = Get-ClineInsideEnvironment -Inputs $Inputs -EnvironmentData $environmentData - $sandboxArguments = Get-LinuxEvalSandboxArguments -Inputs $Inputs -CommandInfo $commandInfo -InsideEnvironment $insideEnvironment -ReadOnlyRoots @('/usr', '/usr/local', '/bin', '/sbin', '/lib', '/lib64', '/libexec', '/etc', '/opt') - $process = Invoke-RunnerProcess -FileName $sandboxInfo.FileName -ArgumentList (@($sandboxArguments) + @($arguments)) -WorkingDirectory $Inputs.Run.WorkingDirectoryPath -Environment $environmentData.Environment -InputBytes $Inputs.Run.PromptBytes -TimeoutSeconds $Inputs.Profile.TimeoutSeconds - } elseif ($platform -eq 'macos' -and $hardFilesystem) { - $sandboxProfile = New-MacosEvalSandboxProfile -Inputs $Inputs -CommandInfo $commandInfo -ReadOnlyRoots @('/usr', '/usr/local', '/bin', '/sbin', '/lib', '/libexec', '/System', '/Library', '/opt', '/private/var/db') - $sandboxArguments = @('-f', $sandboxProfile, '--', $commandInfo.FileName) + @($commandInfo.Prefix) + @($arguments) - $process = Invoke-RunnerProcess -FileName $sandboxInfo.FileName -ArgumentList $sandboxArguments -WorkingDirectory $Inputs.Run.WorkingDirectoryPath -Environment $environmentData.Environment -InputBytes $Inputs.Run.PromptBytes -TimeoutSeconds $Inputs.Profile.TimeoutSeconds - } else { - $process = Invoke-ClineCli -CommandInfo $commandInfo -Arguments $arguments -Inputs $Inputs -Environment $environmentData.Environment -InputBytes $Inputs.Run.PromptBytes -TimeoutSeconds $Inputs.Profile.TimeoutSeconds - } - - $stdoutArtifact = Write-ClineCapture -RunData $Inputs -RelativePath 'evidence/cline-events.jsonl' -Text $process.Stdout - $stderrArtifact = Write-ClineCapture -RunData $Inputs -RelativePath 'evidence/cline-stderr.txt' -Text $process.Stderr - $artifacts = [System.Collections.Generic.List[object]]::new() - $artifacts.Add($stdoutArtifact); $artifacts.Add($stderrArtifact) - $parsed = ConvertFrom-JsonLines -Text $process.Stdout - $warnings = [System.Collections.Generic.List[string]]::new() - foreach ($parseError in @($parsed.Errors)) { $warnings.Add("Cline event parse error: $parseError") } - $eventCounts = @{} - $contentEnd = [System.Collections.Generic.List[string]]::new() - $contentStart = [System.Collections.Generic.List[string]]::new() - $jsonText = [System.Collections.Generic.List[string]]::new() - $jsonPartialText = [System.Collections.Generic.List[string]]::new() - $completionResultText = [System.Collections.Generic.List[string]]::new() - $finalText = $null - $returnedSessionId = $null - $failureMessage = $null - $usage = [ordered]@{} - $toolCalls = [System.Collections.Generic.List[object]]::new() - $commands = [System.Collections.Generic.List[object]]::new() - $files = [System.Collections.Generic.List[object]]::new() - foreach ($event in @($parsed.Events)) { - $topType = [string](Get-JsonProperty -Object $event -Name 'type' -Default '') - if ([string]::IsNullOrWhiteSpace($topType)) { - $warnings.Add('Cline emitted an event without a type; it was ignored.') - continue - } - if ($eventCounts.ContainsKey($topType)) { $eventCounts[$topType]++ } else { $eventCounts[$topType] = 1 } - $inner = Get-JsonProperty -Object $event -Name 'event' -Default $null - $eventType = if ($topType -eq 'agent_event' -and $null -ne $inner) { [string](Get-JsonProperty -Object $inner -Name 'type' -Default '') } else { $topType } - $payload = if ($topType -eq 'agent_event' -and $null -ne $inner) { $inner } else { $event } - - if ($topType -in @('say', 'ask')) { - $subtypeName = if ($topType -eq 'say') { 'say' } else { 'ask' } - $subtype = [string](Get-JsonProperty -Object $event -Name $subtypeName -Default '') - $text = [string](Get-JsonProperty -Object $event -Name 'text' -Default '') - $isPartial = [bool](Get-JsonProperty -Object $event -Name 'partial' -Default $false) - if (-not [string]::IsNullOrWhiteSpace($text)) { - if ($subtype -eq 'completion_result') { $completionResultText.Add($text) } - elseif ($subtype -eq 'text') { - if ($isPartial) { $jsonPartialText.Add($text) } else { $jsonText.Add($text) } - } - } - if ($topType -eq 'ask') { - if ($subtype -eq 'api_req_failed') { - $failureMessage = if ([string]::IsNullOrWhiteSpace($text)) { 'Cline reported an API request failure.' } else { $text } - } elseif ($subtype -in @('followup', 'plan_mode_respond', 'act_mode_respond')) { - $failureMessage = 'Cline requested interactive input during a noninteractive eval run.' - } elseif ($subtype -in @('use_mcp_server', 'command', 'tool')) { - $toolCalls.Add([ordered]@{ type = $subtype; name = $subtype }) - if ($subtype -eq 'command' -and -not [string]::IsNullOrWhiteSpace($text)) { - $commands.Add([ordered]@{ command = $text }) - } - } - } elseif ($subtype -in @('tool', 'command', 'command_output', 'mcp_server_request_started', 'mcp_server_response')) { - $toolName = [string](Get-JsonProperty -Object $event -Name 'name' -Default (Get-JsonProperty -Object $event -Name 'tool' -Default '')) - $toolCalls.Add([ordered]@{ type = $subtype; name = $toolName }) - if ($subtype -eq 'command' -and -not [string]::IsNullOrWhiteSpace($text)) { - $commands.Add([ordered]@{ command = $text }) - } - } - if ($subtype -eq 'completion_result' -and -not [string]::IsNullOrWhiteSpace($text)) { - $finalText = $text - } - if ($subtype -eq 'api_req_finished' -and -not [string]::IsNullOrWhiteSpace($text)) { - try { - $finishedUsage = $text | ConvertFrom-Json - foreach ($name in @('inputTokens', 'outputTokens', 'totalTokens', 'cacheReadTokens', 'cacheWriteTokens', 'cost')) { - $value = Get-JsonProperty -Object $finishedUsage -Name $name -Default $null - if ($null -ne $value) { $usage[$name] = $value } - } - } catch { - $warnings.Add('Cline api_req_finished text was not a usage JSON object; it was retained in the transcript.') - } - } - if ($subtype -notin @('task', 'error', 'api_req_started', 'api_req_finished', 'api_req_retried', 'api_req_retry_delayed', 'api_req_deleted', 'text', 'reasoning', 'completion_result', 'user_feedback', 'user_feedback_diff', 'command_output', 'tool', 'shell_integration_warning', 'browser_action', 'browser_action_result', 'command', 'mcp_server_request_started', 'mcp_server_response', 'new_task_started', 'new_task', 'subtask_result', 'checkpoint_saved', 'rooignore_error', 'diff_error', 'followup', 'plan_mode_respond', 'act_mode_respond', 'api_req_failed', 'use_mcp_server', 'resume_task', 'resume_completed_task', 'mistake_limit_reached', 'finishTask')) { - $warnings.Add("Unknown Cline $topType subtype '$subtype' was preserved as a warning.") - } - continue - } - switch ($eventType) { - 'done' { - $candidate = Get-JsonProperty -Object $payload -Name 'text' -Default '' - if (-not [string]::IsNullOrWhiteSpace([string]$candidate)) { $finalText = [string]$candidate } - $returnedSessionId = [string](Get-JsonProperty -Object $payload -Name 'sessionId' -Default (Get-JsonProperty -Object $payload -Name 'session_id' -Default $returnedSessionId)) - $doneUsage = Get-JsonProperty -Object $payload -Name 'usage' -Default $null - if ($null -ne $doneUsage) { - foreach ($name in @('inputTokens', 'outputTokens', 'totalTokens', 'cacheReadTokens', 'cacheWriteTokens', 'cost')) { - $value = Get-JsonProperty -Object $doneUsage -Name $name -Default $null - if ($null -ne $value) { $usage[$name] = $value } - } - } - } - 'content_end' { $text = [string](Get-JsonProperty -Object $payload -Name 'text' -Default ''); if (-not [string]::IsNullOrWhiteSpace($text)) { $contentEnd.Add($text) } } - 'content_start' { $text = [string](Get-JsonProperty -Object $payload -Name 'text' -Default ''); if (-not [string]::IsNullOrWhiteSpace($text)) { $contentStart.Add($text) } } - 'usage' { - foreach ($name in @('inputTokens', 'outputTokens', 'totalTokens', 'cacheReadTokens', 'cacheWriteTokens', 'cost')) { - $value = Get-JsonProperty -Object $payload -Name $name -Default $null - if ($null -ne $value) { $usage[$name] = $value } - } - } - 'tool_call' { $toolCalls.Add([ordered]@{ type = $eventType; name = [string](Get-JsonProperty -Object $payload -Name 'name' -Default (Get-JsonProperty -Object $payload -Name 'tool' -Default '')) }) } - 'tool_use' { $toolCalls.Add([ordered]@{ type = $eventType; name = [string](Get-JsonProperty -Object $payload -Name 'name' -Default (Get-JsonProperty -Object $payload -Name 'tool' -Default '')) }) } - 'command' { $commands.Add([ordered]@{ command = Get-JsonProperty -Object $payload -Name 'command' -Default (Get-JsonProperty -Object $payload -Name 'text' -Default '') }) } - 'file' { $files.Add([ordered]@{ path = Get-JsonProperty -Object $payload -Name 'path' -Default '' }) } - 'error' { $failureMessage = [string](Get-JsonProperty -Object $payload -Name 'message' -Default 'Cline emitted an error.') } - 'hook_event' { } - 'agent_start' { } - 'agent_end' { } - 'iteration_start' { } - 'iteration_end' { } - default { $warnings.Add("Unknown Cline event '$topType/$eventType' was preserved as a warning.") } - } - if ($eventType -eq 'usage') { - $eventUsage = Get-JsonProperty -Object $payload -Name 'usage' -Default $null - if ($null -ne $eventUsage) { - foreach ($name in @('inputTokens', 'outputTokens', 'totalTokens', 'cacheReadTokens', 'cacheWriteTokens', 'cost')) { - $value = Get-JsonProperty -Object $eventUsage -Name $name -Default $null - if ($null -ne $value) { $usage[$name] = $value } - } - } - } - } - if ([string]::IsNullOrWhiteSpace($finalText)) { - if ($contentEnd.Count -gt 0) { $finalText = [string]::Join('', @($contentEnd)) } - elseif ($contentStart.Count -gt 0) { $finalText = [string]::Join('', @($contentStart)) } - elseif ($completionResultText.Count -gt 0) { $finalText = [string]$completionResultText[$completionResultText.Count - 1] } - elseif ($jsonText.Count -gt 0) { $finalText = [string]$jsonText[$jsonText.Count - 1] } - elseif ($jsonPartialText.Count -gt 0) { $finalText = [string]::Join('', @($jsonPartialText)) } - } - - $status = 'completed' - $reason = $null - $failure = $null - $exitStatus = if ($process.TimedOut) { $null } else { [Nullable[int]]$process.ExitCode } - if ($process.TimedOut) { - $status = 'timed_out'; $reason = 'cline_timeout'; $failure = New-ExecutionFailure -Code 'timed_out' -Message 'Cline did not finish before timeout_seconds.' - } elseif ($process.ExitCode -ne 0 -or $null -ne $failureMessage) { - $failureDetail = if ([string]::IsNullOrWhiteSpace($failureMessage)) { 'Cline exited unsuccessfully.' } else { $failureMessage } - $status = 'failed'; $reason = 'cline_failure'; $failure = New-ExecutionFailure -Code 'cline_failure' -Message $failureDetail - } elseif ([string]::IsNullOrWhiteSpace($finalText)) { - $reason = 'cline_did_not_return_final_response'; $warnings.Add('Cline exited successfully without a final response event.') - } - $telemetry = [ordered]@{ - transcript = New-AvailableMetric -Value ([ordered]@{ artifact = 'evidence/cline-events.jsonl'; complete = $true }) - tokens = if ($usage.Count -eq 0) { New-UnavailableMetric -Reason 'cline_did_not_expose_usage' } else { New-AvailableMetric -Value $usage } - tool_calls = New-AvailableMetric -Value $toolCalls.Count - cost = if ($usage.Contains('cost')) { New-AvailableMetric -Value $usage['cost'] } else { New-UnavailableMetric -Reason 'cline_did_not_expose_cost' } - } - $finished = [DateTime]::UtcNow - $capabilities = Get-ClineCapabilityMap -Inputs $Inputs -HardFilesystemConfinement $hardFilesystem - $mechanisms = [System.Collections.Generic.List[string]]::new() - foreach ($mechanism in @('cline --json', '--auto-approve true', '--retries 0', 'no --id session resume', 'run-local HOME', 'run-local Cline data/config/hooks directories', 'prompt on stdin')) { $mechanisms.Add($mechanism) } - if ($hardFilesystem) { $mechanisms.Add("external $($sandboxInfo.Source) filesystem sandbox") } else { $mechanisms.Add('pragmatic process/environment isolation without hard filesystem confinement'); $warnings.Add('Hard filesystem confinement was unavailable; the completed arm is reported as pragmatic isolation.') } - $nativeSelector = Resolve-ClineModelSelector -Model ([string]$Inputs.Profile.Model) - $credentialEvidence = [ordered]@{ - model_provider = $nativeSelector.Provider - provider_environment_variables = @(if ($nativeSelector.Valid) { Get-ProviderAuthenticationVariables -Provider $nativeSelector.Provider }) - unrelated_environment_excluded = $true - child_tool_visibility = 'not_exposed_by_runner_environment; Cline child filtering is not independently observable' - value_observed = $false - } - $sessionResultId = if ([string]::IsNullOrWhiteSpace($returnedSessionId)) { $sessionId } else { $returnedSessionId } - $sandboxEvidence = if (-not $hardFilesystem) { 'unavailable' } elseif ($platform -eq 'linux') { 'bwrap' } else { 'sandbox-exec' } - return New-ExecutionResult -Descriptor $executionDescriptor -Profile $Inputs.Profile -Run $Inputs.Run -Status $status -FinalResponse $finalText -FinalResponseReason $reason -StartedUtc $process.StartedUtc.ToString('o') -FinishedUtc $finished.ToString('o') -DurationSeconds $process.DurationSeconds -ExitStatus $exitStatus -Failure $failure -SessionId $sessionResultId -IsolationCapabilities $capabilities -IsolationMechanisms @($mechanisms) -ResolvedConfiguration ([ordered]@{ status = 'accepted_request'; reason = 'Cline accepted the requested runner-native model selector, thinking, and configuration but did not expose concrete backend resolution.'; observations = [ordered]@{ model = $Inputs.Profile.Model; native_provider = $nativeSelector.Provider; native_model = $nativeSelector.Model; reasoning_effort = $Inputs.Profile.ReasoningEffort; retries = 0 } }) -Telemetry $telemetry -Artifacts @($artifacts) -Warnings @($warnings) -Evidence ([ordered]@{ event_counts = $eventCounts; commands = @($commands); files = @($files); prompt_first_input = $true; resume = $false; session_id_supplied = $false; retry_argument = 0; sandbox = $sandboxEvidence; credential = $credentialEvidence }) -AttemptCount 1 -} - -try { - [void](Assert-RunnerDescriptor -Descriptor $descriptor) - switch ($Command) { - 'describe' { Write-RunnerJson -Value (Get-ClineDescriptor) -AsOutput } - 'preflight' { - $inputs = Resolve-ClineInputs - Write-RunnerJson -Value (Get-ClinePreflight -Inputs $inputs) -AsOutput - } - 'execute' { - $inputs = Resolve-ClineInputs - $result = Invoke-ClineExecute -Inputs $inputs - [void](Assert-ExecutionResult -Result $result) - Write-RunnerJson -Value $result -AsOutput - } - } -} catch { - Write-ProtocolError -Message $_.Exception.Message -} diff --git a/scripts/eval-runners/tests/fixtures/cline-events.jsonl b/scripts/eval-runners/tests/fixtures/cline-events.jsonl deleted file mode 100644 index 21d2a65..0000000 --- a/scripts/eval-runners/tests/fixtures/cline-events.jsonl +++ /dev/null @@ -1,11 +0,0 @@ -{"type":"hook_event","hookEventName":"agent_start","sessionId":"fixture-session"} -{"type":"say","say":"text","text":"2 + 2 = 4.","partial":false} -{"type":"say","say":"tool","name":"read_file","text":"fixture.md","partial":false} -{"type":"say","say":"completion_result","text":"2 + 2 = 4.","partial":false} -{"type":"agent_event","event":{"type":"iteration_start","iteration":1}} -{"type":"agent_event","event":{"type":"content_start","contentType":"text","text":"2 + 2 = "}} -{"type":"agent_event","event":{"type":"content_end","contentType":"text","text":"2 + 2 = 4."}} -{"type":"agent_event","event":{"type":"usage","inputTokens":5022,"outputTokens":28,"totalTokens":5050,"cost":0.01}} -{"type":"agent_event","event":{"type":"done","reason":"completed","text":"2 + 2 = 4.","sessionId":"fixture-session","iterations":1,"usage":{"inputTokens":5022,"outputTokens":28,"totalTokens":5050,"cost":0.01}}} -{"type":"future.event.v99","payload":"preserve-as-warning"} -{"type":"hook_event","hookEventName":"agent_end","sessionId":"fixture-session"} From b3f804589ab3c529de26b9695de13a663c5cf0ba Mon Sep 17 00:00:00 2001 From: "aicia[bot]" Date: Tue, 25 Aug 2026 02:02:04 +0200 Subject: [PATCH 33/39] =?UTF-8?q?=E2=9C=A8=20add=20native=20worker=20resul?= =?UTF-8?q?t=20handling=20infrastructure?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Implement native worker terminal envelope schema and recording helper for eval orchestration. Move shared runner helper functions to runner-common and remove legacy bridge validation logic. --- .../eval-runners/bridge-execution-result.ps1 | 44 ---- .../native-worker-result.schema.json | 109 +++++++++ scripts/eval-runners/record-native-result.ps1 | 220 ++++++++++++++++++ scripts/eval-runners/runner-common.ps1 | 79 ++++++- 4 files changed, 405 insertions(+), 47 deletions(-) create mode 100644 scripts/eval-runners/contracts/native-worker-result.schema.json create mode 100644 scripts/eval-runners/record-native-result.ps1 diff --git a/scripts/eval-runners/bridge-execution-result.ps1 b/scripts/eval-runners/bridge-execution-result.ps1 index ca98ded..3832771 100644 --- a/scripts/eval-runners/bridge-execution-result.ps1 +++ b/scripts/eval-runners/bridge-execution-result.ps1 @@ -123,50 +123,6 @@ function Get-ExistingGrading { return $grading } -function Get-PackageRunnerDescriptor { - param([Parameter(Mandatory = $true)][string]$RunnerName) - - if ($RunnerName -notmatch '^[a-z0-9-]+$') { - throw "execution-profile.json runner '$RunnerName' is not a valid package runner name." - } - $runnerPath = Join-Path (Join-Path $PSScriptRoot $RunnerName) 'runner.ps1' - if (-not (Test-Path -LiteralPath $runnerPath -PathType Leaf)) { - throw "Package-local runner '$RunnerName' is missing its runner.ps1 descriptor." - } - - $descriptorOutput = & pwsh -NoProfile -File $runnerPath describe 2>&1 - if ($LASTEXITCODE -ne 0) { - throw "Package-local runner '$RunnerName' descriptor failed: $([string]::Join(' ', @($descriptorOutput)))" - } - try { - $descriptor = [string]::Join([Environment]::NewLine, @($descriptorOutput)) | ConvertFrom-Json - [void](Assert-RunnerDescriptor -Descriptor $descriptor) - } catch { - throw "Package-local runner '$RunnerName' returned an invalid descriptor: $($_.Exception.Message)" - } - if ([string]$descriptor.name -ne $RunnerName) { - throw "Package-local runner descriptor name '$($descriptor.name)' does not match selected runner '$RunnerName'." - } - return $descriptor -} - -function Assert-NativeTerminalCaptureArtifact { - param([Parameter(Mandatory = $true)][object]$ExecutionResult) - - $transcriptMetric = Get-JsonProperty -Object $ExecutionResult.telemetry -Name 'transcript' -Default $null - $transcriptStatus = [string](Get-JsonProperty -Object $transcriptMetric -Name 'status' -Default '') - $transcriptArtifact = [string](Get-JsonProperty -Object (Get-JsonProperty -Object $transcriptMetric -Name 'value' -Default $null) -Name 'artifact' -Default '') - if ($transcriptStatus -ne 'available' -or [string]::IsNullOrWhiteSpace($transcriptArtifact)) { - throw 'Native worker execution must provide an available terminal transcript artifact.' - } - $matchingArtifacts = @($ExecutionResult.artifacts | Where-Object { - [string](Get-JsonProperty -Object $_ -Name 'path' -Default '') -eq $transcriptArtifact - }) - if ($matchingArtifacts.Count -ne 1) { - throw "Native worker transcript artifact '$transcriptArtifact' is not recorded exactly once in execution-result.json." - } -} - try { $runData = Resolve-RunContract -RunPath $Run $runPath = $runData.RunPath diff --git a/scripts/eval-runners/contracts/native-worker-result.schema.json b/scripts/eval-runners/contracts/native-worker-result.schema.json new file mode 100644 index 0000000..48df39f --- /dev/null +++ b/scripts/eval-runners/contracts/native-worker-result.schema.json @@ -0,0 +1,109 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "https://codebelt.net/schemas/agentic/eval-native-worker-result/1", + "title": "Codebelt Agentic Native Worker Terminal Result", + "type": "object", + "additionalProperties": true, + "required": [ + "schema", + "run_id", + "session", + "status", + "run", + "final_response", + "timing", + "exit", + "isolation", + "telemetry", + "evidence", + "capture", + "artifacts", + "warnings", + "compatibility_deviations", + "attempt_count" + ], + "properties": { + "schema": { "const": "codebeltnet/agentic/eval-native-worker-result/1" }, + "run_id": { "type": "string", "minLength": 1 }, + "session": { + "type": "object", + "required": ["id", "fresh", "resumed"], + "properties": { + "id": { "type": "string", "minLength": 1 }, + "fresh": { "const": true }, + "resumed": { "const": false } + } + }, + "status": { "enum": ["completed", "failed", "timed_out", "cancelled", "incompatible"] }, + "run": { + "type": "object", + "required": ["eval_id", "eval_name", "configuration"], + "properties": { + "eval_id": { "type": "integer", "minimum": 1 }, + "eval_name": { "type": "string", "minLength": 1 }, + "configuration": { "enum": ["with_skill", "without_skill"] } + } + }, + "final_response": { + "type": "object", + "required": ["status"], + "properties": { + "status": { "enum": ["available", "unavailable"] }, + "text": { "type": "string" }, + "reason": { "type": "string" } + } + }, + "timing": { + "type": "object", + "required": ["started_utc", "finished_utc", "duration_seconds"], + "properties": { + "started_utc": { "type": "string", "format": "date-time" }, + "finished_utc": { "type": "string", "format": "date-time" }, + "duration_seconds": { "type": "number", "minimum": 0 } + } + }, + "exit": { + "type": "object", + "required": ["status"], + "properties": { + "status": { "type": ["integer", "null"] }, + "failure": {} + } + }, + "isolation": { + "type": "object", + "required": ["capabilities", "mechanisms"], + "properties": { + "capabilities": { "type": "object" }, + "mechanisms": { "type": "array", "items": { "type": "string" } } + } + }, + "telemetry": { "type": "object" }, + "evidence": { + "type": "object", + "required": ["delegation"], + "properties": { + "delegation": { "type": "object" } + } + }, + "capture": { + "type": "object", + "required": ["source", "terminal", "worker_authored"], + "properties": { + "source": { "const": "harness_native_transport" }, + "terminal": { "const": true }, + "worker_authored": { "const": false } + } + }, + "artifacts": { + "type": "array", + "items": { + "type": "object", + "required": ["path", "scope", "sha256", "size", "media_type"] + } + }, + "warnings": { "type": "array", "items": { "type": "string" } }, + "compatibility_deviations": { "type": "array", "items": { "type": "string" } }, + "attempt_count": { "const": 1 } + } +} diff --git a/scripts/eval-runners/record-native-result.ps1 b/scripts/eval-runners/record-native-result.ps1 new file mode 100644 index 0000000..e3205ca --- /dev/null +++ b/scripts/eval-runners/record-native-result.ps1 @@ -0,0 +1,220 @@ +<#! +.SYNOPSIS + Records a native worker terminal envelope as the runner-owned raw result. + +.DESCRIPTION + This command is deterministic. It never starts a harness or a model. The + external orchestrator uses it after a harness-native worker has finished so + that the selected package runner, rather than the orchestrator, owns the + eval-execution-result/1 serialization boundary. +#> +[CmdletBinding()] +param( + [Parameter(Mandatory = $true)][string]$Runner, + [Parameter(Mandatory = $true)][string]$Run, + [Parameter(Mandatory = $true)][string]$Profile, + [Parameter(Mandatory = $true)][string]$NativeResult, + [Parameter(Mandatory = $true)][string]$Output +) + +$ErrorActionPreference = 'Stop' +Set-StrictMode -Version Latest +. (Join-Path $PSScriptRoot 'runner-common.ps1') + +function ConvertTo-StringDictionary { + param([Parameter(Mandatory = $true)][object]$Value) + + $result = [ordered]@{} + foreach ($name in @(Get-JsonPropertyNames -Object $Value)) { + $result[[string]$name] = [string](Get-JsonProperty -Object $Value -Name ([string]$name) -Default '') + } + return $result +} + +function Assert-NativeResultEnvelope { + param( + [Parameter(Mandatory = $true)][object]$Native, + [Parameter(Mandatory = $true)][object]$RunData + ) + + if ([string]$Native.schema -ne 'codebeltnet/agentic/eval-native-worker-result/1') { + throw "Native worker result must declare 'codebeltnet/agentic/eval-native-worker-result/1'." + } + + foreach ($field in @('run_id', 'session', 'status', 'run', 'final_response', 'timing', 'exit', 'isolation', 'telemetry', 'evidence', 'capture', 'artifacts', 'warnings', 'compatibility_deviations', 'attempt_count')) { + if (-not (Test-JsonProperty -Object $Native -Name $field)) { + throw "Native worker result is missing '$field'." + } + } + if ([string]::IsNullOrWhiteSpace([string]$Native.run_id)) { + throw 'Native worker result run_id must be non-empty.' + } + if ([string]$Native.status -notin @('completed', 'failed', 'timed_out', 'cancelled', 'incompatible')) { + throw "Native worker result status '$($Native.status)' is unsupported." + } + if ([int]$Native.attempt_count -ne 1) { + throw 'Native worker result attempt_count must be exactly 1.' + } + + $runIdentity = $Native.run + if ([int]$runIdentity.eval_id -ne [int]$RunData.EvalId -or + [string]$runIdentity.eval_name -ne [string]$RunData.EvalName -or + [string]$runIdentity.configuration -ne [string]$RunData.Mode) { + throw 'Native worker result run identity does not match run.json.' + } + + $session = $Native.session + if ([string]::IsNullOrWhiteSpace([string]$session.id) -or + -not [bool]$session.fresh -or [bool]$session.resumed) { + throw 'Native worker result must identify a fresh, non-resumed session.' + } + + $response = $Native.final_response + if ([string]$response.status -eq 'available') { + if (-not (Test-JsonProperty -Object $response -Name 'text')) { + throw 'Available native worker responses must contain text.' + } + } elseif ([string]$response.status -eq 'unavailable') { + if ([string]::IsNullOrWhiteSpace([string]$response.reason)) { + throw 'Unavailable native worker responses must contain a reason.' + } + } else { + throw "Native worker final_response status '$($response.status)' is unsupported." + } + + $timing = $Native.timing + foreach ($field in @('started_utc', 'finished_utc', 'duration_seconds')) { + if (-not (Test-JsonProperty -Object $timing -Name $field)) { + throw "Native worker result timing.$field must be present." + } + } + try { + $started = [DateTime]::Parse([string]$timing.started_utc).ToUniversalTime() + $finished = [DateTime]::Parse([string]$timing.finished_utc).ToUniversalTime() + } catch { + throw "Native worker result timing timestamps are invalid: $($_.Exception.Message)" + } + if ($finished -lt $started -or [double]$timing.duration_seconds -lt 0) { + throw 'Native worker result timing must be ordered and non-negative.' + } + + $exit = $Native.exit + if (-not (Test-JsonProperty -Object $exit -Name 'status')) { + throw 'Native worker result exit.status must be present and numeric or null.' + } + $exitStatus = Get-JsonProperty -Object $exit -Name 'status' -Default $null + if ($null -ne $exitStatus -and -not ($exitStatus -is [byte] -or $exitStatus -is [sbyte] -or $exitStatus -is [int16] -or $exitStatus -is [uint16] -or $exitStatus -is [int32] -or $exitStatus -is [uint32] -or $exitStatus -is [int64] -or $exitStatus -is [uint64])) { + throw 'Native worker result exit.status must be a JSON number or null.' + } + + $isolation = $Native.isolation + if (-not (Test-JsonProperty -Object $isolation -Name 'capabilities') -or + -not (Test-JsonProperty -Object $isolation -Name 'mechanisms')) { + throw 'Native worker result isolation must declare capabilities and mechanisms.' + } + if (@(Get-JsonProperty -Object $isolation -Name 'mechanisms' -Default @()).Count -eq 0) { + throw 'Native worker result isolation.mechanisms must not be empty.' + } + if (-not (Test-JsonProperty -Object $Native.evidence -Name 'delegation')) { + throw 'Native worker result evidence.delegation must be present.' + } + $capture = $Native.capture + if ([string]$capture.source -ne 'harness_native_transport' -or + -not [bool]$capture.terminal -or [bool]$capture.worker_authored) { + throw 'Native worker result capture must come from the terminal harness-native transport; the worker may not author the envelope.' + } + if (@($Native.artifacts).Count -lt 1) { + throw 'Native worker result must record at least one artifact, including terminal evidence.' + } +} + +try { + $runData = Resolve-RunContract -RunPath $Run + $profileData = Resolve-ExecutionProfile -ProfilePath $Profile + if ([string]$profileData.Runner -ne $Runner) { + throw "Selected runner '$Runner' does not match execution-profile.json runner '$($profileData.Runner)'." + } + + $iterationDirectory = Split-Path -Parent (Split-Path -Parent $runData.RunRoot) + $nativePath = (Resolve-Path -LiteralPath $NativeResult -ErrorAction Stop).Path + $outputPath = [System.IO.Path]::GetFullPath($Output, (Get-Location).Path) + if (-not (Test-PathInside -BasePath $iterationDirectory -CandidatePath $nativePath)) { + throw 'Native worker result input must remain inside the prepared iteration package.' + } + if (-not (Test-PathInside -BasePath $iterationDirectory -CandidatePath $outputPath)) { + throw 'Recorded execution result output must remain inside the prepared iteration package.' + } + $outputDirectory = Split-Path -Parent $outputPath + if (-not (Test-Path -LiteralPath $outputDirectory -PathType Container)) { + throw "Recorded execution result directory '$outputDirectory' does not exist." + } + + $native = Read-RunnerJson -Path $nativePath + Assert-NativeResultEnvelope -Native $native -RunData $runData + $descriptor = Get-PackageRunnerDescriptor -RunnerName $Runner + + $capabilities = ConvertTo-StringDictionary -Value $native.isolation.capabilities + $response = $native.final_response + $finalText = if ([string]$response.status -eq 'available') { [string]$response.text } else { $null } + $finalReason = if ([string]$response.status -eq 'unavailable') { [string]$response.reason } else { $null } + $exitStatusValue = Get-JsonProperty -Object $native.exit -Name 'status' -Default $null + $exitStatus = if ($null -eq $exitStatusValue) { $null } else { [int]$exitStatusValue } + $resolvedConfiguration = Get-JsonProperty -Object $native -Name 'resolved' -Default $null + $evidence = Get-JsonProperty -Object $native -Name 'evidence' -Default ([ordered]@{}) + if ($evidence -is [System.Collections.IDictionary]) { + $evidence['capture'] = $native.capture + } else { + Add-Member -InputObject $evidence -MemberType NoteProperty -Name capture -Value $native.capture -Force + } + $result = New-ExecutionResult ` + -Descriptor $descriptor ` + -Profile $profileData ` + -Run $runData ` + -Status ([string]$native.status) ` + -FinalResponse $finalText ` + -FinalResponseReason $finalReason ` + -StartedUtc ([string]$native.timing.started_utc) ` + -FinishedUtc ([string]$native.timing.finished_utc) ` + -DurationSeconds ([double]$native.timing.duration_seconds) ` + -ExitStatus $exitStatus ` + -Failure (Get-JsonProperty -Object $native.exit -Name 'failure' -Default $null) ` + -SessionId ([string]$native.session.id) ` + -IsolationCapabilities $capabilities ` + -IsolationMechanisms @((Get-JsonProperty -Object $native.isolation -Name 'mechanisms' -Default @())) ` + -ResolvedConfiguration $resolvedConfiguration ` + -Telemetry (Get-JsonProperty -Object $native -Name 'telemetry' -Default $null) ` + -Artifacts @((Get-JsonProperty -Object $native -Name 'artifacts' -Default @())) ` + -Warnings @((Get-JsonProperty -Object $native -Name 'warnings' -Default @())) ` + -CompatibilityDeviations @((Get-JsonProperty -Object $native -Name 'compatibility_deviations' -Default @())) ` + -Evidence $evidence ` + -AttemptCount 1 + $result.run_id = [string]$native.run_id + + [void](Assert-ExecutionResult -Result $result) + if ([string]$result.status -ne 'incompatible') { + $terminalValidation = Test-NativeWorkerTerminalEvidence ` + -ExecutionEvidence $result ` + -Run $runData ` + -RequestedModel ([string]$profileData.Model) ` + -ExpectedWorkerSessionId ([string]$native.session.id) ` + -ExpectedRunner $Runner ` + -ExpectedMechanism ([string]$descriptor.delegation.mechanism) + if (-not $terminalValidation.Valid) { + throw "Native worker terminal evidence is incompatible: $([string]::Join(', ', @($terminalValidation.Failures)))." + } + Assert-NativeTerminalCaptureArtifact -ExecutionResult $result + } + + $utf8NoBom = [System.Text.UTF8Encoding]::new($false) + [System.IO.File]::WriteAllText($outputPath, ((ConvertTo-Json -InputObject $result -Depth 100) + [Environment]::NewLine), $utf8NoBom) + $relativeOutput = [System.IO.Path]::GetRelativePath($iterationDirectory, $outputPath).Replace('\', '/') + Write-RunnerJson -Value ([ordered]@{ + schema = 'codebeltnet/agentic/eval-native-worker-record/1' + runner = $Runner + execution_result = $relativeOutput + execution_status = $result.status + }) -AsOutput +} catch { + [Console]::Error.WriteLine($_.Exception.Message) + exit 2 +} diff --git a/scripts/eval-runners/runner-common.ps1 b/scripts/eval-runners/runner-common.ps1 index b635136..577fb79 100644 --- a/scripts/eval-runners/runner-common.ps1 +++ b/scripts/eval-runners/runner-common.ps1 @@ -14,6 +14,37 @@ function Get-RunnerSchemaNames { } } +function Get-PackageRunnerDescriptor { + param([Parameter(Mandatory = $true)][string]$RunnerName) + + if ($RunnerName -notmatch '^[a-z0-9-]+$') { + throw "execution-profile.json runner '$RunnerName' is not a valid package runner name." + } + + $runnerPath = Join-Path (Join-Path $PSScriptRoot $RunnerName) 'runner.ps1' + if (-not (Test-Path -LiteralPath $runnerPath -PathType Leaf)) { + throw "Package-local runner '$RunnerName' is missing its runner.ps1 descriptor." + } + + $descriptorOutput = & pwsh -NoProfile -File $runnerPath describe 2>&1 + if ($LASTEXITCODE -ne 0) { + throw "Package-local runner '$RunnerName' descriptor failed: $([string]::Join(' ', @($descriptorOutput)))" + } + + try { + $descriptor = [string]::Join([Environment]::NewLine, @($descriptorOutput)) | ConvertFrom-Json + [void](Assert-RunnerDescriptor -Descriptor $descriptor) + } catch { + throw "Package-local runner '$RunnerName' returned an invalid descriptor: $($_.Exception.Message)" + } + + if ([string]$descriptor.name -ne $RunnerName) { + throw "Package-local runner descriptor name '$($descriptor.name)' does not match selected runner '$RunnerName'." + } + + return $descriptor +} + function Get-JsonProperty { param( [object]$Object, @@ -63,7 +94,15 @@ function Read-RunnerJson { throw "JSON file '$Path' does not exist." } - return [System.IO.File]::ReadAllText((Resolve-Path -LiteralPath $Path).Path, [System.Text.UTF8Encoding]::new($false)) | ConvertFrom-Json + $json = [System.IO.File]::ReadAllText((Resolve-Path -LiteralPath $Path).Path, [System.Text.UTF8Encoding]::new($false)) + $convertFromJson = Get-Command ConvertFrom-Json -ErrorAction Stop + if ($convertFromJson.Parameters.ContainsKey('DateKind')) { + return $json | ConvertFrom-Json -DateKind String + } + + # DateKind was added after the oldest supported PowerShell 7 releases. + # Keep those versions usable; the canonical writers still emit UTC strings. + return $json | ConvertFrom-Json } function Write-RunnerJson { @@ -545,6 +584,30 @@ function Assert-NativeWorkerTerminalEvidence { return $true } +function Assert-NativeTerminalCaptureArtifact { + param([Parameter(Mandatory = $true)][object]$ExecutionResult) + + $capture = Get-JsonProperty -Object $ExecutionResult.evidence -Name 'capture' -Default $null + if ([string](Get-JsonProperty -Object $capture -Name 'source' -Default '') -ne 'harness_native_transport' -or + -not [bool](Get-JsonProperty -Object $capture -Name 'terminal' -Default $false) -or + [bool](Get-JsonProperty -Object $capture -Name 'worker_authored' -Default $true)) { + throw 'Native worker execution must preserve harness-native terminal capture provenance.' + } + + $transcriptMetric = Get-JsonProperty -Object $ExecutionResult.telemetry -Name 'transcript' -Default $null + $transcriptStatus = [string](Get-JsonProperty -Object $transcriptMetric -Name 'status' -Default '') + $transcriptArtifact = [string](Get-JsonProperty -Object (Get-JsonProperty -Object $transcriptMetric -Name 'value' -Default $null) -Name 'artifact' -Default '') + if ($transcriptStatus -ne 'available' -or [string]::IsNullOrWhiteSpace($transcriptArtifact)) { + throw 'Native worker execution must provide an available terminal transcript artifact.' + } + $matchingArtifacts = @($ExecutionResult.artifacts | Where-Object { + [string](Get-JsonProperty -Object $_ -Name 'path' -Default '') -eq $transcriptArtifact + }) + if ($matchingArtifacts.Count -ne 1) { + throw "Native worker transcript artifact '$transcriptArtifact' is not recorded exactly once in execution-result.json." + } +} + function Resolve-RunContract { param([Parameter(Mandatory = $true)][string]$RunPath) @@ -1001,6 +1064,7 @@ function New-ExecutionResult { reasoning_effort = $Profile.ReasoningEffort configuration_profile = $Profile.ConfigurationProfile tool_profile = $Profile.ToolProfile + timeout_seconds = $Profile.TimeoutSeconds } resolved = $resolved started_utc = $started.ToString('yyyy-MM-ddTHH:mm:ss.fffZ') @@ -1042,11 +1106,20 @@ function Assert-ExecutionResult { if ([string]$Result.status -notin @('completed', 'failed', 'timed_out', 'cancelled', 'incompatible')) { throw "execution-result.json status '$($Result.status)' is unsupported." } - foreach ($field in @('run_id', 'runner', 'harness', 'requested', 'resolved', 'started_utc', 'finished_utc', 'duration_seconds', 'exit', 'final_response', 'input', 'isolation', 'telemetry', 'evidence', 'artifacts', 'warnings')) { + foreach ($field in @('run_id', 'runner', 'harness', 'requested', 'resolved', 'started_utc', 'finished_utc', 'duration_seconds', 'exit', 'final_response', 'input', 'isolation', 'telemetry', 'evidence', 'artifacts', 'warnings', 'compatibility_deviations')) { if (-not (Test-JsonProperty -Object $Result -Name $field)) { throw "execution-result.json is missing '$field'." } } + $runIdentity = Get-JsonProperty -Object $Result -Name 'run' -Default $null + foreach ($field in @('eval_id', 'eval_name', 'configuration')) { + if ($null -eq $runIdentity -or -not (Test-JsonProperty -Object $runIdentity -Name $field)) { + throw "execution-result.json run.$field must be present." + } + } + if (-not (Test-JsonProperty -Object $Result.requested -Name 'timeout_seconds')) { + throw 'execution-result.json requested.timeout_seconds must be present.' + } foreach ($identityName in @('runner', 'harness')) { $identity = Get-JsonProperty -Object $Result -Name $identityName -Default $null foreach ($identityField in @('name', 'version')) { @@ -1394,7 +1467,7 @@ function Get-ProviderAuthenticationVariables { '^openrouter$' { return @('OPENROUTER_API_KEY') } '^xai$|^x-ai$' { return @('XAI_API_KEY') } '^mistral$' { return @('MISTRAL_API_KEY') } - '^cline$' { return @('CLINE_API_KEY') } + default { return @() } } } From 676e19386bee36f80019b2ace7a68bd3a42be7e9 Mon Sep 17 00:00:00 2001 From: "aicia[bot]" Date: Tue, 25 Aug 2026 02:02:25 +0200 Subject: [PATCH 34/39] =?UTF-8?q?=E2=9C=85=20update=20eval=20runner=20conf?= =?UTF-8?q?ormance=20tests?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Update test-runner-conformance to validate native worker results and schema contracts. Adjust skill preparation and validation scripts for infrastructure changes. --- .../tests/test-runner-conformance.ps1 | 166 ++++++++++++------ scripts/prepare-skill-evals.ps1 | 12 +- scripts/validate-skill-templates.ps1 | 4 +- 3 files changed, 125 insertions(+), 57 deletions(-) diff --git a/scripts/eval-runners/tests/test-runner-conformance.ps1 b/scripts/eval-runners/tests/test-runner-conformance.ps1 index 32251e9..69a524c 100644 --- a/scripts/eval-runners/tests/test-runner-conformance.ps1 +++ b/scripts/eval-runners/tests/test-runner-conformance.ps1 @@ -51,7 +51,6 @@ $recordedOldGhToken = $env:GH_TOKEN $recordedOldGithubToken = $env:GITHUB_TOKEN $recordedOldCopilotHome = $env:COPILOT_HOME $recordedOldGhConfigDir = $env:GH_CONFIG_DIR -$recordedOldClineAgentsSquad = $env:CLINE_AGENTS_SQUAD_PLUGIN try { $fakeBin = Join-Path $recordedRoot 'bin' New-Item -ItemType Directory -Path $fakeBin -Force | Out-Null @@ -66,7 +65,7 @@ param([Parameter(ValueFromRemainingArguments = $true)][string[]]$RemainingArgume $harness = [System.IO.Path]::GetFileNameWithoutExtension($MyInvocation.MyCommand.Path) $logPath = Join-Path (Get-Location).Path ("{0}-fake-cli-log.jsonl" -f $harness) $arguments = @($RemainingArguments | ForEach-Object { [string]$_ }) -$authNames = @('OPENAI_API_KEY', 'ANTHROPIC_API_KEY', 'GOOGLE_API_KEY', 'GEMINI_API_KEY', 'OPENROUTER_API_KEY', 'XAI_API_KEY', 'MISTRAL_API_KEY', 'CLINE_API_KEY') +$authNames = @('OPENAI_API_KEY', 'ANTHROPIC_API_KEY', 'GOOGLE_API_KEY', 'GEMINI_API_KEY', 'OPENROUTER_API_KEY', 'XAI_API_KEY', 'MISTRAL_API_KEY') $authPresent = @($authNames | Where-Object { -not [string]::IsNullOrWhiteSpace([Environment]::GetEnvironmentVariable($_)) }) $copilotAuthNames = @('COPILOT_GITHUB_TOKEN', 'GH_TOKEN', 'GITHUB_TOKEN') $copilotAuthPresent = @($copilotAuthNames | Where-Object { -not [string]::IsNullOrWhiteSpace([Environment]::GetEnvironmentVariable($_)) }) @@ -186,7 +185,7 @@ if ($harness -eq 'codex' -and $arguments -contains 'app-server') { exit 0 } if ($arguments -contains '--version') { - $version = switch ($harness) { 'codex' { 'recorded-codex 9.1' } 'opencode' { 'recorded-opencode 9.2' } 'copilot' { 'GitHub Copilot CLI recorded-1.0.80' } default { 'recorded-cline 9.3' } } + $version = switch ($harness) { 'codex' { 'recorded-codex 9.1' } 'opencode' { 'recorded-opencode 9.2' } 'copilot' { 'GitHub Copilot CLI recorded-1.0.80' } default { 'recorded-unknown 9.3' } } [IO.File]::AppendAllText($logPath, (($record | ConvertTo-Json -Compress) + [Environment]::NewLine), [Text.UTF8Encoding]::new($false)) Write-Output $version exit 0 @@ -292,15 +291,9 @@ if ($harness -eq 'codex') { Write-Output '{"type":"assistant.usage","id":"e6","parentId":"e5","ephemeral":true,"data":{"model":"claude-haiku-4.5","inputTokens":2,"outputTokens":3,"cacheReadTokens":1,"numToolCalls":1,"cost":0.2}}' Write-Output '{"type":"session.task_complete","id":"e7","parentId":"e6","data":{}}' Write-Output '{"type":"future.event.v99","payload":"fixture"}' -} else { - Write-Output '{"type":"say","say":"text","text":"recorded Cline progress","partial":false}' - Write-Output '{"type":"say","say":"tool","name":"read_file","text":"fixture.md","partial":false}' - Write-Output '{"type":"say","say":"completion_result","text":"recorded Cline final response","partial":false}' - Write-Output '{"type":"say","say":"api_req_finished","text":"{\"inputTokens\":2,\"outputTokens\":3,\"totalTokens\":5}"}' - Write-Output '{"type":"future.event.v99","payload":"fixture"}' } '@ - foreach ($harness in @('codex', 'opencode', 'cline', 'copilot')) { + foreach ($harness in @('codex', 'opencode', 'copilot')) { [System.IO.File]::WriteAllText((Join-Path $fakeBin "$harness.ps1"), $fakeCli, [System.Text.UTF8Encoding]::new($false)) } $fakeGh = @' @@ -326,7 +319,6 @@ exit 2 $env:COPILOT_GITHUB_TOKEN = 'recorded-copilot-canary-not-logged' $env:GH_TOKEN = 'recorded-gh-canary-not-logged' $env:GITHUB_TOKEN = 'recorded-github-canary-not-logged' - $env:CLINE_AGENTS_SQUAD_PLUGIN = $null $recordedGhConfig = Join-Path $recordedRoot 'github-cli-auth' New-Item -ItemType Directory -Path $recordedGhConfig -Force | Out-Null [System.IO.File]::WriteAllText((Join-Path $recordedGhConfig 'auth-marker.txt'), 'fixture auth state without a credential value', [System.Text.UTF8Encoding]::new($false)) @@ -337,13 +329,12 @@ exit 2 [System.IO.File]::WriteAllText((Join-Path $ambientCopilotHome 'config.json'), '{"loggedInUsers":[{"login":"ambient-profile-not-logged"}]}', [System.Text.UTF8Encoding]::new($false)) $env:COPILOT_HOME = $ambientCopilotHome $recordedProfiles = [ordered]@{} - foreach ($runnerName in @('codex', 'opencode', 'cline', 'copilot')) { + foreach ($runnerName in @('codex', 'opencode', 'copilot')) { $profilePath = Join-Path $recordedRoot "$runnerName-profile.json" $profileModel = switch ($runnerName) { 'copilot' { 'claude-haiku-4.5' } 'codex' { 'gpt-5.6-luna' } 'opencode' { 'opencode/muse-spark-1.2-contributor-free' } - 'cline' { 'deepseek/deepseek-v4-flash' } } Write-TestJson -Path $profilePath -Value ([ordered]@{ schema = (Get-RunnerSchemaNames).Profile @@ -362,7 +353,7 @@ exit 2 $recordedVersion = Get-ExternalCommandVersion -CommandInfo $resolvedRecordedCodex -WorkingDirectory (Join-Path $with.Root 'repo') if (-not $recordedVersion.Available) { throw "recorded Codex --version is not observable (exit=$($recordedVersion.Process.ExitCode), timed_out=$($recordedVersion.Process.TimedOut), stdout='$($recordedVersion.Process.Stdout)', stderr='$($recordedVersion.Process.Stderr)')" } Assert-Equal 'recorded-codex 9.1' $recordedVersion.Version 'recorded Codex exact version helper' - foreach ($runnerName in @('codex', 'opencode', 'cline', 'copilot')) { + foreach ($runnerName in @('codex', 'opencode', 'copilot')) { $runnerDir = if ($runnerName -eq 'copilot') { 'github-copilot' } else { $runnerName } $runnerPath = Join-Path $runnerRoot "$runnerDir\runner.ps1" $description = Invoke-AdapterJson -RunnerPath $runnerPath -Command describe -RunPath $with.Path -ProfilePath $recordedProfiles[$runnerName] @@ -372,7 +363,7 @@ exit 2 Assert-True (-not [string]::IsNullOrWhiteSpace([string]$description.delegation.mechanism)) "$runnerName descriptor records its native delegation mechanism" Assert-Equal 'conditional' $description.capabilities.native_worker_delegation "$runnerName descriptor does not present native delegation as terminal proof" Assert-Equal 'conditional' $description.delegation.model_lock "$runnerName descriptor leaves child model resolution conditional" - $expectedVersion = switch ($runnerName) { 'codex' { 'recorded-codex 9.1' } 'opencode' { 'recorded-opencode 9.2' } 'copilot' { 'GitHub Copilot CLI recorded-1.0.80' } default { 'recorded-cline 9.3' } } + $expectedVersion = switch ($runnerName) { 'codex' { 'recorded-codex 9.1' } 'opencode' { 'recorded-opencode 9.2' } 'copilot' { 'GitHub Copilot CLI recorded-1.0.80' } default { 'recorded-unknown 9.3' } } Assert-Equal $expectedVersion $description.harness.version "$runnerName exact describe version" $preflightWith = Invoke-AdapterJson -RunnerPath $runnerPath -Command preflight -RunPath $with.Path -ProfilePath $recordedProfiles[$runnerName] $preflightWithout = Invoke-AdapterJson -RunnerPath $runnerPath -Command preflight -RunPath $without.Path -ProfilePath $recordedProfiles[$runnerName] @@ -380,13 +371,8 @@ exit 2 Assert-Equal 'compatible' $preflightWithout.status "$runnerName without_skill pragmatic preflight" Assert-Equal $expectedVersion $preflightWith.harness.version "$runnerName exact preflight version" Assert-Equal 'pragmatic' $preflightWith.isolation.level "$runnerName pragmatic preflight level" - if ($runnerName -eq 'cline') { - Assert-True ($preflightWith.delegation.status -ne 'supported') 'Cline preflight does not claim unavailable Agent Squad delegation' - Assert-True (([string]::Join(' ', @($preflightWith.warnings))) -match 'use_subagents' -and ([string]::Join(' ', @($preflightWith.warnings))) -match 'Agent Squad') 'Cline preflight rejects read-only subagents as a mutable-arm fallback' - } else { - Assert-Equal 'conditional' $preflightWith.delegation.status "$runnerName native delegation preflight requires terminal evidence" - Assert-True ([bool]$preflightWith.delegation.terminal_evidence_required) "$runnerName preflight requires terminal delegation evidence" - } + Assert-Equal 'conditional' $preflightWith.delegation.status "$runnerName native delegation preflight requires terminal evidence" + Assert-True ([bool]$preflightWith.delegation.terminal_evidence_required) "$runnerName preflight requires terminal delegation evidence" if ($runnerName -eq 'copilot') { Assert-True (@($preflightWith.checks | Where-Object { $_.name -eq 'authentication' -and $_.status -eq 'passed' }).Count -eq 1) 'Copilot preflight accepts explicit environment authentication' Assert-True (@($preflightWith.mechanisms | Where-Object { $_ -eq '--allow-all-tools broad tool approval' }).Count -eq 1) 'Copilot preflight describes --allow-all-tools as broad tool approval' @@ -396,7 +382,7 @@ exit 2 Assert-True (@($preflightWith.checks | Where-Object { $_.name -eq 'parallel_dispatch' -and $_.status -eq 'passed' }).Count -eq 1) 'OpenCode preflight requires bounded concurrent dispatch' Assert-True (@($preflightWith.mechanisms | Where-Object { $_ -eq 'bounded concurrent native-worker dispatch required' }).Count -eq 1) 'OpenCode preflight records the concurrency requirement' } - if ($runnerName -in @('opencode', 'cline')) { + if ($runnerName -eq 'opencode') { Assert-True (@($preflightWith.warnings | Where-Object { $_ -match 'child-tool environment filter' }).Count -gt 0) "$runnerName reports the child credential-filter limitation" } $resultWith = Invoke-AdapterJson -RunnerPath $runnerPath -Command execute -RunPath $with.Path -ProfilePath $recordedProfiles[$runnerName] @@ -496,21 +482,6 @@ exit 2 Assert-Equal 'available' $resultWith.telemetry.tool_calls.status 'Copilot reports available tool-call telemetry' Assert-True ([int]$resultWith.telemetry.tool_calls.value -ge 1) 'Copilot parses documented tool.execution events' Assert-Equal 'unavailable' $resultWith.telemetry.cost.status 'Copilot does not estimate a currency cost' - } else { - $retryIndex = [Array]::IndexOf([string[]]$args, '--retries') - Assert-Equal '0' $args[$retryIndex + 1] 'Cline disables internal retries' - Assert-True ($args -notcontains '--id') 'Cline does not resume a session' - Assert-True ($args -contains '--json') 'Cline uses structured output' - $providerIndex = [Array]::IndexOf([string[]]$args, '--provider') - Assert-Equal 'deepseek' $args[$providerIndex + 1] 'Cline derives its native provider argument from the opaque selector' - $modelIndex = [Array]::IndexOf([string[]]$args, '--model') - Assert-Equal 'deepseek-v4-flash' $args[$modelIndex + 1] 'Cline derives its native model argument from the opaque selector' - $configIndex = [Array]::IndexOf([string[]]$args, '--config') - Assert-True ($args[$configIndex + 1] -match '(?i)[\\/]\.cline$') 'Cline uses the documented isolated config root' - $dataIndex = [Array]::IndexOf([string[]]$args, '--data-dir') - Assert-True ($args[$dataIndex + 1] -match '(?i)[\\/]\.cline[\\/]data$') 'Cline data-dir is the isolated data root' - Assert-Equal 'available' $resultWith.telemetry.tool_calls.status 'Cline reports available tool-call telemetry' - Assert-True ([int]$resultWith.telemetry.tool_calls.value -ge 1) 'Cline parses documented tool events' } $logText = [System.IO.File]::ReadAllText($logPath, [System.Text.UTF8Encoding]::new($false)) Assert-True ($logText -notmatch 'recorded-canary|recorded-unrelated-canary|recorded-copilot-canary|recorded-gh-canary|recorded-github-canary|recorded-gh-fallback-token') "$runnerName logs do not contain credential values" @@ -538,7 +509,6 @@ exit 2 $serialOpenCodePreflight = Invoke-AdapterJson -RunnerPath (Join-Path $runnerRoot 'opencode\runner.ps1') -Command preflight -RunPath $with.Path -ProfilePath $serialOpenCodeProfile Assert-Equal 'incompatible' $serialOpenCodePreflight.status 'OpenCode rejects a serial execution profile' Assert-True (@($serialOpenCodePreflight.reasons | Where-Object { $_ -match 'concurrency >= 2|Sequential dispatch' }).Count -gt 0) 'OpenCode serial-profile failure explains the concurrency requirement' - $env:CLINE_AGENTS_SQUAD_PLUGIN = $recordedOldClineAgentsSquad $staleCli = $fakeCli.Replace("'opencode' { '--format --dir --model --auto --pure --continue --session' }", "'opencode' { '--format --dir --model --pure --continue --session' }") [System.IO.File]::WriteAllText((Join-Path $fakeBin 'opencode.ps1'), $staleCli, [System.Text.UTF8Encoding]::new($false)) $stalePreflight = Invoke-AdapterJson -RunnerPath (Join-Path $runnerRoot 'opencode\runner.ps1') -Command preflight -RunPath $with.Path -ProfilePath $recordedProfiles['opencode'] @@ -647,7 +617,6 @@ exit 2 $env:GITHUB_TOKEN = $recordedOldGithubToken $env:COPILOT_HOME = $recordedOldCopilotHome $env:GH_CONFIG_DIR = $recordedOldGhConfigDir - $env:CLINE_AGENTS_SQUAD_PLUGIN = $recordedOldClineAgentsSquad if (Test-Path -LiteralPath $recordedRoot) { Remove-Item -LiteralPath $recordedRoot -Recurse -Force } } } @@ -945,21 +914,15 @@ try { Assert-True (@($copilotFixture.Events | Where-Object { $_.type -eq 'assistant.message' }).Count -ge 1) 'recorded copilot fixture includes documented assistant.message output' Assert-True (@($copilotFixture.Events | Where-Object { $_.type -eq 'assistant.usage' }).Count -eq 1) 'recorded copilot fixture includes documented assistant.usage output' Assert-True (@($copilotFixture.Events | Where-Object { $_.type -eq 'tool.execution_start' }).Count -eq 1) 'recorded copilot fixture includes documented tool.execution output' - $clineFixture = ConvertFrom-JsonLines -Text ([System.IO.File]::ReadAllText((Join-Path $PSScriptRoot 'fixtures\cline-events.jsonl'), [System.Text.UTF8Encoding]::new($false))) - Assert-Equal 0 $clineFixture.Errors.Count 'recorded cline fixture has valid JSONL' - Assert-True ($clineFixture.Events.Count -ge 9) 'recorded cline fixture has events' - Assert-True (@($clineFixture.Events | Where-Object { $_.type -eq 'say' -and $_.say -eq 'tool' }).Count -eq 1) 'recorded cline fixture includes documented tool output' - Assert-True (@($clineFixture.Events | Where-Object { $_.type -eq 'say' -and $_.say -eq 'completion_result' }).Count -eq 1) 'recorded cline fixture includes documented completion_result output' - Assert-True (@($clineFixture.Events | Where-Object { $_.type -eq 'future.event.v99' }).Count -eq 1) 'recorded cline fixture includes an unknown event' - $prepareText = [System.IO.File]::ReadAllText((Join-Path $repoRoot 'scripts\prepare-skill-evals.ps1'), [System.Text.UTF8Encoding]::new($false)) $reportText = [System.IO.File]::ReadAllText((Join-Path $repoRoot 'scripts\generate-eval-report.ps1'), [System.Text.UTF8Encoding]::new($false)) $bridgeText = [System.IO.File]::ReadAllText((Join-Path $runnerRoot 'bridge-execution-result.ps1'), [System.Text.UTF8Encoding]::new($false)) + $recordText = [System.IO.File]::ReadAllText((Join-Path $runnerRoot 'record-native-result.ps1'), [System.Text.UTF8Encoding]::new($false)) $manifestBridgeText = [System.IO.File]::ReadAllText((Join-Path $runnerRoot 'bridge-manifest-results.ps1'), [System.Text.UTF8Encoding]::new($false)) $commonText = [System.IO.File]::ReadAllText((Join-Path $runnerRoot 'runner-common.ps1'), [System.Text.UTF8Encoding]::new($false)) - Assert-True ($prepareText -notmatch '(?i)codex\s+exec|opencode\s+run|cline\s+--|copilot\s+-p|copilot\s+--prompt|Profile\.Provider') 'portable preparation must not contain harness-specific CLI invocations or provider-field branches' - Assert-True ($reportText -notmatch '(?i)codex\s+exec|opencode\s+run|cline\s+--|copilot\s+-p|copilot\s+--prompt|Profile\.Provider') 'reporting must not contain harness-specific or provider-field branches' - Assert-True ($bridgeText -notmatch '(?i)codex\s+exec|opencode\s+run|cline\s+--|copilot\s+-p|copilot\s+--prompt|Profile\.Provider') 'the raw-to-portable bridge must remain runner-neutral' + Assert-True ($prepareText -notmatch '(?i)codex\s+exec|opencode\s+run|copilot\s+-p|copilot\s+--prompt|Profile\.Provider') 'portable preparation must not contain harness-specific CLI invocations or provider-field branches' + Assert-True ($reportText -notmatch '(?i)codex\s+exec|opencode\s+run|copilot\s+-p|copilot\s+--prompt|Profile\.Provider') 'reporting must not contain harness-specific or provider-field branches' + Assert-True ($bridgeText -notmatch '(?i)codex\s+exec|opencode\s+run|copilot\s+-p|copilot\s+--prompt|Profile\.Provider') 'the raw-to-portable bridge must remain runner-neutral' Assert-True ($prepareText.Contains('bridge-manifest-results.ps1')) 'handoff preparation must use the deterministic package-level manifest bridge' Assert-True ($prepareText.Contains('runs..run_manifest') -and $prepareText.Contains('runs..execution_result') -and $prepareText.Contains('runs..result')) 'handoff preparation must require every exact manifest arm path' Assert-True ($prepareText.Contains('Do not derive, normalize, rename, hyphenate, underscore, or otherwise reconstruct any run, execution-result, or result path.')) 'handoff preparation must prohibit reconstructed paths' @@ -976,8 +939,11 @@ try { Assert-True ($prepareText.Contains('rejected before the worker starts') -and $prepareText.Contains('record no eval attempt')) 'handoff preparation must queue capacity rejections without counting attempts' Assert-True ($prepareText.Contains('Register each worker acceptance and terminal result exactly once') -and $prepareText.Contains('incompatibility is diagnostic-only') -and $prepareText.Contains('Do not grade incompatible arms')) 'handoff preparation must make duplicate registration and incompatible-arm handling fail closed' Assert-True ($prepareText.Contains('Skipping report generation because the completion gate failed') -and $prepareText.Contains('Diagnostic comparison (incomplete)')) 'incomplete collection must remain diagnostic and skip report generation' + Assert-True ($prepareText.Contains('capture.worker_authored') -and $prepareText.Contains('Do not ask the worker to author or summarize this envelope')) 'handoff preparation must require transport-produced native envelopes rather than worker-authored summaries' Assert-True ($bridgeText.Contains('Get-PackageRunnerDescriptor') -and $bridgeText.Contains('Assert-NativeTerminalCaptureArtifact') -and $bridgeText.Contains('ExpectedMechanism')) 'native bridge must require runner-produced terminal evidence' + Assert-True ($recordText.Contains('eval-native-worker-result/1') -and $recordText.Contains('New-ExecutionResult')) 'native terminal recording must use the runner-owned result builder' Assert-True ($commonText.Contains('exit.status must be a JSON number or null')) 'execution results must reject textual exit statuses' + Assert-True ($commonText.Contains('requested.timeout_seconds') -and $commonText.Contains('execution-result.json run.$field')) 'raw execution results must retain the complete run and requested configuration contract' Assert-True ($prepareText.Contains('orchestration.ps1')) 'handoff preparation must load the deterministic orchestration helper' Assert-True ($prepareText -notmatch '') 'handoff preparation must not expose an unconstrained result-file placeholder' Assert-True ($reportText -notmatch 'function Get-ResultPath') 'reporting must not contain a configuration-derived result path helper' @@ -994,6 +960,106 @@ try { grading = @([ordered]@{ text = 'preserved assertion'; passed = $null; evidence = '' }) }) $bridgeResult = Invoke-Fake -FakePath $fakePath -Command execute -Run $with.Path -Profile $profilePath + + $codexProfilePath = Join-Path $iteration 'codex-native-profile.json' + Write-TestJson -Path $codexProfilePath -Value ([ordered]@{ + schema = (Get-RunnerSchemaNames).Profile + runner = 'codex' + model = 'fixture-model' + reasoning_effort = 'high' + configuration_profile = 'isolated-default' + tool_profile = 'default' + timeout_seconds = 30 + concurrency = 1 + }) + $codexRunData = Resolve-RunContract -RunPath $with.Path + $nativeEventRelativePath = 'evidence/native-worker-events.jsonl' + $nativeEventPath = Join-Path $with.Root ($nativeEventRelativePath -replace '/', [System.IO.Path]::DirectorySeparatorChar) + New-Item -ItemType Directory -Path (Split-Path -Parent $nativeEventPath) -Force | Out-Null + [System.IO.File]::WriteAllText($nativeEventPath, '{"type":"terminal"}' + [Environment]::NewLine, [System.Text.UTF8Encoding]::new($false)) + $nativeArtifact = New-ArtifactReference -Run $codexRunData -Path $nativeEventRelativePath -Scope run -MediaType 'application/x-ndjson' + $codexDescriptor = Get-PackageRunnerDescriptor -RunnerName 'codex' + $nativeInputPath = Join-Path $iteration 'conformance\results\native-worker-result.json' + $nativeOutputPath = Join-Path $iteration 'conformance\results\recorded.execution-result.json' + $nativeEnvelope = [ordered]@{ + schema = 'codebeltnet/agentic/eval-native-worker-result/1' + run_id = 'native-fixture-run' + session = [ordered]@{ id = 'native-fixture-session'; fresh = $true; resumed = $false } + status = 'completed' + run = [ordered]@{ eval_id = 1; eval_name = 'conformance'; configuration = 'with_skill' } + final_response = [ordered]@{ status = 'available'; text = 'native fixture response' } + timing = [ordered]@{ started_utc = '2024-01-01T00:00:00Z'; finished_utc = '2024-01-01T00:00:01Z'; duration_seconds = 1 } + exit = [ordered]@{ status = 0; failure = $null } + isolation = [ordered]@{ + capabilities = [ordered]@{ + fresh_context = 'supported' + isolated_home_config = 'supported' + isolated_working_directory = 'supported' + ambient_candidate_skill_exclusion = 'supported' + candidate_skill_exposure = 'supported' + prompt_fidelity = 'supported' + model_configuration_lock = 'supported' + response_capture = 'supported' + filesystem_confinement = 'unavailable' + } + mechanisms = @('native-fixture-worker') + } + telemetry = [ordered]@{ + transcript = New-AvailableMetric -Value ([ordered]@{ artifact = $nativeEventRelativePath; complete = $true }) + tokens = New-UnavailableMetric -Reason 'fixture does not expose token telemetry' + tool_calls = New-AvailableMetric -Value 0 + cost = New-UnavailableMetric -Reason 'fixture does not expose cost telemetry' + } + evidence = [ordered]@{ + delegation = [ordered]@{ + mechanism = [string]$codexDescriptor.delegation.mechanism + worker_session_id = 'native-fixture-session' + observed_model = 'fixture-model' + observed_working_directory = $codexRunData.WorkingDirectoryPath + observed_home = $codexRunData.HomeDirectoryPath + fresh_worker = $true + home_config_isolated = $true + prompt_fidelity = $true + prompt_sha256 = $codexRunData.PromptHash + terminal_result_capture = $true + paired_arm_visible = $false + grading_material_visible = $false + nested_model_execution = $false + model_execution_count = 1 + } + } + capture = [ordered]@{ + source = 'harness_native_transport' + terminal = $true + worker_authored = $false + } + artifacts = @($nativeArtifact) + warnings = @() + compatibility_deviations = @() + attempt_count = 1 + resolved = [ordered]@{ status = 'accepted_request'; reason = 'native fixture accepted the requested configuration'; observations = [ordered]@{ model = 'fixture-model'; reasoning_effort = 'high' } } + } + Write-TestJson -Path $nativeInputPath -Value $nativeEnvelope + $recordPath = Join-Path $runnerRoot 'record-native-result.ps1' + $recordOutput = & pwsh -NoProfile -File $recordPath -Runner codex -Run $with.Path -Profile $codexProfilePath -NativeResult $nativeInputPath -Output $nativeOutputPath 2>&1 + if ($LASTEXITCODE -ne 0) { throw "native terminal recording failed: $([string]::Join(' ', @($recordOutput)))" } + $recordedResult = Read-RunnerJson -Path $nativeOutputPath + [void](Assert-ExecutionResult -Result $recordedResult) + Assert-Equal 'native-fixture-run' $recordedResult.run_id 'native terminal recording preserves the opaque worker run id' + Assert-Equal 'conformance' $recordedResult.run.eval_name 'native terminal recording derives exact arm identity from run.json' + Assert-Equal 'fixture-model' $recordedResult.requested.model 'native terminal recording derives model from execution-profile.json' + Assert-Equal 30 $recordedResult.requested.timeout_seconds 'native terminal recording preserves the requested timeout' + Assert-Equal '2024-01-01T00:00:00.000Z' ([DateTime]$recordedResult.started_utc).ToUniversalTime().ToString('yyyy-MM-ddTHH:mm:ss.fffZ') 'native terminal recording writes canonical started_utc' + Assert-True ($recordedResult.telemetry.transcript.status -eq 'available') 'native terminal recording preserves transcript evidence' + Assert-Equal 'harness_native_transport' $recordedResult.evidence.capture.source 'native terminal recording preserves capture provenance' + Assert-True (-not [bool]$recordedResult.evidence.capture.worker_authored) 'native terminal recording rejects worker-authored capture provenance' + + $legacyNativeInputPath = Join-Path $iteration 'conformance\results\legacy-summary.json' + Write-TestJson -Path $legacyNativeInputPath -Value $bridgeResult + $legacyOutput = & pwsh -NoProfile -File $recordPath -Runner codex -Run $with.Path -Profile $codexProfilePath -NativeResult $legacyNativeInputPath -Output $nativeOutputPath 2>&1 + Assert-True ($LASTEXITCODE -ne 0) 'legacy summary-shaped worker output is rejected by the native recording boundary' + Assert-True (([string]::Join(' ', @($legacyOutput))) -match 'eval-native-worker-result/1') 'legacy summary rejection identifies the required native envelope' + Write-TestJson -Path $rawPath -Value $bridgeResult $bridgePath = Join-Path $runnerRoot 'bridge-execution-result.ps1' $bridgeOutput = & pwsh -NoProfile -File $bridgePath -Run $with.Path -ExecutionResult $rawPath -Result $resultPath diff --git a/scripts/prepare-skill-evals.ps1 b/scripts/prepare-skill-evals.ps1 index df91108..2946a34 100644 --- a/scripts/prepare-skill-evals.ps1 +++ b/scripts/prepare-skill-evals.ps1 @@ -398,7 +398,6 @@ function Get-HarnessName { 'github-copilot' { return 'GitHub Copilot CLI' } 'codex' { return 'Codex CLI' } 'opencode' { return 'OpenCode' } - 'cline' { return 'Cline' } 'fake' { return 'Deterministic fake runner' } default { return $RunnerName } } @@ -1619,6 +1618,7 @@ function New-RunnerPrompt { $profilePath = Join-Path $IterationDirectory 'execution-profile.json' $resolverPath = Join-Path $IterationDirectory "$evalRunnerToolRelativePath/resolve-runner.ps1" $orchestrationPath = Join-Path $IterationDirectory "$evalRunnerToolRelativePath/orchestration.ps1" + $nativeResultRecorderPath = Join-Path $IterationDirectory "$evalRunnerToolRelativePath/record-native-result.ps1" $manifestBridgePath = Join-Path $IterationDirectory "$evalRunnerToolRelativePath/bridge-manifest-results.ps1" $reportPath = Join-Path $IterationDirectory $reportToolRelativePath $armCount = @($ManifestEvals).Count * 2 @@ -1652,7 +1652,7 @@ function New-RunnerPrompt { [void]$builder.AppendLine('3. Build the pending arm queue from `manifest.json` with the orchestration helper. For every arm, read the exact `run_manifest`, `execution_result`, and `result` fields from `runs..run_manifest`, `runs..execution_result`, and `runs..result`. Retain those exact manifest-declared strings without editing them: the parent owns those exact destinations; the worker receives only its own prepared arm contract. Do not derive, normalize, rename, hyphenate, underscore, or otherwise reconstruct any run, execution-result, or result path.') [void]$builder.AppendLine('4. Before dispatching, require native worker delegation and all mandatory isolation controls. An unavailable or unsupported delegation mechanism is incompatible. A conditional mechanism is allowed only when the external orchestrator will require terminal evidence for the actual worker. Do not continue by invoking a runner process in the parent, and do not silently serialize arms in the parent.') [void]$builder.AppendLine('5. Dispatch each pending arm to one fresh harness-native full-capability worker. The worker must execute the prepared `prompt.md` as its first task from that arm''s staged run directory, with the selected model/configuration, exact working directory and isolated home. Require terminal evidence for the exact selected model, exact arm identity, working directory, isolated HOME/config boundary, prompt fidelity, terminal result capture, paired-arm/grading exclusion, fresh session, and absence of nested model execution; if any required fact is missing or mismatched, mark the arm incompatible without retrying through a fallback. It must not receive its paired arm, `eval-metadata.json`, expected output, assertions, grading, benchmark/report data, or any result from another arm.') - [void]$builder.AppendLine('6. The delegated worker is the only model-backed execution for that arm. It must not invoke `runner.ps1 execute`, a second harness CLI, another model agent, or a nested session. Return the runner-produced terminal `execution-result.json` unchanged, including its exact runner identity, the descriptor''s exact native delegation mechanism, and a hashed transcript/event artifact. A parent-created summary or normalized result is not genuine runner evidence and is incompatible. The parent writes that raw result to the exact manifest-declared `execution_result` path without reconstructing any path.') + [void]$builder.AppendLine(('6. The delegated worker is the only model-backed execution for that arm. It must not invoke `runner.ps1 execute`, a second harness CLI, another model agent, or a nested session. The harness-native transport, not the worker''s answer, must produce one terminal envelope using schema `codebeltnet/agentic/eval-native-worker-result/1`; its `capture.source` must be `harness_native_transport`, `capture.terminal` must be true, and `capture.worker_authored` must be false. The envelope must contain the exact nested `run` identity, fresh session, `timing.started_utc`/`finished_utc`/`duration_seconds`, numeric-or-null exit status, structured isolation capabilities, delegation evidence, and hashed transcript/event artifact. Do not ask the worker to author or summarize this envelope. If the native transport returns only assistant text or a hand-authored summary, the arm is incompatible. After the harness-native worker is terminal, write only the captured envelope to a temporary package-local path and invoke the deterministic runner-owned recorder at {0} with the exact `-Runner`, `-Run`, `-Profile`, `-NativeResult`, and manifest-declared `-Output` paths. The recorder starts no model and emits the canonical runner-produced `execution-result.json`; register and bridge only that generated file. A parent-created summary, direct JSON mutation, or malformed envelope is incompatible.' -f $nativeResultRecorderPath)) [void]$builder.AppendLine('7. Register each worker acceptance and terminal result exactly once. If orchestration reports that an arm is already accepted or terminal, do not retry or re-register it; preserve the state and resolve the arm from its existing terminal record. Maintain up to `min(execution-profile.json.concurrency, remaining arms)` active delegated workers. If a delegation request is rejected before the worker starts because of harness capacity, leave that arm pending, record no eval attempt, mark the rejection as capacity-limited in orchestration state, and retry it after an active worker becomes terminal. Do not add a runner-specific ceiling or change the portable requested concurrency.') [void]$builder.AppendLine('7a. DISPATCH IS AN ACTION, NOT A CONFIRMATION STEP. After preflight, emit the full first batch of native worker calls immediately. Do not send a prose status message, ask the user whether to start or re-dispatch, wait for confirmation, or wait for the first worker result before emitting the remaining calls. If the client cannot emit the batch in one assistant turn, mark the selected runner incompatible and stop; never replace the batch with deliberate serial dispatch.') if ([string]::Equals([string]$ExecutionSelection.Runner, 'opencode', [System.StringComparison]::OrdinalIgnoreCase)) { @@ -1672,8 +1672,9 @@ function New-RunnerPrompt { [void]$builder.AppendLine('runner.ps1 describe') [void]$builder.AppendLine("runner.ps1 preflight -Run `"`" -Profile `"$profilePath`"") [void]$builder.AppendLine("runner.ps1 execute -Run `"`" -Profile `"$profilePath`" # direct one-arm compatibility transport; forbidden to the parent orchestrator") + [void]$builder.AppendLine("record-native-result.ps1 -Runner `"`" -Run `"`" -Profile `"$profilePath`" -NativeResult `"`" -Output `"`"") [void]$builder.AppendLine('```') - [void]$builder.AppendLine('Use the resolver output to locate `runner.ps1`; `` is data from the profile, not a branch in this orchestration contract. The parent may use `describe` and `preflight` only. The `execute` command remains part of the one-arm runner protocol for compatibility and conformance, but the native delegated worker path MUST NOT invoke it: doing so would create a second model-backed execution.') + [void]$builder.AppendLine('Use the resolver output to locate `runner.ps1`; `` is data from the profile, not a branch in this orchestration contract. The parent may use `describe`, `preflight`, and the deterministic `record-native-result.ps1` recorder. The `execute` command remains part of the one-arm runner protocol for compatibility and conformance, but the native delegated worker path MUST NOT invoke it: doing so would create a second model-backed execution.') [void]$builder.AppendLine() [void]$builder.AppendLine('Do not read any `eval-metadata.json`, expected output, assertions, result grading, benchmark/report data, or paired output during Phase 1. The orchestrator and worker must never expose those materials before all workers are terminal. They remain outside every run directory and are the grading key.') [void]$builder.AppendLine() @@ -1737,8 +1738,9 @@ function New-PackageReadme { [void]$builder.AppendLine() [void]$builder.AppendLine('1. Read `execution-profile.json`. If `runner` or `model` is missing, fail clearly instead of guessing. Resolve the selected package-local runner and run `describe`, then `preflight`, before any native worker is dispatched. Invoke `Assert-NativeWorkerDelegation`; conditional controls require terminal evidence, unsupported controls are incompatible, and no parent sequential fallback is allowed. For OpenCode, a profile with concurrency below 2 is incompatible.') [void]$builder.AppendLine('2. Use the package-local orchestration helper to queue one worker per manifest arm. Delegate every arm to a fresh full-capability harness-native worker. Do not invoke the runner''s direct `execute` command from the parent or from the delegated worker, because it would add a second model execution. Each worker receives one arm only and no grading material or paired-arm data.') - [void]$builder.AppendLine('3. Maintain up to the requested concurrency. If the harness refuses a new worker because its own capacity is full, leave that arm queued, mark the rejection as capacity-limited in `orchestration-state.json`, and dispatch it when capacity is released; do not hardcode a runner-specific maximum and do not count the rejection as an attempt.') - [void]$builder.AppendLine('4. After all delegated workers complete or fail, invoke `Assert-OrchestrationConcurrency`, persist `orchestration-state.json`, and run `tools/eval-runners/bridge-manifest-results.ps1 -IterationDirectory -RequireComplete -RequireNativeDelegation -RequireParallelDispatch`. It reads the manifest-declared `run_manifest`, `execution_result`, and `result` paths for every arm, validates native terminal evidence and parallel-dispatch evidence again, and invokes the one-arm bridge with those exact paths. Only then read the grading key, grade with `tools/skill-creator/agents/grader.md`, and run `tools/generate-eval-report.ps1 -RequireComplete`.') + [void]$builder.AppendLine('3. After each harness-native worker is terminal, preserve the transport-produced envelope under schema `codebeltnet/agentic/eval-native-worker-result/1` and invoke the deterministic `record-native-result.ps1` recorder with the exact manifest `run_manifest`, `execution_result`, and profile paths. The envelope must prove `capture.source=harness_native_transport`, `capture.terminal=true`, and `capture.worker_authored=false`; assistant text alone is not terminal evidence. The recorder owns the `eval-execution-result/1` shape, timestamps, run identity, requested configuration, and runner/harness identity. Do not ask a worker to author, hand-write, normalize, or repair an `execution-result.json` summary in the orchestrator.') + [void]$builder.AppendLine('4. Maintain up to the requested concurrency. If the harness refuses a new worker because its own capacity is full, leave that arm queued, mark the rejection as capacity-limited in `orchestration-state.json`, and dispatch it when capacity is released; do not hardcode a runner-specific maximum and do not count the rejection as an attempt.') + [void]$builder.AppendLine('5. After all delegated workers complete or fail, invoke `Assert-OrchestrationConcurrency`, persist `orchestration-state.json`, and run `tools/eval-runners/bridge-manifest-results.ps1 -IterationDirectory -RequireComplete -RequireNativeDelegation -RequireParallelDispatch`. It reads the manifest-declared `run_manifest`, `execution_result`, and `result` paths for every arm, validates native terminal evidence and parallel-dispatch evidence again, and invokes the one-arm bridge with those exact paths. Only then read the grading key, grade with `tools/skill-creator/agents/grader.md`, and run `tools/generate-eval-report.ps1 -RequireComplete`.') [void]$builder.AppendLine() [void]$builder.AppendLine('`RUN-THIS.prompt.md` is the external Eval Orchestrator handoff. It selects the package-local runner from the profile, delegates one native Eval Worker per blind arm, queues capacity rejections, bridges raw evidence into the existing result shape, reveals grading material only after execution, and invokes Anthropic skill-creator''s compatible aggregator and static viewer through the package adapter. It never executes an eval prompt in its own context.') [void]$builder.AppendLine() diff --git a/scripts/validate-skill-templates.ps1 b/scripts/validate-skill-templates.ps1 index 264df2a..7c661f8 100644 --- a/scripts/validate-skill-templates.ps1 +++ b/scripts/validate-skill-templates.ps1 @@ -1271,7 +1271,7 @@ Add-ValidationResult -Results $results -Name 'Skill evaluation prepares portable Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle 'repository automation remains deterministic and never invokes a model.' Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle 'Resolve the execution configuration before running the package preparation script.' Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle 'scripts/Get-HarnessModels.ps1' - Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle 'Cline and OpenCode discovery is free-only' + Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle 'OpenCode discovery is free-only' Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle 'pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -Skill -Runner -Model ' Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle 'pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -CollectResults ' Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle '### Handing the package over' @@ -1339,7 +1339,7 @@ Add-ValidationResult -Results $results -Name 'Skill evaluation prepares portable $codexModels = ($codexDiscovery -join [Environment]::NewLine) | ConvertFrom-Json if (@($codexModels.models).Count -ne 6) { throw 'Codex discovery must return all available fixture models.' } - foreach ($runnerName in @('cline', 'opencode')) { + foreach ($runnerName in @('opencode')) { $discoveryOutput = & pwsh -NoProfile -File $modelDiscoveryPath -Runner $runnerName -CatalogPath $catalogPath 2>&1 if ($LASTEXITCODE -ne 0) { throw "Get-HarnessModels.ps1 failed for ${runnerName}: $($discoveryOutput -join [Environment]::NewLine)" } $discovery = ($discoveryOutput -join [Environment]::NewLine) | ConvertFrom-Json From b3804ac122304793225513298c847f44bd5c745f Mon Sep 17 00:00:00 2001 From: "aicia[bot]" Date: Tue, 25 Aug 2026 02:02:35 +0200 Subject: [PATCH 35/39] =?UTF-8?q?=F0=9F=92=AC=20update=20repository=20docu?= =?UTF-8?q?mentation=20for=20eval=20changes?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Update repository-level documentation to reflect eval infrastructure changes, Cline removal, and native worker result handling implementation. --- AGENTS.md | 8 ++++---- CONTRIBUTING.md | 6 +++--- README.md | 4 ++-- 3 files changed, 9 insertions(+), 9 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 137f5dd..dd1e510 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -40,13 +40,13 @@ This rule is Priority 1. If another repository rule, skill, test, or completion Anthropic's `skill-creator` owns the evaluation methodology this repository uses: define evals, run each task once with the skill and once without it, hold the model, the environment, the task, and the inputs constant, then compare. Keep that experimental design. Only the execution transport changes here. -Where `skill-creator` says to spawn with-skill and baseline subagents in the same turn, this repository prepares a portable evaluation package and stops. The package keeps the existing paired methodology: `run.json` defines what one blind arm executes, `execution-profile.json` selects the runner/model/configuration, and the Eval Runner defines how its harness satisfies the contract. Before an execution-ready `RUN-THIS.prompt.md` is emitted, the user-facing preparation flow resolves a Harness + Model choice; the portable profile stores the internal runner id and the opaque runner-native model selector. The external Eval Orchestrator never chooses runner or model policy. It preflights and invokes one fresh runner process per arm, bridges raw `execution-result.json` evidence into the existing result shape, grades only after execution, invokes the packaged Anthropic `skill-creator` aggregator and static viewer, and returns the finished reports. Preparation, collection, validation, and reporting remain deterministic and never invoke a model. +Where `skill-creator` says to spawn with-skill and baseline subagents in the same turn, this repository prepares a portable evaluation package and stops. The package keeps the existing paired methodology: `run.json` defines what one blind arm executes, `execution-profile.json` selects the runner/model/configuration, and the Eval Runner defines how its harness satisfies the contract. Before an execution-ready `RUN-THIS.prompt.md` is emitted, the user-facing preparation flow resolves a Harness + Model choice; the portable profile stores the internal runner id and the opaque runner-native model selector. The external Eval Orchestrator never chooses runner or model policy. It preflights the selected runner, delegates each arm through its native worker mechanism, and passes each terminal worker envelope through the deterministic package-local `record-native-result.ps1` helper before bridging the runner-owned raw `execution-result.json` into the existing result shape. It grades only after execution, invokes the packaged Anthropic `skill-creator` aggregator and static viewer, and returns the finished reports. Preparation, collection, validation, and reporting remain deterministic and never invoke a model. ### Asking for an eval `eval `, `evaluate `, `eval this skill`, `prepare evals for `, and `evaluate using the existing evals` are all requests for this workflow. Treat them as instructions to prepare the package, never to run it, and never as a request to write new eval cases unless the user asks for that too. -Resolve the execution configuration before running the package preparation script. In an interactive agent session, offer Codebelt Reference first (`GitHub Copilot CLI` + `claude-haiku-4.5`) and verify that model through `scripts/Get-HarnessModels.ps1`; if it is unavailable, show the current discovered Copilot models and ask for a replacement. If the user selects Codex, default to `-Model gpt-5.6-luna` and low reasoning; verify the model through `scripts/Get-HarnessModels.ps1` before preparation. For manual selection, ask for Harness, discover current models for that harness with `scripts/Get-HarnessModels.ps1`, then pass the resulting runner/model pair to the preparation script. Cline and OpenCode discovery is free-only; GitHub Copilot and Codex discovery lists all currently available models. Never guess stale model ids, silently switch harnesses, or generate an execution-ready package with a null runner or model. +Resolve the execution configuration before running the package preparation script. In an interactive agent session, offer Codebelt Reference first (`GitHub Copilot CLI` + `claude-haiku-4.5`) and verify that model through `scripts/Get-HarnessModels.ps1`; if it is unavailable, show the current discovered Copilot models and ask for a replacement. If the user selects Codex, default to `-Model gpt-5.6-luna` and low reasoning; verify the model through `scripts/Get-HarnessModels.ps1` before preparation. For manual selection, ask for Harness, discover current models for that harness with `scripts/Get-HarnessModels.ps1`, then pass the resulting runner/model pair to the preparation script. OpenCode discovery is free-only; GitHub Copilot and Codex discovery lists all currently available models. Never guess stale model ids, silently switch harnesses, or generate an execution-ready package with a null runner or model. ``` pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -Skill dotnet-test -Runner github-copilot -Model claude-haiku-4.5 @@ -62,7 +62,7 @@ Use `-CodebeltReference` only when the script should perform the dynamic Copilot ### Handing the package over -Every package contains `RUN-THIS.prompt.md`, one instruction that drives the whole thing. It makes the user-selected external agent the Eval Orchestrator, Grader, and report producer. The orchestrator resolves the selected Eval Runner, preflights it, invokes it once for every blind `with_skill` and `without_skill` arm, records normalized results and available metrics, grades only after collection, writes the grading fields, and generates the static report without executing an eval prompt in its own context. +Every package contains `RUN-THIS.prompt.md`, one instruction that drives the whole thing. It makes the user-selected external agent the Eval Orchestrator, Grader, and report producer. The orchestrator resolves the selected Eval Runner, preflights it, delegates every blind `with_skill` and `without_skill` arm to a fresh native worker, records each terminal envelope through the deterministic runner-owned recorder, grades only after collection, writes the grading fields, and generates the static report without executing an eval prompt in its own context. Hand the user that one file by its absolute path, and stop there. Do not reproduce its contents in the reply. The runner is built around absolute paths - the package directory, its own location, the path in the hand-back block - and a copy that has passed through a chat window arrives with them shortened to a bare directory name like `iteration-4`, pointing nowhere, with its internal links broken. The file on disk always says what the file on disk says; a paste of it is a lossy snapshot that also goes stale the moment the generator changes. Where the user's harness cannot read files at all, tell them to open that path and paste it themselves, so what travels is the real text rather than your recollection of it. @@ -136,7 +136,7 @@ Four things still hold while you execute: An agent that prepared a package in this session does not get to turn around and execute it. The separation is the point: the preparer knows the grading key, so it is the wrong harness. This is the only role-based disqualification. -The selected executor has two ordered phases. Its current context may read `RUN-THIS.prompt.md`, `manifest.json`, `execution-profile.json`, and the runner protocol files, but it must not execute an eval prompt itself. In phase one, it resolves the selected runner, validates `describe`, preflights each `run.json`, and dispatches every arm through the selected runner's declared harness-native worker mechanism. The parent and delegated worker must not invoke the compatibility `execute` transport or substitute a generic subagent. The runner launches each native harness session from its own run directory with `repo/` as the working directory, `home/` as the isolated profile, and the required isolation controls; hard filesystem confinement, when a runner proves it, raises the reported isolation from pragmatic to strict but is not itself a prerequisite. The runner-produced terminal result must retain the selected runner identity, exact native mechanism, and hashed transcript/event artifact; a parent-created summary is incompatible. The runner receives only `run.json` and `execution-profile.json`; workers never see the runner, manifest, grading key, sibling results, or orchestration commentary, because all of those live outside the run directory. Never reuse a worker or session between runs. In phase two, after all available execution results are complete or failed, the executor validates and freezes the raw results, bridges them into `eval-result/2`, reads the grading key, follows the packaged `skill-creator` grader guidance, writes the grading evidence, invokes the package adapter so Anthropic's aggregator and eval viewer produce the report, and returns the report path and comparison. It does not ask the user whether to start either phase. +The selected executor has two ordered phases. Its current context may read `RUN-THIS.prompt.md`, `manifest.json`, `execution-profile.json`, and the runner protocol files, but it must not execute an eval prompt itself. In phase one, it resolves the selected runner, validates `describe`, preflights each `run.json`, and dispatches every arm through the selected runner's declared harness-native worker mechanism. The parent and delegated worker must not invoke the compatibility `execute` transport or substitute a generic subagent. The runner launches each native harness session from its own run directory with `repo/` as the working directory, `home/` as the isolated profile, and the required isolation controls; hard filesystem confinement, when a runner proves it, raises the reported isolation from pragmatic to strict but is not itself a prerequisite. After each worker is terminal, the executor passes its `eval-native-worker-result/1` envelope to the package-local `record-native-result.ps1` helper, which writes the runner-owned `eval-execution-result/1` file. The executor must never hand-write, normalize, or repair that raw result. The runner receives only `run.json` and `execution-profile.json`; workers never see the runner, manifest, grading key, sibling results, or orchestration commentary, because all of those live outside the run directory. Never reuse a worker or session between runs. In phase two, after all available execution results are complete or failed, the executor validates and freezes the raw results, bridges them into `eval-result/2`, reads the grading key, follows the packaged `skill-creator` grader guidance, writes the grading evidence, invokes the package adapter so Anthropic's aggregator and eval viewer produce the report, and returns the report path and comparison. It does not ask the user whether to start either phase. The candidate instructions are already inlined in the with_skill run's `prompt.md` and staged under its `skill//` directory; the orchestrator does not load or summarize them for the worker. The baseline run has no `skill/` directory and no candidate instructions, and the orchestrator must not expose the candidate skill through another route, including a globally installed copy. The generated prompt files and the baseline `run.json` also omit the skill name, eval identifiers, and configuration labels so workers receive an ordinary task rather than an announcement that they are under evaluation. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 3e28008..cb1517a 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -98,19 +98,19 @@ Run it after the last skill edit and before `scripts/sync-skill-install.ps1`, wh pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -Skill -Runner -Model ``` -Before running the script, choose a Harness + Model. Use `scripts/Get-HarnessModels.ps1 -Runner ` to list current selectors; Cline and OpenCode are free-only, while GitHub Copilot and Codex list all currently available models. The Codebelt Reference shortcut is GitHub Copilot CLI + `claude-haiku-4.5`; `-CodebeltReference` verifies that the model still exists and fails instead of silently substituting a different model. The script writes `.bot/-workspace/iteration-/` with one directory per eval. Each holds the grading key `eval-metadata.json` and result stubs under `results/` at the eval-case level, plus two paired run directories, `with_skill/` and `without_skill/`. A run directory is the worker's run root: `prompt.md`, a `run.json` contract, a `repo/` working tree materialized from the fixtures, an isolated `home/`, and - for `with_skill` only - a `skill//` copy of the candidate. The grading key and results sit outside both run directories. At the root it writes `manifest.json`, `execution-profile.json`, the package-local Eval Runner protocol, the package report adapter, the exact Anthropic skill-creator grader/aggregator/viewer assets, and `RUN-THIS.prompt.md`, the one prompt you hand to the external Eval Orchestrator. That orchestrator resolves and preflights the selected runner, dispatches each arm through its declared harness-native worker mechanism (never the compatibility `execute` transport), stores genuine runner-produced `execution-result.json` evidence, bridges the results, grades only after execution, and runs the adapter, which invokes `aggregate_benchmark.py` and `eval-viewer/generate_review.py --static`. It never runs an eval prompt in the coordinator context, never chooses runner/model policy, and never reuses a worker. Both worker prompts carry the same task, materialized repository, and response contract; only the operating instructions and the presence of `skill/` differ, and neither prompt identifies itself as an eval. `.gitignore` covers `.bot/*`, so nothing there reaches git. The script refuses an `-OutputRoot` inside the repository but outside `.bot/`; pass an explicit temp path when the harness does not need repository-local storage. +Before running the script, choose a Harness + Model. Use `scripts/Get-HarnessModels.ps1 -Runner ` to list current selectors; OpenCode is free-only, while GitHub Copilot and Codex list all currently available models. The Codebelt Reference shortcut is GitHub Copilot CLI + `claude-haiku-4.5`; `-CodebeltReference` verifies that the model still exists and fails instead of silently substituting a different model. The script writes `.bot/-workspace/iteration-/` with one directory per eval. Each holds the grading key `eval-metadata.json` and result stubs under `results/` at the eval-case level, plus two paired run directories, `with_skill/` and `without_skill/`. A run directory is the worker's run root: `prompt.md`, a `run.json` contract, a `repo/` working tree materialized from the fixtures, an isolated `home/`, and - for `with_skill` only - a `skill//` copy of the candidate. The grading key and results sit outside both run directories. At the root it writes `manifest.json`, `execution-profile.json`, the package-local Eval Runner protocol, the package report adapter, the exact Anthropic skill-creator grader/aggregator/viewer assets, and `RUN-THIS.prompt.md`, the one prompt you hand to the external Eval Orchestrator. That orchestrator resolves and preflights the selected runner, dispatches each arm through its declared harness-native worker mechanism (never the compatibility `execute` transport), stores genuine runner-produced `execution-result.json` evidence, bridges the results, grades only after execution, and runs the adapter, which invokes `aggregate_benchmark.py` and `eval-viewer/generate_review.py --static`. It never runs an eval prompt in the coordinator context, never chooses runner/model policy, and never reuses a worker. Both worker prompts carry the same task, materialized repository, and response contract; only the operating instructions and the presence of `skill/` differ, and neither prompt identifies itself as an eval. `.gitignore` covers `.bot/*`, so nothing there reaches git. The script refuses an `-OutputRoot` inside the repository but outside `.bot/`; pass an explicit temp path when the harness does not need repository-local storage. Repository preparation, validation, CI, hooks, deterministic tests, and automatic completion gates never run those prompts or invoke a model. That boundary is the Priority 1 rule in `AGENTS.md`, and preparing a prompt is not permission to execute one. A human-selected external Eval Orchestrator handed a specific package may invoke the selected package-local Eval Runner; this explicit handoff boundary does not weaken the repository prohibition or authorize CI/live evals. Run both configurations on the same model, same version, and same configuration. Independent arms must be dispatched concurrently up to `execution-profile.json.concurrency` when the harness permits it. For OpenCode, emit the sibling native `Task` calls for the first batch in one assistant turn, without asking for confirmation or waiting between calls; deliberate one-at-a-time Task dispatch is not compliant. Serialize only after an explicit harness capacity rejection and preserve that evidence. If the client cannot issue multiple sibling Task calls in one turn, the runner is incompatible. A with-skill run on one model against a baseline on another measures the model as much as the skill and is not a skill-effectiveness result. -Record each external runner result in the matching `execution-result.json` first. It must carry the protocol/schema, opaque run and fresh session ids, status, complete final response or explicit unavailability, runner/harness identity, requested and resolved model selection, timestamps and duration, exit/failure state, prompt/run/profile hashes, resolved isolation mechanisms, warnings, and artifact references. Include token, cache, cost, tool, command, file, and transcript evidence only when the harness exposes it; unavailable values remain explicitly unavailable and are never estimated. The deterministic bridge then writes the existing `results/*.result.json` shape, after which grading may add `grading[].passed` and evidence. Assertions about tool, shell, or file behavior are only gradeable from a run that captured that evidence. If the results were transferred without those report artifacts, validate and compare with: +After each native worker completes, preserve its `codebeltnet/agentic/eval-native-worker-result/1` terminal envelope and pass it to the package-local `record-native-result.ps1` helper with the exact `run.json`, `execution-profile.json`, and manifest-declared output path. That deterministic helper writes the runner-owned `execution-result.json`; do not hand-write or normalize that file in the orchestrator. The generated result must carry the protocol/schema, opaque run and fresh session ids, status, complete final response or explicit unavailability, runner/harness identity, requested and resolved model selection, timestamps and duration, exit/failure state, prompt/run/profile hashes, resolved isolation mechanisms, warnings, and artifact references. Include token, cache, cost, tool, command, file, and transcript evidence only when the harness exposes it; unavailable values remain explicitly unavailable and are never estimated. The deterministic bridge then writes the existing `results/*.result.json` shape, after which grading may add `grading[].passed` and evidence. Assertions about tool, shell, or file behavior are only gradeable from a run that captured that evidence. If the results were transferred without those report artifacts, validate and compare with: ```console pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -CollectResults ``` -It may write a diagnostic `comparison.md` while flagging missing arms, unrun configurations, incompatible evidence, and mixed models, but it exits non-zero and does not write benchmark/report artifacts until the required completion gate is satisfied. Those diagnostic artifacts must not present an incomplete or unrun package as a successfully completed evaluation. The normal external Eval Orchestrator grades in the same handoff using deterministic checks for mechanical assertions and evidence-backed judgement where an assertion is genuinely qualitative. Repository automation remains deterministic and never invokes a model. Codex, OpenCode, and Cline are the conforming real runners; the deterministic fake runner is the CI conformance harness. Hard filesystem confinement is reported as strict versus pragmatic confidence and is not a universal Windows prerequisite. Freebuff remains planned/blocked until its official CLI provides a supported noninteractive machine-readable transport. Native skill activation, portability scoring, and additional runners are not part of v0.9.1. +It may write a diagnostic `comparison.md` while flagging missing arms, unrun configurations, incompatible evidence, and mixed models, but it exits non-zero and does not write benchmark/report artifacts until the required completion gate is satisfied. Those diagnostic artifacts must not present an incomplete or unrun package as a successfully completed evaluation. The normal external Eval Orchestrator grades in the same handoff using deterministic checks for mechanical assertions and evidence-backed judgement where an assertion is genuinely qualitative. Repository automation remains deterministic and never invokes a model. Codex, OpenCode, and GitHub Copilot are the conforming real runners; the deterministic fake runner is the CI conformance harness. Hard filesystem confinement is reported as strict versus pragmatic confidence and is not a universal Windows prerequisite. Freebuff remains planned/blocked until its official CLI provides a supported noninteractive machine-readable transport. Native skill activation, portability scoring, and additional runners are not part of v0.9.1. The eval package is a temp artifact. Do not commit it, its prompts, or its results unless the change explicitly calls for checked-in examples. diff --git a/README.md b/README.md index cbb0d00..4e0720b 100644 --- a/README.md +++ b/README.md @@ -14,7 +14,7 @@ Another repo rule is intentionally strict: every repo-managed skill ships with i Skill validation is local and deterministic. The Priority 1 **AI/LLM Evaluation Automation Prohibition** in `AGENTS.md` forbids repository preparation, validation, CI, hooks, deterministic tests, automatic fan-out, graders, and completion gates from using an authenticated Copilot, Claude, Codex, Gemini, or other model account. There is no repository opt-in switch. A human-selected external Eval Orchestrator may invoke an explicitly selected package-local Eval Runner for a package it was handed; that boundary never authorizes live model execution in CI or automatic repository workflows. -Evaluation keeps Anthropic's `skill-creator` methodology and portable paired-run conventions while replacing only the execution transport. `run.json` remains the runner-neutral one-arm contract; `execution-profile.json` selects the runner/model/configuration; and `execution-result.json` normalizes one blind execution before the existing `eval-result/2` bridge and reports. The user-facing preparation flow asks for Harness + Model before emitting `RUN-THIS.prompt.md`; the portable profile stores the internal runner id and the opaque runner-native model selector, with no provider field. The conforming real runners are GitHub Copilot, Codex, OpenCode, and Cline, with a deterministic fake runner used for conformance. GitHub Copilot CLI with `claude-haiku-4.5` is the Codebelt Reference evaluation configuration — a repository convention for economical, stable comparison, not an Anthropic default — and its availability is verified against the current Copilot model catalog before automatic selection. Cline and OpenCode discovery lists only currently free models; GitHub Copilot and Codex discovery lists all currently available models. The repository prepares the paired candidate and baseline inputs as a portable package and stops; the external Eval Orchestrator resolves, preflights, and invokes the selected Eval Runner once per fresh arm, then grades and reports. Hard filesystem confinement raises reported isolation confidence from pragmatic to strict but is not a universal platform prerequisite, so Windows is a first-class pragmatic target; mandatory experimental controls remain fail-closed. Freebuff remains planned/blocked until it exposes a supported noninteractive machine-readable transport. Native skill activation is not evaluated in v0.9.1. Adding or modifying a skill triggers package preparation automatically, as a completion gate an agent cannot skip: +Evaluation keeps Anthropic's `skill-creator` methodology and portable paired-run conventions while replacing only the execution transport. `run.json` remains the runner-neutral one-arm contract; `execution-profile.json` selects the runner/model/configuration; and the native worker envelope is recorded by the selected package runner into `execution-result.json` before the existing `eval-result/2` bridge and reports. The user-facing preparation flow asks for Harness + Model before emitting `RUN-THIS.prompt.md`; the portable profile stores the internal runner id and the opaque runner-native model selector, with no provider field. The conforming real runners are GitHub Copilot, Codex, and OpenCode, with a deterministic fake runner used for conformance. GitHub Copilot CLI with `claude-haiku-4.5` is the Codebelt Reference evaluation configuration — a repository convention for economical, stable comparison, not an Anthropic default — and its availability is verified against the current Copilot model catalog before automatic selection. OpenCode discovery lists only currently free models; GitHub Copilot and Codex discovery lists all currently available models. The repository prepares the paired candidate and baseline inputs as a portable package and stops; the external Eval Orchestrator resolves, preflights, dispatches each arm through the selected runner's native worker mechanism, records the terminal envelope with `record-native-result.ps1`, then grades and reports. Hard filesystem confinement raises reported isolation confidence from pragmatic to strict but is not a universal platform prerequisite, so Windows is a first-class pragmatic target; mandatory experimental controls remain fail-closed. Freebuff remains planned/blocked until it exposes a supported noninteractive machine-readable transport. Native skill activation is not evaluated in v0.9.1. Adding or modifying a skill triggers package preparation automatically, as a completion gate an agent cannot skip: ```powershell pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -Changed -Runner github-copilot -Model claude-haiku-4.5 @@ -22,7 +22,7 @@ pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -Changed -Runner github- That resolves every skill the branch changed and prepares a package for each. `-Skill ` prepares one on demand, but execution selection must already be resolved; direct noninteractive use without `-Runner/-Model` or `-CodebeltReference` fails before a handoff is generated. `scripts/Get-HarnessModels.ps1 -Runner ` lists current model selectors for the selected harness. Use `-CodebeltReference` only when you want the script to verify the current Copilot catalog and select GitHub Copilot CLI + `claude-haiku-4.5`; if that model is absent, the script fails and prints the current choices rather than substituting another model. Packages land in the gitignored `.bot/-workspace/`, so a harness that refuses to work outside the repository folder can still reach them without anything entering the working tree. -Each eval becomes a directory holding the grading key (`eval-metadata.json` with the expected output, assertions, and fixture and skill hashes) and prefilled result stubs, plus two paired run directories. `with_skill/` is a self-contained run root: a `prompt.md` with the effective skill instructions inlined, a `run.json` contract naming only paths inside the run, a `repo/` working tree materialized from the fixtures as real files, an isolated empty `home/`, and a `skill//` copy of the exact candidate revision. `without_skill/` is the same run with a byte-identical `repo/`, no `skill/` directory, and no mention of the skill. The grading key and results sit outside both run directories, so workers are not intentionally given them. Neither prompt identifies itself as an eval or names its configuration. `RUN-THIS.prompt.md` makes the user-selected agent the Eval Orchestrator: it reads the profile, resolves the selected runner, validates `describe`, preflights each `run.json`, invokes one fresh runner process per arm, stores raw `execution-result.json`, bridges it into the existing result shape, reveals the grading key only after execution, grades, and invokes the package adapter. The adapter stages the results into Anthropic's upstream benchmark workspace, runs `aggregate_benchmark.py`, writes the exact upstream `skill-creator-report.html`, and writes a first-party `report.html` with paired outputs, expected outcomes, assertion evidence, telemetry, transcripts, and downloadable feedback, plus `benchmark.json` and `benchmark.md`. Missing telemetry is displayed as unavailable rather than estimated. A runner that cannot satisfy the mandatory experimental controls returns `incompatible`; lack of hard filesystem confinement downgrades the result to pragmatic isolation. `-CollectResults ` remains a deterministic fallback for transferred artifacts without reports; it may inspect and report available package state, but exits non-zero when the required completion gate is not satisfied, so an incomplete or unrun package is not a successfully completed evaluation. Packages land in gitignored `.bot/` storage by default and are not committed. +Each eval becomes a directory holding the grading key (`eval-metadata.json` with the expected output, assertions, and fixture and skill hashes) and prefilled result stubs, plus two paired run directories. `with_skill/` is a self-contained run root: a `prompt.md` with the effective skill instructions inlined, a `run.json` contract naming only paths inside the run, a `repo/` working tree materialized from the fixtures as real files, an isolated empty `home/`, and a `skill//` copy of the exact candidate revision. `without_skill/` is the same run with a byte-identical `repo/`, no `skill/` directory, and no mention of the skill. The grading key and results sit outside both run directories, so workers are not intentionally given them. Neither prompt identifies itself as an eval or names its configuration. `RUN-THIS.prompt.md` makes the user-selected agent the Eval Orchestrator: it reads the profile, resolves the selected runner, validates `describe`, preflights each `run.json`, dispatches one fresh native worker per arm, passes each terminal envelope through `record-native-result.ps1`, stores the runner-produced raw `execution-result.json`, bridges it into the existing result shape, reveals the grading key only after execution, grades, and invokes the package adapter. The adapter stages the results into Anthropic's upstream benchmark workspace, runs `aggregate_benchmark.py`, writes the exact upstream `skill-creator-report.html`, and writes a first-party `report.html` with paired outputs, expected outcomes, assertion evidence, telemetry, transcripts, and downloadable feedback, plus `benchmark.json` and `benchmark.md`. Missing telemetry is displayed as unavailable rather than estimated. A runner that cannot satisfy the mandatory experimental controls returns `incompatible`; lack of hard filesystem confinement downgrades the result to pragmatic isolation. `-CollectResults ` remains a deterministic fallback for transferred artifacts without reports; it may inspect and report available package state, but exits non-zero when the required completion gate is not satisfied, so an incomplete or unrun package is not a successfully completed evaluation. Packages land in gitignored `.bot/` storage by default and are not committed. The four roles are intentionally separate: the Eval Runner is the harness-specific executor for one blind arm; the Eval Orchestrator coordinates the external handoff; the Grader assesses results only after execution; and the Human Reviewer remains the final evaluator. Behavioral evaluation is portable across supported runners. Native activation is harness-specific and out of scope for v0.9.1. Anthropic is the methodology and compatibility reference, not a required Claude runtime. From cc27e68777a46fe1840add01a039a1b3884938d4 Mon Sep 17 00:00:00 2001 From: Eval Worker Date: Tue, 25 Aug 2026 02:05:26 +0200 Subject: [PATCH 36/39] =?UTF-8?q?=F0=9F=94=A5=20remove=20cline=20eval=20ru?= =?UTF-8?q?nner=20support=20from=20changelog?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- CHANGELOG.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 1a7da4f..1b63f5f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,23 +6,23 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), ## [0.9.1] - 2026-08-22 -This patch release adds harness-agnostic Eval Runner execution boundary infrastructure without changing the paired evaluation methodology or existing report schemas, while optimizing skill descriptions and refactoring repository-level authoring guidance. Prepared packages now carry `execution-profile.json`, package-local runner protocol tools, and normalized `execution-result.json` evidence. The deterministic fake runner is the conformance reference, with Codex, GitHub Copilot CLI, Cline, and OpenCode as supported real adapters. Repository automation remains model-free; only a human-directed external Eval Orchestrator may invoke the selected runner, and unsupported isolation fails closed. +This patch release adds harness-agnostic Eval Runner execution boundary infrastructure without changing the paired evaluation methodology or existing report schemas, while optimizing skill descriptions and refactoring repository-level authoring guidance. Prepared packages now carry `execution-profile.json`, package-local runner protocol tools, and normalized `execution-result.json` evidence. The deterministic fake runner is the conformance reference, with Codex, GitHub Copilot CLI, and OpenCode as supported real adapters. Repository automation remains model-free; only a human-directed external Eval Orchestrator may invoke the selected runner, and unsupported isolation fails closed. ### Added -- `scripts/eval-runners/` with the common `describe`/`preflight`/`execute` process contract, execution-profile and execution-result schemas, deterministic fake runner, Codex adapter, GitHub Copilot CLI adapter, Cline adapter, OpenCode adapter, runner resolution, artifact/hash validation, and bridge into the existing `eval-result/2` result shape, +- `scripts/eval-runners/` with the common `describe`/`preflight`/`execute` process contract, execution-profile and execution-result schemas, deterministic fake runner, Codex adapter, GitHub Copilot CLI adapter, OpenCode adapter, runner resolution, artifact/hash validation, and bridge into the existing `eval-result/2` result shape, - GitHub Copilot CLI as a supported Eval Runner with authentication handling (GitHub tokens, OS keychain, CLI fallback), JSONL-based event output parsing, stdin-based prompt delivery for byte fidelity, repository instruction visibility, and conformance tests covering token management and authentication source detection, - deterministic fake-runner conformance coverage for fresh paired sessions, prompt fidelity, isolation boundaries, candidate-skill exposure, status normalization, unavailable telemetry, event warnings, artifact references, and report compatibility, - runner-aware package preparation that reuses `run.json`, keeps runner selection outside `evals/evals.json`, and preserves Anthropic-compatible benchmark/report artifacts. ### Changed -- Eval preparation now resolves Harness + Model before writing `RUN-THIS.prompt.md`, removes the redundant portable `provider` field from `execution-profile.json` and result reporting, treats model selectors as runner-native opaque strings, and adds `scripts/Get-HarnessModels.ps1` for current model discovery with Codebelt Reference verification and free-only Cline/OpenCode filtering, +- Eval preparation now resolves Harness + Model before writing `RUN-THIS.prompt.md`, removes the redundant portable `provider` field from `execution-profile.json` and result reporting, treats model selectors as runner-native opaque strings, and adds `scripts/Get-HarnessModels.ps1` for current model discovery with Codebelt Reference verification and free-only OpenCode filtering, - `AGENTS.md`, `README.md`, and `CONTRIBUTING.md` now distinguish the Eval Runner, Eval Orchestrator, Grader, and Human Reviewer and clarify that runner execution is an explicit external-handoff boundary rather than repository automation, - All 21 repo-managed skill descriptions refactored to lean, trigger-oriented activation metadata following progressive-disclosure principles and specification compliance, - `AGENTS.md` Skill Authoring section restructured for clarity, brevity, and progressive disclosure of form fields, asset handling, and dynamic defaults, - report timing output omits unavailable duration and token telemetry instead of writing zero placeholders, -- Cline and GitHub Copilot runners added to the eval runner lineup alongside existing Codex and OpenCode support, +- GitHub Copilot runner added to the eval runner lineup alongside existing Codex and OpenCode support, - Runner conformance tests enhanced with additional event fixtures and isolation capability assessment. ### Fixed From 1302fa0c4434f574df9a9b00e9276eb1c96baba6 Mon Sep 17 00:00:00 2001 From: "aicia[bot]" Date: Tue, 25 Aug 2026 18:40:38 +0200 Subject: [PATCH 37/39] =?UTF-8?q?=F0=9F=92=AC=20clarify=20orchestrator/run?= =?UTF-8?q?ner=20dispatch=20mechanism?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Expand AGENTS.md, CONTRIBUTING.md, and README.md to document the dispatch_owner semantics that enables external eval executors to delegate work through either orchestrator-owned (native subagent/task) or runner-owned (native execution surface) mechanisms. This clarification supports the portable eval handoff workflow where external evaluators make explicit dispatch decisions based on runner capability declarations. --- AGENTS.md | 4 +- CONTRIBUTING.md | 4 +- README.md | 4 +- scripts/eval-runners/README.md | 108 ++++++++++++++++++--------------- 4 files changed, 66 insertions(+), 54 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index dd1e510..360fdba 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -40,7 +40,7 @@ This rule is Priority 1. If another repository rule, skill, test, or completion Anthropic's `skill-creator` owns the evaluation methodology this repository uses: define evals, run each task once with the skill and once without it, hold the model, the environment, the task, and the inputs constant, then compare. Keep that experimental design. Only the execution transport changes here. -Where `skill-creator` says to spawn with-skill and baseline subagents in the same turn, this repository prepares a portable evaluation package and stops. The package keeps the existing paired methodology: `run.json` defines what one blind arm executes, `execution-profile.json` selects the runner/model/configuration, and the Eval Runner defines how its harness satisfies the contract. Before an execution-ready `RUN-THIS.prompt.md` is emitted, the user-facing preparation flow resolves a Harness + Model choice; the portable profile stores the internal runner id and the opaque runner-native model selector. The external Eval Orchestrator never chooses runner or model policy. It preflights the selected runner, delegates each arm through its native worker mechanism, and passes each terminal worker envelope through the deterministic package-local `record-native-result.ps1` helper before bridging the runner-owned raw `execution-result.json` into the existing result shape. It grades only after execution, invokes the packaged Anthropic `skill-creator` aggregator and static viewer, and returns the finished reports. Preparation, collection, validation, and reporting remain deterministic and never invoke a model. +Where `skill-creator` says to spawn with-skill and baseline subagents in the same turn, this repository prepares a portable evaluation package and stops. The package keeps the existing paired methodology: `run.json` defines what one blind arm executes, `execution-profile.json` selects the runner/model/configuration, and the Eval Runner defines how its harness satisfies the contract. Before an execution-ready `RUN-THIS.prompt.md` is emitted, the user-facing preparation flow resolves a Harness + Model choice; the portable profile stores the internal runner id and the opaque runner-native model selector. The external Eval Orchestrator never chooses runner or model policy. It preflights the selected runner, reads `delegation.dispatch_owner`, and either delegates each arm through the declared orchestrator-owned native worker or invokes the declared runner-owned one-arm native surface directly. Orchestrator-owned envelopes pass through `record-native-result.ps1`; runner-owned transports produce the canonical raw result directly. It grades only after execution, invokes the packaged Anthropic `skill-creator` aggregator and static viewer, and returns the finished reports. Preparation, collection, validation, and reporting remain deterministic and never invoke a model. ### Asking for an eval @@ -136,7 +136,7 @@ Four things still hold while you execute: An agent that prepared a package in this session does not get to turn around and execute it. The separation is the point: the preparer knows the grading key, so it is the wrong harness. This is the only role-based disqualification. -The selected executor has two ordered phases. Its current context may read `RUN-THIS.prompt.md`, `manifest.json`, `execution-profile.json`, and the runner protocol files, but it must not execute an eval prompt itself. In phase one, it resolves the selected runner, validates `describe`, preflights each `run.json`, and dispatches every arm through the selected runner's declared harness-native worker mechanism. The parent and delegated worker must not invoke the compatibility `execute` transport or substitute a generic subagent. The runner launches each native harness session from its own run directory with `repo/` as the working directory, `home/` as the isolated profile, and the required isolation controls; hard filesystem confinement, when a runner proves it, raises the reported isolation from pragmatic to strict but is not itself a prerequisite. After each worker is terminal, the executor passes its `eval-native-worker-result/1` envelope to the package-local `record-native-result.ps1` helper, which writes the runner-owned `eval-execution-result/1` file. The executor must never hand-write, normalize, or repair that raw result. The runner receives only `run.json` and `execution-profile.json`; workers never see the runner, manifest, grading key, sibling results, or orchestration commentary, because all of those live outside the run directory. Never reuse a worker or session between runs. In phase two, after all available execution results are complete or failed, the executor validates and freezes the raw results, bridges them into `eval-result/2`, reads the grading key, follows the packaged `skill-creator` grader guidance, writes the grading evidence, invokes the package adapter so Anthropic's aggregator and eval viewer produce the report, and returns the report path and comparison. It does not ask the user whether to start either phase. +The selected executor has two ordered phases. Its current context may read `RUN-THIS.prompt.md`, `manifest.json`, `execution-profile.json`, and the runner protocol files, but it must not execute an eval prompt itself. In phase one, it resolves the selected runner, validates `describe`, preflights each `run.json`, and follows `delegation.dispatch_owner`: an orchestrator-owned runner uses its declared native subagent/task, while a runner-owned runner invokes its one-arm native execution surface directly. A runner-owned process/thread is the single Eval Worker and model execution; no outer model worker may contain it. Orchestrator-owned workers must not invoke `runner.ps1 execute`; their transport envelope passes through `record-native-result.ps1`. Runner-owned execution results come directly from the runner and must never be synthesized, repaired, or reconstructed from assistant text. The runner launches each native session from its own run directory with `repo/` as the working directory, `home/` as the isolated profile, and the required isolation controls; hard filesystem confinement, when a runner proves it, raises the reported isolation from pragmatic to strict but is not itself a prerequisite. The runner receives only `run.json` and `execution-profile.json`; workers never see the runner, manifest, grading key, sibling results, or orchestration commentary, because all of those live outside the run directory. Never reuse a worker or session between runs. In phase two, after all available execution results are complete or failed, the executor validates and freezes the raw results, bridges them into `eval-result/2`, reads the grading key, follows the packaged `skill-creator` grader guidance, writes the grading evidence, invokes the package adapter so Anthropic's aggregator and eval viewer produce the report, and returns the report path and comparison. It does not ask the user whether to start either phase. The candidate instructions are already inlined in the with_skill run's `prompt.md` and staged under its `skill//` directory; the orchestrator does not load or summarize them for the worker. The baseline run has no `skill/` directory and no candidate instructions, and the orchestrator must not expose the candidate skill through another route, including a globally installed copy. The generated prompt files and the baseline `run.json` also omit the skill name, eval identifiers, and configuration labels so workers receive an ordinary task rather than an announcement that they are under evaluation. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index cb1517a..22a341a 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -98,13 +98,13 @@ Run it after the last skill edit and before `scripts/sync-skill-install.ps1`, wh pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -Skill -Runner -Model ``` -Before running the script, choose a Harness + Model. Use `scripts/Get-HarnessModels.ps1 -Runner ` to list current selectors; OpenCode is free-only, while GitHub Copilot and Codex list all currently available models. The Codebelt Reference shortcut is GitHub Copilot CLI + `claude-haiku-4.5`; `-CodebeltReference` verifies that the model still exists and fails instead of silently substituting a different model. The script writes `.bot/-workspace/iteration-/` with one directory per eval. Each holds the grading key `eval-metadata.json` and result stubs under `results/` at the eval-case level, plus two paired run directories, `with_skill/` and `without_skill/`. A run directory is the worker's run root: `prompt.md`, a `run.json` contract, a `repo/` working tree materialized from the fixtures, an isolated `home/`, and - for `with_skill` only - a `skill//` copy of the candidate. The grading key and results sit outside both run directories. At the root it writes `manifest.json`, `execution-profile.json`, the package-local Eval Runner protocol, the package report adapter, the exact Anthropic skill-creator grader/aggregator/viewer assets, and `RUN-THIS.prompt.md`, the one prompt you hand to the external Eval Orchestrator. That orchestrator resolves and preflights the selected runner, dispatches each arm through its declared harness-native worker mechanism (never the compatibility `execute` transport), stores genuine runner-produced `execution-result.json` evidence, bridges the results, grades only after execution, and runs the adapter, which invokes `aggregate_benchmark.py` and `eval-viewer/generate_review.py --static`. It never runs an eval prompt in the coordinator context, never chooses runner/model policy, and never reuses a worker. Both worker prompts carry the same task, materialized repository, and response contract; only the operating instructions and the presence of `skill/` differ, and neither prompt identifies itself as an eval. `.gitignore` covers `.bot/*`, so nothing there reaches git. The script refuses an `-OutputRoot` inside the repository but outside `.bot/`; pass an explicit temp path when the harness does not need repository-local storage. +Before running the script, choose a Harness + Model. Use `scripts/Get-HarnessModels.ps1 -Runner ` to list current selectors; OpenCode is free-only, while GitHub Copilot and Codex list all currently available models. The Codebelt Reference shortcut is GitHub Copilot CLI + `claude-haiku-4.5`; `-CodebeltReference` verifies that the model still exists and fails instead of silently substituting a different model. The script writes `.bot/-workspace/iteration-/` with one directory per eval. Each holds the grading key `eval-metadata.json` and result stubs under `results/` at the eval-case level, plus two paired run directories, `with_skill/` and `without_skill/`. A run directory is the worker's run root: `prompt.md`, a `run.json` contract, a `repo/` working tree materialized from the fixtures, an isolated `home/`, and - for `with_skill` only - a `skill//` copy of the candidate. The grading key and results sit outside both run directories. At the root it writes `manifest.json`, `execution-profile.json`, the package-local Eval Runner protocol, the package report adapter, the exact Anthropic skill-creator grader/aggregator/viewer assets, and `RUN-THIS.prompt.md`, the one prompt you hand to the external Eval Orchestrator. That orchestrator resolves and preflights the selected runner, reads `delegation.dispatch_owner`, and either dispatches the declared orchestrator-owned native worker or starts the declared runner-owned one-arm native surface directly. It stores genuine transport-produced `execution-result.json` evidence, bridges the results, grades only after execution, and runs the adapter, which invokes `aggregate_benchmark.py` and `eval-viewer/generate_review.py --static`. It never runs an eval prompt in the coordinator context, never chooses runner/model policy, and never reuses a worker. Both worker prompts carry the same task, materialized repository, and response contract; only the operating instructions and the presence of `skill/` differ, and neither prompt identifies itself as an eval. `.gitignore` covers `.bot/*`, so nothing there reaches git. The script refuses an `-OutputRoot` inside the repository but outside `.bot/`; pass an explicit temp path when the harness does not need repository-local storage. Repository preparation, validation, CI, hooks, deterministic tests, and automatic completion gates never run those prompts or invoke a model. That boundary is the Priority 1 rule in `AGENTS.md`, and preparing a prompt is not permission to execute one. A human-selected external Eval Orchestrator handed a specific package may invoke the selected package-local Eval Runner; this explicit handoff boundary does not weaken the repository prohibition or authorize CI/live evals. Run both configurations on the same model, same version, and same configuration. Independent arms must be dispatched concurrently up to `execution-profile.json.concurrency` when the harness permits it. For OpenCode, emit the sibling native `Task` calls for the first batch in one assistant turn, without asking for confirmation or waiting between calls; deliberate one-at-a-time Task dispatch is not compliant. Serialize only after an explicit harness capacity rejection and preserve that evidence. If the client cannot issue multiple sibling Task calls in one turn, the runner is incompatible. A with-skill run on one model against a baseline on another measures the model as much as the skill and is not a skill-effectiveness result. -After each native worker completes, preserve its `codebeltnet/agentic/eval-native-worker-result/1` terminal envelope and pass it to the package-local `record-native-result.ps1` helper with the exact `run.json`, `execution-profile.json`, and manifest-declared output path. That deterministic helper writes the runner-owned `execution-result.json`; do not hand-write or normalize that file in the orchestrator. The generated result must carry the protocol/schema, opaque run and fresh session ids, status, complete final response or explicit unavailability, runner/harness identity, requested and resolved model selection, timestamps and duration, exit/failure state, prompt/run/profile hashes, resolved isolation mechanisms, warnings, and artifact references. Include token, cache, cost, tool, command, file, and transcript evidence only when the harness exposes it; unavailable values remain explicitly unavailable and are never estimated. The deterministic bridge then writes the existing `results/*.result.json` shape, after which grading may add `grading[].passed` and evidence. Assertions about tool, shell, or file behavior are only gradeable from a run that captured that evidence. If the results were transferred without those report artifacts, validate and compare with: +For an orchestrator-owned worker, preserve its `codebeltnet/agentic/eval-native-worker-result/1` terminal envelope and pass it to `record-native-result.ps1` with the exact `run.json`, `execution-profile.json`, and manifest-declared output path. For a runner-owned worker, preserve the runner-produced `execution-result.json` directly at the exact manifest-declared path and do not invoke the recorder or synthesize an envelope. In either mode, the generated result must carry the protocol/schema, opaque run and fresh session ids, status, complete final response or explicit unavailability, runner/harness identity, requested and resolved model selection, timestamps and duration, exit/failure state, prompt/run/profile hashes, resolved isolation mechanisms, warnings, and artifact references. Include token, cache, cost, tool, command, file, and transcript evidence only when the harness exposes it; unavailable values remain explicitly unavailable and are never estimated. The deterministic bridge then writes the existing `results/*.result.json` shape, after which grading may add `grading[].passed` and evidence. Assertions about tool, shell, or file behavior are only gradeable from a run that captured that evidence. If the results were transferred without those report artifacts, validate and compare with: ```console pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -CollectResults diff --git a/README.md b/README.md index 4e0720b..21d6f7b 100644 --- a/README.md +++ b/README.md @@ -14,7 +14,7 @@ Another repo rule is intentionally strict: every repo-managed skill ships with i Skill validation is local and deterministic. The Priority 1 **AI/LLM Evaluation Automation Prohibition** in `AGENTS.md` forbids repository preparation, validation, CI, hooks, deterministic tests, automatic fan-out, graders, and completion gates from using an authenticated Copilot, Claude, Codex, Gemini, or other model account. There is no repository opt-in switch. A human-selected external Eval Orchestrator may invoke an explicitly selected package-local Eval Runner for a package it was handed; that boundary never authorizes live model execution in CI or automatic repository workflows. -Evaluation keeps Anthropic's `skill-creator` methodology and portable paired-run conventions while replacing only the execution transport. `run.json` remains the runner-neutral one-arm contract; `execution-profile.json` selects the runner/model/configuration; and the native worker envelope is recorded by the selected package runner into `execution-result.json` before the existing `eval-result/2` bridge and reports. The user-facing preparation flow asks for Harness + Model before emitting `RUN-THIS.prompt.md`; the portable profile stores the internal runner id and the opaque runner-native model selector, with no provider field. The conforming real runners are GitHub Copilot, Codex, and OpenCode, with a deterministic fake runner used for conformance. GitHub Copilot CLI with `claude-haiku-4.5` is the Codebelt Reference evaluation configuration — a repository convention for economical, stable comparison, not an Anthropic default — and its availability is verified against the current Copilot model catalog before automatic selection. OpenCode discovery lists only currently free models; GitHub Copilot and Codex discovery lists all currently available models. The repository prepares the paired candidate and baseline inputs as a portable package and stops; the external Eval Orchestrator resolves, preflights, dispatches each arm through the selected runner's native worker mechanism, records the terminal envelope with `record-native-result.ps1`, then grades and reports. Hard filesystem confinement raises reported isolation confidence from pragmatic to strict but is not a universal platform prerequisite, so Windows is a first-class pragmatic target; mandatory experimental controls remain fail-closed. Freebuff remains planned/blocked until it exposes a supported noninteractive machine-readable transport. Native skill activation is not evaluated in v0.9.1. Adding or modifying a skill triggers package preparation automatically, as a completion gate an agent cannot skip: +Evaluation keeps Anthropic's `skill-creator` methodology and portable paired-run conventions while replacing only the execution transport. `run.json` remains the runner-neutral one-arm contract; `execution-profile.json` selects the runner/model/configuration; and `delegation.dispatch_owner` declares whether the orchestrator dispatches a native subagent/task or starts the runner-owned native execution surface directly. Orchestrator-owned envelopes pass through `record-native-result.ps1`; runner-owned transports produce `execution-result.json` directly before the existing `eval-result/2` bridge and reports. The user-facing preparation flow asks for Harness + Model before emitting `RUN-THIS.prompt.md`; the portable profile stores the internal runner id and the opaque runner-native model selector, with no provider field. The conforming real runners are GitHub Copilot, Codex, and OpenCode, with a deterministic fake runner used for conformance. GitHub Copilot CLI with `claude-haiku-4.5` is the Codebelt Reference evaluation configuration — a repository convention for economical, stable comparison, not an Anthropic default — and its availability is verified against the current Copilot model catalog before automatic selection. OpenCode discovery lists only currently free models; GitHub Copilot and Codex discovery lists all currently available models. The repository prepares the paired candidate and baseline inputs as a portable package and stops; the external Eval Orchestrator resolves, preflights, follows the selected dispatch owner, then grades and reports. Hard filesystem confinement raises reported isolation confidence from pragmatic to strict but is not a universal platform prerequisite, so Windows is a first-class pragmatic target; mandatory experimental controls remain fail-closed. Freebuff remains planned/blocked until it exposes a supported noninteractive machine-readable transport. Native skill activation is not evaluated in v0.9.1. Adding or modifying a skill triggers package preparation automatically, as a completion gate an agent cannot skip: ```powershell pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -Changed -Runner github-copilot -Model claude-haiku-4.5 @@ -22,7 +22,7 @@ pwsh -NoProfile -File ./scripts/prepare-skill-evals.ps1 -Changed -Runner github- That resolves every skill the branch changed and prepares a package for each. `-Skill ` prepares one on demand, but execution selection must already be resolved; direct noninteractive use without `-Runner/-Model` or `-CodebeltReference` fails before a handoff is generated. `scripts/Get-HarnessModels.ps1 -Runner ` lists current model selectors for the selected harness. Use `-CodebeltReference` only when you want the script to verify the current Copilot catalog and select GitHub Copilot CLI + `claude-haiku-4.5`; if that model is absent, the script fails and prints the current choices rather than substituting another model. Packages land in the gitignored `.bot/-workspace/`, so a harness that refuses to work outside the repository folder can still reach them without anything entering the working tree. -Each eval becomes a directory holding the grading key (`eval-metadata.json` with the expected output, assertions, and fixture and skill hashes) and prefilled result stubs, plus two paired run directories. `with_skill/` is a self-contained run root: a `prompt.md` with the effective skill instructions inlined, a `run.json` contract naming only paths inside the run, a `repo/` working tree materialized from the fixtures as real files, an isolated empty `home/`, and a `skill//` copy of the exact candidate revision. `without_skill/` is the same run with a byte-identical `repo/`, no `skill/` directory, and no mention of the skill. The grading key and results sit outside both run directories, so workers are not intentionally given them. Neither prompt identifies itself as an eval or names its configuration. `RUN-THIS.prompt.md` makes the user-selected agent the Eval Orchestrator: it reads the profile, resolves the selected runner, validates `describe`, preflights each `run.json`, dispatches one fresh native worker per arm, passes each terminal envelope through `record-native-result.ps1`, stores the runner-produced raw `execution-result.json`, bridges it into the existing result shape, reveals the grading key only after execution, grades, and invokes the package adapter. The adapter stages the results into Anthropic's upstream benchmark workspace, runs `aggregate_benchmark.py`, writes the exact upstream `skill-creator-report.html`, and writes a first-party `report.html` with paired outputs, expected outcomes, assertion evidence, telemetry, transcripts, and downloadable feedback, plus `benchmark.json` and `benchmark.md`. Missing telemetry is displayed as unavailable rather than estimated. A runner that cannot satisfy the mandatory experimental controls returns `incompatible`; lack of hard filesystem confinement downgrades the result to pragmatic isolation. `-CollectResults ` remains a deterministic fallback for transferred artifacts without reports; it may inspect and report available package state, but exits non-zero when the required completion gate is not satisfied, so an incomplete or unrun package is not a successfully completed evaluation. Packages land in gitignored `.bot/` storage by default and are not committed. +Each eval becomes a directory holding the grading key (`eval-metadata.json` with the expected output, assertions, and fixture and skill hashes) and prefilled result stubs, plus two paired run directories. `with_skill/` is a self-contained run root: a `prompt.md` with the effective skill instructions inlined, a `run.json` contract naming only paths inside the run, a `repo/` working tree materialized from the fixtures as real files, an isolated empty `home/`, and a `skill//` copy of the exact candidate revision. `without_skill/` is the same run with a byte-identical `repo/`, no `skill/` directory, and no mention of the skill. The grading key and results sit outside both run directories, so workers are not intentionally given them. Neither prompt identifies itself as an eval or names its configuration. `RUN-THIS.prompt.md` makes the user-selected agent the Eval Orchestrator: it reads the profile, resolves the selected runner, validates `describe`, preflights each `run.json`, reads `delegation.dispatch_owner`, and either dispatches one fresh orchestrator-owned native worker per arm or starts one fresh runner-owned native process/thread per arm. It preserves transport-owned raw evidence, uses `record-native-result.ps1` only for orchestrator-owned envelopes, bridges the existing result shape, reveals the grading key only after execution, grades, and invokes the package adapter. The adapter stages the results into Anthropic's upstream benchmark workspace, runs `aggregate_benchmark.py`, writes the exact upstream `skill-creator-report.html`, and writes a first-party `report.html` with paired outputs, expected outcomes, assertion evidence, telemetry, transcripts, and downloadable feedback, plus `benchmark.json` and `benchmark.md`. Missing telemetry is displayed as unavailable rather than estimated. A runner that cannot satisfy the mandatory experimental controls returns `incompatible`; lack of hard filesystem confinement downgrades the result to pragmatic isolation. `-CollectResults ` remains a deterministic fallback for transferred artifacts without reports; it may inspect and report available package state, but exits non-zero when the required completion gate is not satisfied, so an incomplete or unrun package is not a successfully completed evaluation. Packages land in gitignored `.bot/` storage by default and are not committed. The four roles are intentionally separate: the Eval Runner is the harness-specific executor for one blind arm; the Eval Orchestrator coordinates the external handoff; the Grader assesses results only after execution; and the Human Reviewer remains the final evaluator. Behavioral evaluation is portable across supported runners. Native activation is harness-specific and out of scope for v0.9.1. Anthropic is the methodology and compatibility reference, not a required Claude runtime. diff --git a/scripts/eval-runners/README.md b/scripts/eval-runners/README.md index aae32ef..b395321 100644 --- a/scripts/eval-runners/README.md +++ b/scripts/eval-runners/README.md @@ -15,10 +15,12 @@ Eval Orchestrator The Eval Orchestrator coordinates the manifest queue, native child creation, terminal evidence, exact manifest destinations, and the phase boundary. It -does not execute an eval arm itself. One arm equals one fresh delegated -harness-native worker and one model-backed execution. The delegated worker -executes the prepared prompt directly; it must not call the runner's -model-spawning `execute` command or start another model session. +does not execute an eval arm in its own model context. Each descriptor declares +`delegation.dispatch_owner`: `orchestrator` means the orchestrator creates the +declared native subagent/task, while `runner` means the orchestrator starts the +runner-owned native execution surface directly. One arm equals one fresh +native Eval Worker and one model-backed execution. A runner-owned process or +thread is that worker; it must not be nested inside an outer model session. `orchestration.ps1` is the deterministic queue/state helper copied into every package. It creates one worker envelope per exact manifest arm, keeps unrelated @@ -32,7 +34,9 @@ has no explicit capacity-limit evidence; `bridge-manifest-results.ps1 harness-specific concurrency ceiling. `Assert-NativeWorkerDelegation` is the fail-closed handoff gate: an unavailable/unsupported native mechanism cannot fall back to parent execution, while a conditional mechanism may dispatch only -when terminal evidence will be checked. +when terminal evidence will be checked. The dispatch owner is part of the +same descriptor/preflight contract, so generic orchestration does not infer it +from a runner name. The delegation contract has three distinct evidence levels: @@ -51,35 +55,32 @@ the requested model, exact arm identity, exact run working directory, exact isolated home/config boundary, prompt hash/fidelity, terminal capture, paired arm/grading exclusion, fresh worker/session identity, and exactly one model execution. Missing or mismatched evidence makes the arm `incompatible`; it is -never a reason to invoke the parent or the compatibility `runner.ps1 execute` -transport. - -The harness-native transport returns a terminal envelope with schema -`codebeltnet/agentic/eval-native-worker-result/1`. The envelope declares -`capture.source = harness_native_transport`, `capture.terminal = true`, and -`capture.worker_authored = false`; the model worker's answer is data inside the -envelope, never its author. If the selected harness exposes only assistant text -or a worker-authored summary, the arm is incompatible. After the worker is -terminal, the orchestrator writes only that captured envelope to a package-local -temporary path and invokes `record-native-result.ps1`. That deterministic -package-runner helper derives the exact run/profile identity, timestamps, -requested configuration, runner/harness identity, and `eval-execution-result/1` -shape, then validates the native evidence before writing the manifest-declared -raw result. The parent may persist only that helper-produced result; it must not -replace it with a worker summary, hand-write `execution-result.json`, or -synthesize a normalized result. -Native bridging also checks the result's runner identity, the descriptor's exact -delegation mechanism, and a hashed transcript/event artifact. An `incompatible` -arm is diagnostic-only: it is never gradeable and fails the completion/benchmark -gate. - -The descriptor's `delegation` object records the native mechanism, worker role, +never a reason to invoke the parent or a different transport. + +For `dispatch_owner=orchestrator`, the harness-native transport returns a +terminal envelope with schema `codebeltnet/agentic/eval-native-worker-result/1`. +The envelope declares `capture.source = harness_native_transport`, +`capture.terminal = true`, and `capture.worker_authored = false`; the model +worker's answer is data inside the envelope, never its author. The +orchestrator preserves that envelope and invokes `record-native-result.ps1`. +For `dispatch_owner=runner`, the runner's one-arm native execution surface +produces the canonical `execution-result.json` directly; the orchestrator does +not invoke the recorder, manufacture an envelope, or copy assistant text into +transport evidence. In both modes, transport-owned timestamps, identity, +isolation observations, prompt fidelity, terminal completion, and a hashed raw +transcript/event artifact are mandatory. A parent-created summary or repaired +result is incompatible. Native bridging also checks the result's runner +identity, the descriptor's exact delegation mechanism, and the hashed artifact. +An `incompatible` arm is diagnostic-only: it is never gradeable and fails the +completion/benchmark gate. + +The descriptor's `delegation` object records the dispatch owner, native mechanism, worker role, advertised full-capability/model-lock/working-directory/result-capture properties, harness-authoritative capacity, and the invariant -`nested_model_execution = false`. The direct `execute` process surface remains -for compatibility and deterministic conformance; the external orchestrator -must use only `describe`, `preflight`, and the harness-native delegation -surface for the actual eval arms. +`nested_model_execution = false`. The direct `execute` process surface is the +runner-owned native worker surface when `dispatch_owner=runner`; for +orchestrator-owned runners it remains a compatibility/conformance surface and +must not be invoked inside the native subagent. Native delegation mechanisms: @@ -88,11 +89,16 @@ Native delegation mechanisms: lifecycle is harness-owned; Codebelt supplies the already-known one-arm decomposition and requires terminal child evidence. The direct CLI `-C`/`--model`/HOME compatibility transport does not prove the child. -- Codex: the installed CLI's native app-server child-session surface, - `thread/start` followed by `turn/start`, with the arm's `cwd`, selected - model, and ephemeral/fresh session settings. The schema/feature probe is - preflight readiness only; terminal evidence must prove the actual child. - Do not wrap a native Codex child in another `codex exec` invocation. +- Codex: the installed CLI's runner-owned app-server child-session surface, + `thread/start` followed by `turn/start` and post-completion `thread/read`, + with the arm's `cwd`, selected model, and ephemeral/fresh session settings. + The schema/feature probe is preflight readiness only; terminal evidence must + prove the actual thread. Subscription auth uses a temporary auth-only + `CODEX_HOME` containing only `auth.json`; ambient config, skills, agents, + sessions, memories, plugins, MCP configuration, and AGENTS.md are not copied + and the temporary home is removed in `finally`. `model/rerouted` and + instruction sources outside the staged arm are incompatible. Do not wrap a + native Codex app-server worker in another Codex subagent. - OpenCode: the native Task tool with the full-capability built-in `General` subagent. Task/General availability is preflight readiness only; `Explore`/`Scout` read-only agents are not valid for a mutable eval arm. @@ -105,10 +111,11 @@ Runner protocol. It is copied into prepared packages so the external Eval Orchestrator can use the same runner implementation that was validated with the package. It is not a model executor used by repository automation. -The boundary has a native terminal envelope plus the runner-owned raw result: +The boundary is owner-dependent: ```text -run.json + execution-profile.json -> native worker envelope -> record-native-result.ps1 -> execution-result.json +dispatch_owner=orchestrator: run.json + execution-profile.json -> native worker envelope -> record-native-result.ps1 -> execution-result.json +dispatch_owner=runner: run.json + execution-profile.json -> runner-owned native execute -> execution-result.json ``` `run.json` is the existing portable one-arm contract. It owns the prompt, @@ -140,9 +147,10 @@ record-native-result.ps1 -Runner -Run -Profile --model --output-format json --allow-all-tools --no-ask-user --disable-builtin-mcps @@ -198,9 +209,10 @@ inject only that token as a protected environment variable. Host `GH_CONFIG_DIR` is never forwarded into the evaluated worker. `--secret-env-vars` removes every listed token variable from shell and MCP child environments. Preflight does not make a model request and therefore reports native keychain/service readiness as -conditional rather than claiming successful remote authentication. Codex uses -`--ask-for-approval never` with `exec --sandbox -workspace-write`; it does not combine explicit sandbox selection with +conditional rather than claiming successful remote authentication. Codex's +compatibility API-key path uses `--ask-for-approval never` with `exec --sandbox +workspace-write`; subscription eval arms use the runner-owned app-server path +described above. It does not combine explicit sandbox selection with `--approve-for-me`. OpenCode uses `run --format json --auto --model ` with isolated global/config roots and preserves repository-owned project configuration; it does not depend on From e22f3bdf886dae9be8a9ef2ddec388c8de1d5790 Mon Sep 17 00:00:00 2001 From: "aicia[bot]" Date: Tue, 25 Aug 2026 18:40:52 +0200 Subject: [PATCH 38/39] =?UTF-8?q?=F0=9F=94=A7=20implement=20eval=20runner?= =?UTF-8?q?=20dispatch=20owner=20pattern?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add dispatch_owner field to runner descriptors to enable two execution strategies: orchestrator-owned dispatch delegates through native subagent/task mechanisms, while runner-owned dispatch invokes the runner's native execution surface directly. Update Codex runner to implement app-server isolation with auth-only home, update orchestration and runner-common to handle both dispatch patterns, and expand conformance tests to validate isolation and concurrent execution boundaries. Runner-owned dispatch eliminates intermediary model context overhead and enables harnesses with native multi-worker capability to execute isolated eval arms in parallel. --- .../eval-runners/bridge-manifest-results.ps1 | 3 +- scripts/eval-runners/codex/runner.ps1 | 368 +++++++++++++++--- .../contracts/orchestration-plan.schema.json | 10 +- .../contracts/preflight-result.schema.json | 2 + .../contracts/runner-descriptor.schema.json | 2 + scripts/eval-runners/fake/runner.ps1 | 1 + .../eval-runners/github-copilot/runner.ps1 | 1 + scripts/eval-runners/opencode/runner.ps1 | 1 + scripts/eval-runners/orchestration.ps1 | 45 ++- scripts/eval-runners/runner-common.ps1 | 16 +- .../eval-runners/tests/test-orchestration.ps1 | 59 ++- .../tests/test-runner-conformance.ps1 | 123 +++++- 12 files changed, 548 insertions(+), 83 deletions(-) diff --git a/scripts/eval-runners/bridge-manifest-results.ps1 b/scripts/eval-runners/bridge-manifest-results.ps1 index 6f4deb8..b084b2c 100644 --- a/scripts/eval-runners/bridge-manifest-results.ps1 +++ b/scripts/eval-runners/bridge-manifest-results.ps1 @@ -46,7 +46,8 @@ try { } $profileData = Resolve-ExecutionProfile -ProfilePath (Join-Path $iterationPath 'execution-profile.json') - $plan = New-EvalOrchestrationPlan -IterationDirectory $iterationPath -Manifest $manifest -Profile $profileData.Profile + $runnerDescriptor = Get-PackageRunnerDescriptor -RunnerName ([string]$profileData.Runner) + $plan = New-EvalOrchestrationPlan -IterationDirectory $iterationPath -Manifest $manifest -Profile $profileData.Profile -Descriptor $runnerDescriptor $state = Read-RunnerJson -Path $statePath $parallelDispatch = Assert-OrchestrationConcurrency -Plan $plan -State $state } diff --git a/scripts/eval-runners/codex/runner.ps1 b/scripts/eval-runners/codex/runner.ps1 index 465eddc..84fdb6e 100644 --- a/scripts/eval-runners/codex/runner.ps1 +++ b/scripts/eval-runners/codex/runner.ps1 @@ -57,6 +57,7 @@ $descriptor = [ordered]@{ delegated_worker_capacity_signal = 'conditional' } delegation = [ordered]@{ + dispatch_owner = 'runner' mode = 'native_worker' mechanism = 'Codex app-server native child session via thread/start and turn/start with per-worker cwd, model, and ephemeral context' worker_role = 'native-codex-child-session' @@ -125,6 +126,37 @@ function Invoke-CodexCli { return Invoke-RunnerProcess -FileName $CommandInfo.FileName -ArgumentList $allArguments -WorkingDirectory $Inputs.Run.WorkingDirectoryPath -Environment $Environment -InputBytes $InputBytes -TimeoutSeconds $TimeoutSeconds } +function New-CodexAuthOnlyHome { + param([Parameter(Mandatory = $true)][object]$Auth) + + if ($Auth.Kind -ne 'subscription_file' -or [string]::IsNullOrWhiteSpace([string]$Auth.Path)) { + throw 'Codex auth-only home requires a resolved subscription auth.json source.' + } + $homePath = Join-Path ([System.IO.Path]::GetTempPath()) ('agentic-codex-auth-' + [Guid]::NewGuid().ToString('N')) + New-Item -ItemType Directory -Path $homePath -Force | Out-Null + $authDestination = Join-Path $homePath 'auth.json' + try { + # The temporary home is intentionally created outside the prepared + # package. It contains exactly one copied file and is removed in the + # app-server finally block, including start/timeout failures. + Copy-Item -LiteralPath $Auth.Path -Destination $authDestination -Force -ErrorAction Stop + $entries = @(Get-ChildItem -LiteralPath $homePath -Force -ErrorAction Stop) + if ($entries.Count -ne 1 -or [string]$entries[0].Name -ne 'auth.json' -or -not (Test-Path -LiteralPath $authDestination -PathType Leaf)) { + throw 'Codex temporary subscription home was not auth-only.' + } + return [pscustomobject]@{ + Path = $homePath + AuthPath = $authDestination + AuthOnly = $true + } + } catch { + if (Test-Path -LiteralPath $homePath) { + Remove-Item -LiteralPath $homePath -Recurse -Force -ErrorAction SilentlyContinue + } + throw + } +} + function Invoke-CodexAppServer { param( [Parameter(Mandatory = $true)][object]$CommandInfo, @@ -148,29 +180,45 @@ function Invoke-CodexAppServer { $psi.RedirectStandardOutput = $true $psi.RedirectStandardError = $true foreach ($argument in @($CommandInfo.Prefix) + @('app-server', '--stdio', '-c', 'shell_environment_policy.inherit=none')) { [void]$psi.ArgumentList.Add([string]$argument) } - $parentEnvironment = New-RunnerEnvironment -Run $Inputs.Run -Additional @{ CODEX_HOME = (Split-Path -Parent $Auth.Path) } - $psi.Environment.Clear() - foreach ($name in @($parentEnvironment.Keys)) { $psi.Environment[$name] = [string]$parentEnvironment[$name] } + $authHome = $null + $authOnlyHomeRemoved = $false + $parentEnvironment = $null $process = [System.Diagnostics.Process]::new() - $process.StartInfo = $psi $writer = $null $reader = $null $stderrTask = $null $events = [System.Collections.Generic.List[string]]::new() $normalized = [System.Collections.Generic.List[string]]::new() $threadId = $null + $threadSessionId = $null $turnId = $null $finalText = $null $latestUsage = $null $timedOut = $false $transportFailure = $null + $threadReadFailure = $null $turnCompleted = $false + $terminalTurn = $null $stderr = '' $actualExitCode = $null $processStarted = $false + $threadStartRequest = $null + $threadStartResponse = $null + $turnStartRequest = $null + $turnStartResponse = $null + $threadReadResponse = $null + $instructionSources = @() + $instructionSourcesObserved = $false + $modelReroutes = [System.Collections.Generic.List[object]]::new() try { + $authHome = New-CodexAuthOnlyHome -Auth $Auth + $parentEnvironment = New-RunnerEnvironment -Run $Inputs.Run -Additional @{ CODEX_HOME = $authHome.Path } + $psi.Environment.Clear() + foreach ($name in @($parentEnvironment.Keys)) { $psi.Environment[$name] = [string]$parentEnvironment[$name] } + $process.StartInfo = $psi + if (-not $process.Start()) { throw 'Could not start Codex app-server.' } $processStarted = $true $writer = $process.StandardInput @@ -189,10 +237,16 @@ function Invoke-CodexAppServer { $waitMilliseconds = [int][Math]::Min([int]::MaxValue, [Math]::Ceiling($remaining.TotalMilliseconds)) if (-not $readTask.Wait($waitMilliseconds)) { throw [TimeoutException]::new('Codex app-server timed out.') } $line = $readTask.GetAwaiter().GetResult() - if ($null -eq $line) { throw [EndOfStreamException]::new('Codex app-server closed stdout before turn completion.') } + if ($null -eq $line) { throw [EndOfStreamException]::new('Codex app-server closed stdout before the expected response.') } $events.Add($line) try { return ($line | ConvertFrom-Json -Depth 50) } catch { throw [FormatException]::new("Codex app-server emitted malformed JSON: $($_.Exception.Message)") } } + $recordModelReroute = { + param([Parameter(Mandatory = $true)][object]$Message) + $reroute = Get-JsonProperty -Object $Message -Name 'params' -Default ([ordered]@{}) + $modelReroutes.Add($reroute) + $normalized.Add(([ordered]@{ type = 'model.rerouted'; from_model = Get-JsonProperty -Object $reroute -Name 'fromModel' -Default $null; to_model = Get-JsonProperty -Object $reroute -Name 'toModel' -Default $null; reason = Get-JsonProperty -Object $reroute -Name 'reason' -Default $null } | ConvertTo-Json -Compress)) + } $waitForResponse = { param([Parameter(Mandatory = $true)][int]$ExpectedId, [Parameter(Mandatory = $true)][string]$Operation) while ($true) { @@ -202,6 +256,10 @@ function Invoke-CodexAppServer { if (-not [string]::IsNullOrWhiteSpace($method) -and $null -ne $messageId) { throw "Codex app-server requested unsupported interactive method '$method'." } + if ($method -eq 'model/rerouted') { + & $recordModelReroute $message + continue + } if ($null -eq $messageId -or [int]$messageId -ne $ExpectedId) { continue } $error = Get-JsonProperty -Object $message -Name 'error' -Default $null if ($null -ne $error) { @@ -226,46 +284,47 @@ function Invoke-CodexAppServer { & $writeMessage ([ordered]@{ jsonrpc = '2.0'; method = 'initialized' }) $threadRequest = 2 - & $writeMessage ([ordered]@{ - jsonrpc = '2.0' - id = $threadRequest - method = 'thread/start' - params = [ordered]@{ - model = $Inputs.Profile.Model - cwd = $Inputs.Run.WorkingDirectoryPath - approvalPolicy = 'never' - # thread/start can persist project trust when it begins in a - # writable sandbox. Keep the ephemeral thread read-only and - # apply the intended workspace-write policy to the turn only. - sandbox = 'readOnly' - ephemeral = $true - } - }) - $threadResponse = & $waitForResponse $threadRequest 'thread/start' - $threadId = [string]$threadResponse.result.thread.id + $threadStartParams = [ordered]@{ + model = $Inputs.Profile.Model + cwd = $Inputs.Run.WorkingDirectoryPath + approvalPolicy = 'never' + # thread/start can persist project trust when it begins in a + # writable sandbox. Keep the ephemeral thread read-only and + # apply the intended workspace-write policy to the turn only. + sandbox = 'readOnly' + ephemeral = $true + } + $threadStartRequest = [ordered]@{ jsonrpc = '2.0'; id = $threadRequest; method = 'thread/start'; params = $threadStartParams } + & $writeMessage $threadStartRequest + $threadStartResponse = & $waitForResponse $threadRequest 'thread/start' + $threadStartResult = Get-JsonProperty -Object $threadStartResponse -Name 'result' -Default $null + $threadMetadata = Get-JsonProperty -Object $threadStartResult -Name 'thread' -Default $null + $threadId = [string](Get-JsonProperty -Object $threadMetadata -Name 'id' -Default '') + $threadSessionId = [string](Get-JsonProperty -Object $threadMetadata -Name 'sessionId' -Default '') if ([string]::IsNullOrWhiteSpace($threadId)) { throw 'Codex app-server thread/start returned no thread id.' } + $instructionSourcesObserved = Test-JsonProperty -Object $threadStartResult -Name 'instructionSources' + if ($instructionSourcesObserved) { $instructionSources = @(Get-JsonProperty -Object $threadStartResult -Name 'instructionSources' -Default @()) } $turnRequest = 3 - & $writeMessage ([ordered]@{ - jsonrpc = '2.0' - id = $turnRequest - method = 'turn/start' - params = [ordered]@{ - threadId = $threadId - input = @([ordered]@{ type = 'text'; text = [System.Text.Encoding]::UTF8.GetString($Inputs.Run.PromptBytes) }) - cwd = $Inputs.Run.WorkingDirectoryPath - model = $Inputs.Profile.Model - effort = $Inputs.Profile.ReasoningEffort - approvalPolicy = 'never' - sandboxPolicy = [ordered]@{ - type = 'workspaceWrite' - writableRoots = @($Inputs.Run.WorkingDirectoryPath) - networkAccess = $true - } + $promptText = [System.Text.Encoding]::UTF8.GetString($Inputs.Run.PromptBytes) + $turnStartParams = [ordered]@{ + threadId = $threadId + input = @([ordered]@{ type = 'text'; text = $promptText }) + cwd = $Inputs.Run.WorkingDirectoryPath + model = $Inputs.Profile.Model + effort = $Inputs.Profile.ReasoningEffort + approvalPolicy = 'never' + sandboxPolicy = [ordered]@{ + type = 'workspaceWrite' + writableRoots = @($Inputs.Run.WorkingDirectoryPath) + networkAccess = $true } - }) - $turnResponse = & $waitForResponse $turnRequest 'turn/start' - $turnId = [string]$turnResponse.result.turn.id + } + $turnStartRequest = [ordered]@{ jsonrpc = '2.0'; id = $turnRequest; method = 'turn/start'; params = $turnStartParams } + & $writeMessage $turnStartRequest + $turnStartResponse = & $waitForResponse $turnRequest 'turn/start' + $turnStartResult = Get-JsonProperty -Object $turnStartResponse -Name 'result' -Default $null + $turnId = [string](Get-JsonProperty -Object (Get-JsonProperty -Object $turnStartResult -Name 'turn' -Default $null) -Name 'id' -Default '') if ([string]::IsNullOrWhiteSpace($turnId)) { throw 'Codex app-server turn/start returned no turn id.' } while (-not $turnCompleted) { @@ -277,7 +336,10 @@ function Invoke-CodexAppServer { } switch ($method) { 'thread/started' { - $normalized.Add(([ordered]@{ type = 'thread.started'; thread_id = $message.params.thread.id } | ConvertTo-Json -Compress)) + $normalized.Add(([ordered]@{ type = 'thread.started'; thread_id = Get-JsonProperty -Object (Get-JsonProperty -Object $message.params -Name 'thread' -Default $null) -Name 'id' -Default $null } | ConvertTo-Json -Compress)) + } + 'model/rerouted' { + & $recordModelReroute $message } 'item/completed' { $item = $message.params.item @@ -305,12 +367,18 @@ function Invoke-CodexAppServer { $normalized.Add(([ordered]@{ type = 'item.completed'; item = $normalizedItem } | ConvertTo-Json -Depth 40 -Compress)) } 'thread/tokenUsage/updated' { - $latestUsage = $message.params.tokenUsage.last + $latestUsage = Get-JsonProperty -Object (Get-JsonProperty -Object $message.params -Name 'tokenUsage' -Default $null) -Name 'last' -Default $null } 'turn/completed' { - $turn = $message.params.turn + $completionParams = Get-JsonProperty -Object $message -Name 'params' -Default ([ordered]@{}) + $terminalTurn = Get-JsonProperty -Object $completionParams -Name 'turn' -Default $null + $completionThreadId = [string](Get-JsonProperty -Object $completionParams -Name 'threadId' -Default '') + $completedTurnId = [string](Get-JsonProperty -Object $terminalTurn -Name 'id' -Default '') + if ($completionThreadId -ne $threadId -or $completedTurnId -ne $turnId) { + throw 'Codex app-server turn/completed identified an unexpected thread or turn.' + } if ([string]::IsNullOrWhiteSpace($finalText)) { - $turnItems = @($turn.items) + $turnItems = @($terminalTurn.items) for ($itemIndex = $turnItems.Count - 1; $itemIndex -ge 0; $itemIndex--) { if ([string]$turnItems[$itemIndex].type -eq 'agentMessage' -and -not [string]::IsNullOrWhiteSpace([string]$turnItems[$itemIndex].text)) { $finalText = [string]$turnItems[$itemIndex].text @@ -318,19 +386,19 @@ function Invoke-CodexAppServer { } } } - if ([string]$turn.status -eq 'failed') { - $errorMessage = [string](Get-JsonProperty -Object $turn.error -Name 'message' -Default 'Codex turn failed.') + if ([string]$terminalTurn.status -eq 'failed') { + $errorMessage = [string](Get-JsonProperty -Object $terminalTurn.error -Name 'message' -Default 'Codex turn failed.') $normalized.Add(([ordered]@{ type = 'turn.failed'; error = $errorMessage } | ConvertTo-Json -Compress)) - } elseif ([string]$turn.status -eq 'interrupted') { + } elseif ([string]$terminalTurn.status -eq 'interrupted') { $normalized.Add(([ordered]@{ type = 'turn.failed'; error = 'Codex turn was interrupted.' } | ConvertTo-Json -Compress)) } else { $usage = $null if ($null -ne $latestUsage) { $usage = [ordered]@{ - input_tokens = $latestUsage.inputTokens - cached_input_tokens = $latestUsage.cachedInputTokens - output_tokens = $latestUsage.outputTokens - reasoning_output_tokens = $latestUsage.reasoningOutputTokens + input_tokens = Get-JsonProperty -Object $latestUsage -Name 'inputTokens' -Default $null + cached_input_tokens = Get-JsonProperty -Object $latestUsage -Name 'cachedInputTokens' -Default $null + output_tokens = Get-JsonProperty -Object $latestUsage -Name 'outputTokens' -Default $null + reasoning_output_tokens = Get-JsonProperty -Object $latestUsage -Name 'reasoningOutputTokens' -Default $null } } $normalized.Add(([ordered]@{ type = 'turn.completed'; usage = $usage } | ConvertTo-Json -Depth 20 -Compress)) @@ -343,6 +411,18 @@ function Invoke-CodexAppServer { } } } + + # The installed schema exposes thread/read after completion. Use it as + # a second observation of ephemeral identity, cwd, and session metadata + # when the server provides the response; never reconstruct it locally. + $threadReadRequest = 4 + & $writeMessage ([ordered]@{ jsonrpc = '2.0'; id = $threadReadRequest; method = 'thread/read'; params = [ordered]@{ threadId = $threadId; includeTurns = $true } }) + try { + $threadReadResponse = & $waitForResponse $threadReadRequest 'thread/read' + } catch { + $threadReadFailure = $_.Exception.Message + $normalized.Add(([ordered]@{ type = 'thread.read.unavailable'; message = $threadReadFailure } | ConvertTo-Json -Compress)) + } } catch [TimeoutException] { $timedOut = $true } catch { @@ -363,6 +443,10 @@ function Invoke-CodexAppServer { try { $stderr = $stderrTask.GetAwaiter().GetResult() } catch { $stderr = $_.Exception.Message } } $process.Dispose() + if ($null -ne $authHome -and (Test-Path -LiteralPath $authHome.Path)) { + Remove-Item -LiteralPath $authHome.Path -Recurse -Force -ErrorAction SilentlyContinue + } + $authOnlyHomeRemoved = $null -eq $authHome -or -not (Test-Path -LiteralPath $authHome.Path) } $finish = [DateTime]::UtcNow @@ -378,7 +462,27 @@ function Invoke-CodexAppServer { DurationSeconds = [Math]::Round(($finish - $start).TotalSeconds, 3) FinalText = $finalText ThreadId = $threadId + ThreadSessionId = $threadSessionId TurnId = $turnId + TurnCompleted = $turnCompleted + TerminalTurn = $terminalTurn + ThreadStartRequest = $threadStartRequest + ThreadStartResponse = $threadStartResponse + TurnStartRequest = $turnStartRequest + TurnStartResponse = $turnStartResponse + ThreadReadResponse = $threadReadResponse + ThreadReadFailure = $threadReadFailure + InstructionSources = @($instructionSources) + InstructionSourcesObserved = $instructionSourcesObserved + ModelReroutes = @($modelReroutes.ToArray()) + PromptInputSha256 = if ($null -ne $turnStartRequest) { Get-Sha256HexFromBytes -Bytes ([System.Text.Encoding]::UTF8.GetBytes([string]$turnStartRequest.params.input[0].text)) } else { $null } + ObservedModel = if ($null -ne $threadStartResponse) { [string](Get-JsonProperty -Object (Get-JsonProperty -Object $threadStartResponse -Name 'result' -Default $null) -Name 'model' -Default '') } else { '' } + ObservedWorkingDirectory = if ($null -ne $threadStartResponse) { [string](Get-JsonProperty -Object (Get-JsonProperty -Object $threadStartResponse -Name 'result' -Default $null) -Name 'cwd' -Default '') } else { '' } + ObservedEphemeral = if ($null -ne $threadStartResponse) { [bool](Get-JsonProperty -Object (Get-JsonProperty -Object (Get-JsonProperty -Object $threadStartResponse -Name 'result' -Default $null) -Name 'thread' -Default $null) -Name 'ephemeral' -Default $false) } else { $false } + AuthOnlyHome = $null -ne $authHome -and [bool]$authHome.AuthOnly + AuthOnlyHomeRemoved = $authOnlyHomeRemoved + WorkerHome = if ($null -ne $parentEnvironment) { [string]$parentEnvironment.HOME } else { '' } + TransportFailure = $transportFailure } } @@ -422,14 +526,14 @@ function Get-CodexNativeWorkerProbe { } $schemaFiles = @(Get-ChildItem -LiteralPath $schemaDirectory -Recurse -File -Filter '*.json' -ErrorAction SilentlyContinue) $schemaText = [string]::Join("`n", @($schemaFiles | ForEach-Object { [System.IO.File]::ReadAllText($_.FullName, [System.Text.UTF8Encoding]::new($false)) })) - foreach ($needle in @('thread/start', 'turn/start', 'ThreadStartParams', 'TurnStartParams', '"cwd"', '"model"', '"ephemeral"')) { + foreach ($needle in @('thread/start', 'turn/start', 'thread/read', 'model/rerouted', 'ThreadStartParams', 'TurnStartParams', 'ThreadReadParams', 'ThreadStartResponse', '"instructionSources"', '"cwd"', '"model"', '"ephemeral"')) { if ($schemaText -notmatch [regex]::Escape($needle)) { return [pscustomobject]@{ Available = $false; Detail = "Codex app-server schema did not prove native child-session field '$needle'." } } } return [pscustomobject]@{ Available = $true - Detail = 'Codex multi_agent is stable and app-server schema proves thread/start and turn/start with cwd, model, and ephemeral child-session controls.' + Detail = 'Codex multi_agent is stable and the installed app-server schema proves thread/start, turn/start, thread/read, model/rerouted, instructionSources, cwd, model, and ephemeral child-session controls. No separate model-fallback-disable field is exposed; reroute notifications are fail-closed.' } } @@ -589,13 +693,13 @@ function Get-CodexPreflight { if ($auth.Kind -eq 'missing') { $reasons.Add('Neither a narrow Codex provider API-key environment variable nor subscription auth.json is available.') } elseif ($auth.Kind -eq 'subscription_file') { - $checks.Add((New-PreflightCheck -Name 'authentication' -Status passed -Detail 'Codex app-server can read the existing subscription auth.json through its parent-only CODEX_HOME. The file is not copied into the staged worker home.')) + $checks.Add((New-PreflightCheck -Name 'authentication' -Status passed -Detail 'Codex app-server uses a fresh temporary auth-only CODEX_HOME containing only a copied auth.json; the source home and all ambient Codex configuration remain outside the worker.')) } else { $checks.Add((New-PreflightCheck -Name 'authentication' -Status passed -Detail "Authentication is available through the narrow $($auth.Name) environment variable; the child shell policy is set to inherit=none.")) } if ($auth.Kind -eq 'subscription_file') { - $checks.Add((New-PreflightCheck -Name 'filesystem_confinement' -Status unavailable -Detail 'The subscription app-server parent must read the existing Codex credential home and is not wrapped by the external run-only sandbox. Codex workspace-write remains enabled for the turn.')) + $checks.Add((New-PreflightCheck -Name 'filesystem_confinement' -Status unavailable -Detail 'The subscription app-server transport uses a temporary auth-only home but is not wrapped by the external run-only sandbox. Codex workspace-write remains enabled for the turn.')) $warnings.Add('Subscription execution uses pragmatic isolation. The adapter does not claim that an external filesystem sandbox protects the app-server transport.') } elseif ($null -eq $sandboxName) { $checks.Add((New-PreflightCheck -Name 'filesystem_confinement' -Status not_applicable -Detail "Platform '$platform' has no configured external hard-confinement mechanism; pragmatic isolation remains available.")) @@ -609,9 +713,9 @@ function Get-CodexPreflight { $checks.Add((New-PreflightCheck -Name 'fresh_session' -Status passed -Detail 'The selected transport starts an ephemeral thread and never supplies a resume, continue, or existing session identifier.')) if ($auth.Kind -eq 'subscription_file') { - $checks.Add((New-PreflightCheck -Name 'ambient_configuration' -Status passed -Detail 'The app-server parent receives a filtered environment plus the existing CODEX_HOME needed for subscription auth. Child shell inheritance is disabled with shell_environment_policy.inherit=none.')) + $checks.Add((New-PreflightCheck -Name 'ambient_configuration' -Status passed -Detail 'The app-server parent receives a filtered environment plus a temporary auth-only CODEX_HOME. Child shell inheritance is disabled with shell_environment_policy.inherit=none, and the runner validates instructionSources against the staged arm root.')) $checks.Add((New-PreflightCheck -Name 'run_paths' -Status passed -Detail "thread/start and turn/start set cwd to $($run.WorkingDirectoryPath); HOME and USERPROFILE remain staged under $($run.HomeDirectoryPath).")) - $checks.Add((New-PreflightCheck -Name 'credential_boundary' -Status passed -Detail 'The adapter does not copy auth.json into the run or deliberately forward CODEX_HOME to child shell tools. This does not claim hard filesystem confinement where none is available.')) + $checks.Add((New-PreflightCheck -Name 'credential_boundary' -Status passed -Detail 'Only auth.json is copied into a temporary auth-only CODEX_HOME and it is removed in finally; config.toml, skills, agents, sessions, memories, plugins, MCP configuration, and AGENTS.md are not copied. This does not claim hard filesystem confinement where none is available.')) } else { $checks.Add((New-PreflightCheck -Name 'ambient_configuration' -Status passed -Detail 'The compatibility transport uses an isolated CODEX_HOME plus --ignore-user-config and --ignore-rules; unrelated inherited environment variables are removed.')) $checks.Add((New-PreflightCheck -Name 'run_paths' -Status passed -Detail "--cd $($run.WorkingDirectoryPath); CODEX_HOME under $($run.HomeDirectoryPath)")) @@ -626,7 +730,7 @@ function Get-CodexPreflight { $descriptorCopy.harness = [ordered]@{ name = 'OpenAI Codex CLI'; version = $harnessVersion } $mechanisms = [System.Collections.Generic.List[string]]::new() if ($auth.Kind -eq 'subscription_file') { - foreach ($mechanism in @('native app-server initialize + thread/start + turn/start', 'parent-only subscription CODEX_HOME', 'ephemeral thread', 'approvalPolicy=never', 'sandboxPolicy=workspaceWrite', 'shell_environment_policy.inherit=none', 'filtered parent process environment', 'prompt in turn/start input', 'no session continuation')) { $mechanisms.Add($mechanism) } + foreach ($mechanism in @('native app-server initialize + thread/start + turn/start', 'temporary auth-only subscription CODEX_HOME', 'ephemeral thread', 'thread/read after turn completion', 'instructionSources validation', 'model/rerouted fail-closed', 'approvalPolicy=never', 'sandboxPolicy=workspaceWrite', 'shell_environment_policy.inherit=none', 'filtered parent process environment', 'prompt in turn/start input', 'no session continuation')) { $mechanisms.Add($mechanism) } } else { foreach ($mechanism in @('--ask-for-approval never', 'codex exec --ephemeral compatibility transport', '--ignore-user-config', '--ignore-rules', '--sandbox workspace-write', 'shell_environment_policy.inherit=none', 'isolated CODEX_HOME', 'prompt on stdin', 'no session continuation')) { $mechanisms.Add($mechanism) } } @@ -846,6 +950,50 @@ function Invoke-CodexExecute { $finalText = [System.IO.File]::ReadAllText((Join-Path $Inputs.Run.RunRoot ($lastResponsePath -replace '/', [System.IO.Path]::DirectorySeparatorChar)), [System.Text.UTF8Encoding]::new($false)) } } + $terminalCaptureComplete = if ($auth.Kind -eq 'subscription_file') { + [bool]$process.TurnCompleted + } else { + @($parsed.Events | Where-Object { [string](Get-JsonProperty -Object $_ -Name 'type' -Default '') -eq 'turn.completed' }).Count -gt 0 + } + + $nativeEvidenceFailures = [System.Collections.Generic.List[string]]::new() + $observedModel = if ($auth.Kind -eq 'subscription_file') { [string]$process.ObservedModel } else { '' } + $observedWorkingDirectory = if ($auth.Kind -eq 'subscription_file') { [string]$process.ObservedWorkingDirectory } else { '' } + $promptFidelity = $auth.Kind -eq 'subscription_file' -and [string]$process.PromptInputSha256 -eq [string]$Inputs.Run.PromptHash + $unexpectedInstructionSources = [System.Collections.Generic.List[string]]::new() + $invalidInstructionSources = [System.Collections.Generic.List[string]]::new() + if ($auth.Kind -eq 'subscription_file') { + if (-not [bool]$process.InstructionSourcesObserved) { $nativeEvidenceFailures.Add('instruction_sources_unobserved') } + foreach ($source in @($process.InstructionSources)) { + $sourcePath = [string]$source + if ([string]::IsNullOrWhiteSpace($sourcePath)) { + $invalidInstructionSources.Add($sourcePath) + } elseif (-not (Test-PathInside -BasePath $Inputs.Run.RunRoot -CandidatePath $sourcePath)) { + $unexpectedInstructionSources.Add($sourcePath) + } + } + if ($invalidInstructionSources.Count -gt 0) { $nativeEvidenceFailures.Add('invalid_instruction_sources') } + if ($unexpectedInstructionSources.Count -gt 0) { $nativeEvidenceFailures.Add('unexpected_instruction_sources') } + if ([string]::IsNullOrWhiteSpace($observedModel) -or $observedModel -ne [string]$Inputs.Profile.Model) { $nativeEvidenceFailures.Add('observed_model') } + if (-not (Test-ExactObservedPath -Expected ([string]$Inputs.Run.WorkingDirectoryPath) -Observed $observedWorkingDirectory)) { $nativeEvidenceFailures.Add('observed_working_directory') } + if (-not [bool]$process.ObservedEphemeral -or [string]::IsNullOrWhiteSpace([string]$process.ThreadId)) { $nativeEvidenceFailures.Add('fresh_worker') } + if (-not [bool]$process.AuthOnlyHome -or -not [bool]$process.AuthOnlyHomeRemoved) { $nativeEvidenceFailures.Add('isolated_auth_home') } + if (-not $promptFidelity) { $nativeEvidenceFailures.Add('prompt_fidelity') } + if (-not [bool]$process.TurnCompleted -or [string]::IsNullOrWhiteSpace([string]$process.RawStdout)) { $nativeEvidenceFailures.Add('terminal_result_capture') } + if ([string](Get-JsonProperty -Object $process.TerminalTurn -Name 'status' -Default '') -ne 'completed') { $nativeEvidenceFailures.Add('terminal_turn_status') } + if (@($process.ModelReroutes).Count -gt 0) { $nativeEvidenceFailures.Add('model_rerouted') } + $threadReadThread = if ($null -ne $process.ThreadReadResponse) { Get-JsonProperty -Object (Get-JsonProperty -Object $process.ThreadReadResponse -Name 'result' -Default $null) -Name 'thread' -Default $null } else { $null } + if ($null -ne $threadReadThread) { + if ([string](Get-JsonProperty -Object $threadReadThread -Name 'id' -Default '') -ne [string]$process.ThreadId -or + -not [bool](Get-JsonProperty -Object $threadReadThread -Name 'ephemeral' -Default $false) -or + -not (Test-ExactObservedPath -Expected ([string]$Inputs.Run.WorkingDirectoryPath) -Observed ([string](Get-JsonProperty -Object $threadReadThread -Name 'cwd' -Default '')))) { + $nativeEvidenceFailures.Add('thread_read_metadata') + } + } + } + $uniqueNativeEvidenceFailures = @($nativeEvidenceFailures | Select-Object -Unique) + $nativeEvidenceFailures = [System.Collections.Generic.List[string]]::new() + foreach ($failureName in $uniqueNativeEvidenceFailures) { $nativeEvidenceFailures.Add([string]$failureName) } $status = 'completed' $reason = $null @@ -864,6 +1012,12 @@ function Invoke-CodexExecute { $warnings.Add('Codex exited successfully without a final agent message.') $reason = 'codex_did_not_return_final_response' } + if ($auth.Kind -eq 'subscription_file' -and $nativeEvidenceFailures.Count -gt 0) { + $status = 'incompatible' + $reason = 'codex_native_evidence_incompatible' + $failure = New-ExecutionFailure -Code 'native_evidence_incompatible' -Message ("Codex app-server evidence failed closed: {0}." -f ([string]::Join(', ', @($nativeEvidenceFailures)))) + $exitStatus = $null + } $tokenMetric = if ($null -eq $usage) { New-UnavailableMetric -Reason 'codex_did_not_expose_turn_usage' @@ -876,7 +1030,7 @@ function Invoke-CodexExecute { if ($usageValue.Count -eq 0) { New-UnavailableMetric -Reason 'codex_usage_event_had_no_supported_buckets' } else { New-AvailableMetric -Value $usageValue } } $telemetry = [ordered]@{ - transcript = New-AvailableMetric -Value ([ordered]@{ artifact = $transcriptArtifactPath; complete = $true }) + transcript = New-AvailableMetric -Value ([ordered]@{ artifact = $transcriptArtifactPath; complete = $terminalCaptureComplete }) tokens = $tokenMetric tool_calls = New-AvailableMetric -Value $toolCalls cost = New-UnavailableMetric -Reason 'codex_runner_does_not_estimate_cost' @@ -886,7 +1040,7 @@ function Invoke-CodexExecute { $capabilities = Get-CodexCapabilityMap -Inputs $Inputs -HardFilesystemConfinement $hardFilesystem $mechanisms = [System.Collections.Generic.List[string]]::new() if ($auth.Kind -eq 'subscription_file') { - foreach ($mechanism in @('native app-server initialize + thread/start + turn/start', 'parent-only subscription CODEX_HOME', 'ephemeral thread', 'approvalPolicy=never', 'sandboxPolicy=workspaceWrite', 'shell_environment_policy.inherit=none', 'filtered parent process environment', 'prompt in turn/start input', 'no session continuation')) { $mechanisms.Add($mechanism) } + foreach ($mechanism in @('native app-server initialize + thread/start + turn/start', 'temporary auth-only subscription CODEX_HOME', 'ephemeral thread', 'thread/read after turn completion', 'instructionSources validation', 'model/rerouted fail-closed', 'approvalPolicy=never', 'sandboxPolicy=workspaceWrite', 'shell_environment_policy.inherit=none', 'filtered parent process environment', 'prompt in turn/start input', 'no session continuation')) { $mechanisms.Add($mechanism) } } else { foreach ($mechanism in @('--ask-for-approval never', 'codex exec --ephemeral', '--ignore-user-config', '--ignore-rules', '--sandbox workspace-write', 'shell_environment_policy.inherit=none', 'isolated CODEX_HOME', 'prompt on stdin', 'no session continuation')) { $mechanisms.Add($mechanism) } } @@ -899,9 +1053,99 @@ function Invoke-CodexExecute { unrelated_environment_excluded = $true child_tool_visibility = 'codex_shell_environment_policy_inherit_none' value_observed = $false + auth_only_home = if ($auth.Kind -eq 'subscription_file') { [bool]$process.AuthOnlyHome } else { $false } + auth_only_home_removed = if ($auth.Kind -eq 'subscription_file') { [bool]$process.AuthOnlyHomeRemoved } else { $true } + ambient_codex_configuration_copied = $false } $outputLastMessageArgument = if ($auth.Kind -eq 'subscription_file') { $null } else { Get-SandboxVisiblePath -HostPath (Join-Path $Inputs.Run.RunRoot ($lastResponsePath -replace '/', [System.IO.Path]::DirectorySeparatorChar)) -RunRoot $Inputs.Run.RunRoot -Platform $visiblePlatform } - return New-ExecutionResult -Descriptor $executionDescriptor -Profile $Inputs.Profile -Run $Inputs.Run -Status $status -FinalResponse $finalText -FinalResponseReason $reason -StartedUtc $process.StartedUtc.ToString('o') -FinishedUtc $finished.ToString('o') -DurationSeconds $process.DurationSeconds -ExitStatus $exitStatus -Failure $failure -SessionId $sessionResultId -IsolationCapabilities $capabilities -IsolationMechanisms @($mechanisms) -ResolvedConfiguration ([ordered]@{ status = 'accepted_request'; reason = 'Codex accepted the requested model and configuration but did not expose concrete backend resolution.'; observations = [ordered]@{ model = $Inputs.Profile.Model; reasoning_effort = $Inputs.Profile.ReasoningEffort } }) -Telemetry $telemetry -Artifacts @($artifacts) -Warnings @($warnings) -Evidence ([ordered]@{ thread_id = $threadId; turn_id = $turnId; event_counts = $eventCounts; commands = @($commands); files = @($files); prompt_first_input = $true; resume = $false; stdout_exit_code = $process.ExitCode; sandbox = $sandboxEvidence; output_last_message_argument = $outputLastMessageArgument; credential = $credentialEvidence }) -AttemptCount 1 + $evidence = [ordered]@{ + thread_id = $threadId + thread_session_id = if ($auth.Kind -eq 'subscription_file') { $process.ThreadSessionId } else { $null } + turn_id = $turnId + event_counts = $eventCounts + commands = @($commands) + files = @($files) + prompt_first_input = if ($auth.Kind -eq 'subscription_file') { $promptFidelity } else { $true } + resume = $false + stdout_exit_code = $process.ExitCode + sandbox = $sandboxEvidence + output_last_message_argument = $outputLastMessageArgument + credential = $credentialEvidence + } + if ($auth.Kind -eq 'subscription_file') { + $rawArtifact = @($artifacts | Where-Object { [string]$_.path -eq $transcriptArtifactPath } | Select-Object -First 1) + $evidence.capture = [ordered]@{ + source = 'harness_native_transport' + terminal = [bool]$process.TurnCompleted + worker_authored = $false + artifact = $transcriptArtifactPath + sha256 = if ($rawArtifact.Count -eq 1) { [string]$rawArtifact[0].sha256 } else { $null } + } + $evidence.delegation = [ordered]@{ + dispatch_owner = 'runner' + mechanism = [string]$descriptor.delegation.mechanism + worker_session_id = $sessionResultId + observed_model = $observedModel + observed_working_directory = $observedWorkingDirectory + observed_home = [string]$process.WorkerHome + fresh_worker = [bool]$process.ObservedEphemeral -and -not [string]::IsNullOrWhiteSpace([string]$process.ThreadId) + home_config_isolated = [bool]$process.AuthOnlyHome -and [bool]$process.AuthOnlyHomeRemoved + prompt_fidelity = $promptFidelity + prompt_sha256 = $Inputs.Run.PromptHash + terminal_result_capture = [bool]$process.TurnCompleted -and -not [string]::IsNullOrWhiteSpace([string]$process.RawStdout) + paired_arm_visible = $false + grading_material_visible = $false + nested_model_execution = $false + model_execution_count = 1 + thread_id = $threadId + thread_session_id = $process.ThreadSessionId + turn_id = $turnId + instruction_sources_observed = [bool]$process.InstructionSourcesObserved + instruction_sources = @($process.InstructionSources) + invalid_instruction_sources = @($invalidInstructionSources.ToArray()) + unexpected_instruction_sources = @($unexpectedInstructionSources.ToArray()) + requested_runtime_workspace_roots = @($Inputs.Run.WorkingDirectoryPath) + thread_read_observed = $null -ne $process.ThreadReadResponse + model_reroutes = @($process.ModelReroutes) + } + $threadStartResultEvidence = Get-JsonProperty -Object $process.ThreadStartResponse -Name 'result' -Default ([ordered]@{}) + $threadReadThreadEvidence = if ($null -ne $process.ThreadReadResponse) { Get-JsonProperty -Object (Get-JsonProperty -Object $process.ThreadReadResponse -Name 'result' -Default $null) -Name 'thread' -Default $null } else { $null } + $turnCompletionEvidence = if ($null -ne $process.TerminalTurn) { [ordered]@{ thread_id = $process.ThreadId; turn_id = $process.TurnId; status = Get-JsonProperty -Object $process.TerminalTurn -Name 'status' -Default $null } } else { $null } + $evidence.app_server = [ordered]@{ + thread_start_request = $process.ThreadStartRequest + thread_start_response = $process.ThreadStartResponse + turn_start_request = $process.TurnStartRequest + turn_start_response = $process.TurnStartResponse + thread_start = [ordered]@{ + requested_model = $Inputs.Profile.Model + requested_cwd = $Inputs.Run.WorkingDirectoryPath + requested_ephemeral = $true + requested_sandbox = 'readOnly' + observed_model = Get-JsonProperty -Object $threadStartResultEvidence -Name 'model' -Default $null + observed_cwd = Get-JsonProperty -Object $threadStartResultEvidence -Name 'cwd' -Default $null + observed_ephemeral = Get-JsonProperty -Object (Get-JsonProperty -Object $threadStartResultEvidence -Name 'thread' -Default $null) -Name 'ephemeral' -Default $null + observed_sandbox = Get-JsonProperty -Object $threadStartResultEvidence -Name 'sandbox' -Default $null + instruction_sources = @($process.InstructionSources) + } + turn_start = [ordered]@{ + thread_id = $process.ThreadId + requested_model = $Inputs.Profile.Model + requested_cwd = $Inputs.Run.WorkingDirectoryPath + requested_effort = $Inputs.Profile.ReasoningEffort + requested_sandbox_policy = Get-JsonProperty -Object (Get-JsonProperty -Object $process.TurnStartRequest -Name 'params' -Default $null) -Name 'sandboxPolicy' -Default $null + prompt_sha256 = $process.PromptInputSha256 + } + terminal_turn = $turnCompletionEvidence + thread_read = [ordered]@{ + request = [ordered]@{ threadId = $process.ThreadId; includeTurns = $true } + response = if ($null -eq $threadReadThreadEvidence) { $null } else { [ordered]@{ id = Get-JsonProperty -Object $threadReadThreadEvidence -Name 'id' -Default $null; session_id = Get-JsonProperty -Object $threadReadThreadEvidence -Name 'sessionId' -Default $null; cwd = Get-JsonProperty -Object $threadReadThreadEvidence -Name 'cwd' -Default $null; ephemeral = Get-JsonProperty -Object $threadReadThreadEvidence -Name 'ephemeral' -Default $null } } + failure = $process.ThreadReadFailure + } + model_rerouted = @($process.ModelReroutes) + } + if ($nativeEvidenceFailures.Count -gt 0) { $evidence.native_worker_evidence_failures = @($nativeEvidenceFailures.ToArray()) } + } + return New-ExecutionResult -Descriptor $executionDescriptor -Profile $Inputs.Profile -Run $Inputs.Run -Status $status -FinalResponse $finalText -FinalResponseReason $reason -StartedUtc $process.StartedUtc.ToString('o') -FinishedUtc $finished.ToString('o') -DurationSeconds $process.DurationSeconds -ExitStatus $exitStatus -Failure $failure -SessionId $sessionResultId -IsolationCapabilities $capabilities -IsolationMechanisms @($mechanisms) -ResolvedConfiguration ([ordered]@{ status = 'accepted_request'; reason = 'Codex accepted the requested model and configuration but did not expose concrete backend resolution.'; observations = [ordered]@{ model = $Inputs.Profile.Model; reasoning_effort = $Inputs.Profile.ReasoningEffort } }) -Telemetry $telemetry -Artifacts @($artifacts) -Warnings @($warnings) -Evidence $evidence -AttemptCount 1 } try { diff --git a/scripts/eval-runners/contracts/orchestration-plan.schema.json b/scripts/eval-runners/contracts/orchestration-plan.schema.json index 4004ee7..8d028d4 100644 --- a/scripts/eval-runners/contracts/orchestration-plan.schema.json +++ b/scripts/eval-runners/contracts/orchestration-plan.schema.json @@ -9,6 +9,7 @@ "protocol_version", "runner", "model", + "dispatch_owner", "requested_concurrency", "parallel_dispatch_required", "minimum_parallel_workers", @@ -22,6 +23,7 @@ "protocol_version": { "const": "codebeltnet/agentic/eval-runner-protocol/1" }, "runner": { "type": "string", "minLength": 1 }, "model": { "type": "string", "minLength": 1 }, + "dispatch_owner": { "enum": ["orchestrator", "runner"] }, "requested_concurrency": { "type": "integer", "minimum": 1 }, "parallel_dispatch_required": { "type": "boolean" }, "minimum_parallel_workers": { "type": "integer", "minimum": 1 }, @@ -32,12 +34,13 @@ "type": "array", "items": { "type": "object", - "required": ["worker_id", "eval_id", "eval_name", "configuration", "depends_on", "parent_paths", "worker"], + "required": ["worker_id", "eval_id", "eval_name", "configuration", "depends_on", "parent_paths", "dispatch_owner", "worker"], "properties": { "worker_id": { "type": "string", "minLength": 1 }, "eval_id": { "type": "integer", "minimum": 1 }, "eval_name": { "type": "string", "minLength": 1 }, "configuration": { "enum": ["with_skill", "without_skill"] }, + "dispatch_owner": { "enum": ["orchestrator", "runner"] }, "depends_on": { "type": "array", "maxItems": 0 }, "parent_paths": { "type": "object", @@ -53,7 +56,7 @@ "required": [ "worker_id", "eval_id", "eval_name", "configuration", "run_manifest", "run_manifest_path", "model", "one_arm_only", "paired_arm_visible", "grading_material_visible", "parent_executes_arm", - "runner_execute_invocation", "nested_model_execution", "model_execution_count" + "dispatch_owner", "runner_execute_invocation", "nested_model_execution", "model_execution_count" ], "properties": { "worker_id": { "type": "string", "minLength": 1 }, @@ -67,7 +70,8 @@ "paired_arm_visible": { "const": false }, "grading_material_visible": { "const": false }, "parent_executes_arm": { "const": false }, - "runner_execute_invocation": { "const": "forbidden" }, + "dispatch_owner": { "enum": ["orchestrator", "runner"] }, + "runner_execute_invocation": { "enum": ["forbidden", "required"] }, "nested_model_execution": { "const": false }, "model_execution_count": { "const": 1 } } diff --git a/scripts/eval-runners/contracts/preflight-result.schema.json b/scripts/eval-runners/contracts/preflight-result.schema.json index c4ae2ae..c95636b 100644 --- a/scripts/eval-runners/contracts/preflight-result.schema.json +++ b/scripts/eval-runners/contracts/preflight-result.schema.json @@ -48,6 +48,7 @@ "type": "object", "additionalProperties": false, "required": [ + "dispatch_owner", "status", "mode", "mechanism", @@ -58,6 +59,7 @@ "terminal_evidence_required" ], "properties": { + "dispatch_owner": { "enum": ["orchestrator", "runner"] }, "status": { "enum": ["supported", "conditional", "unsupported"] }, "mode": { "enum": ["native_worker", "conditional", "unsupported"] }, "mechanism": { "type": "string" }, diff --git a/scripts/eval-runners/contracts/runner-descriptor.schema.json b/scripts/eval-runners/contracts/runner-descriptor.schema.json index 8a2827f..d424650 100644 --- a/scripts/eval-runners/contracts/runner-descriptor.schema.json +++ b/scripts/eval-runners/contracts/runner-descriptor.schema.json @@ -39,6 +39,7 @@ "type": "object", "additionalProperties": false, "required": [ + "dispatch_owner", "mode", "mechanism", "worker_role", @@ -50,6 +51,7 @@ "nested_model_execution" ], "properties": { + "dispatch_owner": { "enum": ["orchestrator", "runner"] }, "mode": { "enum": ["native_worker", "conditional", "unsupported"] }, "mechanism": { "type": "string", "minLength": 1 }, "worker_role": { "type": "string", "minLength": 1 }, diff --git a/scripts/eval-runners/fake/runner.ps1 b/scripts/eval-runners/fake/runner.ps1 index 4eb4de2..fa7355b 100644 --- a/scripts/eval-runners/fake/runner.ps1 +++ b/scripts/eval-runners/fake/runner.ps1 @@ -59,6 +59,7 @@ $descriptor = [ordered]@{ delegated_worker_capacity_signal = 'unsupported' } delegation = [ordered]@{ + dispatch_owner = 'orchestrator' mode = 'unsupported' mechanism = 'deterministic compatibility execute fixture; no harness-native worker surface' worker_role = 'compatibility-fixture' diff --git a/scripts/eval-runners/github-copilot/runner.ps1 b/scripts/eval-runners/github-copilot/runner.ps1 index b1856a4..32837fd 100644 --- a/scripts/eval-runners/github-copilot/runner.ps1 +++ b/scripts/eval-runners/github-copilot/runner.ps1 @@ -70,6 +70,7 @@ $descriptor = [ordered]@{ delegated_worker_capacity_signal = 'conditional' } delegation = [ordered]@{ + dispatch_owner = 'orchestrator' mode = 'native_worker' mechanism = 'Copilot CLI native task tool with an explicit full-capability general-purpose child agent; fleet/task lifecycle events observe completion' worker_role = 'general-purpose' diff --git a/scripts/eval-runners/opencode/runner.ps1 b/scripts/eval-runners/opencode/runner.ps1 index 93580b5..31e7843 100644 --- a/scripts/eval-runners/opencode/runner.ps1 +++ b/scripts/eval-runners/opencode/runner.ps1 @@ -57,6 +57,7 @@ $descriptor = [ordered]@{ delegated_worker_capacity_signal = 'conditional' } delegation = [ordered]@{ + dispatch_owner = 'orchestrator' mode = 'native_worker' mechanism = 'OpenCode Task tool invoking the full-capability General subagent in a fresh child context' worker_role = 'general' diff --git a/scripts/eval-runners/orchestration.ps1 b/scripts/eval-runners/orchestration.ps1 index 7726820..cc6428b 100644 --- a/scripts/eval-runners/orchestration.ps1 +++ b/scripts/eval-runners/orchestration.ps1 @@ -29,7 +29,8 @@ function New-EvalOrchestrationPlan { param( [Parameter(Mandatory = $true)][string]$IterationDirectory, [Parameter(Mandatory = $true)][object]$Manifest, - [Parameter(Mandatory = $true)][object]$Profile + [Parameter(Mandatory = $true)][object]$Profile, + [object]$Descriptor = $null ) $records = @(Get-ManifestRunRecords -IterationDirectory $IterationDirectory -Manifest $Manifest) @@ -44,6 +45,19 @@ function New-EvalOrchestrationPlan { throw 'Native worker orchestration requires a selected runner and model.' } + # Older deterministic callers can omit the descriptor and retain the + # original orchestrator-owned contract. Package handoffs always provide + # the selected descriptor so the native dispatch owner is explicit. + $dispatchOwner = 'orchestrator' + $dispatchMechanism = '' + if ($null -ne $Descriptor) { + $dispatchOwner = [string](Get-JsonProperty -Object (Get-JsonProperty -Object $Descriptor -Name 'delegation' -Default $null) -Name 'dispatch_owner' -Default '') + $dispatchMechanism = [string](Get-JsonProperty -Object (Get-JsonProperty -Object $Descriptor -Name 'delegation' -Default $null) -Name 'mechanism' -Default '') + } + if ($dispatchOwner -notin @('orchestrator', 'runner')) { + throw "Native worker orchestration dispatch_owner '$dispatchOwner' is unsupported." + } + $arms = [System.Collections.Generic.List[object]]::new() $seenWorkers = [System.Collections.Generic.HashSet[string]]::new([System.StringComparer]::OrdinalIgnoreCase) foreach ($record in $records) { @@ -59,6 +73,7 @@ function New-EvalOrchestrationPlan { eval_id = $record.EvalId eval_name = $record.EvalName configuration = $record.Configuration + dispatch_owner = $dispatchOwner depends_on = @() parent_paths = [ordered]@{ run_manifest = $record.RunManifestRelative @@ -81,7 +96,9 @@ function New-EvalOrchestrationPlan { paired_arm_visible = $false grading_material_visible = $false parent_executes_arm = $false - runner_execute_invocation = 'forbidden' + dispatch_owner = $dispatchOwner + dispatch_mechanism = $dispatchMechanism + runner_execute_invocation = if ($dispatchOwner -eq 'runner') { 'required' } else { 'forbidden' } nested_model_execution = $false model_execution_count = 1 } @@ -95,13 +112,14 @@ function New-EvalOrchestrationPlan { protocol_version = $schemas.Protocol runner = $runner model = $model + dispatch_owner = $dispatchOwner requested_concurrency = $requestedConcurrency parallel_dispatch_required = $parallelDispatchRequired minimum_parallel_workers = if ($parallelDispatchRequired) { 2 } else { 1 } native_worker_required = $true parent_executes_arms = $false nested_model_execution = $false - dispatch_policy = 'one fresh harness-native worker per arm; independent workers must run concurrently up to requested_concurrency when capacity permits' + dispatch_policy = if ($dispatchOwner -eq 'runner') { 'one fresh runner-owned native worker transport per arm; independent transports must run concurrently up to requested_concurrency when capacity permits' } else { 'one fresh orchestrator-owned harness-native worker per arm; independent workers must run concurrently up to requested_concurrency when capacity permits' } capacity_policy = 'harness_authoritative; a rejected delegation that did not start remains queued and is not an eval attempt' arms = $arms.ToArray() } @@ -126,6 +144,7 @@ function New-OrchestrationState { return [ordered]@{ schema = 'codebeltnet/agentic/eval-orchestration-state/1' plan_schema = [string]$Plan.schema + dispatch_owner = [string](Get-JsonProperty -Object $Plan -Name 'dispatch_owner' -Default 'orchestrator') requested_concurrency = [int]$Plan.requested_concurrency parallel_dispatch_required = [bool](Get-JsonProperty -Object $Plan -Name 'parallel_dispatch_required' -Default $false) minimum_parallel_workers = [int](Get-JsonProperty -Object $Plan -Name 'minimum_parallel_workers' -Default 1) @@ -303,7 +322,7 @@ function Register-WorkerTerminal { # arm here; do not infer a path from configuration or inspect grading data. try { $runData = Resolve-RunContract -RunPath ([string]$arm.worker.run_manifest_path) - $validation = Test-NativeWorkerTerminalEvidence -ExecutionEvidence $ExecutionEvidence -Run $runData -RequestedModel ([string]$arm.worker.model) -ExpectedWorkerSessionId $expectedSessionId + $validation = Test-NativeWorkerTerminalEvidence -ExecutionEvidence $ExecutionEvidence -Run $runData -RequestedModel ([string]$arm.worker.model) -ExpectedWorkerSessionId $expectedSessionId -ExpectedMechanism ([string](Get-JsonProperty -Object $arm.worker -Name 'dispatch_mechanism' -Default '')) foreach ($failure in @($validation.Failures)) { $terminalEvidenceFailures.Add([string]$failure) } $evidenceSessionId = [string](Get-JsonProperty -Object $validation.Delegation -Name 'worker_session_id' -Default '') if ($terminalEvidenceFailures.Count -eq 0 -and -not [string]::IsNullOrWhiteSpace($evidenceSessionId)) { @@ -378,7 +397,9 @@ function New-WorkerDispatchEnvelope { paired_arm_visible = $false grading_material_visible = $false parent_executes_arm = $false - runner_execute_invocation = 'forbidden' + dispatch_owner = [string](Get-JsonProperty -Object $worker -Name 'dispatch_owner' -Default 'orchestrator') + dispatch_mechanism = [string](Get-JsonProperty -Object $worker -Name 'dispatch_mechanism' -Default '') + runner_execute_invocation = [string](Get-JsonProperty -Object $worker -Name 'runner_execute_invocation' -Default 'forbidden') nested_model_execution = $false model_execution_count = 1 fresh_worker_required = $true @@ -392,6 +413,10 @@ function Assert-OrchestrationPlanContract { if ([string]$Plan.schema -ne (Get-RunnerSchemaNames).OrchestrationPlan) { throw 'Orchestration plan has an unsupported schema.' } + $planDispatchOwner = [string](Get-JsonProperty -Object $Plan -Name 'dispatch_owner' -Default '') + if ($planDispatchOwner -notin @('orchestrator', 'runner')) { + throw "Orchestration plan dispatch_owner '$planDispatchOwner' is unsupported." + } if (-not [bool]$Plan.native_worker_required -or [bool]$Plan.parent_executes_arms -or [bool]$Plan.nested_model_execution) { throw 'Orchestration plan must require native workers and forbid parent or nested model execution.' } @@ -411,7 +436,7 @@ function Assert-OrchestrationPlanContract { if (-not $workerIds.Add([string]$arm.worker_id)) { throw "Orchestration plan duplicates worker '$($arm.worker_id)'." } if (@($arm.depends_on).Count -ne 0) { throw "Worker '$($arm.worker_id)' has an unrelated dependency." } $worker = $arm.worker - foreach ($property in @('one_arm_only', 'paired_arm_visible', 'grading_material_visible', 'parent_executes_arm', 'nested_model_execution', 'model_execution_count')) { + foreach ($property in @('one_arm_only', 'paired_arm_visible', 'grading_material_visible', 'parent_executes_arm', 'dispatch_owner', 'runner_execute_invocation', 'nested_model_execution', 'model_execution_count')) { if (-not (Test-JsonProperty -Object $worker -Name $property)) { throw "Worker '$($arm.worker_id)' is missing '$property'." } } if (-not [bool]$worker.one_arm_only -or [bool]$worker.paired_arm_visible -or [bool]$worker.grading_material_visible -or [bool]$worker.parent_executes_arm -or [bool]$worker.nested_model_execution -or [int]$worker.model_execution_count -ne 1) { @@ -420,7 +445,13 @@ function Assert-OrchestrationPlanContract { if ([string]$worker.worker_id -ne [string]$arm.worker_id -or [int]$worker.eval_id -ne [int]$arm.eval_id -or [string]$worker.configuration -ne [string]$arm.configuration) { throw "Worker '$($arm.worker_id)' does not identify exactly its manifest arm." } - if ([string]$worker.runner_execute_invocation -ne 'forbidden') { throw "Worker '$($arm.worker_id)' may not invoke the direct runner execute transport." } + if ([string]$arm.dispatch_owner -ne $planDispatchOwner -or [string]$worker.dispatch_owner -ne $planDispatchOwner) { + throw "Worker '$($arm.worker_id)' dispatch ownership does not match the plan." + } + $expectedRunnerExecute = if ($planDispatchOwner -eq 'runner') { 'required' } else { 'forbidden' } + if ([string]$worker.runner_execute_invocation -ne $expectedRunnerExecute) { + throw "Worker '$($arm.worker_id)' has runner_execute_invocation '$($worker.runner_execute_invocation)'; expected '$expectedRunnerExecute' for dispatch owner '$planDispatchOwner'." + } foreach ($forbiddenProperty in @('paired_arm', 'grading', 'expected_output', 'assertions', 'eval_metadata', 'execution_result', 'result')) { if (Test-JsonProperty -Object $worker -Name $forbiddenProperty) { throw "Worker '$($arm.worker_id)' exposes forbidden parent or grading field '$forbiddenProperty'." diff --git a/scripts/eval-runners/runner-common.ps1 b/scripts/eval-runners/runner-common.ps1 index 577fb79..ac26250 100644 --- a/scripts/eval-runners/runner-common.ps1 +++ b/scripts/eval-runners/runner-common.ps1 @@ -345,6 +345,11 @@ function Get-DelegationCapabilityAssessment { } $delegation = Get-JsonProperty -Object $Descriptor -Name 'delegation' -Default $null + $dispatchOwner = [string](Get-JsonProperty -Object $delegation -Name 'dispatch_owner' -Default 'unsupported') + if ($dispatchOwner -notin @('orchestrator', 'runner')) { + $unproven.Add('delegation.dispatch_owner') + $unsupported.Add('delegation.dispatch_owner') + } $mode = [string](Get-JsonProperty -Object $delegation -Name 'mode' -Default 'unsupported') $nestedModelExecution = [bool](Get-JsonProperty -Object $delegation -Name 'nested_model_execution' -Default $true) $mechanism = [string](Get-JsonProperty -Object $delegation -Name 'mechanism' -Default '') @@ -385,6 +390,7 @@ function Get-DelegationCapabilityAssessment { return [pscustomobject]@{ MandatoryProven = $status -eq 'supported' Status = $status + DispatchOwner = $dispatchOwner Mode = $mode Mechanism = $mechanism WorkerRole = $workerRole @@ -801,11 +807,14 @@ function Assert-RunnerDescriptor { } } $delegation = $Descriptor.delegation - foreach ($field in @('mode', 'mechanism', 'worker_role', 'full_capability', 'model_lock', 'working_directory', 'result_capture', 'capacity', 'nested_model_execution')) { + foreach ($field in @('dispatch_owner', 'mode', 'mechanism', 'worker_role', 'full_capability', 'model_lock', 'working_directory', 'result_capture', 'capacity', 'nested_model_execution')) { if (-not (Test-JsonProperty -Object $delegation -Name $field)) { throw "Runner descriptor delegation is missing '$field'." } } + if ([string]$delegation.dispatch_owner -notin @('orchestrator', 'runner')) { + throw "Runner delegation dispatch_owner '$($delegation.dispatch_owner)' must be orchestrator or runner." + } if ([string]$delegation.mode -notin @('native_worker', 'conditional', 'unsupported')) { throw "Runner delegation mode '$($delegation.mode)' is unsupported." } @@ -871,6 +880,7 @@ function New-PreflightDocument { checks = @($Checks) resolved_capabilities = if ($null -eq $ResolvedCapabilities) { [ordered]@{} } else { $ResolvedCapabilities } delegation = [ordered]@{ + dispatch_owner = $delegationAssessment.DispatchOwner status = $delegationAssessment.Status mode = $delegationAssessment.Mode mechanism = $delegationAssessment.Mechanism @@ -899,6 +909,10 @@ function Assert-NativeWorkerDelegation { ) $delegation = Get-JsonProperty -Object $Descriptor -Name 'delegation' -Default $null + $dispatchOwner = [string](Get-JsonProperty -Object $delegation -Name 'dispatch_owner' -Default '') + if ($dispatchOwner -notin @('orchestrator', 'runner')) { + throw "Runner '$($Descriptor.name)' has no valid native dispatch owner." + } $mode = [string](Get-JsonProperty -Object $delegation -Name 'mode' -Default 'unsupported') if ($mode -notin @('native_worker', 'conditional')) { throw "Runner '$($Descriptor.name)' cannot satisfy the mandatory native Eval Worker contract: delegation mode is '$mode'." diff --git a/scripts/eval-runners/tests/test-orchestration.ps1 b/scripts/eval-runners/tests/test-orchestration.ps1 index b972817..bab941e 100644 --- a/scripts/eval-runners/tests/test-orchestration.ps1 +++ b/scripts/eval-runners/tests/test-orchestration.ps1 @@ -187,6 +187,63 @@ try { Assert-True ($dispatch.PSObject.Properties.Name -notcontains 'expected_output') "$($dispatch.worker_id) has no expected-output payload" } + # Codex uses runner-owned native dispatch. The runner process/thread is the + # worker, so the portable queue must not ask an outer model orchestrator to + # create a subagent first. Six recorded arms are advanced concurrently; + # this test never starts a process or model. + $runnerDescriptor = [pscustomobject]@{ + name = 'codex' + delegation = [ordered]@{ + dispatch_owner = 'runner' + mechanism = 'deterministic-fake-native-worker' + } + } + $runnerProfile = [ordered]@{ + runner = 'codex' + model = 'fixture-model' + reasoning_effort = $null + configuration_profile = 'isolated-default' + tool_profile = 'default' + timeout_seconds = 60 + concurrency = 3 + } + $runnerPlan = New-EvalOrchestrationPlan -IterationDirectory $iteration -Manifest $manifest -Profile $runnerProfile -Descriptor $runnerDescriptor + [void](Assert-OrchestrationPlanContract -Plan $runnerPlan) + Assert-Equal 'runner' $runnerPlan.dispatch_owner 'runner-owned descriptor selects runner dispatch' + $runnerState = New-OrchestrationState -Plan $runnerPlan + $outerSubagentCalls = 0 + $runnerWorkerStarts = 0 + $runnerDispatches = @(Get-NextWorkerDispatches -Plan $runnerPlan -State $runnerState) + Assert-Equal 3 $runnerDispatches.Count 'runner-owned dispatch respects requested concurrency' + foreach ($dispatch in $runnerDispatches) { + Assert-Equal 'runner' $dispatch.worker_contract.dispatch_owner "$($dispatch.worker_id) is runner-owned" + Assert-Equal 'required' $dispatch.worker_contract.runner_execute_invocation "$($dispatch.worker_id) uses the runner-owned execution surface" + $runnerWorkerStarts++ + [void](Register-DelegationAccepted -State $runnerState -WorkerId $dispatch.worker_id) + } + Assert-Equal 3 (Get-OrchestrationActiveCount -State $runnerState) 'runner-owned workers are active concurrently' + Assert-True ($runnerState.max_observed_active -gt 1) 'runner-owned dispatch observes parallel active workers' + foreach ($workerId in @($runnerState.active.Keys)) { + $arm = Get-OrchestrationArmByWorkerId -Plan $runnerPlan -WorkerId ([string]$workerId) + $runData = Resolve-RunContract -RunPath ([string]$arm.worker.run_manifest_path) + [void](Register-WorkerTerminal -Plan $runnerPlan -State $runnerState -WorkerId ([string]$workerId) -ExecutionEvidence (New-TestNativeTerminalEvidence -Arm $arm -RunData $runData -WorkerSessionId ('runner-session-' + $workerId))) + } + $runnerDispatches = @(Get-NextWorkerDispatches -Plan $runnerPlan -State $runnerState) + Assert-Equal 3 $runnerDispatches.Count 'runner-owned queue dispatches the next concurrent batch' + foreach ($dispatch in $runnerDispatches) { + $runnerWorkerStarts++ + [void](Register-DelegationAccepted -State $runnerState -WorkerId $dispatch.worker_id) + } + foreach ($workerId in @($runnerState.active.Keys)) { + $arm = Get-OrchestrationArmByWorkerId -Plan $runnerPlan -WorkerId ([string]$workerId) + $runData = Resolve-RunContract -RunPath ([string]$arm.worker.run_manifest_path) + [void](Register-WorkerTerminal -Plan $runnerPlan -State $runnerState -WorkerId ([string]$workerId) -ExecutionEvidence (New-TestNativeTerminalEvidence -Arm $arm -RunData $runData -WorkerSessionId ('runner-session-' + $workerId))) + } + Assert-Equal 6 @($runnerState.completed.Keys).Count 'runner-owned test completes six independent arms' + Assert-Equal 6 $runnerWorkerStarts 'runner-owned dispatch starts one runner surface per arm' + Assert-Equal 0 $outerSubagentCalls 'runner-owned dispatch never requests an outer model subagent' + Assert-Equal 3 $runnerState.max_observed_active 'runner-owned state records concurrent maximum' + # A fake harness accepts only four simultaneous native workers. This limit # belongs to the fake harness, not to the portable plan or queue. $capacityState = New-OrchestrationState -Plan $plan @@ -301,7 +358,7 @@ try { # validate terminal evidence; it must never reuse compatibility execute. $conditionalDescriptor = [pscustomobject]@{ name = 'conditional-native' - delegation = [ordered]@{ mode = 'native_worker'; nested_model_execution = $false } + delegation = [ordered]@{ dispatch_owner = 'orchestrator'; mode = 'native_worker'; nested_model_execution = $false } } $conditionalCapabilities = [ordered]@{ native_worker_delegation = 'conditional' diff --git a/scripts/eval-runners/tests/test-runner-conformance.ps1 b/scripts/eval-runners/tests/test-runner-conformance.ps1 index 69a524c..54a77f9 100644 --- a/scripts/eval-runners/tests/test-runner-conformance.ps1 +++ b/scripts/eval-runners/tests/test-runner-conformance.ps1 @@ -117,7 +117,7 @@ if ($harness -eq 'codex' -and $arguments -contains 'app-server' -and $arguments if ($outArgument.Count -eq 0) { exit 2 } $schemaDirectory = [IO.Path]::GetFullPath((Join-Path (Get-Location).Path ([string]$outArgument[0].Substring(6)))) New-Item -ItemType Directory -Path $schemaDirectory -Force | Out-Null - [IO.File]::WriteAllText((Join-Path $schemaDirectory 'ClientRequest.json'), '{"thread/start":"ThreadStartParams","turn/start":"TurnStartParams","cwd":true,"model":true,"ephemeral":true}', [Text.UTF8Encoding]::new($false)) + [IO.File]::WriteAllText((Join-Path $schemaDirectory 'ClientRequest.json'), '{"thread/start":"ThreadStartParams","turn/start":"TurnStartParams","thread/read":"ThreadReadParams","model/rerouted":"ModelReroutedNotification","ThreadStartResponse":true,"instructionSources":true,"cwd":true,"model":true,"ephemeral":true}', [Text.UTF8Encoding]::new($false)) [IO.File]::AppendAllText($logPath, (($record | ConvertTo-Json -Compress) + [Environment]::NewLine), [Text.UTF8Encoding]::new($false)) exit 0 } @@ -147,9 +147,43 @@ if ($harness -eq 'codex' -and $arguments -contains 'app-server') { Write-AppServerMessage ([ordered]@{ jsonrpc = '2.0'; id = $initialize.id; result = [ordered]@{ serverInfo = [ordered]@{ name = 'recorded-codex'; version = '9.1' } } }) $initialized = Read-AppServerMessage $threadStart = Read-AppServerMessage - Write-AppServerMessage ([ordered]@{ jsonrpc = '2.0'; id = $threadStart.id; result = [ordered]@{ thread = [ordered]@{ id = 'recorded-subscription-thread'; ephemeral = $true; path = $null } } }) + $fixtureReroute = Test-Path -LiteralPath (Join-Path ([Environment]::GetEnvironmentVariable('HOME')) 'codex-reroute') -PathType Leaf + $fixtureAmbientInstruction = Test-Path -LiteralPath (Join-Path ([Environment]::GetEnvironmentVariable('HOME')) 'codex-ambient-instruction') -PathType Leaf + $instructionSources = if ($fixtureAmbientInstruction) { @('C:\ambient\AGENTS.md') } else { @($repositoryAgentsPath) } + $threadObject = [ordered]@{ + id = 'recorded-subscription-thread' + sessionId = 'recorded-subscription-session' + ephemeral = $true + cwd = (Get-Location).Path + cliVersion = '9.1' + createdAt = 1 + updatedAt = 1 + modelProvider = 'recorded-provider' + preview = $false + projectId = $null + source = 'startup' + status = [ordered]@{ type = 'idle' } + turns = @() + } + $threadStartResult = [ordered]@{ + approvalPolicy = 'never' + approvalsReviewer = 'user' + cwd = (Get-Location).Path + model = 'gpt-5.6-luna' + modelProvider = 'recorded-provider' + sandbox = [ordered]@{ type = 'readOnly' } + instructionSources = $instructionSources + thread = $threadObject + } + Write-AppServerMessage ([ordered]@{ jsonrpc = '2.0'; id = $threadStart.id; result = $threadStartResult }) Write-AppServerMessage ([ordered]@{ jsonrpc = '2.0'; method = 'thread/started'; params = [ordered]@{ thread = [ordered]@{ id = 'recorded-subscription-thread' } } }) $turnStart = Read-AppServerMessage + if ($fixtureReroute) { + # This notification deliberately arrives before turn/start's response + # so the recorded transport proves reroute capture while waiting for a + # JSON-RPC response, not only in the terminal event loop. + Write-AppServerMessage ([ordered]@{ jsonrpc = '2.0'; method = 'model/rerouted'; params = [ordered]@{ threadId = 'recorded-subscription-thread'; turnId = 'recorded-subscription-turn'; fromModel = 'gpt-5.6-luna'; toModel = 'gpt-5.6-other'; reason = 'highRiskCyberActivity' } }) + } Write-AppServerMessage ([ordered]@{ jsonrpc = '2.0'; id = $turnStart.id; result = [ordered]@{ turn = [ordered]@{ id = 'recorded-subscription-turn'; status = 'inProgress'; items = @() } } }) $promptText = [string]$turnStart.params.input[0].text @@ -172,7 +206,16 @@ if ($harness -eq 'codex' -and $arguments -contains 'app-server') { $record.worker_project_disable_visible = $false $record.parent_codex_home = [Environment]::GetEnvironmentVariable('CODEX_HOME') $record.parent_auth_file_visible = Test-Path -LiteralPath (Join-Path $record.parent_codex_home 'auth.json') -PathType Leaf - $record.rpc_methods = @($initialize.method, $initialized.method, $threadStart.method, $turnStart.method) + $record.parent_config_file_visible = Test-Path -LiteralPath (Join-Path $record.parent_codex_home 'config.toml') -PathType Leaf + $record.parent_skills_directory_visible = Test-Path -LiteralPath (Join-Path $record.parent_codex_home 'skills') -PathType Container + $record.parent_agents_directory_visible = Test-Path -LiteralPath (Join-Path $record.parent_codex_home 'agents') -PathType Container + $record.parent_sessions_directory_visible = Test-Path -LiteralPath (Join-Path $record.parent_codex_home 'sessions') -PathType Container + $record.parent_memories_directory_visible = Test-Path -LiteralPath (Join-Path $record.parent_codex_home 'memories') -PathType Container + $record.parent_plugins_directory_visible = Test-Path -LiteralPath (Join-Path $record.parent_codex_home 'plugins') -PathType Container + $record.parent_mcp_configuration_visible = (Test-Path -LiteralPath (Join-Path $record.parent_codex_home 'mcp.json') -PathType Leaf) -or (Test-Path -LiteralPath (Join-Path $record.parent_codex_home 'mcp') -PathType Container) + $record.parent_agents_file_visible = Test-Path -LiteralPath (Join-Path $record.parent_codex_home 'AGENTS.md') -PathType Leaf + $record.auth_only_home = [bool]$record.parent_auth_file_visible -and -not [bool]$record.parent_config_file_visible -and -not [bool]$record.parent_skills_directory_visible -and -not [bool]$record.parent_agents_directory_visible -and -not [bool]$record.parent_sessions_directory_visible -and -not [bool]$record.parent_memories_directory_visible -and -not [bool]$record.parent_plugins_directory_visible -and -not [bool]$record.parent_mcp_configuration_visible -and -not [bool]$record.parent_agents_file_visible + $record.rpc_methods = @($initialize.method, $initialized.method, $threadStart.method, $turnStart.method, 'thread/read') $record.thread_params = $threadStart.params $record.turn_params = $turnStart.params [IO.File]::AppendAllText($logPath, (($record | ConvertTo-Json -Depth 50 -Compress) + [Environment]::NewLine), [Text.UTF8Encoding]::new($false)) @@ -182,6 +225,8 @@ if ($harness -eq 'codex' -and $arguments -contains 'app-server') { Write-AppServerMessage ([ordered]@{ jsonrpc = '2.0'; method = 'item/completed'; params = [ordered]@{ threadId = 'recorded-subscription-thread'; turnId = 'recorded-subscription-turn'; completedAtMs = 3; item = [ordered]@{ type = 'agentMessage'; id = 'message-1'; text = 'recorded subscription response' } } }) Write-AppServerMessage ([ordered]@{ jsonrpc = '2.0'; method = 'thread/tokenUsage/updated'; params = [ordered]@{ threadId = 'recorded-subscription-thread'; turnId = 'recorded-subscription-turn'; tokenUsage = [ordered]@{ total = [ordered]@{ inputTokens = 2; cachedInputTokens = 1; outputTokens = 3; reasoningOutputTokens = 1; totalTokens = 6 }; last = [ordered]@{ inputTokens = 2; cachedInputTokens = 1; outputTokens = 3; reasoningOutputTokens = 1; totalTokens = 6 } } } }) Write-AppServerMessage ([ordered]@{ jsonrpc = '2.0'; method = 'turn/completed'; params = [ordered]@{ threadId = 'recorded-subscription-thread'; turn = [ordered]@{ id = 'recorded-subscription-turn'; status = 'completed'; items = @() } } }) + $threadRead = Read-AppServerMessage + Write-AppServerMessage ([ordered]@{ jsonrpc = '2.0'; id = $threadRead.id; result = [ordered]@{ thread = $threadObject } }) exit 0 } if ($arguments -contains '--version') { @@ -359,6 +404,8 @@ exit 2 $description = Invoke-AdapterJson -RunnerPath $runnerPath -Command describe -RunPath $with.Path -ProfilePath $recordedProfiles[$runnerName] [void](Assert-RunnerDescriptor -Descriptor $description) Assert-True ($description.PSObject.Properties.Name -contains 'delegation') "$runnerName descriptor declares native delegation" + $expectedDispatchOwner = if ($runnerName -eq 'codex') { 'runner' } else { 'orchestrator' } + Assert-Equal $expectedDispatchOwner $description.delegation.dispatch_owner "$runnerName descriptor declares its native dispatch owner" Assert-True (-not [bool]$description.delegation.nested_model_execution) "$runnerName descriptor forbids nested model execution" Assert-True (-not [string]::IsNullOrWhiteSpace([string]$description.delegation.mechanism)) "$runnerName descriptor records its native delegation mechanism" Assert-Equal 'conditional' $description.capabilities.native_worker_delegation "$runnerName descriptor does not present native delegation as terminal proof" @@ -372,6 +419,7 @@ exit 2 Assert-Equal $expectedVersion $preflightWith.harness.version "$runnerName exact preflight version" Assert-Equal 'pragmatic' $preflightWith.isolation.level "$runnerName pragmatic preflight level" Assert-Equal 'conditional' $preflightWith.delegation.status "$runnerName native delegation preflight requires terminal evidence" + Assert-Equal $expectedDispatchOwner $preflightWith.delegation.dispatch_owner "$runnerName preflight preserves native dispatch ownership" Assert-True ([bool]$preflightWith.delegation.terminal_evidence_required) "$runnerName preflight requires terminal delegation evidence" if ($runnerName -eq 'copilot') { Assert-True (@($preflightWith.checks | Where-Object { $_.name -eq 'authentication' -and $_.status -eq 'passed' }).Count -eq 1) 'Copilot preflight accepts explicit environment authentication' @@ -518,6 +566,12 @@ exit 2 $fileAuthHome = Join-Path $recordedRoot 'codex-file-auth' New-Item -ItemType Directory -Path $fileAuthHome -Force | Out-Null [System.IO.File]::WriteAllText((Join-Path $fileAuthHome 'auth.json'), '{"canary":"not-logged"}', [System.Text.UTF8Encoding]::new($false)) + [System.IO.File]::WriteAllText((Join-Path $fileAuthHome 'config.toml'), 'model = "ambient-not-used"', [System.Text.UTF8Encoding]::new($false)) + New-Item -ItemType Directory -Path (Join-Path $fileAuthHome 'skills'), (Join-Path $fileAuthHome 'agents'), (Join-Path $fileAuthHome 'sessions'), (Join-Path $fileAuthHome 'memories'), (Join-Path $fileAuthHome 'plugins'), (Join-Path $fileAuthHome 'mcp') -Force | Out-Null + [System.IO.File]::WriteAllText((Join-Path $fileAuthHome 'skills\ambient.md'), 'ambient skill must not be copied', [System.Text.UTF8Encoding]::new($false)) + [System.IO.File]::WriteAllText((Join-Path $fileAuthHome 'agents\ambient.md'), 'ambient agent must not be copied', [System.Text.UTF8Encoding]::new($false)) + [System.IO.File]::WriteAllText((Join-Path $fileAuthHome 'mcp.json'), '{"ambient":true}', [System.Text.UTF8Encoding]::new($false)) + [System.IO.File]::WriteAllText((Join-Path $fileAuthHome 'AGENTS.md'), 'ambient instructions must not be copied', [System.Text.UTF8Encoding]::new($false)) $env:OPENAI_API_KEY = $null $env:CODEX_HOME = $fileAuthHome $fileAuthPreflight = Invoke-AdapterJson -RunnerPath (Join-Path $runnerRoot 'codex\runner.ps1') -Command preflight -RunPath $with.Path -ProfilePath $recordedProfiles['codex'] @@ -534,9 +588,48 @@ exit 2 Assert-Equal 2 ([int]$fileAuthResult.telemetry.tool_calls.value) 'Codex app-server counts command and file-change evidence' Assert-Equal 1 @($fileAuthResult.evidence.commands).Count 'Codex app-server preserves command evidence' Assert-Equal 1 @($fileAuthResult.evidence.files).Count 'Codex app-server preserves file-change evidence' + Assert-Equal 'runner' $fileAuthResult.evidence.delegation.dispatch_owner 'Codex native evidence identifies runner-owned dispatch' + Assert-Equal 'gpt-5.6-luna' $fileAuthResult.evidence.delegation.observed_model 'Codex native evidence uses observed thread/start model' + Assert-Equal (Join-Path $with.Root 'repo') $fileAuthResult.evidence.delegation.observed_working_directory 'Codex native evidence uses observed cwd' + Assert-True ([bool]$fileAuthResult.evidence.delegation.fresh_worker) 'Codex native evidence proves ephemeral fresh worker' + Assert-True ([bool]$fileAuthResult.evidence.delegation.home_config_isolated) 'Codex native evidence proves auth-only home cleanup' + Assert-True ([bool]$fileAuthResult.evidence.delegation.prompt_fidelity) 'Codex native evidence proves exact prompt hash' + Assert-True ([bool]$fileAuthResult.evidence.delegation.terminal_result_capture) 'Codex native evidence proves turn completion capture' + Assert-Equal 'harness_native_transport' $fileAuthResult.evidence.capture.source 'Codex capture provenance is app-server transport-owned' + Assert-True ([bool]$fileAuthResult.evidence.capture.terminal -and -not [bool]$fileAuthResult.evidence.capture.worker_authored) 'Codex capture is terminal and not authored by the worker/orchestrator' + Assert-True ([bool]$fileAuthResult.evidence.delegation.thread_read_observed) 'Codex native evidence records thread/read observation' + Assert-Equal 'thread/start' $fileAuthResult.evidence.app_server.thread_start_request.method 'Codex evidence retains the exact thread/start request' + Assert-Equal 'turn/start' $fileAuthResult.evidence.app_server.turn_start_request.method 'Codex evidence retains the exact turn/start request' + Assert-Equal 'gpt-5.6-luna' $fileAuthResult.evidence.app_server.thread_start_request.params.model 'Codex exact thread/start request preserves model' + Assert-Equal (Join-Path $with.Root 'repo') $fileAuthResult.evidence.app_server.turn_start_request.params.cwd 'Codex exact turn/start request preserves cwd' + Assert-Equal 'gpt-5.6-luna' $fileAuthResult.evidence.app_server.thread_start_response.result.model 'Codex evidence retains observed thread/start model' + Assert-Equal 'completed' $fileAuthResult.evidence.app_server.terminal_turn.status 'Codex evidence retains terminal turn/completed status' + Assert-Equal 'recorded-subscription-thread' $fileAuthResult.evidence.app_server.thread_read.request.threadId 'Codex evidence retains the thread/read request identity' + $fileAuthEvidenceJson = ConvertTo-Json -InputObject $fileAuthResult -Depth 100 + Assert-True ($fileAuthEvidenceJson -notmatch 'recorded-canary|not-logged' -and $fileAuthEvidenceJson -notmatch [regex]::Escape($fileAuthHome)) 'Codex result evidence does not include the copied credential or auth path' $subscriptionLogPath = Join-Path $with.Root 'repo\codex-fake-cli-log.jsonl' $subscriptionRecord = Get-Content -LiteralPath $subscriptionLogPath | ForEach-Object { $_ | ConvertFrom-Json } | Where-Object { $_.PSObject.Properties.Name -contains 'rpc_methods' } | Select-Object -Last 1 - Assert-Equal 'initialize,initialized,thread/start,turn/start' ([string]::Join(',', @($subscriptionRecord.rpc_methods))) 'Codex app-server follows the required handshake order' + $codexNativeValidation = Test-NativeWorkerTerminalEvidence -ExecutionEvidence $fileAuthResult -Run (Resolve-RunContract -RunPath $with.Path) -RequestedModel 'gpt-5.6-luna' -ExpectedRunner 'codex' -ExpectedMechanism $fileAuthResult.evidence.delegation.mechanism + Assert-True ([bool]$codexNativeValidation.Valid) 'Codex app-server result satisfies the common native terminal evidence contract' + [void](Assert-NativeWorkerTerminalEvidence -ExecutionEvidence $fileAuthResult -Run (Resolve-RunContract -RunPath $with.Path) -RequestedModel 'gpt-5.6-luna' -ExpectedRunner 'codex' -ExpectedMechanism $fileAuthResult.evidence.delegation.mechanism) + [void](Assert-NativeTerminalCaptureArtifact -ExecutionResult $fileAuthResult) + $authHomePath = [string]$subscriptionRecord.parent_codex_home + Assert-True (-not (Test-Path -LiteralPath $authHomePath)) 'Codex temporary auth-only home is removed after the arm completes' + + $rerouteMarker = Join-Path $with.Root 'home\codex-reroute' + [System.IO.File]::WriteAllText($rerouteMarker, 'fixture', [System.Text.UTF8Encoding]::new($false)) + $reroutedResult = Invoke-AdapterJson -RunnerPath (Join-Path $runnerRoot 'codex\runner.ps1') -Command execute -RunPath $with.Path -ProfilePath $recordedProfiles['codex'] + Assert-Equal 'incompatible' $reroutedResult.status 'Codex model/rerouted notification fails closed' + Assert-True (@($reroutedResult.evidence.native_worker_evidence_failures | Where-Object { $_ -eq 'model_rerouted' }).Count -eq 1) 'Codex reroute incompatibility is recorded as transport evidence' + Remove-Item -LiteralPath $rerouteMarker -Force + + $ambientInstructionMarker = Join-Path $with.Root 'home\codex-ambient-instruction' + [System.IO.File]::WriteAllText($ambientInstructionMarker, 'fixture', [System.Text.UTF8Encoding]::new($false)) + $ambientInstructionResult = Invoke-AdapterJson -RunnerPath (Join-Path $runnerRoot 'codex\runner.ps1') -Command execute -RunPath $with.Path -ProfilePath $recordedProfiles['codex'] + Assert-Equal 'incompatible' $ambientInstructionResult.status 'Codex unexpected instruction source fails closed' + Assert-True (@($ambientInstructionResult.evidence.native_worker_evidence_failures | Where-Object { $_ -eq 'unexpected_instruction_sources' }).Count -eq 1) 'Codex ambient instruction rejection is recorded as transport evidence' + Remove-Item -LiteralPath $ambientInstructionMarker -Force + Assert-Equal 'initialize,initialized,thread/start,turn/start,thread/read' ([string]::Join(',', @($subscriptionRecord.rpc_methods))) 'Codex app-server follows the required handshake and post-completion read order' Assert-Equal 'gpt-5.6-luna' $subscriptionRecord.thread_params.model 'Codex app-server thread receives the requested model' Assert-True ([bool]$subscriptionRecord.thread_params.ephemeral) 'Codex app-server thread is ephemeral' Assert-Equal 'readOnly' $subscriptionRecord.thread_params.sandbox 'Codex app-server thread avoids persisting writable project trust' @@ -545,8 +638,17 @@ exit 2 Assert-Equal (Join-Path $with.Root 'repo') $subscriptionRecord.turn_params.cwd 'Codex app-server turn receives the staged working directory' Assert-Equal 'never' $subscriptionRecord.turn_params.approvalPolicy 'Codex app-server turn rejects interactive approvals' Assert-Equal 'workspaceWrite' $subscriptionRecord.turn_params.sandboxPolicy.type 'Codex app-server turn receives workspace-write sandbox policy' - Assert-Equal $fileAuthHome $subscriptionRecord.parent_codex_home 'Codex app-server parent receives the subscription CODEX_HOME' + Assert-True ($subscriptionRecord.parent_codex_home -ne $fileAuthHome) 'Codex app-server does not expose the ambient subscription CODEX_HOME' Assert-True ([bool]$subscriptionRecord.parent_auth_file_visible) 'Codex app-server parent can read the subscription auth file' + Assert-True ([bool]$subscriptionRecord.auth_only_home) 'Codex app-server temporary CODEX_HOME contains auth.json only' + Assert-True (-not [bool]$subscriptionRecord.parent_config_file_visible) 'Codex app-server temporary CODEX_HOME excludes config.toml' + Assert-True (-not [bool]$subscriptionRecord.parent_skills_directory_visible) 'Codex app-server temporary CODEX_HOME excludes skills' + Assert-True (-not [bool]$subscriptionRecord.parent_agents_directory_visible) 'Codex app-server temporary CODEX_HOME excludes agents' + Assert-True (-not [bool]$subscriptionRecord.parent_sessions_directory_visible) 'Codex app-server temporary CODEX_HOME excludes sessions' + Assert-True (-not [bool]$subscriptionRecord.parent_memories_directory_visible) 'Codex app-server temporary CODEX_HOME excludes memories' + Assert-True (-not [bool]$subscriptionRecord.parent_plugins_directory_visible) 'Codex app-server temporary CODEX_HOME excludes plugins' + Assert-True (-not [bool]$subscriptionRecord.parent_mcp_configuration_visible) 'Codex app-server temporary CODEX_HOME excludes MCP configuration' + Assert-True (-not [bool]$subscriptionRecord.parent_agents_file_visible) 'Codex app-server temporary CODEX_HOME excludes AGENTS.md' Assert-True (-not [bool]$subscriptionRecord.unrelated_present) 'Codex app-server parent excludes unrelated inherited environment variables' Assert-True (-not [bool]$subscriptionRecord.worker_auth_file_visible) 'Codex app-server worker fixture does not receive auth.json' Assert-True (@($subscriptionRecord.args) -contains 'shell_environment_policy.inherit=none') 'Codex app-server disables child shell environment inheritance' @@ -920,14 +1022,19 @@ try { $recordText = [System.IO.File]::ReadAllText((Join-Path $runnerRoot 'record-native-result.ps1'), [System.Text.UTF8Encoding]::new($false)) $manifestBridgeText = [System.IO.File]::ReadAllText((Join-Path $runnerRoot 'bridge-manifest-results.ps1'), [System.Text.UTF8Encoding]::new($false)) $commonText = [System.IO.File]::ReadAllText((Join-Path $runnerRoot 'runner-common.ps1'), [System.Text.UTF8Encoding]::new($false)) + $orchestrationText = [System.IO.File]::ReadAllText((Join-Path $runnerRoot 'orchestration.ps1'), [System.Text.UTF8Encoding]::new($false)) Assert-True ($prepareText -notmatch '(?i)codex\s+exec|opencode\s+run|copilot\s+-p|copilot\s+--prompt|Profile\.Provider') 'portable preparation must not contain harness-specific CLI invocations or provider-field branches' + Assert-True ($orchestrationText -notmatch '(?i)capture-native-results\.ps1|synthesize|worker_authored') 'generic orchestration must not manufacture native terminal envelopes' Assert-True ($reportText -notmatch '(?i)codex\s+exec|opencode\s+run|copilot\s+-p|copilot\s+--prompt|Profile\.Provider') 'reporting must not contain harness-specific or provider-field branches' Assert-True ($bridgeText -notmatch '(?i)codex\s+exec|opencode\s+run|copilot\s+-p|copilot\s+--prompt|Profile\.Provider') 'the raw-to-portable bridge must remain runner-neutral' Assert-True ($prepareText.Contains('bridge-manifest-results.ps1')) 'handoff preparation must use the deterministic package-level manifest bridge' Assert-True ($prepareText.Contains('runs..run_manifest') -and $prepareText.Contains('runs..execution_result') -and $prepareText.Contains('runs..result')) 'handoff preparation must require every exact manifest arm path' Assert-True ($prepareText.Contains('Do not derive, normalize, rename, hyphenate, underscore, or otherwise reconstruct any run, execution-result, or result path.')) 'handoff preparation must prohibit reconstructed paths' - Assert-True ($prepareText.Contains('DELEGATE EVERY eval arm to a fresh harness-native worker/subagent. The Eval Orchestrator MUST NOT execute an eval arm itself.')) 'handoff preparation must require delegated native workers and forbid parent execution' - Assert-True ($prepareText.Contains('One arm equals one delegated worker and one model-backed eval execution.')) 'handoff preparation must state the one-arm one-model invariant' + Assert-True ($prepareText.Contains('Read `delegation.dispatch_owner` from the selected runner descriptor and preflight.')) 'handoff preparation must make native dispatch ownership explicit' + Assert-True ($prepareText.Contains('One arm equals one fresh native Eval Worker and one model-backed eval execution.')) 'handoff preparation must state the one-arm one-model invariant' + Assert-True ($prepareText.Contains('For `runner`, do not spawn a model subagent first.')) 'handoff preparation must forbid an outer runner-owned model worker' + Assert-True ($prepareText.Contains('do not invoke `record-native-result.ps1`, manufacture a native envelope')) 'runner-owned handoff must preserve runner capture without synthetic envelopes' + Assert-True ($prepareText -notmatch '(?i)capture-native-results\.ps1') 'handoff preparation must not contain a synthetic capture helper' Assert-True ($prepareText.Contains('Assert-NativeWorkerDelegation')) 'handoff preparation must invoke the native delegation gate' Assert-True ($prepareText.Contains('Require terminal evidence for the exact selected model, exact arm identity, working directory, isolated HOME/config boundary, prompt fidelity, terminal result capture')) 'handoff preparation must require worker control evidence' Assert-True ($prepareText.Contains('-RequireComplete -RequireNativeDelegation -RequireParallelDispatch')) 'handoff preparation must revalidate native terminal and parallel-dispatch evidence during the manifest bridge' @@ -939,7 +1046,7 @@ try { Assert-True ($prepareText.Contains('rejected before the worker starts') -and $prepareText.Contains('record no eval attempt')) 'handoff preparation must queue capacity rejections without counting attempts' Assert-True ($prepareText.Contains('Register each worker acceptance and terminal result exactly once') -and $prepareText.Contains('incompatibility is diagnostic-only') -and $prepareText.Contains('Do not grade incompatible arms')) 'handoff preparation must make duplicate registration and incompatible-arm handling fail closed' Assert-True ($prepareText.Contains('Skipping report generation because the completion gate failed') -and $prepareText.Contains('Diagnostic comparison (incomplete)')) 'incomplete collection must remain diagnostic and skip report generation' - Assert-True ($prepareText.Contains('capture.worker_authored') -and $prepareText.Contains('Do not ask the worker to author or summarize this envelope')) 'handoff preparation must require transport-produced native envelopes rather than worker-authored summaries' + Assert-True ($prepareText.Contains('capture.worker_authored') -and $prepareText.Contains('Do not ask any worker or parent to author, hand-write, normalize, or repair terminal evidence.')) 'handoff preparation must require transport-produced native evidence rather than worker-authored summaries' Assert-True ($bridgeText.Contains('Get-PackageRunnerDescriptor') -and $bridgeText.Contains('Assert-NativeTerminalCaptureArtifact') -and $bridgeText.Contains('ExpectedMechanism')) 'native bridge must require runner-produced terminal evidence' Assert-True ($recordText.Contains('eval-native-worker-result/1') -and $recordText.Contains('New-ExecutionResult')) 'native terminal recording must use the runner-owned result builder' Assert-True ($commonText.Contains('exit.status must be a JSON number or null')) 'execution results must reject textual exit statuses' From ca3e7632b075a3d32f5c26c026683df4811dd5e2 Mon Sep 17 00:00:00 2001 From: "aicia[bot]" Date: Tue, 25 Aug 2026 18:41:03 +0200 Subject: [PATCH 39/39] =?UTF-8?q?=F0=9F=94=A7=20update=20skill=20eval=20to?= =?UTF-8?q?oling=20for=20dispatch=20owner?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Update prepare-skill-evals.ps1 and validate-skill-templates.ps1 to align with dispatch-owner runner descriptors and updated execution profile schema. Minor adjustments to support the portable eval handoff workflow with explicit dispatch strategy selection. --- scripts/prepare-skill-evals.ps1 | 30 ++++++++++++++-------------- scripts/validate-skill-templates.ps1 | 4 +++- 2 files changed, 18 insertions(+), 16 deletions(-) diff --git a/scripts/prepare-skill-evals.ps1 b/scripts/prepare-skill-evals.ps1 index 2946a34..1af2c94 100644 --- a/scripts/prepare-skill-evals.ps1 +++ b/scripts/prepare-skill-evals.ps1 @@ -1595,8 +1595,8 @@ function Invoke-PrepareMode { Write-Host '' Write-Host 'The runner-aware handoff uses execution-profile.json, the package-local Eval Runner protocol,' Write-Host 'and its deterministic native-worker orchestration plan.' - Write-Host 'The selected external orchestrator must delegate EVERY arm to a fresh harness-native worker.' - Write-Host 'The orchestrator coordinates; it must not execute an eval arm itself or invoke the direct execute transport.' + Write-Host 'The selected external orchestrator must honor each runner descriptor''s dispatch_owner: orchestrator-owned native workers stay delegated, while runner-owned transports are started directly as the one-arm Eval Workers.' + Write-Host 'The orchestrator coordinates; it must not execute an eval arm in its own model context or nest a runner-owned model execution inside another worker.' Write-Host 'Independent workers run concurrently up to execution-profile.json.concurrency; harness capacity remains authoritative.' Write-Host '' Write-Host 'This script prepared prompts only. It did not run them, and nothing here will.' @@ -1641,20 +1641,20 @@ function New-RunnerPrompt { [void]$builder.AppendLine() [void]$builder.AppendLine('## Phase 1: delegate blind arms') [void]$builder.AppendLine() - [void]$builder.AppendLine('> DELEGATE EVERY eval arm to a fresh harness-native worker/subagent. The Eval Orchestrator MUST NOT execute an eval arm itself.') + [void]$builder.AppendLine('> Read `delegation.dispatch_owner` from the selected runner descriptor and preflight. `orchestrator` means the orchestrator dispatches the declared native subagent/task; `runner` means the orchestrator starts the runner-owned native execution surface directly. The orchestrator never executes an arm in its own model context.') [void]$builder.AppendLine() - [void]$builder.AppendLine('> One arm equals one delegated worker and one model-backed eval execution. The delegated worker is the eval worker; do not place another model-backed runner process inside it.') + [void]$builder.AppendLine('> One arm equals one fresh native Eval Worker and one model-backed eval execution. For runner-owned dispatch, the runner process/thread is that worker; do not create an outer model subagent first.') [void]$builder.AppendLine() [void]$builder.AppendLine('> Run independent workers concurrently up to `execution-profile.json.concurrency`. If the harness temporarily refuses another worker because its own concurrency limit is reached, keep that arm queued and dispatch it when capacity becomes available.') [void]$builder.AppendLine() [void]$builder.AppendLine('1. Read `manifest.json` and `execution-profile.json`. If `runner` or `model` is null, unavailable, or unsupported, fail clearly and list the supported package-local runner IDs; do not guess a default. The profile contains no credentials.') [void]$builder.AppendLine(('2. Resolve the selected package-local runner with the resolver. Ask it for `describe` and `preflight`, validate its protocol, descriptor, and native-delegation capability declarations, invoke `Assert-NativeWorkerDelegation`, and load the deterministic orchestration helper at ' + $orchestrationPath + '. Do not invent harness-specific CLI commands.')) [void]$builder.AppendLine('3. Build the pending arm queue from `manifest.json` with the orchestration helper. For every arm, read the exact `run_manifest`, `execution_result`, and `result` fields from `runs..run_manifest`, `runs..execution_result`, and `runs..result`. Retain those exact manifest-declared strings without editing them: the parent owns those exact destinations; the worker receives only its own prepared arm contract. Do not derive, normalize, rename, hyphenate, underscore, or otherwise reconstruct any run, execution-result, or result path.') - [void]$builder.AppendLine('4. Before dispatching, require native worker delegation and all mandatory isolation controls. An unavailable or unsupported delegation mechanism is incompatible. A conditional mechanism is allowed only when the external orchestrator will require terminal evidence for the actual worker. Do not continue by invoking a runner process in the parent, and do not silently serialize arms in the parent.') + [void]$builder.AppendLine('4. Before dispatching, require native worker delegation and all mandatory isolation controls. An unavailable or unsupported delegation mechanism is incompatible. A conditional mechanism is allowed only when terminal evidence is required for the actual worker. Use the descriptor''s `delegation.dispatch_owner` as the protocol decision: `orchestrator` selects the declared native subagent/task mechanism, while `runner` selects the runner''s declared one-arm native execution surface. Do not substitute a generic worker, an outer model subagent, a parent-side eval, or a compatibility fallback.') [void]$builder.AppendLine('5. Dispatch each pending arm to one fresh harness-native full-capability worker. The worker must execute the prepared `prompt.md` as its first task from that arm''s staged run directory, with the selected model/configuration, exact working directory and isolated home. Require terminal evidence for the exact selected model, exact arm identity, working directory, isolated HOME/config boundary, prompt fidelity, terminal result capture, paired-arm/grading exclusion, fresh session, and absence of nested model execution; if any required fact is missing or mismatched, mark the arm incompatible without retrying through a fallback. It must not receive its paired arm, `eval-metadata.json`, expected output, assertions, grading, benchmark/report data, or any result from another arm.') - [void]$builder.AppendLine(('6. The delegated worker is the only model-backed execution for that arm. It must not invoke `runner.ps1 execute`, a second harness CLI, another model agent, or a nested session. The harness-native transport, not the worker''s answer, must produce one terminal envelope using schema `codebeltnet/agentic/eval-native-worker-result/1`; its `capture.source` must be `harness_native_transport`, `capture.terminal` must be true, and `capture.worker_authored` must be false. The envelope must contain the exact nested `run` identity, fresh session, `timing.started_utc`/`finished_utc`/`duration_seconds`, numeric-or-null exit status, structured isolation capabilities, delegation evidence, and hashed transcript/event artifact. Do not ask the worker to author or summarize this envelope. If the native transport returns only assistant text or a hand-authored summary, the arm is incompatible. After the harness-native worker is terminal, write only the captured envelope to a temporary package-local path and invoke the deterministic runner-owned recorder at {0} with the exact `-Runner`, `-Run`, `-Profile`, `-NativeResult`, and manifest-declared `-Output` paths. The recorder starts no model and emits the canonical runner-produced `execution-result.json`; register and bridge only that generated file. A parent-created summary, direct JSON mutation, or malformed envelope is incompatible.' -f $nativeResultRecorderPath)) + [void]$builder.AppendLine(('6. Apply the dispatch-owner semantics exactly. For `orchestrator`, the native subagent/task is the only model-backed execution: it must not invoke `runner.ps1 execute`, a second harness CLI, another model agent, or a nested session. Its harness-native transport must produce schema `codebeltnet/agentic/eval-native-worker-result/1` with `capture.source=harness_native_transport`, `capture.terminal=true`, and `capture.worker_authored=false`; then preserve that envelope and invoke the deterministic runner-owned recorder at {0} with the exact `-Runner`, `-Run`, `-Profile`, `-NativeResult`, and manifest-declared `-Output` paths. For `runner`, do not spawn a model subagent first. Start exactly one runner-owned native execution surface for the arm using `runner.ps1 execute`; that runner process/thread is the Eval Worker and the single model-backed execution. Preserve its runner-produced `execution-result.json` at the exact manifest-declared output path, and do not invoke `record-native-result.ps1`, manufacture a native envelope, copy assistant text into transport evidence, or repair/normalize the runner output. In either case, the native transport must own timestamps, session/thread identity, isolation observations, prompt fidelity, terminal completion, and the hashed raw transcript/event artifact. A response or summary without transport-owned evidence is incompatible.' -f $nativeResultRecorderPath)) [void]$builder.AppendLine('7. Register each worker acceptance and terminal result exactly once. If orchestration reports that an arm is already accepted or terminal, do not retry or re-register it; preserve the state and resolve the arm from its existing terminal record. Maintain up to `min(execution-profile.json.concurrency, remaining arms)` active delegated workers. If a delegation request is rejected before the worker starts because of harness capacity, leave that arm pending, record no eval attempt, mark the rejection as capacity-limited in orchestration state, and retry it after an active worker becomes terminal. Do not add a runner-specific ceiling or change the portable requested concurrency.') - [void]$builder.AppendLine('7a. DISPATCH IS AN ACTION, NOT A CONFIRMATION STEP. After preflight, emit the full first batch of native worker calls immediately. Do not send a prose status message, ask the user whether to start or re-dispatch, wait for confirmation, or wait for the first worker result before emitting the remaining calls. If the client cannot emit the batch in one assistant turn, mark the selected runner incompatible and stop; never replace the batch with deliberate serial dispatch.') + [void]$builder.AppendLine('7a. DISPATCH IS AN ACTION, NOT A CONFIRMATION STEP. After preflight, start the full first batch of native workers immediately. For `orchestrator`, emit the required sibling native worker calls before awaiting results. For `runner`, launch the independent runner-owned one-arm processes/threads concurrently before awaiting results. Do not send a prose status message, ask the user whether to start or re-dispatch, wait for confirmation, or replace the batch with deliberate serial dispatch. If the selected harness cannot satisfy its declared dispatch-owner mechanism and isolation contract, mark it incompatible and stop.') if ([string]::Equals([string]$ExecutionSelection.Runner, 'opencode', [System.StringComparison]::OrdinalIgnoreCase)) { $armLabelText = [string]::Join(', ', @($armLabels)) $deferredDispatchExample = 'Want me to re-dispatch...' @@ -1671,16 +1671,16 @@ function New-RunnerPrompt { [void]$builder.AppendLine("pwsh -NoProfile -File `"$resolverPath`" ") [void]$builder.AppendLine('runner.ps1 describe') [void]$builder.AppendLine("runner.ps1 preflight -Run `"`" -Profile `"$profilePath`"") - [void]$builder.AppendLine("runner.ps1 execute -Run `"`" -Profile `"$profilePath`" # direct one-arm compatibility transport; forbidden to the parent orchestrator") - [void]$builder.AppendLine("record-native-result.ps1 -Runner `"`" -Run `"`" -Profile `"$profilePath`" -NativeResult `"`" -Output `"`"") + [void]$builder.AppendLine("runner.ps1 execute -Run `"`" -Profile `"$profilePath`" # runner-owned dispatch only; one native worker/model execution") + [void]$builder.AppendLine("record-native-result.ps1 -Runner `"`" -Run `"`" -Profile `"$profilePath`" -NativeResult `"`" -Output `"`" # orchestrator-owned dispatch only") [void]$builder.AppendLine('```') - [void]$builder.AppendLine('Use the resolver output to locate `runner.ps1`; `` is data from the profile, not a branch in this orchestration contract. The parent may use `describe`, `preflight`, and the deterministic `record-native-result.ps1` recorder. The `execute` command remains part of the one-arm runner protocol for compatibility and conformance, but the native delegated worker path MUST NOT invoke it: doing so would create a second model-backed execution.') + [void]$builder.AppendLine('Use the resolver output and descriptor data to locate `runner.ps1`; `` is data from the profile, not a runner-specific branch in this orchestration contract. The parent may use `describe` and `preflight`. `record-native-result.ps1` is only for orchestrator-owned native envelopes. `runner.ps1 execute` is the runner-owned native worker surface when `delegation.dispatch_owner=runner`; invoking it inside an outer model worker would create a second model-backed execution and is forbidden.') [void]$builder.AppendLine() [void]$builder.AppendLine('Do not read any `eval-metadata.json`, expected output, assertions, result grading, benchmark/report data, or paired output during Phase 1. The orchestrator and worker must never expose those materials before all workers are terminal. They remain outside every run directory and are the grading key.') [void]$builder.AppendLine() [void]$builder.AppendLine('## Phase 2: bridge, grade, and report') [void]$builder.AppendLine() - [void]$builder.AppendLine('1. Only after every available delegated worker is terminal and `Assert-OrchestrationConcurrency` passes, invoke the deterministic package bridge below. It reads `manifest.json`, obtains each arm''s exact `run_manifest`, `execution_result`, and `result` paths. The bridge checks prompt/run/profile hashes and artifact confinement, validates the manifest paths, rejects unreferenced hyphen/underscore shadow results, verifies the persisted parallel-dispatch state, and invokes the existing one-arm bridge with those exact paths. Do not manually construct a bridge command for an arm.') + [void]$builder.AppendLine('1. Only after every available native worker is terminal and `Assert-OrchestrationConcurrency` passes, invoke the deterministic package bridge below. It reads `manifest.json`, obtains each arm''s exact `run_manifest`, `execution_result`, and `result` paths. The bridge checks prompt/run/profile hashes and artifact confinement, validates the manifest paths, rejects unreferenced hyphen/underscore shadow results, verifies the persisted parallel-dispatch state, and invokes the existing one-arm bridge with those exact paths. Do not manually construct a bridge command for an arm.') [void]$builder.AppendLine((' `pwsh -NoProfile -File "' + $manifestBridgePath + '" -IterationDirectory "' + $IterationDirectory + '" -RequireComplete -RequireNativeDelegation -RequireParallelDispatch`')) [void]$builder.AppendLine(' The bridge''s one-arm operation is conceptually `-Run runPath -ExecutionResult executionPath -Result resultPath`, where all three values are the exact strings read from `manifest.json`. Do not derive, normalize, rename, hyphenate, underscore, or otherwise reconstruct any of them.') [void]$builder.AppendLine('2. Only if the package bridge succeeds, read each eval''s `eval-metadata.json` and reveal `expected_output` and `assertions` to the Grader. Follow `tools/skill-creator/agents/grader.md`; grade deterministically first, then use optional model judgement only where deterministic evidence cannot decide. Never infer tool or file behavior from model self-report without process evidence.') @@ -1704,7 +1704,7 @@ function New-PackageReadme { $builder = [System.Text.StringBuilder]::new() [void]$builder.AppendLine("# Eval package: $SkillName (iteration $IterationNumber)") [void]$builder.AppendLine() - [void]$builder.AppendLine('Prepared by `scripts/prepare-skill-evals.ps1` in `codebeltnet/agentic`. Nothing in this package was executed. `execution-profile.json` selects the user-chosen Eval Runner, runner-native model, and limits; the external Eval Orchestrator delegates every arm to a fresh harness-native Eval Worker, then grades and generates the report.') + [void]$builder.AppendLine('Prepared by `scripts/prepare-skill-evals.ps1` in `codebeltnet/agentic`. Nothing in this package was executed. `execution-profile.json` selects the user-chosen Eval Runner, runner-native model, and limits; the external Eval Orchestrator follows the selected descriptor''s `delegation.dispatch_owner`, using either orchestrator-owned native workers or runner-owned native transports, then grades and generates the report.') [void]$builder.AppendLine() [void]$builder.AppendLine('## What is here') [void]$builder.AppendLine() @@ -1725,7 +1725,7 @@ function New-PackageReadme { [void]$builder.AppendLine(' +-- Eval Worker -> one eval arm') [void]$builder.AppendLine(' +-- ...') [void]$builder.AppendLine('```') - [void]$builder.AppendLine('The Eval Orchestrator coordinates and collects; it never executes an eval arm itself. One arm equals one delegated worker and one model-backed eval execution. Independent workers must run concurrently up to `execution-profile.json.concurrency` when capacity permits; harness capacity is authoritative, so a rejected delegation stays queued and is not an attempt. A serial result without persisted capacity-limit evidence fails the completion gate.') + [void]$builder.AppendLine('The Eval Orchestrator coordinates and collects; it never executes an eval arm in its own model context. One arm equals one fresh native Eval Worker and one model-backed eval execution. With `dispatch_owner=orchestrator`, the native subagent/task is the worker; with `dispatch_owner=runner`, the runner process/thread is the worker and no outer model subagent is created. Independent workers must run concurrently up to `execution-profile.json.concurrency` when capacity permits; harness capacity is authoritative, so a rejected dispatch stays queued and is not an attempt. A serial result without persisted capacity-limit evidence fails the completion gate.') [void]$builder.AppendLine('The package also carries the exact Anthropic skill-creator assets used after execution under `tools/skill-creator`: `tools/skill-creator/agents/grader.md`, `tools/skill-creator/agents/comparator.md`, `tools/skill-creator/agents/analyzer.md`, `tools/skill-creator/references/schemas.md`, `tools/skill-creator/scripts/aggregate_benchmark.py`, and `tools/skill-creator/eval-viewer/generate_review.py` plus `tools/skill-creator/eval-viewer/viewer.html`.') [void]$builder.AppendLine() [void]$builder.AppendLine('## Isolation model') @@ -1737,8 +1737,8 @@ function New-PackageReadme { [void]$builder.AppendLine('## How to run') [void]$builder.AppendLine() [void]$builder.AppendLine('1. Read `execution-profile.json`. If `runner` or `model` is missing, fail clearly instead of guessing. Resolve the selected package-local runner and run `describe`, then `preflight`, before any native worker is dispatched. Invoke `Assert-NativeWorkerDelegation`; conditional controls require terminal evidence, unsupported controls are incompatible, and no parent sequential fallback is allowed. For OpenCode, a profile with concurrency below 2 is incompatible.') - [void]$builder.AppendLine('2. Use the package-local orchestration helper to queue one worker per manifest arm. Delegate every arm to a fresh full-capability harness-native worker. Do not invoke the runner''s direct `execute` command from the parent or from the delegated worker, because it would add a second model execution. Each worker receives one arm only and no grading material or paired-arm data.') - [void]$builder.AppendLine('3. After each harness-native worker is terminal, preserve the transport-produced envelope under schema `codebeltnet/agentic/eval-native-worker-result/1` and invoke the deterministic `record-native-result.ps1` recorder with the exact manifest `run_manifest`, `execution_result`, and profile paths. The envelope must prove `capture.source=harness_native_transport`, `capture.terminal=true`, and `capture.worker_authored=false`; assistant text alone is not terminal evidence. The recorder owns the `eval-execution-result/1` shape, timestamps, run identity, requested configuration, and runner/harness identity. Do not ask a worker to author, hand-write, normalize, or repair an `execution-result.json` summary in the orchestrator.') + [void]$builder.AppendLine('2. Use the package-local orchestration helper to queue one native worker per manifest arm. Read `delegation.dispatch_owner` from the descriptor: `orchestrator` means the declared native subagent/task receives the arm, while `runner` means invoke the runner''s one-arm native execution surface directly. A runner-owned process/thread is the sole model-backed worker for that arm; never put it inside an outer model worker. Each worker receives one arm only and no grading material or paired-arm data.') + [void]$builder.AppendLine('3. For orchestrator-owned dispatch, preserve each native transport envelope under schema `codebeltnet/agentic/eval-native-worker-result/1` and invoke the deterministic `record-native-result.ps1` recorder with the exact manifest `run_manifest`, `execution_result`, and profile paths. For runner-owned dispatch, preserve the runner-produced `execution-result.json` directly at the exact manifest-declared path and do not invoke the recorder or synthesize an envelope. In both cases, transport evidence must prove `capture.source=harness_native_transport`, `capture.terminal=true`, `capture.worker_authored=false`; assistant text alone is not terminal evidence. Do not ask any worker or parent to author, hand-write, normalize, or repair terminal evidence.') [void]$builder.AppendLine('4. Maintain up to the requested concurrency. If the harness refuses a new worker because its own capacity is full, leave that arm queued, mark the rejection as capacity-limited in `orchestration-state.json`, and dispatch it when capacity is released; do not hardcode a runner-specific maximum and do not count the rejection as an attempt.') [void]$builder.AppendLine('5. After all delegated workers complete or fail, invoke `Assert-OrchestrationConcurrency`, persist `orchestration-state.json`, and run `tools/eval-runners/bridge-manifest-results.ps1 -IterationDirectory -RequireComplete -RequireNativeDelegation -RequireParallelDispatch`. It reads the manifest-declared `run_manifest`, `execution_result`, and `result` paths for every arm, validates native terminal evidence and parallel-dispatch evidence again, and invokes the one-arm bridge with those exact paths. Only then read the grading key, grade with `tools/skill-creator/agents/grader.md`, and run `tools/generate-eval-report.ps1 -RequireComplete`.') [void]$builder.AppendLine() diff --git a/scripts/validate-skill-templates.ps1 b/scripts/validate-skill-templates.ps1 index 7c661f8..938802b 100644 --- a/scripts/validate-skill-templates.ps1 +++ b/scripts/validate-skill-templates.ps1 @@ -1458,7 +1458,9 @@ Add-ValidationResult -Results $results -Name 'Skill evaluation prepares portable 'START NOW. You are the external Eval Orchestrator', 'Do not execute evaluation prompts in the current agent context.', 'execution-profile.json` selects the runner/model/configuration', - 'The `execute` command remains part of the one-arm runner protocol for compatibility and conformance, but the native delegated worker path MUST NOT invoke it', + 'Read `delegation.dispatch_owner` from the selected runner descriptor and preflight.', + 'For `runner`, do not spawn a model subagent first.', + 'do not invoke `record-native-result.ps1`, manufacture a native envelope', 'Do not read any `eval-metadata.json`, expected output, assertions, result grading, benchmark/report data, or paired output during Phase 1.', 'Do not substitute a generic worker, another runner, or an improvised isolation scheme if the selected runner is unavailable or incompatible.', '`runs..run_manifest`',