feat(plugin): ship coder_eval as a Claude Code plugin + marketplace

.claude-plugin/marketplace.json

@@ -0,0 +1,17 @@
+{
+  "name": "coder-eval",
+  "owner": {
+    "name": "UiPath",
+    "url": "https://github.com/UiPath/coder_eval"
+  },
+  "description": "Evaluate and benchmark AI coding agents and Claude Code skills.",
+  "plugins": [
+    {
+      "name": "coder-eval",
+      "source": "./plugins/coder-eval",
+      "description": "Author, run, and analyze coder-eval suites — including whether your Claude Code skills actually trigger.",
+      "category": "testing",
+      "keywords": ["evaluation", "testing", "skills", "benchmark", "ci"]
+    }
+  ]
+}

.claude/commands/coder-eval-task-create.md

@@ -63,10 +63,18 @@ Choose criteria types based on what needs to be verified:
 | Observed vs expected label | `classification_match` | File-based label match for classification suites (emits P/R/F1) |
 | UiPath agent eval | `uipath_eval` | UiPath agent evaluation results |
 
-**Criteria design rules:**
-- Every task needs at least one criterion that validates the **output content**, not just existence
+Before choosing criteria, read `plugins/coder-eval/reference/task-rubric.md` — the shared
+adversarial checklist ("could this pass for the wrong reason?", fixture lifecycle, scope
+match). It is the single declaration for all three consumers: this command and the
+plugin's `task` and `lint-tasks` skills.
+
+The rubric owns the *correctness* checks — that something validates output content, that
+`require_success: true` is set whenever a command's success is what you are grading, that a
+criterion cannot pass for the wrong reason. Do not restate them here; apply them from there.
+
+**Criteria design rules** (conventions the rubric does not cover):
 - Use `run_command` with `expected_stdout` + `stdout_match: regex` to validate script output
-- Use `command_executed` sparingly — only when verifying the agent used a specific tool matters. Set `require_success: false` unless the command must succeed.
+- Use `command_executed` sparingly — only when verifying the agent used a specific tool matters
 - Use `file_check` instead of separate `file_exists` + `file_contains` when checking the same file
 - Set `weight` to reflect importance: 0.5 for nice-to-have, 1.0 for standard, 1.5-2.0 for critical
 - Default `pass_threshold: 0.9` is fine for most criteria. Use `1.0` only for binary checks.

.claude/harness-candidates.md

@@ -107,3 +107,39 @@ Deferred lint/test guardrails surfaced during reviews. Promote to a `CExxx` rule
   caught them. The cleanup plan explicitly deferred this as YAGNI for the
   one-time purge, but any future doc rename/deletion re-opens the same blind
   spot — caught in the 2026-07-03 open-source-docs-cleanup implementation run.
+
+- [ ] **Plugin skills must not name a file that exists only in THIS repo** — the
+  `test_bundled_files_reference_no_repo_paths` denylist (`docs/`, `src/`,
+  `.claude/shared/`, `.claude/commands/`, `uv run`, `../`) deliberately allows
+  `tasks/` and `.claude/skills/`, because those are user-workspace paths the
+  skills legitimately scan and scaffold. So a skill body naming a specific repo
+  file (e.g. `tasks/hello_date.yaml`) would slip past the guard even though an
+  installed plugin is copied to `~/.claude/plugins/cache/` without it. The
+  obvious rule — "extract path-shaped tokens, fail if the path exists at the repo
+  root" — is NOT cheap: `init` legitimately tells users to scan `pyproject.toml`
+  and `package.json`, and `pyproject.toml` exists here, so the heuristic
+  false-positives on correct prose. Needs a token classifier that distinguishes
+  "a file to look for in the user's repo" from "a file in ours", which is a
+  design problem, not a 30-minute one. No skill violates it today (grepped) —
+  caught in the 2026-08-04 claude-code-plugin-marketplace implementation run.
+  *Update (2026-08-04, plugin-audit-p0-p1 run): the guard was renamed and widened
+  from `skills/*/SKILL.md` to every shipped text file under `plugins/coder-eval/`
+  (`PLUGIN_TEXT_FILES`), which closed the coverage half of this gap — a bundled
+  reference now cannot name a repo path either. The token-classifier problem
+  described above is unchanged and still deferred.*
+
+## From 2026-08-04 plugin-audit-p0-p1 run
+
+- [ ] **A skill's advertised `description` must not promise a check that no bundled
+  reference declares.** `lint-tasks` ships a user-facing description claiming it
+  finds "prompts that give away the answer", but that check was declared only in
+  `skills/task/SKILL.md` prose — a file `lint-tasks` never reads — so the two
+  rubric readers had already forked on it before the skill shipped. Caught by a
+  reviewer, not by a test; fixed by promoting it to rubric check 7. A guard would
+  have to map claim-phrases in a description onto declarations in
+  `reference/task-rubric.md`, which is natural-language matching, not a token
+  grep — the phrasings are deliberately different (a description sells, a rubric
+  check instructs), so any cheap version either misses the real case or fails on
+  correct prose. Needs a fixed vocabulary of claim tags shared between the two
+  files to become mechanical, which is a design change rather than a 30-minute
+  rule — caught in the 2026-08-04 plugin-audit-p0-p1 implementation run.

.claude/shared/run-layout.md

@@ -1,8 +1,9 @@
-# Run layout (shared)
+<!-- Mirrored verbatim at plugins/coder-eval/reference/run-layout.md — update both together. -->
+# Run layout
 
-The on-disk structure of a coder_eval evaluation run — the factual contract that
-`coder-eval-run-analysis` and `coder-eval-review` both read. If the run directory
-structure changes, update it here and every consumer follows.
+The on-disk structure of a coder_eval evaluation run — the factual contract every
+run-reading command and skill follows. If the run directory structure changes, update it
+here and every consumer follows.
 
 ```
 runs/<run_id>/<variant_id>/<task_id>/<NN>/{task.json, task.log, artifacts/}

.github/workflows/pr-checks.yml

@@ -157,6 +157,66 @@ jobs:
           echo "✅ Quality gate complete!"
           echo "📊 All checks passed: formatting, linting, types, security, tests"
 
+  plugin-validate:
+    # Proves the Claude Code plugin marketplace is installable and that the suite
+    # `skill-check` scaffolds is real: the manifests pass strict validation, and the
+    # bundled activation template both schema-validates and expands to one task per
+    # dataset row. Needs no credentials — nothing here invokes a model.
+    name: Claude Code Plugin (manifests + offline scaffold)
+    runs-on: ubuntu-latest
+    timeout-minutes: 10
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+
+      # `claude plugin validate` ships in the Claude Code npm package.
+      - name: Set up Node.js 20
+        uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4.4.0
+        with:
+          node-version: "20"
+      - name: Install Claude CLI
+        run: npm install -g @anthropic-ai/claude-code
+
+      - name: Validate plugin manifest (strict)
+        run: claude plugin validate ./plugins/coder-eval --strict
+
+      - name: Validate marketplace manifest (strict)
+        run: claude plugin validate . --strict
+
+      - name: Install uv
+        uses: astral-sh/setup-uv@38f3f104447c67c051c4a08e39b64a148898af3a # v4.2.0
+
+      # Runs OUTSIDE the source tree, the way a user's copy of the template does:
+      # no experiments/, no tasks/, no coder_eval checkout on the path. `plan` is a
+      # schema check only (it exits 0 even when dataset.paths names a nonexistent
+      # file), so the row-count assert goes through expand_dataset — otherwise this
+      # step would pass even if activation-rows.jsonl were never copied.
+      # Reproduce locally with: SCRATCH=$(mktemp -d) VENV=$(mktemp -d)/venv
+      - name: Scaffold assert (no source tree)
+        run: |
+          set -euo pipefail
+          SCRATCH="$RUNNER_TEMP/scratch"
+          VENV="$RUNNER_TEMP/venv"
+          mkdir -p "$SCRATCH"
+          cp plugins/coder-eval/reference/templates/activation.yaml "$SCRATCH/"
+          cp plugins/coder-eval/reference/templates/activation-rows.jsonl "$SCRATCH/"
+          # A venv (not `uv tool install`) because the expansion assert needs
+          # `coder_eval` importable, not just the `coder-eval` CLI on PATH.
+          uv venv "$VENV"
+          VIRTUAL_ENV="$VENV" uv pip install .
+          cd "$SCRATCH"
+          "$VENV/bin/coder-eval" plan activation.yaml
+          "$VENV/bin/python" - <<'PY'
+          from pathlib import Path
+          from coder_eval.orchestration.task_loader import expand_dataset, load_task
+          task, _ = load_task(Path("activation.yaml"))
+          rows = expand_dataset(task, Path("."))
+          assert len(rows) == 6, f"expected 6 row-tasks, got {len(rows)}"
+          labels = {c.expected_skill for t in rows for c in t.success_criteria}
+          assert labels == {"my-skill", ""}, labels
+          print(f"ok: {len(rows)} row-tasks")
+          PY
+
   no-uipath-extra:
     # Proves that `pip install coder-eval` (without the optional `[uipath]`
     # extra) yields a working framework: imports succeed, the criterion

.github/workflows/release.yml

@@ -185,7 +185,7 @@ jobs:
           echo "version=$V" >> "$GITHUB_OUTPUT"
           echo "Publishing version: $V"
 
-      - name: Regenerate uv.lock, bump action.yml pin, and amend release commit
+      - name: Regenerate uv.lock, bump action.yml + plugin.json pins, and amend release commit
         if: steps.mode.outputs.prerelease != 'true' && steps.release.outputs.version != ''
         env:
           # Passed via env (not interpolated into the script) per GitHub's
@@ -203,10 +203,18 @@ jobs:
           sed -i -E 's/^([[:space:]]*default: ")[0-9]+\.[0-9]+\.[0-9]+("   # <-- kept in sync)/\1'"${VERSION}"'\2/' action.yml
           grep -q "default: \"${VERSION}\"" action.yml || { echo "action.yml version bump failed"; exit 1; }
           git add action.yml
+          # Keep the Claude Code plugin manifest's version in lockstep. `claude
+          # plugin validate --strict` (run in pr-checks) rejects a manifest with
+          # no version, and a stale one strands users on a cached copy.
+          sed -i -E 's/^([[:space:]]*"version": ")[0-9]+\.[0-9]+\.[0-9]+(",)/\1'"${VERSION}"'\2/' \
+            plugins/coder-eval/.claude-plugin/plugin.json
+          grep -q "\"version\": \"${VERSION}\"" plugins/coder-eval/.claude-plugin/plugin.json \
+            || { echo "plugin.json version bump failed"; exit 1; }
+          git add plugins/coder-eval/.claude-plugin/plugin.json
           # Regenerate the lock too; stage it (a no-op if unchanged).
           uv lock
           git add uv.lock
-          # Amend only if action.yml/uv.lock actually changed the tree.
+          # Amend only if action.yml/plugin.json/uv.lock actually changed the tree.
           if ! git diff --cached --quiet; then
             git commit --amend --no-edit
             # Amend replaced the commit the tag points at; re-point it before pushing.
@@ -215,7 +223,11 @@ jobs:
 
       - name: Push release commit and tags
         if: steps.mode.outputs.prerelease != 'true' && steps.release.outputs.version != ''
-        run: git push origin main "v${{ steps.release.outputs.version }}"
+        env:
+          # Passed via env (not interpolated into the script) per GitHub's
+          # injection guidance — matching the step above.
+          VERSION: ${{ steps.release.outputs.version }}
+        run: git push origin main "v${VERSION}"
 
       - name: Move major action tag (vN -> this release)
         if: steps.release.outputs.version != ''

CLAUDE.md

@@ -122,6 +122,8 @@ tasks/                             # Task definition YAML files
 tests/                             # Test suite
 docs/                              # Documentation
 templates/                         # Sandbox template directories
+.claude-plugin/marketplace.json    # Makes this repo a Claude Code plugin marketplace (`/plugin marketplace add UiPath/coder_eval`); lists the one plugin below.
+plugins/coder-eval/                # The published Claude Code plugin: `.claude-plugin/plugin.json` (its `version` is a derived pin of pyproject's, bumped by release.yml, guarded by tests/test_action_version_pin.py), `skills/<name>/SKILL.md` × 6 (`/coder-eval:init`, `/coder-eval:skill-check`, `/coder-eval:task`, `/coder-eval:lint-tasks`, `/coder-eval:analyze`, `/coder-eval:ci`), and `reference/` — everything a skill reads must live here, since an installed plugin is copied to ~/.claude/plugins/cache/ WITHOUT its parent dirs (address it via `${CLAUDE_PLUGIN_ROOT}`). `reference/criteria.md` is generated (`make plugin-reference`, CE032); `reference/run-layout.md` is a verbatim mirror of `.claude/shared/run-layout.md`; `reference/task-rubric.md` is the shared task-quality rubric that `task` and `lint-tasks` both read (plugin-only — no repo-side twin). Every skill must appear in all four surfaces in `SKILL_DOC_SURFACES` (derived test), and their combined frontmatter `description` length is capped (`SKILL_LISTING_BUDGET_CHARS`) because the skill listing's budget is shared with every skill the user has installed. Distinct from `.claude/commands/`, which stays repo-local contributor tooling.
 action.yml                         # Published composite GitHub Action (coder-eval as a CI gate). release.yml maintains its `version:` default + the moving `v<major>` tag.
 ```
 
@@ -198,9 +200,13 @@ make typecheck   # pyright
 make test        # pytest
 make lint        # custom architectural lint rules (CE001+)
 make verify      # All of the above + coverage check (CI equivalent)
+
+# Regenerate a generated surface (both are CE-guarded; never hand-edit the output)
+make docs-indexes      # README/docs index tables from the mkdocs nav (CE028)
+make plugin-reference  # the plugin's bundled criteria reference from the models (CE032)
 ```
 
-When fixing a bug, ask: *could a custom lint rule have prevented this?* If the root cause is a mechanically detectable pattern (e.g., "always import from `coder_eval.models`", "never call blocking IO in async"), add a rule to `tests/lint/rules/` following the CE001+ pattern and wire it up in `tests/lint/runner.py`. This turns a one-time fix into permanent enforcement. See `tests/test_custom_lint.py` for how rules are tested. (Doc-surface / whole-tree rules that reason over Markdown/YAML or the entire `src/` tree rather than one `.py` AST at a time — CE026–CE031 — are not `BaseRule`s in the runner; they are wired as dedicated `@pytest.mark.lint` test classes. CE031 guards against dead config: a behavior-driving field on `SimulationConfig`/`RunLimits`/`Dataset` that no code reads by name. CE026 keeps the GitHub Action's three onboarding surfaces honest: a page's *first* Action snippet must show the agent-runtime prerequisite steps (pinned to the `action-dogfood` job that proves them in CI), a zero-install absolute next to such a snippet must name the channel it means, and every `github.com/marketplace/actions/<slug>` link plus the shields badge label must match `action.yml`'s `name:`.)
+When fixing a bug, ask: *could a custom lint rule have prevented this?* If the root cause is a mechanically detectable pattern (e.g., "always import from `coder_eval.models`", "never call blocking IO in async"), add a rule to `tests/lint/rules/` following the CE001+ pattern and wire it up in `tests/lint/runner.py`. This turns a one-time fix into permanent enforcement. See `tests/test_custom_lint.py` for how rules are tested. (Doc-surface / whole-tree rules that reason over Markdown/YAML or the entire `src/` tree rather than one `.py` AST at a time — CE026–CE032 — are not `BaseRule`s in the runner; they are wired as dedicated `@pytest.mark.lint` test classes. CE032 keeps the plugin's bundled `reference/criteria.md` in parity with the `SuccessCriterion` union that generates it (`make plugin-reference` writes it; the rule re-renders and diffs — never hand-edit the file). CE031 guards against dead config: a behavior-driving field on `SimulationConfig`/`RunLimits`/`Dataset` that no code reads by name. CE026 keeps the GitHub Action's onboarding surfaces honest — `README.md`, `docs/CI_GATE.md`, `docs/tutorials/02-ci-pipeline.md`, and the plugin's `ci` skill, whose emitted workflow users copy into their own repos: a page's *first* Action snippet must show the agent-runtime prerequisite steps (pinned to the `action-dogfood` job that proves them in CI), a zero-install absolute next to such a snippet must name the channel it means, every `github.com/marketplace/actions/<slug>` link plus the shields badge label must match `action.yml`'s `name:`, and every `with:` key on a snippet's action step must be a real `action.yml` input (GitHub ignores unknown inputs, so a rename would silently degrade every copied workflow). Renaming an action input or changing its runtime prerequisites therefore means updating the skill too.)
 
 Adding a user-facing field to one of the models CE030 tracks (`TaskDefinition`, `RunLimits`, `Dataset`, `SimulationConfig` — see `tests/lint/doc_schema_parity.py`) means documenting it in its guide (mention the field name as inline code) or adding an `EXEMPT` entry with a reason it is not user-authored. `make lint` fails otherwise.
 

Makefile

@@ -1,4 +1,4 @@
-.PHONY: help install format check typecheck test test-live test-smoke verify verify-noextra clean run lint docs-indexes docker-image docker-image-full coder-eval-runtime docker-images
+.PHONY: help install format check typecheck test test-live test-smoke verify verify-noextra clean run lint docs-indexes plugin-reference docker-image docker-image-full coder-eval-runtime docker-images
 
 # Single source of the installed coder-eval version (used to tag the docker
 # images). Referenced lazily inside the docker recipes, so it doesn't run on
@@ -33,6 +33,9 @@ lint:  ## Run custom architectural lint rules (CE001+)
 docs-indexes:  ## Regenerate README/docs indexes from the mkdocs nav (SSOT)
 	uv run python -m tests.lint.doc_indexes
 
+plugin-reference:  ## Regenerate the plugin's bundled criteria reference from the models (SSOT)
+	uv run python -m tests.lint.plugin_reference
+
 typecheck:  ## Run type checking with pyright
 	uv run pyright