diff --git a/docs/BENCHMARKS.md b/docs/BENCHMARKS.md deleted file mode 100644 index e69de29bb..000000000 diff --git a/rosetta-benchmarks/.gitignore b/rosetta-benchmarks/.gitignore new file mode 100644 index 000000000..daab50967 --- /dev/null +++ b/rosetta-benchmarks/.gitignore @@ -0,0 +1,3 @@ +# Raw Claude Code session logs: large, and they contain local access tokens and passwords. +# Keep them locally; commit only tokens.md (the breakdown made from them). +saleor/runs/**/session/ diff --git a/rosetta-benchmarks/BENCHMARKS.md b/rosetta-benchmarks/BENCHMARKS.md new file mode 100644 index 000000000..092bab014 --- /dev/null +++ b/rosetta-benchmarks/BENCHMARKS.md @@ -0,0 +1,540 @@ +# Saleor benchmark + +Compare seven AI coding approaches on one real e-commerce platform (Saleor: core API, dashboard, storefront), on two tasks, in two agent runners: **8 methods × 2 runners × 2 tasks = 32 runs**. The full run list is in [saleor/EXPERIMENTS.md](./saleor/EXPERIMENTS.md). + +A run counts only if the result is merge-ready, not a draft that needs cleanup. + +## 1. Methods + +| # | Method | What is installed for the run | +| --: | --------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| 1 | Baseline | Only the shared team rules file (see 3.3). No method, no plugins. Runs in the runner's built-in **plan mode** (plan → questions → approve → implement): [saleor/methods/baseline.md](./saleor/methods/baseline.md) | +| 2 | Grill-with-docs | [mattpocock/skills](https://github.com/mattpocock/skills/tree/main/skills/engineering/grill-with-docs) `grill-with-docs` ([docs](https://github.com/mattpocock/skills/blob/main/docs/engineering/grill-with-docs.md)) plugin `mattpocock-skills` from Claude Code's official marketplace (`claude-plugins-official`, which pins it to a commit, `49dd158` on 2026-10-22); Copilot: the README's own installer `npx skills@latest add mattpocock/skills`. S-CC-02 used a local marketplace at `447ca70` | +| 3 | Superpowers | [obra/superpowers](https://github.com/obra/superpowers) **v6.4.2** (2026-09-25) | +| 4 | Spec Kit | [github/spec-kit](https://github.com/github/spec-kit) **v1.0.13** (2026-09-29) + one constitution per repo | +| 5 | AWS AI-DLC | [awslabs/aidlc-workflows](https://github.com/awslabs/aidlc-workflows) **v2.10.0** (2026-09-24) | +| 6 | Rosetta | Plugin `rosetta` **3.1.17** (2026-10-19) only | +| 7 | Rosetta Light | Plugin `rosetta-light` **3.1.17** (2026-10-19) only; skills `hitl` + `tech-specs` (large task: + `orchestration`) and `USE ROSETTA SUBAGENTS.` named in the prompt, no `/coding-flow` | +| 8 | Rosetta Skills | Plugin `rosetta` **3.1.17** (2026-10-19) only; skills `hitl` + `tech-specs` named in the prompt (large task: + `orchestration`), no `/coding-flow` | + +Versions were chosen on 2026-10-15 as the newest stable release at least two weeks old. Newer releases (Superpowers v7.0.0, Spec Kit v1.1.x–v1.2.0, AI-DLC v2.11.x) are too fresh, and the AI-DLC v2.11.1 builds are previews. **Exception: Rosetta, Rosetta Light and Rosetta Skills use 3.1.17**, the current release (2026-10-19), although it is newer than the two-week rule allows (3.1.13 would qualify). The study measures the current Rosetta; the exception is stated here so the results can be read with it in mind. Compared with 3.1.13, the gates and phases are the same; the subagent models changed (full: architect `claude-opus-5-5`, reviewers `claude-sonnet-5-5`; light: implementation and reviews on `claude-haiku-5-5`). Do not update any method during the study. + +Methods 1–6 come from the presentation slide "Six approaches, one more if time allows". BMAD-METHOD is optional (one task, if time allows) and not part of the 32. GSD is not run (archived upstream in May 2026). + +**Isolation:** every method runs alone, with no help from any other AI. See 3.5, "Each method runs in a vacuum". + +## 2. Runners, models, permissions + +| Runner | Model | Effort | +| --- | --- | --- | +| Claude Code **2.1.305** (`npm install -g @anthropic-ai/claude-code@2.1.305`, under Node 24) | Sonnet 5.5 (`claude-sonnet-5-5`) | medium | +| GitHub Copilot (VS Code) | GPT-5.6 Terra | medium | + +- No other agent products (Codex, Cursor, API scripts, …). +- Automatic model switching / Auto model selection is OFF. +- The two runners use different models, so a Claude Code vs Copilot difference is a product + model difference. Methods are compared **within** one runner. +- **Full-access (YOLO) mode, sandbox disabled**, so permission prompts don't affect the test: Claude Code `claude --dangerously-skip-permissions` (Baseline: `--permission-mode plan --allow-dangerously-skip-permissions`, full access is switched on when the plan is approved); Copilot `chat.tools.autoApprove: true` with terminal confirmations off. **This is not normal usage. Use it only for this test**, only in disposable benchmark clones with synthetic data, and revert it after the study. + +## 3. Environment setup (once) + +Everything is done by one script: [saleor/scripts/bench.sh](./saleor/scripts/bench.sh). Don't type the setup or reset commands by hand. + +### 3.1 Tools (macOS) + +Git, Docker Desktop (running) with Docker Compose, Node.js 24 + Corepack. + +```bash +brew install git +brew install --cask docker +brew install libmagic # in advance: an agent that runs Saleor's backend tests on the host needs it, so it does not have to install it or ask +curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.39.7/install.sh | bash +source ~/.nvm/nvm.sh && nvm install 24 && nvm use 24 && corepack enable +docker compose version # must print a version; Docker Desktop includes it +``` + +If `docker compose version` fails (Docker CLI without Docker Desktop, e.g. Colima), install the Compose plugin: + +```bash +brew install docker-compose +mkdir -p ~/.docker/cli-plugins && ln -sfn "$(brew --prefix)/opt/docker-compose/bin/docker-compose" ~/.docker/cli-plugins/docker-compose +docker compose version +``` + +The script also accepts the standalone `docker-compose` binary if the plugin is missing. + +Run `nvm use 24` in every new terminal before the script; it stops with a hint if Node isn't 24. + +### 3.2 Run the setup script + +```bash +B=/rosetta-benchmarks/saleor/scripts/bench.sh +$B setup # once, about 2 minutes (longer the first time Docker builds the image) +$B start # backend API :8000, storefront :3000, dashboard :9000, in the background +$B health # all three must say OK (wait ~1 minute after start) +$B stop # when you're done +``` + +Repos go to `$HOME/saleor-benchmark`. To use another folder, set `BENCH_DIR`, e.g. `BENCH_DIR=~/work/saleor-benchmark $B setup`. Dev server logs are in `$BENCH_DIR/logs/`. + +What `setup` does: + +1. Downloads the three repos at pinned commits that work together (current `main` branches don't): `saleor` 3.23.41 (`28a1227b…`), `saleor-dashboard` 3.23.39 (`bc75e988…`), `react-storefront` Paper (`05641e1f…`). Don't use the deprecated `saleor/saleor-storefront`. +2. Removes Saleor's own AI instructions and adds our rules file (3.3), commits, and tags `baseline-v1` in all three repos. +3. Builds and starts the backend in Docker, runs migrations and `populatedb --createsuperuser`. +4. Puts the shared database `test-library/saleor-seed.dump` into `$BENCH_DIR/seed.dump` and restores it, so every run of every operator starts from exactly the same data (`populatedb` makes random data, so a dump made on each machine would differ). `setup`, `from-zips` and `reset` all check this: a different local `seed.dump` is moved to `$BENCH_DIR/logs/seed.dump.own-