Add Saleor benchmark: plan, experiment matrix, baseline rules, setup … by olepoletaiev-gd · Pull Request #369 · griddynamics/rosetta

olepoletaiev-gd · 2026-10-15T15:44:41Z

What

Benchmark plan for comparing AI coding methods on a real e-commerce platform (Saleor: core API, dashboard, storefront).

7 methods × 2 runners × 2 tasks = 28 runs

Methods: Baseline, Grill-with-docs, Superpowers, Spec Kit, AWS AI-DLC, Rosetta, Rosetta Light (pinned versions)
Runners: Claude Code (Sonnet 5.5, medium), GitHub Copilot (GPT-6.1 Sol, medium)
Tasks: small (product storage conditions) and large (age-restricted products + customer birthday)

Files

rosetta-benchmarks/BENCHMARKS.md: full plan: methods, runners, setup, tasks with hidden questions and answers, run steps, scoring, judge prompts
rosetta-benchmarks/saleor/EXPERIMENTS.md: list of all 28 runs with IDs and status
rosetta-benchmarks/saleor/baseline-instructions.md: simple team rules file, the same start point for every method
rosetta-benchmarks/saleor/scripts/bench.sh: one script for setup, start, health check and reset (code + database) before each run
rosetta-benchmarks/saleor/diffs/: folder for run results
Removes the empty docs/BENCHMARKS.md placeholder

Key points

Every method runs in isolation, with no help from other AI tools or plugins
All methods start from the same baseline-v1 tag and the same database snapshot
Hidden questions are never in the prompt; we check whether the AI raises them on its own
Scoring: follow-ups, hidden-question coverage, time, usage, and a blinded AI judge (8 categories, 1–5)

Tested

bench.sh setup from scratch: clones and pins the 3 repos, removes Saleor's AI files, adds the rules file, tags baseline-v1, seeds the database, saves seed.dump
start / health: API, storefront and dashboard all OK
reset baseline-v1: code and database changes fully reverted
stop: all containers stopped

Not done yet

Installing each method into its own environment (method tags)
The 28 runs themselves

…script

isolomatov-gd · 2026-10-19T12:20:43Z

Converted to the draft so that we don't accidentally merge work in progress

isolomatov-gd · 2026-10-19T12:33:11Z

Once everything is ready, please rename rosetta-benchmarks to benchmarks.

- Run report templates (small/large) with a data block, metric definitions and measurement rules; one runs/<task>/<runner>/<method>/ folder per run (replaces diffs/, protocols/, usage/); pilot runs go to <method>.pilot/ - Common Claude Code settings for every method's config dir (claude.ai plugin/skill/connector sync off); isolation check now checks clean repos and gives a PASS/FAIL result - Grill-with-docs install and run guide; extra-answers table for questions outside the hidden-question tables - bench.sh: remove nested agent instruction files, regenerate the dashboard schema once into baseline-v1, clean the workspace root on reset, start dev servers without AI-agent env vars - session-usage.py: token breakdown of one Claude Code session - Contacting the Rosetta team applies to Rosetta runs only Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Pilot on the old baseline (nested upstream AI files still present, Node 20 in the agent shell), so it is not a result; S-CC-02 is re-run on the fixed baseline. Raw session logs stay local (git-ignored). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

- Baseline install and run guide (methods/baseline.md): empty config dir with the common settings, rules-file check, exact texts to paste (task prompt, hidden-question check, post-implementation prompt), and how to make and check the run's files (diffs, tokens.md, plan.md) - Baseline starts in the runner's built-in plan mode: Claude Code `--allow-dangerously-skip-permissions --permission-mode plan`, plan approved with "Yes, and bypass permissions"; Copilot: the chat's Plan mode - Same task prompt, no entry command; the plan approval and any later "Approved, continue." count as method_steps, not follow-ups - Marked in the methods table, permissions, 3.3, method steps, run folder layout, EXPERIMENTS.md, the method_steps definition and both run report templates - New optional run file plan.md (Baseline: the approved plan, if the runner saved it) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

- bench.sh setup: run the dashboard's `pnpm run generate` after fetch-local-schema, so baseline-v1 holds the generated types that `predev` writes on every start - Fixes a dirty saleor-dashboard (4 src/graphql/*.generated.ts files) after the first `start` on a fresh setup; it only worked before on machines where `start` had already run Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

- Common Claude Code settings: the built-ins agents-md (reads AGENTS.md only where a repo has no CLAUDE.md) and diff (the diff panel) stay on; they are part of the runner, not a method - Baseline and Grill-with-docs guides: /plugin check expects built-in plugins switched off except agents-md and diff Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

… tools Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

- saleor/methods/<method>.md for all 7 methods: the method's own commands (with doc sources), setup, and each run step by step (small/large task x Claude Code/Copilot), with the exact text to send at every stop - Superpowers starts with /superpowers:brainstorming (team decision, for repeatable runs); Rosetta and Rosetta Light pinned to 3.1.17 (stated exception to the two-week rule); Copilot model GPT-5.6 Terra - One folder per run ID; a re-run replaces it (git keeps the old one). The earlier Grill run is counted as S-CC-02, with its deviations noted - Post-implementation prompt writes to ~/saleor-benchmark/run-output/, copied into the run folder by the operator - scripts/collect-results.py: one summary table from all run reports; scripts/check-runbooks.py: runs the method guides' commands in a sandbox - Common settings keep hooks on (Superpowers, Rosetta and AI-DLC need them) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…ixes From the Baseline branch (PR #371): - S-CC-01 Baseline run on Claude Code (plan mode), counted as done - bench.sh setup also runs the dashboard's `pnpm run generate`, so baseline-v1 holds the generated types `predev` writes on every start - plan.md: the approved Baseline plan is saved in the run folder - /context note for the isolation check; built-ins agents-md and diff stay on Applied to all 7 method guides: - Claude Code pinned to 2.1.305 for every run - step 2 moves the method's tag to the current baseline-v1 before reset - isolation check explains the bundled skills / IDE tools in /context - /usage at T1 and T2, /context at T2 (context_end_pct) - how to resume a closed session for the post-implementation prompt - Baseline: plan approval option "Yes, and bypass permissions" (as shown in 2.1.305); small-task hidden-question check uses the wording of S-CC-01 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…, Grill-with-docs) - reports/preliminary-results.md: the 2 runs done so far, side by side, with caveats (n = 1, no judge scores yet, setup differences, IDE terminal) - reports/summary.md and reports/runs.csv generated by scripts/collect-results.py Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…nswers - variable checks print ERROR: instead of "!!" (zsh history expansion) - AI-DLC: answers for "Plan size" and "Learnings" stops Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…large task) Benchmark results for the Superpowers method (obra/superpowers v6.4.2) on the small task (S-CC-03, product storage conditions) and the large task (L-CC-03, age-restricted products), Claude Code, Sonnet 5.5, medium effort. Per run: report.md (data block, timeline, hidden questions, notes), the code as <repo>.diff files, tokens.md, and EXPERIMENTS.md status. The small run also has the agent's own report, isolation check, patches and the spec and plan the method wrote. Both runs used the older setup and the plain task prompt (no entry command); the operator checks and scores are not filled in, and the small run's token report is reduced (built from /cost). See section 8 of each report. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

- preliminary-results.md: Superpowers column, reading notes and caveats - summary.md / runs.csv regenerated (S-CC-03, L-CC-03) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Small task (product storage conditions), Rosetta 3.1.17 coding-flow, Claude Code 2.1.305, Sonnet 5.5 medium. T0 11:32, T1 13:32, T2 14:00; $21.30 at T1, $21.37 at T2; 17 subagents; 3 gates, 0 follow-ups. Hidden questions 3 asked / 4 covered / 0 missed; 8/8 Done-when checks; operator score 5, merge-ready yes; 1 red flag (dev-DB writes outside the approved snapshot). Section 8 item 11 (operator messages before T1) awaits a team decision. - run folder: report, agent report, diffs, tokens.md, isolation check, Rosetta's plans/ and agents/TEMP/ files (session logs stay local) - BENCHMARKS.md: Task 1 extra answers 11-18 first asked in S-CC-06 - EXPERIMENTS.md: S-CC-06 done Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…ary results - runs/.../spec-kit: report, diffs vs speckit-v1, tokens.md, agent-report, spec artifacts - preliminary-results.md: Spec Kit and Rosetta (S-CC-06) columns, notes and caveats - Superpowers S-CC-03: tests passed (operator) - summary.md / runs.csv regenerated; EXPERIMENTS.md: S-CC-04 done Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…review) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…w under review)" This reverts commit ed7a46f.

… time Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…ing; Rosetta Light method rebuilt Rosetta Light now follows the Rosetta Skills process: skills hitl + tech-specs and "USE ROSETTA SUBAGENTS." in the T0 prompt, no /coding-flow, start from the two zips. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

…requisite; extra answers 20-26 Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

…uides, operator does), time asked at each step, gap-free time Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

…h.sh from-zips not-initialized test-library/saleor-not-initialized-repos.zip is saleor-initialized-repos.zip reset to baseline-v1 in all three repos, with the rosetta*-v1 tags, the init commit and its objects removed. Non-Rosetta methods can start from it instead of cloning. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

populatedb makes random data, so every machine had its own seed.dump and runs by different operators started from different databases. test-library/saleor-seed.dump (pinned Saleor 3.23.41, 29 products, 20 customers, 20 orders, admin) is now the only seed: setup, from-zips and reset put it into $BENCH_DIR/seed.dump and move a different local dump to logs/. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…, never copies skills Claude Code: mattpocock-skills@claude-plugins-official (the marketplace pins it, 49dd158 / 1.3.1 on 2026-10-22) instead of a local marketplace at 447ca70. Copilot: the README's own installer (npx skills@latest add mattpocock/skills, all skills) instead of copying 3 skills by hand. S-CC-02's report notes it used the local marketplace at 447ca70. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

… rounds 1-2) Ten answers to questions not in the hidden-question table, decided by the operator. The operator's visibility rule from this run (hide restricted products from under-age customers) contradicts hidden #5 and is not added here; it is a task variant noted in the L-CC-02 report only. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Operator decisions that don't depend on this run's visibility variant: site-settings field for the minimum age, drafts with a linked Customer without birthday, audit via CustomerEvent, eligible-flag reads logged, who sees the eligible flag. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…tup on 1.3.1 Extra answers on the birthday audit events. Grill-with-docs part 7: the one-time /setup-matt-pocock-skills on mattpocock-skills 1.3.1 @ 49dd158 (2026-10-22 12:08-12:17, $0.1077) replaces the 1.2.3 setup, with its token breakdown. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…, task variant) Report, diffs, agent report, token breakdown and workspace-root files of the large-task Grill-with-docs run (mattpocock-skills 1.3.1 from claude-plugins-official). Gap-free time to first done 53 min, $9.46 / $10.17, 2 follow-ups, hidden 6 asked / 3 covered / 0 missed. The operator replaced hidden key #5 with an own visibility rule, so the run is a task variant and not comparable with other large runs (report section 8). Operator score, merge-ready and done-checks count are still pending. preliminary-results.md gets a large-task section; summary.md and runs.csv regenerated. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…in code Assistant's checks at the operator's request: support-staff visibility (temporary manage_users staff user), under-age product view, migrations, age boundary and birthday privacy in code. Operator score and merge-ready still pending. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

All Claude Code runs in one table (7 small: Baseline, Grill-with-docs, Superpowers, Spec Kit, Rosetta, Rosetta Light, Rosetta Skills; 2 large: Grill-with-docs variant, Superpowers), numbers from each report's data block; per-task detail tables kept below. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…etta Skills Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

…tocol updates - Run folder: report, diffs, agent report, tokens, workspace-root files - Large task prompts for Rosetta Skills and Rosetta Light now name the orchestration skill - BENCHMARKS.md: assistant role (does setup itself, session log is the authority for stops and waits, full key texts, spec-review corrections, hiding only before T1), cumulative /cost, flaky tests, new extra answers 19-34 for the large task Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

…ize (L-CC-02) /cost counts only Edit/Write tool lines; L-CC-02 wrote its code through Bash and subagents, so /cost shows 18 lines against ~3 976 added lines in 131 files in the diffs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…diffs, tokens - run folder: report, agent-report, diffs for saleor/saleor-dashboard/react-storefront, tokens.md, isolation.txt, workspace-root specs - EXPERIMENTS.md: L-CC-04 set to done - BENCHMARKS.md: extra answer 35 (shop time zone), first asked in L-CC-04 Co-Authored-By: Claude Haiku 5.5 <noreply@anthropic.com>

Numbers from Rustem's L-CC-04 report data block and section 8 (gap-free time). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

- Run folder: report, plan, isolation check, diffs, agent report, tokens - EXPERIMENTS.md: L-CC-01 done - Spec review rule does not apply to Baseline (BENCHMARKS.md §5, methods/baseline.md) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…ified_in_code Co-Authored-By: Claude Haiku 5.5 <noreply@anthropic.com>

…table format fix

Signed-off-by: isolomatov-gd <isolomatov@griddynamics.com>

…ocol updates - Run folder: report, diffs, agent report, tokens, workspace-root files - BENCHMARKS.md: extra answers 35-41 for the large task (agent's recommended option for questions in no table), spec/plan review rule (read documents in full, corrections only for partly covered items), cumulative /cost, flaky tests, start sequence (cleanup once, new session), assistant role - Templates: hidden_matches_answer now counts the original (first-pass) result Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

Add Saleor benchmark: plan, experiment matrix, baseline rules, setup …

e4cff82

…script

olepoletaiev-gd marked this pull request as ready for review October 16, 2026 08:59

olepoletaiev-gd requested a review from isolomatov-gd as a code owner October 16, 2026 08:59

github-actions Bot mentioned this pull request Oct 16, 2026

[SECURITY ALERT] Suspicious activity detected in GitHub pull_request_target #369 #370

Open

isolomatov-gd previously approved these changes Oct 19, 2026

View reviewed changes

Comment thread rosetta-benchmarks/BENCHMARKS.md Outdated

Comment thread rosetta-benchmarks/BENCHMARKS.md Outdated

Update rosetta-benchmarks/BENCHMARKS.md

1b7f0a5

isolomatov-gd dismissed their stale review via 1b7f0a5 October 19, 2026 12:19

Update rosetta-benchmarks/BENCHMARKS.md

c49e027

isolomatov-gd marked this pull request as draft October 19, 2026 12:20

olepoletaiev-gd and others added 11 commits October 20, 2026 13:10

bench(saleor): isolation check explains extra /context skills and IDE…

8b28eff

… tools Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

bench(saleor): S-CC-01 Baseline run on Claude Code

51e8822

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

bench(saleor): method guides: zsh-safe variable checks, AI-DLC stop a…

fea80b7

…nswers - variable checks print ERROR: instead of "!!" (zsh history expansion) - AI-DLC: answers for "Plan size" and "Learnings" stops Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

olepoletaiev-gd force-pushed the benchmarks/saleor-v2 branch from e706935 to fea80b7 Compare October 21, 2026 12:24

anagavrilita and others added 7 commits October 21, 2026 15:31

bench(saleor): add Superpowers S-CC-03 to preliminary results

32d1c58

- preliminary-results.md: Superpowers column, reading notes and caveats - summary.md / runs.csv regenerated (S-CC-03, L-CC-03) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

bench(saleor): preliminary results without Rosetta (entry flow under …

ed7a46f

…review) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Revert "bench(saleor): preliminary results without Rosetta (entry flo…

e443a14

…w under review)" This reverts commit ed7a46f.

bench(saleor): preliminary results: times in hours, Spec Kit operator…

86b30b3

… time Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

isolomatov-gd and others added 30 commits October 21, 2026 15:38

bench(saleor): S-CC-07 Rosetta Light run on Claude Code; libmagic pre…

4e4e1e7

…requisite; extra answers 20-26 Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

bench(saleor): protocol for working with an AI assistant (assistant g…

9a2d9e6

…uides, operator does), time asked at each step, gap-free time Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

bench(saleor): L-CC-02 operator score 4, not merge-ready

1c417d4

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

bench(saleor): preliminary results intro and next steps include L-CC-02

c023588

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

bench(saleor): preliminary results: full 32-cell table, 9 runs filled

15579b5

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

bench(saleor): preliminary results: list only runs that exist

1e8dfd1

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

bench(saleor): small-task detail table includes Rosetta Light and Ros…

20c3e66

…etta Skills Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

bench(saleor): templates README: gap-free run time rule

eda5acf

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

bench(saleor): preliminary results: add L-CC-08 (Rosetta Skills, large)

c07a498

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

bench(saleor): preliminary results: add L-CC-04 (Spec Kit, large)

c547645

Numbers from Rustem's L-CC-04 report data block and section 8 (gap-free time). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

bench(saleor): L-CC-04 report: red flags, strong and weak points, ver…

1d6c6b7

…ified_in_code Co-Authored-By: Claude Haiku 5.5 <noreply@anthropic.com>

bench(saleor): L-CC-04 report: Done-when item 10 passed (API check), …

831b074

…table format fix

still running light

a70f60b

Signed-off-by: isolomatov-gd <isolomatov@griddynamics.com>

bench(saleor): L-CC-07 report: red flags, strong and weak points

9b3ebcf

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

bench(saleor): L-CC-08 report: hidden_matches_answer 8 (original result)

364137a

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

Navigation Menu

Repository navigation

Add Saleor benchmark: plan, experiment matrix, baseline rules, setup … - #369

Add Saleor benchmark: plan, experiment matrix, baseline rules, setup …#369
olepoletaiev-gd wants to merge 60 commits into
mainfrom
benchmarks/saleor-v2

olepoletaiev-gd commented Oct 15, 2026

Uh oh!

Uh oh!

Uh oh!

isolomatov-gd commented Oct 19, 2026 •

edited

Loading

Uh oh!

isolomatov-gd commented Oct 19, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

olepoletaiev-gd commented Oct 15, 2026

What

Files

Key points

Tested

Not done yet

Uh oh!

Uh oh!

Uh oh!

isolomatov-gd commented Oct 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

isolomatov-gd commented Oct 19, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

isolomatov-gd commented Oct 19, 2026 •

edited

Loading