Repository navigation

Add Saleor benchmark: plan, experiment matrix, baseline rules, setup … - #369

Draft

olepoletaiev-gd wants to merge 60 commits into

mainfrom

benchmarks/saleor-v2

Draft

Add Saleor benchmark: plan, experiment matrix, baseline rules, setup …#369
olepoletaiev-gd wants to merge 60 commits into
mainfrom
benchmarks/saleor-v2

Commits on Oct 15, 2026
- Add Saleor benchmark: plan, experiment matrix, baseline rules, setup script
  olepoletaiev-gd
  committed
  e4cff82
Commits on Oct 19, 2026
- Update rosetta-benchmarks/BENCHMARKS.md
  isolomatov-gd
  authored
  1b7f0a5
- Update rosetta-benchmarks/BENCHMARKS.md
  isolomatov-gd
  authored
  c49e027
Commits on Oct 20, 2026
- bench(saleor): run report templates, Grill-with-docs guide, setup fixes
  
  olepoletaiev-gd
  and
  claude
  committed
  89f4df1
- bench(saleor): S-CC-02 Grill-with-docs pilot run
  
  olepoletaiev-gd
  and
  claude
  committed
  7489f9c
- bench(saleor): Baseline guide, Baseline runs start in plan mode
  
  nina.jakovljevic
  and
  claude
  committed
  f653a41
- bench(saleor): setup puts dashboard codegen output into baseline-v1
  
  nina.jakovljevic
  and
  claude
  committed
  8a9bf1b
- bench(saleor): built-in plugins agents-md and diff stay on
  
  nina.jakovljevic
  and
  claude
  committed
  c924b07
Commits on Oct 21, 2026
- bench(saleor): isolation check explains extra /context skills and IDE tools
  
  nina.jakovljevic
  and
  claude
  committed
  8b28eff
- bench(saleor): S-CC-01 Baseline run on Claude Code
  
  nina.jakovljevic
  and
  claude
  committed
  51e8822
- bench(saleor): one guide per method, run collector, Rosetta 3.1.17
  
  olepoletaiev-gd
  and
  claude
  committed
  59b702e
- Merge benchmarks/saleor-v2-baseline: S-CC-01 Baseline run and setup fixes
  
  olepoletaiev-gd
  and
  claude
  committed
  9a26e17
- bench(saleor): preliminary results (Claude Code, small task: Baseline, Grill-with-docs)
  
  olepoletaiev-gd
  and
  claude
  committed
  1bfba82
- bench(saleor): method guides: zsh-safe variable checks, AI-DLC stop answers
  
  olepoletaiev-gd
  and
  claude
  committed
  fea80b7
- bench(saleor): Superpowers benchmark results (Claude Code, small and large task)
  
  anagavrilita
  and
  claude
  committed
  02a6cb4
- bench(saleor): add Superpowers S-CC-03 to preliminary results
  
  olepoletaiev-gd
  and
  claude
  committed
  32d1c58
- bench(saleor): S-CC-06 Rosetta run on Claude Code
  
  nina.jakovljevic
  and
  claude
  committed
  61515e2
- bench(saleor): S-CC-04 Spec Kit run; Rosetta and Spec Kit in preliminary results
  
  olepoletaiev-gd
  and
  claude
  committed
  0c2ba89
- bench(saleor): preliminary results without Rosetta (entry flow under review)
  
  olepoletaiev-gd
  and
  claude
  committed
  ed7a46f
- Revert "bench(saleor): preliminary results without Rosetta (entry flow under review)"
  olepoletaiev-gd
  committed
  e443a14
- bench(saleor): preliminary results: times in hours, Spec Kit operator time
  
  olepoletaiev-gd
  and
  claude
  committed
  86b30b3
- Fixes and improvements
  isolomatov-gd
  committed
  b602a8c
- Multiple fixes across the board
  isolomatov-gd
  committed
  71663d3
- bench(saleor): Spec Kit method: restore original constitution text, add specify init row
  
  isolomatov-gd
  and
  claude
  committed
  78b38c0
- bench(saleor): Rosetta methods start from test-library/saleor-initialized.zip
  
  isolomatov-gd
  and
  claude
  committed
  04bee04
- bench(saleor): Rosetta starting point = two zips (repos .git + root files), bench.sh from-zips
  
  isolomatov-gd
  and
  claude
  committed
  400b621
- Fix instructions
  isolomatov-gd
  committed
  671f98a
- bench(saleor): S-CC-08 first attempt (user-profile Claude, discarded): round 1 questions recorded
  
  isolomatov-gd
  and
  claude
  committed
  f7fdedf
- bench(saleor): S-CC-08 Rosetta Skills run on Claude Code; follow-up rules; init cost
  
  isolomatov-gd
  and
  claude
  committed
  cd20a66
- bench(saleor): S-CC-08 report: weak point, operator confirms; fix rosetta-skills name in collect-results
  
  isolomatov-gd
  and
  claude
  committed
  34e37c9
- bench(saleor): S-CC-08 report: restore deviations, create-dialog wording; Rosetta Light method rebuilt
  
  isolomatov-gd
  and
  claude
  committed
  0250783
- bench(saleor): S-CC-07 Rosetta Light run on Claude Code; libmagic prerequisite; extra answers 20-26
  
  isolomatov-gd
  and
  claude
  committed
  4e4e1e7
- bench(saleor): protocol for working with an AI assistant (assistant guides, operator does), time asked at each step, gap-free time
  
  isolomatov-gd
  and
  claude
  committed
  9a2d9e6
Commits on Oct 22, 2026
- bench(saleor): saleor-not-initialized-repos.zip (no Rosetta) and bench.sh from-zips not-initialized
  
  olepoletaiev-gd
  and
  claude
  committed
  2535c05
- bench(saleor): one shared seed database for all operators
  
  olepoletaiev-gd
  and
  claude
  committed
  9268fcc
- bench(saleor): Grill-with-docs installs from the official marketplace, never copies skills
  
  olepoletaiev-gd
  and
  claude
  committed
  f779676
- bench(saleor): Task 2 extra answers from L-CC-02 (operator decisions, rounds 1-2)
  
  olepoletaiev-gd
  and
  claude
  committed
  a885f32
- bench(saleor): Task 2 extra answers 11-15 from L-CC-02 (rounds 3-4)
  
  olepoletaiev-gd
  and
  claude
  committed
  e80ac1f
- bench(saleor): Task 2 extra answers 16-18 (L-CC-02 round 5); Grill setup on 1.3.1
  
  olepoletaiev-gd
  and
  claude
  committed
  94c5bf8
- bench(saleor): L-CC-02 Grill-with-docs run on Claude Code (large task, task variant)
  
  olepoletaiev-gd
  and
  claude
  committed
  b8fdd13
- bench(saleor): L-CC-02 checks completed: 12/12 done-when, hidden 8/8 in code
  
  olepoletaiev-gd
  and
  claude
  committed
  21c3d07
- bench(saleor): L-CC-02 operator score 4, not merge-ready
  
  olepoletaiev-gd
  and
  claude
  committed
  1c417d4
- bench(saleor): preliminary results intro and next steps include L-CC-02
  
  olepoletaiev-gd
  and
  claude
  committed
  c023588
- bench(saleor): preliminary results: one table with all 9 runs
  
  olepoletaiev-gd
  and
  claude
  committed
  a31359c
- bench(saleor): preliminary results: full 32-cell table, 9 runs filled
  
  olepoletaiev-gd
  and
  claude
  committed
  15579b5
- bench(saleor): preliminary results: list only runs that exist
  
  olepoletaiev-gd
  and
  claude
  committed
  1e8dfd1
- bench(saleor): small-task detail table includes Rosetta Light and Rosetta Skills
  
  olepoletaiev-gd
  and
  claude
  committed
  20c3e66
- bench(saleor): templates README: gap-free run time rule
  
  isolomatov-gd
  and
  claude
  committed
  eda5acf
Commits on Oct 23, 2026
- bench(saleor): L-CC-08 run (Rosetta Skills, large, Claude Code) + protocol updates
  
  isolomatov-gd
  and
  claude
  committed
  81a237b
- bench(saleor): note that /cost code-change lines are not the change size (L-CC-02)
  
  olepoletaiev-gd
  and
  claude
  committed
  e34a9f8
- bench(saleor): preliminary results: add L-CC-08 (Rosetta Skills, large)
  
  olepoletaiev-gd
  and
  claude
  committed
  c07a498
- bench(saleor): L-CC-04 Spec Kit large run (age restriction), report, diffs, tokens
  
  fxmidaaa
  and
  claude
  committed
  7ade6cf
- bench(saleor): preliminary results: add L-CC-04 (Spec Kit, large)
  
  olepoletaiev-gd
  and
  claude
  committed
  c547645
- bench(saleor): L-CC-01 run (Baseline, large, Claude Code)
  
  nina.jakovljevic
  and
  claude
  committed
  bd81a25
- bench(saleor): L-CC-04 report: red flags, strong and weak points, verified_in_code
  
  fxmidaaa
  and
  claude
  committed
  1d6c6b7
- bench(saleor): L-CC-04 report: Done-when item 10 passed (API check), table format fix
  fxmidaaa
  committed
  831b074
- still running light
  isolomatov-gd
  committed
  a70f60b
Commits on Oct 24, 2026
- bench(saleor): L-CC-07 run (Rosetta Light, large, Claude Code) + protocol updates
  
  isolomatov-gd
  and
  claude
  committed
  d1c91bf
- bench(saleor): L-CC-07 report: red flags, strong and weak points
  
  isolomatov-gd
  and
  claude
  committed
  9b3ebcf
- bench(saleor): L-CC-08 report: hidden_matches_answer 8 (original result)
  
  isolomatov-gd
  and
  claude
  committed
  364137a

Commits

Commits on Oct 15, 2026

Commits on Oct 19, 2026

Commits on Oct 20, 2026

Commits on Oct 21, 2026

Commits on Oct 22, 2026

Commits on Oct 23, 2026

Commits on Oct 24, 2026