Repository navigation
chore: add a development harness for agents working on Tsim #281
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
12 commits
Select commit
Hold shift + click to select a range
6b9c5dd
chore: add a development harness for agents working on Tsim
rafaelha ff34684
chore: take benchmark circuits and their postselection from tsim-bench
rafaelha 521f5d4
chore: compare sides with their own threads, device or strategy in ts…
rafaelha 04413e4
chore: review fixes for the tsim-bench harness
rafaelha 24575dd
chore: hide development skills from the skill installer
rafaelha b88563b
feat(dev): `tsim-dev bench --width` compares active widths without sa…
rafaelha bbf11ef
chore(dev): remove the GPU options from tsim-dev bench
rafaelha 9d378be
feat(dev): show and check the active width in shot benchmarks
rafaelha e70e43d
docs(dev): scale the shipping workflow to the change
rafaelha 749d4d6
fix(dev): pyright errors in the width harness; type-check dev/ in pre…
rafaelha 2a38216
feat(dev): stream `bench --width` results and give the run a deadline
rafaelha 44541ca
fix(dev): address Codex review of the harness
rafaelha File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| ../.claude/skills |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,50 @@ | ||
| #!/usr/bin/env python3 | ||
| """PreToolUse hook for Bash: block two commands that have taken sessions down. | ||
|
|
||
| Exit code 2 blocks the command and shows stderr to the agent. Everything else | ||
| passes. Keep this list short: it is for rules that must hold every time. | ||
| """ | ||
|
|
||
| import json | ||
| import re | ||
| import sys | ||
|
|
||
| COMPUTE = re.compile(r"\b(python3?|pytest|uv run)\b") | ||
| CAPPED = re.compile(r"\b(tsim-dev|systemd-run)\b") | ||
| DETACHED = re.compile(r"\bnohup\b|\bsetsid\b|&\s*(disown\b|$)", re.MULTILINE) | ||
|
|
||
|
|
||
| def verdict(tool_input: dict) -> str | None: | ||
| """Return why the command is blocked, or None to let it run.""" | ||
| command = tool_input.get("command", "") | ||
| if re.search(r"\bpkill\b[^|;&\n]*\s(-\w*f|--full\b)", command): | ||
| return ( | ||
| "`pkill -f` also matches this shell's own command line and kills it. " | ||
| "Stop a job with `uv run tsim-dev job stop NAME`, or kill by PID." | ||
| ) | ||
| background = tool_input.get("run_in_background") or DETACHED.search(command) | ||
| if background and COMPUTE.search(command) and not CAPPED.search(command): | ||
| return ( | ||
| "Background compute needs a hard memory cap; uncapped jobs have used " | ||
| "55 GB and killed every session on this machine. Start it with " | ||
| "`uv run tsim-dev job start NAME -- <command>` and follow it with " | ||
| "`uv run tsim-dev job wait NAME`." | ||
| ) | ||
| return None | ||
|
|
||
|
|
||
| def main() -> int: | ||
| """Read the hook payload from stdin and decide.""" | ||
| try: | ||
| payload = json.load(sys.stdin) | ||
| except ValueError: | ||
| return 0 | ||
| reason = verdict(payload.get("tool_input") or {}) | ||
| if reason: | ||
| print(reason, file=sys.stderr) | ||
| return 2 | ||
| return 0 | ||
|
|
||
|
|
||
| if __name__ == "__main__": | ||
| sys.exit(main()) | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,35 @@ | ||
| { | ||
| "permissions": { | ||
| "allow": [ | ||
| "Bash(uv run tsim-dev *)", | ||
| "Bash(uv sync)", | ||
| "Bash(git status *)", | ||
| "Bash(git diff *)", | ||
| "Bash(git log *)", | ||
| "Bash(git show *)", | ||
| "Bash(gh pr view *)", | ||
| "Bash(gh pr diff *)", | ||
| "Bash(gh pr checks *)", | ||
| "Bash(gh issue view *)", | ||
| "Bash(gh run view *)" | ||
| ], | ||
| "ask": [ | ||
| "Bash(git push *)", | ||
| "Bash(gh pr create *)", | ||
| "Bash(gh pr merge *)" | ||
| ] | ||
| }, | ||
| "hooks": { | ||
| "PreToolUse": [ | ||
| { | ||
| "matcher": "Bash", | ||
| "hooks": [ | ||
| { | ||
| "type": "command", | ||
| "command": "python3 \"$CLAUDE_PROJECT_DIR/.claude/hooks/guard_bash.py\"" | ||
| } | ||
| ] | ||
| } | ||
| ] | ||
| } | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,138 @@ | ||
| --- | ||
| name: benchmarking-tsim | ||
| description: Measures Tsim throughput and compile time with `tsim-dev bench` on the tsim-bench circuits and compares a change against a reference commit or checkout. Use when asked to benchmark Tsim, to check a change for performance regressions, to produce before and after numbers for a pull request, to run the tsim-bench corpus, or to add a benchmark circuit. | ||
| metadata: | ||
| internal: true | ||
| --- | ||
|
|
||
| # Benchmarking Tsim | ||
|
|
||
| Use `uv run tsim-dev bench`. Do not write timing scripts: hand-written ones have | ||
| measured the wrong thing often enough (compile time included, a cold numba cache, | ||
| several threads, a busy machine) that their numbers are not accepted in reviews. | ||
|
|
||
| ## Check a change against main | ||
|
|
||
| ```bash | ||
| git fetch origin main | ||
| uv run tsim-dev bench --ref origin/main # quick suite, about 4 minutes | ||
| ``` | ||
|
|
||
| Use `origin/main`, not `main`: the local branch is often behind. `--ref` also takes | ||
| any commit, or the path of another checkout that has a `.venv` | ||
| (`--ref ../tsim-other-worktree`). | ||
|
|
||
| ``` | ||
| circuit k shots/s kept compile shots/s vs ref (min, max) compile vs ref | ||
| inject_cultivate_d5_p0.001 6 13.9M 0.141 0.13 1.20x [1.19, 1.21] faster 0.96x same | ||
| ``` | ||
|
|
||
| - `kept` is the fraction of shots that survive postselection (`-` when there is none). | ||
| - `k` is the active width of the compiled sampler. With `--ref`, a row that ends in | ||
| `WIDER: k 6 -> 7` or `NARROWER: ...` means the two sides compiled different widths; | ||
| `wider` fails the run. A width that changes between runs of one side is an error. | ||
|
|
||
| - `shots/s vs ref` is new over reference, so above 1 is faster. The bracket is the | ||
| smallest and largest ratio over the rounds. | ||
| - A verdict of `faster` or `slower` needs the mean outside the tolerance (5%) and | ||
| every round on the same side of 1. `noisy` means the rounds disagree: run again | ||
| with `--rounds 9 --min-seconds 1` before you conclude anything. | ||
| - `kept-differs`: the two sides kept different fractions of shots (beyond binomial | ||
| noise), so they did not sample the same thing and the rate is not compared. It | ||
| counts as a failure: a change to postselection or to the sampled distribution. | ||
| - Exit code 0: nothing slower. 1: at least one `slower` in rate or compile time, a | ||
| `kept-differs`, or a wider active width. 2: a circuit could not be measured. | ||
| - The JSON under `.tsim-dev/bench/` has every round. | ||
|
|
||
| ## Active width only (`--width`) | ||
|
|
||
| For changes to compilation, scheduling or the plan, compare widths without sampling: | ||
|
|
||
| ```bash | ||
| uv run tsim-dev bench --width --ref origin/main # whole corpus, compile only | ||
| uv run tsim-dev bench --width --suite full --ref origin/main | ||
| ``` | ||
|
|
||
| Both sides compile every circuit once, at the same time, in `--jobs` workers (default | ||
| 4). A row is printed as soon as a circuit is done on both sides, small circuits first, | ||
| with an `[n/N]` count, so the first rows appear within seconds; the row shows the width | ||
| next to the one recorded in tsim-bench. The verdict is `wider`, `narrower`, `same` or | ||
| `unknown`; any `wider` exits 1. `--case-timeout` (300 s) stops one slow compile; | ||
| `--timeout` (1800 s here) is the deadline for the whole run, after which the circuits | ||
| not yet compiled are reported as errors. A circuit over Tsim's limit shows `>=N`: N is the width | ||
| in Tsim's refusal, a lower bound. Widths do not depend on the machine, so this needs | ||
| no rounds or quiet machine, but it says nothing about speed: a change that keeps the | ||
| width can still be slower. Compile times are shown for each side. | ||
|
|
||
| ## Compare settings: threads, strategy | ||
|
|
||
| `--side REF[:key=value,...]`, given twice, replaces `--ref` when the two sides differ in | ||
| more than the checkout. REF is `.` (this checkout), a git revision, a checkout path, or | ||
| a Python executable (an environment built elsewhere). Keys: `threads=N`, | ||
| `strategy=NAME`. The first side is "new". | ||
|
|
||
| ```bash | ||
| uv run tsim-dev bench --side .:threads=10 --side .:threads=1 # 10 cores vs 1 | ||
| uv run tsim-dev bench --side .:threads=4 --side origin/main:threads=4 | ||
| ``` | ||
|
|
||
| - Compile time is compared too; when the sides differ only in threads, read the rate | ||
| column. | ||
|
|
||
| ## What the command controls for you | ||
|
|
||
| - One thread, pinned to a core (`--threads N` to change; say so when you report). | ||
| - Each measurement is a fresh process with a warm numba cache, under a memory cap. | ||
| Kernels loaded from the cache have behaved differently from freshly compiled | ||
| ones, and the cached case is what users get. | ||
| - Compile time and the first call are excluded from shots/s and reported apart. | ||
| - The two checkouts alternate round by round, so drift affects both. | ||
| - Circuits come from tsim-bench (https://github.com/rafaelha/tsim-bench) at the commit | ||
| pinned in `dev/tsim_dev/corpus.py`, fetched into `~/.cache/tsim-dev` on first use. | ||
| Each circuit's manifest there records its arXiv paper, its sampler and how the paper | ||
| postselects; the benchmark postselects that way by default. `--no-postselect` | ||
| samples every shot instead. shots/s counts requested shots, kept or not. | ||
|
|
||
| ## What you must do | ||
|
|
||
| - Run one benchmark at a time, and nothing else heavy (tests, other agents' jobs) | ||
| next to it. Heed the busy-machine warning. | ||
| - A change must hold across families. Run `--suite full` before claiming a speedup, | ||
| and report the circuits that got slower as well as the ones that got faster. | ||
| - For a single circuit: `uv run tsim-dev bench inject_cultivate_d5_p0.001 --ref origin/main` | ||
| (a file stem or a full id from `tsim-dev corpus list`), or a path to any `.stim` file | ||
| with `--sampler` and `--postselect all|none`. | ||
| - Postselection changes what is measured: a change that speeds up rejected shots shows | ||
| only with it on. Report which you used, and run both when the change touches the | ||
| postselection path. | ||
| - Paste the table in the pull request, with the CPU model (`lscpu | grep 'Model name'`) | ||
| and the thread count. | ||
|
|
||
| ## Suites and the whole corpus | ||
|
|
||
| Suites are defined in tsim-bench's `suites.toml`: `quick` (the default, a few minutes), | ||
| `full` (one or two circuits per family, widths up to about 16) and `hard` (widths 16 to | ||
| 26, such as the tricycle CCZ factories; minutes per circuit). `--suite corpus` is every | ||
| circuit, over 190 from 28 papers, so run it as a job: | ||
|
|
||
| ```bash | ||
| uv run tsim-dev corpus list --max-width 12 # what would run | ||
| uv run tsim-dev job start corpus --timeout 14400 -- \ | ||
| uv run tsim-dev bench --suite corpus --max-width 12 --ref origin/main | ||
| uv run tsim-dev job wait corpus # repeat until it reports the end | ||
| ``` | ||
|
|
||
| `--max-width K` skips circuits whose recorded active width is larger than K, and those | ||
| with no recorded width; `--width-of clifft` (or another backend in tsim-bench's | ||
| `k_max.json`) filters on that backend's width instead, for backend work that targets a | ||
| different regime. `--only SUBSTRING` keeps circuits whose id contains it. | ||
|
|
||
| ## Limits | ||
|
|
||
| - Tsim on the CPU only. Comparisons with other simulators (Clifft, SymFT, Magna) are | ||
| run from the tsim-bench repository, which pins each simulator's environment. | ||
| - To add a circuit or change a suite, change tsim-bench: put the file in its paper's | ||
| folder, add an entry to that folder's `circuits.toml` (arXiv id, sampler, postselection | ||
| and the reason for it, see its `BENCHMARKS.md`), run `python -m tsim_bench check` | ||
| there, then bump `REV` in `dev/tsim_dev/corpus.py`. Until the commit is pushed, point | ||
| the harness at the checkout with `--corpus-dir` or `TSIM_BENCH_DIR`. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,63 @@ | ||
| --- | ||
| name: profiling-tsim | ||
| description: Finds where Tsim spends compile and sampling time with `tsim-dev profile` and `tsim-dev inspect`, and turns a profile into an optimization plan. Use when asked why a circuit is slow, where time goes, what to optimize next, or before starting performance work on Tsim. | ||
| metadata: | ||
| internal: true | ||
| --- | ||
|
|
||
| # Profiling Tsim | ||
|
|
||
| ## 1. Size the circuit first | ||
|
|
||
| ```bash | ||
| uv run tsim-dev inspect cultivation_d5 # or a path to any .stim file | ||
| ``` | ||
|
|
||
| It prints qubits, T count, detectors, `active_width` (k) and `SamplerStats` without | ||
| sampling. Cost per shot grows like 2^k. Do not sample or profile above k ≈ 20 without | ||
| a smaller `--batch-size` and a memory estimate (2^k × 16 B × batch). | ||
|
|
||
| ## 2. Profile one run | ||
|
|
||
| ```bash | ||
| uv run tsim-dev profile cultivation_d5 --seconds 5 | ||
| ``` | ||
|
|
||
| The run is a fresh process, one pinned thread, warm numba cache, under a memory cap. | ||
| The output splits samples by phase (`compile`, `sample`) and prints three tables: | ||
|
|
||
| - **innermost Tsim frame (kernel callers):** read this first. Jitted numba code has | ||
| no symbols, so time inside a kernel is charged to the Python function that called | ||
| it. On cultivation_d5 at 1.0.0 this shows 82% in `sample_words` | ||
| (`noise/channels.py`) and 14% in `_run_chunk` (`statevector/shots.py`): noise | ||
| sampling, not the state vector, is the cost there. | ||
| - **innermost frame:** separates `[jitted numba code]` from numpy, libc (memcpy, | ||
| malloc) and Python overhead. A large non-jitted share means work outside kernels. | ||
| - **Tsim frames, inclusive:** the call tree, for compile time in particular. | ||
|
|
||
| The folded stacks are saved for flame graphs (`speedscope <file>` or | ||
| `flamegraph.pl`). | ||
|
|
||
| Profile at least two families before choosing a target; the hot spot differs | ||
| between circuits with mostly direct detectors (cultivation, surface codes), wide | ||
| quantum components (quantum volume) and many rotations (coherent noise). | ||
|
|
||
| ## 3. Inside a kernel | ||
|
|
||
| The profile stops at the kernel boundary. To compare variants of a kernel, change | ||
| it and use `uv run tsim-dev bench <circuit> --ref origin/main`: that is the | ||
| measurement that counts. Micro-benchmarks of a kernel alone have misled before | ||
| (cache effects, struct passing by value), so confirm any gain end to end. | ||
|
|
||
| Hardware counters (`perf stat`) need `/proc/sys/kernel/perf_event_paranoid` at 1 or | ||
| lower; ask the user rather than changing it. | ||
|
|
||
| ## 4. Turn it into a plan | ||
|
|
||
| - Write down the target (phase, function, share of time) and the circuits it | ||
| affects, before changing code. | ||
| - Estimate the ceiling: removing a function that takes 14% gives at most 1.16x. | ||
| - Prefer one general change over special cases for particular circuits, and drop | ||
| ideas whose gain is small compared with the code they add. | ||
| - After the change, profile again to confirm the share moved, then follow the | ||
| `benchmarking-tsim` skill for the numbers that go in the pull request. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,84 @@ | ||
| --- | ||
| name: shipping-tsim-changes | ||
| description: Takes a change to Tsim's code from a branch to a reviewed local commit, with the checks, evidence and git rules this repository uses, scaled to what the change touches. Use when starting a feature, bug fix or optimization in the Tsim repository, when updating an existing pull request, or before saying that a change is done. | ||
| metadata: | ||
| internal: true | ||
| --- | ||
|
|
||
| # Shipping a change to Tsim | ||
|
|
||
| ## 1. Start | ||
|
|
||
| **New work.** Work in a worktree on a new branch off the newest main, not in the main | ||
| checkout: `git fetch origin main && git worktree add -b <type>/<topic> ../tsim-<topic> | ||
| origin/main`, then `uv sync` there. Stack on another branch only when the change needs it. | ||
|
|
||
| **Updating an existing pull request** (review comments, a follow-up, a fix the user asks | ||
| for on that PR). Keep its branch: find it with `gh pr view <n> --json headRefName`, use | ||
| the worktree that already has it (`git worktree list`) or add one for that branch, and | ||
| commit on top. Do not start a new branch or a new PR for it. Rebase only when the user | ||
| asks or the branch conflicts with its base; a rebase rewrites a pushed branch, so say | ||
| that it needs a force-push. | ||
|
|
||
| **Acceptance criteria.** Infer them from the task and the code, and state them in a line | ||
| or two before you start: the behavior, the test that shows it, and for performance work | ||
| the circuits and the gain that matters. Ask only when they are unclear, for example an | ||
| unstated performance target, two plausible behaviors, or a change to public API. | ||
|
|
||
| ## 2. Build | ||
|
|
||
| - For a bug, write the failing test first and watch it fail. | ||
| - `uv run tsim-dev test --changed` while iterating. | ||
| - Anything long or memory-hungry goes through `uv run tsim-dev run -- <command>` or | ||
| `uv run tsim-dev job start NAME -- <command>`; see AGENTS.md. | ||
|
|
||
| ## 3. Verify | ||
|
|
||
| Run what applies and keep the output. A claim without the command and its result is not | ||
| evidence. | ||
|
|
||
| | Change | Required | | ||
| |---|---| | ||
| | Any | `uv run tsim-dev check` prints PASS for lint and tests | | ||
| | Could affect compile or sampling cost (see below) | `uv run tsim-dev bench --ref origin/main`; any regression investigated and explained | | ||
| | Aims to make Tsim faster | the same with `--suite full`; see the `benchmarking-tsim` skill | | ||
| | Could change the active width (graph, simplification, scheduling, plan) | `uv run tsim-dev bench --width --ref origin/main`; any `wider` explained | | ||
| | Bug fix | the new test fails on `origin/main` and passes on the branch | | ||
| | User-visible | a `CHANGELOG.md` entry under `## [Unreleased]` | | ||
| | Behavior the user skills describe | `skills/` updated; `test/unit/test_skills.py` passes | | ||
|
|
||
| **When to benchmark.** Code on the compile path (parsing, the ZX graph and its | ||
| simplification, components, the plan, scheduling, lowering to the action tape) or the | ||
| sampling path (samplers, noise, the state-vector executor, numba kernels), and changes | ||
| to dependencies or numba settings, can change performance. Docs, tests, type hints, | ||
| error messages, printing and code that runs once per call outside those paths do not. | ||
| When unsure, run the quick suite: it takes a few minutes. | ||
|
|
||
| **A regression is a finding, not a veto.** If the benchmark reports `slower` (exit 1), | ||
| do not stop at the exit code and do not hide it: | ||
|
|
||
| 1. Confirm it: rerun the affected circuits with `--rounds 9 --min-seconds 1`. | ||
| 2. Find the cause: `tsim-dev profile` on one affected circuit, on both checkouts. | ||
| 3. Remove it if the change does not need it. If it does, for example a correctness fix | ||
| that has to do more work, keep it and say so: which circuits, by how much, and why. | ||
| A wrong result is never the faster option. | ||
|
|
||
| Then read your own diff (`git diff origin/main --stat`, then the diff). Remove anything | ||
| the task did not need: special cases for one benchmark, leftover debug code, unrelated | ||
| reformatting. Ask whether a simpler change would do. | ||
|
|
||
| ## 4. Commit and review | ||
|
|
||
| - Commit locally with a Conventional Commits message. | ||
| - Do not push, open a pull request, force-push or merge unless the user asked for that | ||
| step in this conversation. Some branches are not meant to be public yet. When updating | ||
| an existing PR, pushing to its branch still needs the user's go-ahead. | ||
| - For a review, give a fresh subagent the diff and the task, not your conclusions, and | ||
| ask for correctness findings only. Fix what it finds, then run step 3 again. | ||
|
|
||
| ## 5. Report | ||
|
|
||
| Say in a few sentences what changed, the verification commands you ran with their | ||
| results, and what you did not verify. For performance-relevant changes include the bench | ||
| table, the CPU model and the thread count, and the explanation of any regression. Say | ||
| plainly if something failed or was skipped. |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.