Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .agents/skills
42 changes: 42 additions & 0 deletions .claude/agents/tsim-reviewer.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
---
name: tsim-reviewer
description: Reviews a Tsim diff for correctness and performance bugs in a fresh context and reports only findings it has reproduced. Use for each review round of the reviewing-tsim-changes skill.
tools: Read, Grep, Glob, Bash
---

You review a change to Tsim, a sampler for quantum circuits with non-Clifford gates.
The owner will merge this change without reading it, so a bug you miss ships, and a
finding you cannot show wastes a round. Read `AGENTS.md` first for the pipeline and
the invariants.

You are given a diff command, the task the change is meant to do, and findings that
are already known. Judge the change against the task and the invariants; form your
own view of what the code does by reading it and running it.

Look for, in this order:

1. Wrong samples: a changed distribution, a broken invariant from `AGENTS.md` (seeds
independent of threads and `batch_size`, raw parities, direct outputs,
postselection returning only accepted rows), precision loss.
2. Crashes or wrong results on valid input forms the tests do not cover: empty and
single-element cases, zero shots, circuits without detectors or without noise,
merged multi-target instructions, more than 63 outputs, large `batch_size`.
3. Memory or time that grows with shots or 2^k where it did not before.
4. A public API or documented behavior that changed without the task asking for it.
5. Tests that do not test what they claim, or that pass without the change.

For every finding, write a reproducer and run it: a pytest function or a short
script under `.tsim-dev/review/`, run with `uv run tsim-dev run -- uv run python ...`
or `uv run tsim-dev test <file>`. Use small circuits; compare against the dense
references in `test/helpers/`, against Stim on Clifford circuits, or against exact
values. Do not edit files outside `.tsim-dev/review/`, and do not run long jobs.

Report, most severe first, only what you reproduced:

- `file:line`, one sentence stating the defect;
- the input, the expected result and the observed result;
- the path of the reproducer and the command that runs it.

Leave out style, naming, refactoring ideas, missing comments, anything a linter or
the type checker reports, and anything you could not reproduce. If nothing
survives, say "no findings" and list in two or three lines what you checked.
60 changes: 60 additions & 0 deletions .claude/hooks/guard_bash.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
#!/usr/bin/env python3
"""PreToolUse hook for Bash: block commands that have cost sessions or let bugs out.

Exit code 2 blocks the command and shows stderr to the agent. Everything else
passes. Keep this list short: it is for rules that must hold every time.
"""

import json
import re
import sys

COMPUTE = re.compile(r"\b(python3?|pytest|uv run)\b")
CAPPED = re.compile(r"\b(tsim-dev|systemd-run)\b")
MERGE = re.compile(r"\bgh\s+pr\s+merge\b")
GATED = re.compile(
r"\btsim-dev\s+review\s+status\b[^;|\n]*&&[^;|\n]*\bgh\s+pr\s+merge\b"
)
DETACHED = re.compile(r"\bnohup\b|\bsetsid\b|&\s*(disown\b|$)", re.MULTILINE)


def verdict(tool_input: dict) -> str | None:
"""Return why the command is blocked, or None to let it run."""
command = tool_input.get("command", "")
if re.search(r"\bpkill\b[^|;&\n]*\s-\w*f", command):
return (
"`pkill -f` also matches this shell's own command line and kills it. "
"Stop a job with `uv run tsim-dev job stop NAME`, or kill by PID."
)
if MERGE.search(command) and not GATED.search(command):
return (
"Merge only through the gate, from the pull request's branch: "
"`uv run tsim-dev review status && gh pr merge <number> --squash`. "
"Unanswered review comments have been merged past before."
)
background = tool_input.get("run_in_background") or DETACHED.search(command)
if background and COMPUTE.search(command) and not CAPPED.search(command):
return (
"Background compute needs a hard memory cap; uncapped jobs have used "
"55 GB and killed every session on this machine. Start it with "
"`uv run tsim-dev job start NAME -- <command>` and follow it with "
"`uv run tsim-dev job wait NAME`."
)
return None


def main() -> int:
"""Read the hook payload from stdin and decide."""
try:
payload = json.load(sys.stdin)
except ValueError:
return 0
reason = verdict(payload.get("tool_input") or {})
if reason:
print(reason, file=sys.stderr)
return 2
return 0


if __name__ == "__main__":
sys.exit(main())
35 changes: 35 additions & 0 deletions .claude/settings.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
{
"permissions": {
"allow": [
"Bash(uv run tsim-dev *)",
"Bash(uv sync)",
"Bash(git status *)",
"Bash(git diff *)",
"Bash(git log *)",
"Bash(git show *)",
"Bash(gh pr view *)",
"Bash(gh pr diff *)",
"Bash(gh pr checks *)",
"Bash(gh issue view *)",
"Bash(gh run view *)"
],
"ask": [
"Bash(git push *)",
"Bash(gh pr create *)",
"Bash(gh pr merge *)"
]
},
"hooks": {
"PreToolUse": [
{
"matcher": "Bash",
"hooks": [
{
"type": "command",
"command": "python3 \"$CLAUDE_PROJECT_DIR/.claude/hooks/guard_bash.py\""
}
]
}
]
}
}
134 changes: 134 additions & 0 deletions .claude/skills/benchmarking-tsim/SKILL.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,134 @@
---
name: benchmarking-tsim
description: Measures Tsim throughput and compile time with `tsim-dev bench` on the tsim-bench circuits and compares a change against a reference commit or checkout. Use when asked to benchmark Tsim, to check a change for performance regressions, to produce before and after numbers for a pull request, to run the tsim-bench corpus, or to add a benchmark circuit.
metadata:
internal: true
---

# Benchmarking Tsim

Use `uv run tsim-dev bench`. Do not write timing scripts: hand-written ones have
measured the wrong thing often enough (compile time included, a cold numba cache,
several threads, a busy machine) that their numbers are not accepted in reviews.

## Check a change against main

```bash
git fetch origin main
uv run tsim-dev bench --ref origin/main # quick suite, about 4 minutes
```

Use `origin/main`, not `main`: the local branch is often behind. `--ref` also takes
any commit, or the path of another checkout that has a `.venv`
(`--ref ../tsim-other-worktree`).

```
circuit k shots/s kept compile shots/s vs ref (min, max) compile vs ref
inject_cultivate_d5_p0.001 6 13.9M 0.141 0.13 1.20x [1.19, 1.21] faster 0.96x same
```

- `kept` is the fraction of shots that survive postselection (`-` when there is none).
- `k` is the active width of the compiled sampler. With `--ref`, a row that ends in
`WIDER: k 6 -> 7` or `NARROWER: ...` means the two sides compiled different widths;
`wider` fails the run. A width that changes between runs of one side is an error.

- `shots/s vs ref` is new over reference, so above 1 is faster. The bracket is the
smallest and largest ratio over the rounds.
- A verdict of `faster` or `slower` needs the mean outside the tolerance (5%) and
every round on the same side of 1. `noisy` means the rounds disagree: run again
with `--rounds 9 --min-seconds 1` before you conclude anything.
- `kept-differs`: the two sides kept different fractions of shots (beyond binomial
noise), so they did not sample the same thing and the rate is not compared. It
counts as a failure: a change to postselection or to the sampled distribution.
- Exit code 0: nothing slower. 1: at least one `slower` in rate or compile time, a
`kept-differs`, or a wider active width. 2: a circuit could not be measured.
- The JSON under `.tsim-dev/bench/` has every round.

## Active width only (`--width`)

For changes to compilation, scheduling or the plan, compare widths without sampling:

```bash
uv run tsim-dev bench --width --ref origin/main # whole corpus, compile only
uv run tsim-dev bench --width --suite full --ref origin/main
```

Each side compiles every circuit once, in one process, and prints the width next to
the one recorded in tsim-bench. The verdict is `wider`, `narrower`, `same` or
`unknown`; any `wider` exits 1. A circuit over Tsim's limit shows `>=N`: N is the width
in Tsim's refusal, a lower bound. Widths do not depend on the machine, so this needs
no rounds or quiet machine, but it says nothing about speed: a change that keeps the
width can still be slower. Compile times are shown for each side.

## Compare settings: threads, strategy

`--side REF[:key=value,...]`, given twice, replaces `--ref` when the two sides differ in
more than the checkout. REF is `.` (this checkout), a git revision, a checkout path, or
a Python executable (an environment built elsewhere). Keys: `threads=N`,
`strategy=NAME`. The first side is "new".

```bash
uv run tsim-dev bench --side .:threads=10 --side .:threads=1 # 10 cores vs 1
uv run tsim-dev bench --side .:threads=4 --side origin/main:threads=4
```

- Compile time is compared too; when the sides differ only in threads, read the rate
column.

## What the command controls for you

- One thread, pinned to a core (`--threads N` to change; say so when you report).
- Each measurement is a fresh process with a warm numba cache, under a memory cap.
Kernels loaded from the cache have behaved differently from freshly compiled
ones, and the cached case is what users get.
- Compile time and the first call are excluded from shots/s and reported apart.
- The two checkouts alternate round by round, so drift affects both.
- Circuits come from tsim-bench (https://github.com/rafaelha/tsim-bench) at the commit
pinned in `dev/tsim_dev/corpus.py`, fetched into `~/.cache/tsim-dev` on first use.
Each circuit's manifest there records its arXiv paper, its sampler and how the paper
postselects; the benchmark postselects that way by default. `--no-postselect`
samples every shot instead. shots/s counts requested shots, kept or not.

## What you must do

- Run one benchmark at a time, and nothing else heavy (tests, other agents' jobs)
next to it. Heed the busy-machine warning.
- A change must hold across families. Run `--suite full` before claiming a speedup,
and report the circuits that got slower as well as the ones that got faster.
- For a single circuit: `uv run tsim-dev bench inject_cultivate_d5_p0.001 --ref origin/main`
(a file stem or a full id from `tsim-dev corpus list`), or a path to any `.stim` file
with `--sampler` and `--postselect all|none`.
- Postselection changes what is measured: a change that speeds up rejected shots shows
only with it on. Report which you used, and run both when the change touches the
postselection path.
- Paste the table in the pull request, with the CPU model (`lscpu | grep 'Model name'`)
and the thread count.

## Suites and the whole corpus

Suites are defined in tsim-bench's `suites.toml`: `quick` (the default, a few minutes),
`full` (one or two circuits per family, widths up to about 16) and `hard` (widths 16 to
26, such as the tricycle CCZ factories; minutes per circuit). `--suite corpus` is every
circuit, over 190 from 28 papers, so run it as a job:

```bash
uv run tsim-dev corpus list --max-width 12 # what would run
uv run tsim-dev job start corpus --timeout 14400 -- \
uv run tsim-dev bench --suite corpus --max-width 12 --ref origin/main
uv run tsim-dev job wait corpus # repeat until it reports the end
```

`--max-width K` skips circuits whose recorded active width is larger than K, and those
with no recorded width; `--width-of clifft` (or another backend in tsim-bench's
`k_max.json`) filters on that backend's width instead, for backend work that targets a
different regime. `--only SUBSTRING` keeps circuits whose id contains it.

## Limits

- Tsim on the CPU only. Comparisons with other simulators (Clifft, SymFT, Magna) are
run from the tsim-bench repository, which pins each simulator's environment.
- To add a circuit or change a suite, change tsim-bench: put the file in its paper's
folder, add an entry to that folder's `circuits.toml` (arXiv id, sampler, postselection
and the reason for it, see its `BENCHMARKS.md`), run `python -m tsim_bench check`
there, then bump `REV` in `dev/tsim_dev/corpus.py`. Until the commit is pushed, point
the harness at the checkout with `--corpus-dir` or `TSIM_BENCH_DIR`.
63 changes: 63 additions & 0 deletions .claude/skills/profiling-tsim/SKILL.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,63 @@
---
name: profiling-tsim
description: Finds where Tsim spends compile and sampling time with `tsim-dev profile` and `tsim-dev inspect`, and turns a profile into an optimization plan. Use when asked why a circuit is slow, where time goes, what to optimize next, or before starting performance work on Tsim.
metadata:
internal: true
---

# Profiling Tsim

## 1. Size the circuit first

```bash
uv run tsim-dev inspect cultivation_d5 # or a path to any .stim file
```

It prints qubits, T count, detectors, `active_width` (k) and `SamplerStats` without
sampling. Cost per shot grows like 2^k. Do not sample or profile above k ≈ 20 without
a smaller `--batch-size` and a memory estimate (2^k × 16 B × batch).

## 2. Profile one run

```bash
uv run tsim-dev profile cultivation_d5 --seconds 5
```

The run is a fresh process, one pinned thread, warm numba cache, under a memory cap.
The output splits samples by phase (`compile`, `sample`) and prints three tables:

- **innermost Tsim frame (kernel callers):** read this first. Jitted numba code has
no symbols, so time inside a kernel is charged to the Python function that called
it. On cultivation_d5 at 1.0.0 this shows 82% in `sample_words`
(`noise/channels.py`) and 14% in `_run_chunk` (`statevector/shots.py`): noise
sampling, not the state vector, is the cost there.
- **innermost frame:** separates `[jitted numba code]` from numpy, libc (memcpy,
malloc) and Python overhead. A large non-jitted share means work outside kernels.
- **Tsim frames, inclusive:** the call tree, for compile time in particular.

The folded stacks are saved for flame graphs (`speedscope <file>` or
`flamegraph.pl`).

Profile at least two families before choosing a target; the hot spot differs
between circuits with mostly direct detectors (cultivation, surface codes), wide
quantum components (quantum volume) and many rotations (coherent noise).

## 3. Inside a kernel

The profile stops at the kernel boundary. To compare variants of a kernel, change
it and use `uv run tsim-dev bench <circuit> --ref origin/main`: that is the
measurement that counts. Micro-benchmarks of a kernel alone have misled before
(cache effects, struct passing by value), so confirm any gain end to end.

Hardware counters (`perf stat`) need `/proc/sys/kernel/perf_event_paranoid` at 1 or
lower; ask the user rather than changing it.

## 4. Turn it into a plan

- Write down the target (phase, function, share of time) and the circuits it
affects, before changing code.
- Estimate the ceiling: removing a function that takes 14% gives at most 1.16x.
- Prefer one general change over special cases for particular circuits, and drop
ideas whose gain is small compared with the code they add.
- After the change, profile again to confirm the share moved, then follow the
`benchmarking-tsim` skill for the numbers that go in the pull request.
Loading
Loading