From d02d8b51dfb3cb6c1d21bff94e81f3ac692e1922 Mon Sep 17 00:00:00 2001 From: Ruslan Malymon <42340633+ruslan-splynx@users.noreply.github.com> Date: Sun, 18 Oct 2026 21:25:06 +0200 Subject: [PATCH] [P8-00] Consolidate performance recovery evidence and remaining gates --- BACKLOG.md | 700 ++++++++++++++++++++++- ROADMAP.md | 22 + docs/benchmarking.md | 5 + docs/perf/data/p8-00/README.md | 31 + docs/perf/data/p8-00/SHA256SUMS | 10 + docs/perf/data/p8-00/check-1.log | 18 + docs/perf/data/p8-00/check-2.log | 17 + docs/perf/data/p8-00/check-3.log | 30 + docs/perf/data/p8-00/check-4.log | 29 + docs/perf/data/p8-00/check-5.log | 8 + docs/perf/data/p8-00/cli-smoke.txt | 3 + docs/perf/data/p8-00/saved-receipts.json | 44 ++ docs/perf/data/p8-00/status.json | 234 ++++++++ docs/perf/data/p8-00/validation.json | 84 +++ docs/perf/p8-13-public-workflow-gates.md | 2 +- docs/perf/p8-performance-recovery.md | 281 +++++++++ 16 files changed, 1516 insertions(+), 2 deletions(-) create mode 100644 docs/perf/data/p8-00/README.md create mode 100644 docs/perf/data/p8-00/SHA256SUMS create mode 100644 docs/perf/data/p8-00/check-1.log create mode 100644 docs/perf/data/p8-00/check-2.log create mode 100644 docs/perf/data/p8-00/check-3.log create mode 100644 docs/perf/data/p8-00/check-4.log create mode 100644 docs/perf/data/p8-00/check-5.log create mode 100644 docs/perf/data/p8-00/cli-smoke.txt create mode 100644 docs/perf/data/p8-00/saved-receipts.json create mode 100644 docs/perf/data/p8-00/status.json create mode 100644 docs/perf/data/p8-00/validation.json create mode 100644 docs/perf/p8-performance-recovery.md diff --git a/BACKLOG.md b/BACKLOG.md index ed6925bf..15f9bf25 100644 --- a/BACKLOG.md +++ b/BACKLOG.md @@ -1,7 +1,7 @@ # Quantum Simulator — Detailed Backlog > **Source of truth for GitHub Issues.** -> This document contains the mainline phase issues, including Phase 7's cross-track research instruments. Each issue is structured so that an AI agent (e.g., Claude Code) or a human can create a GitHub Issue directly from its contents. The companion QEC backlog is `docs/qec/BACKLOG.md`. +> This document contains the mainline phase issues, including Phase 7's research instruments and Phase 8's performance recovery. Each issue is structured so that an AI agent (e.g., Claude Code) or a human can create a GitHub Issue directly from its contents. The companion QEC backlog is `docs/qec/BACKLOG.md`. ----- @@ -52,6 +52,7 @@ Each issue follows the same template: - Phase 5.5 — Apple/Metal GPU - Phase 6 — Multi-GPU & Distributed - Phase 7 — Research Instruments +- Phase 8 — Performance Audit and Recovery ----- @@ -4864,6 +4865,702 @@ engine, not a universal replacement for state-vector amplitudes or sampling. ----- +# Phase 8 — Performance Audit and Recovery + +Tracker: [#550](https://github.com/aleph-sim/aleph/issues/550). +The [integration report](docs/perf/p8-performance-recovery.md) records the +2026-10-18 snapshot: 13/16 capabilities closed, with P8-01/P8-11/P8-12 open. +This phase remains in progress. Closed task state does not imply default adoption +or a universal speedup; research opt-ins and negative regimes stay explicit. + +The specifications below restore the existing GitHub issues to the local backlog. +Issue links identify the existing items: do not create duplicates. The checkboxes +in each specification are acceptance requirements, not a synchronized status feed. + +----- + +### [P8-00] Performance audit integration and end-to-end recovery report + +**GitHub:** [#550](https://github.com/aleph-sim/aleph/issues/550) — open as of 2026-10-18. + +**Labels:** `area:bench`, `area:docs`, `type:docs`, `priority:high` +**Milestone:** Phase 8 +**Estimate:** L +**Depends on:** P8-01 through P8-16 (for closure; coordination starts immediately) + +**Description** +Track the audit findings, coordinate incremental repairs, and publish an end-to-end report showing which public workflows improved and which hypotheses were rejected. + +**Context** +The October 2026 source audit found gaps between benchmark execution and public frontends, eager full-state downloads, sampling scratch omitted from reach policy, and algorithms that retain avoidable work. Historical negative results identify where another broad rewrite is unlikely to pay. Phase 7 adds research capabilities; Phase 8 makes existing execution reliable and efficient. + +**Technical Details** +- Keep confirmed source findings separate from unmeasured performance hypotheses. The local 12-qubit probe validated GPU-safe fusion semantics (brickwall 295→32 instructions, QFT on a prepared state 90→34, amplitude error <2e-16); these are instruction counts, not measured GPU speedups. +- Coordinate the issue order: P8-08 and P8-01/P8-02 first; P8-03/P8-04 next; P8-05/P8-06/P8-07 for sampling and trajectories. P8-13 supplies measurement discipline throughout. Hardware experiments proceed independently after their baselines are captured. +- Require one focused PR per capability, meaningful oracle/contract checks, and before/after measurements on an idle relevant machine. Avoid new performance claims from the overloaded audit host or unreviewed synthetic-only fixtures. +- Publish `docs/perf/p8-performance-recovery.md` with public API commands, startup/compile/execute/sample/materialize timing, launches, transferred bytes, peak memory, precision, seeds, machine/software identity, and negative results. +- Preserve strong existing practices: sparse algorithms, lazy permutations, pass/traffic reduction, reference implementations, deterministic shots, and numerical fallback guards. Do not remove safety checks to manufacture speed. +- Research tasks may close with a reproducible no-go result under their stated criteria. The phase report must label that result; it must not imply the speed target was reached. + +**Phase Checklist** + +- [ ] [P8-01: Use optimized CPU and Metal execution in the CLI](https://github.com/aleph-sim/aleph/issues/551) +- [x] [P8-02: Integrate CUDA fusion and layered dispatch into public frontends](https://github.com/aleph-sim/aleph/issues/552) +- [x] [P8-03: Materialize Python GPU amplitudes lazily](https://github.com/aleph-sim/aleph/issues/553) +- [x] [P8-04: Plan backend reach from requested operations and peak memory](https://github.com/aleph-sim/aleph/issues/554) +- [x] [P8-05: Replace full-state multi-pass CUDA CDF sampling](https://github.com/aleph-sim/aleph/issues/555) +- [x] [P8-06: Avoid building CPU alias tables for one or few samples](https://github.com/aleph-sim/aleph/issues/556) +- [x] [P8-07: Bound noisy trajectory memory and reuse per-worker state](https://github.com/aleph-sim/aleph/issues/557) +- [x] [P8-08: Probe GPU availability without compiling discarded backends](https://github.com/aleph-sim/aleph/issues/558) +- [x] [P8-09: Coalesce GPU QEC scratch while retaining serial-shot arithmetic](https://github.com/aleph-sim/aleph/issues/559) +- [x] [P8-10: Prototype cooperative GPU BP with explicit accuracy and latency trade-offs](https://github.com/aleph-sim/aleph/issues/560) +- [ ] [P8-11: Prove a competitive Metal MPS factorization before another pipeline rewrite](https://github.com/aleph-sim/aleph/issues/561) +- [ ] [P8-12: Validate communication-aware plans on real multi-GPU workloads](https://github.com/aleph-sim/aleph/issues/562) +- [x] [P8-13: Make public-workflow baselines and experiment gates reproducible](https://github.com/aleph-sim/aleph/issues/563) +- [x] [P8-14: Evaluate combined tile residency and overlapped paging](https://github.com/aleph-sim/aleph/issues/564) +- [x] [P8-15: Bound DEM experiment batches and validate decoder result contracts](https://github.com/aleph-sim/aleph/issues/565) +- [x] [P8-16: Use valid logical-error intervals for zero and rare observed failures](https://github.com/aleph-sim/aleph/issues/566) + +**Acceptance Criteria** +- [ ] Every child has a verified repair or its explicitly allowed, measured research verdict. +- [ ] The report covers at least CPU CLI, CUDA Python counts-only, large-state sampling, and noisy trajectories. +- [ ] Modelled reach, measured allocation limits, benchmark kernel speed, and public API speed are distinguished. +- [ ] Current baselines and historical limitations are linked without treating completed work as missing. + +**Testing Requirements** +- Smoke-run documented commands and reuse child oracle/memory/transfer checks. +- Verify reproducible raw data and current baselines; keep expensive hardware performance campaigns outside correctness CI. + +**References** +- `ROADMAP.md`, Phase 8; `BACKLOG.md`, P8-01 through P8-16. +- `docs/perf/p5.9-gpu-fusion.md`, `phase5.8.md`, `q1-03b-sparse-blossom.md`. +- Phase 7 tracker: https://github.com/aleph-sim/aleph/issues/542. + +----- + +### [P8-01] Use optimized CPU and Metal execution in the CLI + +**GitHub:** [#551](https://github.com/aleph-sim/aleph/issues/551) — open as of 2026-10-18. + +**Labels:** `area:cli`, `area:ir`, `type:optimization`, `priority:high` +**Milestone:** Phase 8 +**Estimate:** M +**Depends on:** P1-09, P1-10, P2-07 + +**Description** +Route supported CPU/Metal CLI execution through the backend-appropriate optimized pipeline instead of the generic raw-gate driver. + +**Context** +`crates/aleph-cli/src/exec.rs::run_with_backend` currently calls `aleph_backend::run` directly. CPU Python uses `run_optimized`; Metal Python uses `MetalSvBackend::run_optimized`. Thus public frontends can execute different pipelines for the same backend while performance reports time optimized drivers. + +**Technical Details** +1. Separate execution from output formatting so the helper does not force every backend through raw `run`. Use the existing CPU optimized driver and the Metal supported wrapper; keep an explicit unfused baseline for A/B work. +2. Preserve CPU/Metal qubit-permutation restoration, amplitude ordering, expectation strings, measurement behavior, barriers, precision flags, and error handling. Do not apply SV fusion to stabilizer/MPS/noisy trajectories. +3. Check the FP32 CPU backend's optimized-instruction support before enabling the default there. Any unsupported subset needs explicit handling and tests rather than a silent raw fallback. +4. Cover measurements carefully: CLI returns final-state views, while DCE is seeded by measurements. Optimizing must preserve the CLI's final-state/sample contract, not merely the retained classical measurement distribution; conservatively skip unsafe passes for those queries. +5. Measure actual CLI wall time including planning and output selection on tiny and large workloads. Record retained instructions and planning cost. + +**Acceptance Criteria** +- [ ] Supported CLI CPU/Metal cases use the intended optimized driver; raw A/B remains available. +- [ ] Final-state, sampling, expectations, permutation and measurement semantics match the documented contract. +- [ ] Unsupported FP32/IR combinations fail clearly or use a documented safe policy. +- [ ] Public CLI before/after measurements show useful workload gains and expose tiny-input planning overhead. + +**Testing Requirements** +- Compare CLI outputs with unfused exact SV and Python equivalents on small fixtures, including nontrivial input preparation, Y terms, controls, barriers and measurements. +- Use deterministic structural checks to prove the optimized path ran, plus idle-machine workload benchmarks. + +**References** +- `crates/aleph-cli/src/exec.rs`, `crates/aleph-backend/src/lib.rs`. +- `crates/aleph-py/src/run.rs`, `crates/aleph-metal/src/sv/backend.rs`. +- `docs/perf/parity.md`. + +----- + +### [P8-02] Integrate CUDA fusion and layered dispatch into public frontends + +**GitHub:** [#552](https://github.com/aleph-sim/aleph/issues/552) — closed as of 2026-10-18. + +**Labels:** `area:backend-gpu`, `area:cli`, `area:python`, `area:ir`, `type:optimization`, `priority:high` +**Milestone:** Phase 8 +**Estimate:** L +**Depends on:** P5.9-01, P5.9-03, P5.9-06 + +**Description** +Make CLI/Python CUDA execution use the existing GPU-safe fusion and layered dispatch mechanisms that the CUDA benchmark suite already measures. + +**Context** +The earlier fusion issue #272 is closed and the kernels/helpers exist. The remaining gap is public integration: Python `run_cuda_py` calls raw `run`, and CLI's generic helper does the same. The audit probe demonstrated substantial instruction-count reduction with the existing GPU-safe passes, but did not measure GPU time. + +**Technical Details** +1. Add a shared CUDA execution-planning/helper boundary used by CLI, Python, and end-to-end benchmarks. Use GPU-safe fusion, not CPU relabel/tile passes, then supported layered execution. Avoid duplicate optimization and retain raw A/B execution. +2. Preserve measure/reset/barrier behavior, final-state output semantics, controls, seed handling and DiagonalPhase dispatch. If a pass is only safe for measurement-distribution queries, exclude it when frontends request final-state views or re-sample the collapsed final state. +3. Keep FP64 and ordinary FP32 accuracy contracts. Width-4 TF32 fusion has a different tolerance and optional hardware/module availability: require an explicit precision policy, and use width 3 when TF32 is unavailable/disabled. Do not silently expand FP32 error to the TF32 budget. +4. Report planning time, resulting gate kinds/launches, execution time, sampling and transfer separately. Include QFT/QPE, brickwall, VQE and QAOA through the actual public entry points. +5. Reuse existing helpers/oracles and make compiled-plan reuse possible without requiring a new compiler architecture in this issue. + +**Acceptance Criteria** +- [ ] CLI/Python use a shared supported CUDA plan and demonstrably emit/execute fusion and layered batches. +- [ ] Small public-API results match raw FP64 SV at each declared precision tolerance. +- [ ] TF32 capability/fallback and precision policy are explicit and tested. +- [ ] Current frontends are benchmarked before/after against the same workloads and hardware, including planning overhead. + +**Testing Requirements** +- Extend fusion/layered oracles with frontend-path checks, measurements, barriers and nontrivial prepared states. +- Test no-TF32 capability and unsupported IR; run transfer/launch counters so correctness-only CI cannot pass via a raw path. +- GPU benchmarks must execute on real supported hardware, not silently skip. + +**References** +- Previous integration issue: https://github.com/aleph-sim/aleph/issues/272 (closed; helper substrate). +- `crates/aleph-cuda/src/fusion.rs`, `sv/backend.rs`, `sv/fp32.rs`. +- `crates/aleph-py/src/run.rs`, `crates/aleph-cli/src/exec.rs`. +- `docs/perf/p5.9-gpu-fusion.md`, `p5.11-05-tf32-fused-block.md`. + +----- + +### [P8-03] Materialize Python GPU amplitudes lazily + +**GitHub:** [#553](https://github.com/aleph-sim/aleph/issues/553) — closed as of 2026-10-18. + +**Labels:** `area:python`, `area:backend-gpu`, `type:optimization`, `priority:high` +**Milestone:** Phase 8 +**Estimate:** L +**Depends on:** P5-05, P5.11-04 + +**Description** +Keep GPU results device-resident for counts-only Python calls and download/widen amplitudes only when statevector() is requested. + +**Context** +CUDA Python always calls `state.amplitudes_vec()` after sampling; Metal Python always calls `to_aos_f64()`. At n=28 FP32 CUDA this downloads 2 GiB and creates a 4 GiB complex128 host vector even when only counts are needed. The core readout's small-result transfer guarantee does not currently cover the frontend. + +**Technical Details** +1. Extend AmpStore/RunResult with feature-gated owned GPU states or a safe lazy materialization holder. Preserve device/context lifetime and satisfy PyO3 sendability/thread ownership without unsafe assertions. +2. Keep sampling before final result construction and hold only the resources required by the result. Make statevector() produce the established complex128 NumPy representation and correct qubit order; define whether repeated calls reuse a host cache. +3. Release the GIL appropriately for device synchronization/copy, and surface device-copy errors as Python errors. Do not change counts-only semantics or force GPU availability in CPU-only builds. +4. Add public-API transfer accounting for CUDA; on Metal use appropriate materialization/allocation counters rather than pretending unified memory transfers are PCIe copies. +5. Measure counts-only versus requested-statevector time and peak host/device memory. Explain the lifetime/memory trade-off of retaining a GPU result. + +**Acceptance Criteria** +- [ ] Counts-only Python GPU runs perform no full-state host materialization. +- [ ] statevector() returns oracle-correct complex128 amplitudes on demand, including repeated calls and object cleanup. +- [ ] Thread/context lifetime and failed-copy paths are tested without new unchecked unsafe assumptions. +- [ ] Public transfer and peak-memory measurements demonstrate the removed work. + +**Testing Requirements** +- CPU-only, CUDA and Metal feature builds; small amplitude/order oracles and lifetime tests. +- Assert counts-only DTOH bytes scale with shots/scalars, not 2^n, and materialization happens only on explicit access. + +**References** +- `crates/aleph-py/src/run.rs` (AmpStore, RunResult, run_cuda_py, Metal arm). +- `crates/aleph-cuda/src/sv/state.rs`, `sv/fp32.rs`. +- `crates/aleph-cuda/tests/transfer.rs`; `docs/perf/p5-05-lazy-transfer.md`. + +----- + +### [P8-04] Plan backend reach from requested operations and peak memory + +**GitHub:** [#554](https://github.com/aleph-sim/aleph/issues/554) — closed as of 2026-10-18. + +**Labels:** `area:backend`, `area:backend-gpu`, `area:cli`, `area:python`, `type:bug`, `priority:high` +**Milestone:** Phase 8 +**Estimate:** L +**Depends on:** P5.11-06 + +**Description** +Replace state-only CUDA reach assumptions with a capability-aware peak-memory plan that includes sampling, outputs, scratch and runtime device capacity. + +**Context** +Reach::for_kind uses fixed 30/31-qubit caps for the reference card. CUDA sampling allocates two 2^n FP64 CDF buffers. At n=30 FP32, 8 GiB state plus 16 GiB CDF already needs at least 24 GiB; at n=31 it needs 48 GiB. These lower bounds cannot fit the reference 20 GiB card, even though its state-only policy selects in-core execution. + +**Technical Details** +1. Model requested operations explicitly: execute-only, counts, expectation, marginals, amplitude export and precision. Estimate peak concurrent allocations, optional retained pool resources, matrix/readout scratch and transfer/materialization buffers with checked arithmetic. +2. Probe runtime total/free device capacity cheaply and reserve documented headroom. A snapshot is advisory because other processes may allocate; retain actionable allocation failures. +3. Keep abstract selection independent of CUDA implementation types. Pass capability/memory data from the frontend and make decisions explainable. Support smaller/larger devices instead of treating one card's cap as universal. +4. For an operation that does not fit, reject before large allocation or select an explicitly supported route. The current paged path is norm-only: never silently replace requested counts/expectations with a norm. +5. Coordinate with P8-05 so estimates use the actual sampler's memory model. Cover host export limits separately from device capacity. + +**Acceptance Criteria** +- [ ] Estimates include current sampling scratch and output requirements with overflow guards. +- [ ] The n=30/n=31 FP32 counts cases cannot be labelled feasible on a 20 GiB budget with the current sampler. +- [ ] Synthetic capability tests cover multiple card sizes and requested views; explanations identify the limiting allocation. +- [ ] Allocation failures remain clear under changing shared-device memory; no requested output is silently discarded. + +**Testing Requirements** +- GPU-free budget/overflow boundary tests and real-device constrained-memory checks without intentionally OOMing the host. +- Compare measured peak allocation with estimates for execution, sampling and export on tractable sizes. + +**References** +- `crates/aleph-backend/src/select.rs` (Reach::for_kind, SelectEnv). +- `crates/aleph-cuda/src/sv/readout.rs`, `sv/readout_f32.rs`. +- `crates/aleph-cli/src/exec.rs::run_cuda_paged`, `crates/aleph-py/src/run.rs`. + +----- + +### [P8-05] Replace full-state multi-pass CUDA CDF sampling + +**GitHub:** [#555](https://github.com/aleph-sim/aleph/issues/555) — closed as of 2026-10-18. + +**Labels:** `area:backend-gpu`, `type:optimization`, `priority:high` +**Milestone:** Phase 8 +**Estimate:** XL +**Depends on:** P5-05, P5.11-04, P8-04 + +**Description** +Replace Hillis-Steele full-state scan with an efficient scan or hierarchical sampling method, reducing work and scratch while preserving the distribution contract. + +**Context** +Both precision readouts allocate two FP64 arrays of 2^n entries and launch scan_step for offsets 1,2,4,... . Building a CDF therefore makes n full-state scan passes even for a small shot count. Device-resident execution avoids PCIe traffic but still does avoidable device-memory work. + +**Technical Details** +1. Baseline the existing sampler separately from gate execution, including check_norm, allocation, scan launches and result download. Sweep state size and shot count. +2. Prototype block-local scan with block totals/offsets, or a hierarchical sampler using block probability totals and selected-block sampling. Compare correctness, memory, read passes and shot-count crossover before selecting a default. +3. CUB DeviceScan is a reference for efficient scans; the current NVRTC/Rust integration may require a compiled shim or a custom supported kernel path. Do not add an unmeasured build dependency merely to copy an API. +4. Preserve finite/norm checks, zero-probability bins, deterministic seeded execution within the chosen algorithm, and FP64 accumulation where needed. Parallel summation may alter exact seed-to-outcome mapping; specify that change and verify distributions instead of promising unchanged bits without evidence. +5. Bound allocation before launch and update P8-04's memory model. Retain the old implementation as an explicit benchmark/reference path until validation completes. + +**Acceptance Criteria** +- [ ] The default avoids n full-array Hillis-Steele passes and has a documented peak-memory bound. +- [ ] Uniform, sparse, concentrated, mixed-phase and boundary distributions pass numerical/statistical oracles. +- [ ] Repeated fixed-seed runs are deterministic under the documented policy. +- [ ] Before/after data cover FP32/FP64 and small/large shot counts, reporting time, passes and peak scratch. + +**Testing Requirements** +- Small exact CDF/reference checks; zero bins, NaNs, norm drift, zero shots and extreme concentration. +- Real-GPU scaling campaigns and transfer checks; meaningful distribution tests with declared sample sizes/tolerances. + +**References** +- `crates/aleph-cuda/src/sv/readout.rs`, `sv/readout_f32.rs`, `sv/readout.cu`, `sv/readout_f32.cu`. +- https://nvidia.github.io/cccl/unstable/cub/api/structcub_1_1DeviceScan.html. +- `docs/perf/p5-05-lazy-transfer.md`. + +----- + +### [P8-06] Avoid building CPU alias tables for one or few samples + +**GitHub:** [#556](https://github.com/aleph-sim/aleph/issues/556) — closed as of 2026-10-18. + +**Labels:** `area:backend-sv`, `type:optimization`, `priority:high` +**Milestone:** Phase 8 +**Estimate:** M +**Depends on:** P0-11 + +**Description** +Add a low-shot CPU sampling path that avoids alias-table construction when its preprocessing cost cannot be amortized. + +**Context** +sample_impl always computes probabilities and builds an AliasTable. Every noisy trajectory calls sample(state, 1), then discards the table. The table builds thresholds, aliases, scaled weights and index stacks over 2^n amplitudes for a single draw. + +**Technical Details** +1. Implement a validated cumulative/streaming weighted draw for shots=1, then measure whether a small-shot threshold is justified. Separate norm/finiteness validation from materializing every probability when possible. +2. Keep the alias path for repeated draws. Preserve normalization drift handling, zero-probability bins, degenerate states and seeded reproducibility within each algorithm. +3. Document any seed-to-outcome change from choosing a different sampling algorithm; do not accidentally change the noisy shot-seed derivation or worker-count independence. +4. Measure sampler-only and end-to-end noisy trajectory workloads, reporting allocation count/bytes, time and crossover by n/shots. Limit the initial PR to the low-shot path rather than rewriting all sampling. + +**Acceptance Criteria** +- [ ] Single-shot execution does not build an alias table and retains required state validation. +- [ ] Empirical distributions and exact degenerate cases agree with the established sampler contract. +- [ ] Noisy results remain deterministic across worker counts for fixed configuration/seed. +- [ ] Allocation/time measurements establish a useful regime and preserve the many-shot baseline. + +**Testing Requirements** +- NaN/invalid norm, tiny valid branches, uniform/Bell/sparse/random states, zero and many shots. +- Reuse sampling distribution oracles; run paired low-shot versus alias benchmarks on an idle CPU. + +**References** +- `crates/aleph-sv/src/measure.rs::sample_impl`, `sampling.rs::AliasTable`. +- `crates/aleph-sv/src/noise/mod.rs::run_one_shot`. +- `crates/aleph-sv/benches/sample.rs`. + +----- + +### [P8-07] Bound noisy trajectory memory and reuse per-worker state + +**GitHub:** [#557](https://github.com/aleph-sim/aleph/issues/557) — closed as of 2026-10-18. + +**Labels:** `area:backend-sv`, `area:python`, `area:cli`, `type:optimization`, `priority:medium` +**Milestone:** Phase 8 +**Estimate:** L +**Depends on:** P4.6-04, P8-06 + +**Description** +Reuse trajectory buffers and choose concurrency from a memory budget rather than allocating a fresh dense state for every active shot. + +**Context** +run_noisy parallelizes over shots; run_one_shot constructs a backend and allocates a state each time. Peak memory scales with active trajectories and sampler scratch, plus a dense 2^n count histogram. Existing channel lookup is already hoisted outside the shot loop, so repeating that optimization would not address this cost. + +**Technical Details** +1. Add a worker/chunk execution context with reusable state and scratch. Reset to |0...0> correctly before each shot and seed stochastic work by the existing absolute shot index, independent of chunking/scheduling. +2. Estimate per-trajectory state/channel/sampler peak memory and fixed result memory. Select bounded concurrency from a declared memory policy; clarify that Rayon uses a shared pool and nested kernel work is not automatically threads-squared oversubscription. +3. Start with the current public histogram format, then expose a sparse counts path to frontends if dense 2^n output itself dominates. Preserve the legacy Rust API via an explicit adapter/compatibility policy. +4. Keep noise attached to original gate boundaries. Generic noiseless fusion cannot cross channel insertions without changing the model. Do not remove unitarity/channel validation; any prevalidated plan must prove its immutable-input contract. +5. Benchmark state reuse, sampler changes and concurrency separately to attribute gains; include small states and large memory-bound trajectories. + +**Acceptance Criteria** +- [ ] Reused buffers do not leak state between shots; fixed-seed results are worker/chunk invariant. +- [ ] Concurrency and result allocations respect a documented budget or fail before excessive allocation. +- [ ] Gate/channel ordering and the legacy output contract remain correct. +- [ ] Before/after measurements include peak memory, allocations, throughput and noise-model identity. + +**Testing Requirements** +- Independent trajectory oracle versus the existing path, including Pauli and Kraus channels, readout noise and zero noise. +- Test consecutive shots with different seeds and constrained concurrency; report dense-result limits. + +**References** +- `crates/aleph-sv/src/noise/mod.rs`, `noise/apply.rs`, `noise/model.rs`. +- `docs/decisions/0014-noise-trajectories.md`. +- `crates/aleph-py/src/run.rs::hist_to_counts`. + +----- + +### [P8-08] Probe GPU availability without compiling discarded backends + +**GitHub:** [#558](https://github.com/aleph-sim/aleph/issues/558) — closed as of 2026-10-18. + +**Labels:** `area:backend-gpu`, `area:cli`, `area:python`, `type:optimization`, `priority:high` +**Milestone:** Phase 8 +**Estimate:** M +**Depends on:** P5.11-06 + +**Description** +Separate device capability probing from backend/kernel construction and avoid unnecessary GPU probes for explicit backend choices. + +**Context** +Both frontends construct SelectEnv before deciding between Auto and Fixed. CUDA availability calls CudaSvBackendF32::new(), compiling/loading modules; CLI Metal availability constructs MetalSvBackend too. A CUDA-enabled explicit CPU run may therefore pay GPU setup, and a selected GPU is then constructed again. + +**Technical Details** +1. Resolve explicit requests without irrelevant availability probes. For auto, use a cheap device/capability query and circuit structure to determine which environment facts are actually needed. +2. Separate capabilities from loaded kernels; construct the selected backend once. If module caching is needed, key it by device/context/precision/build configuration and define invalidation/lifetime/error behavior. +3. Keep auto fallback and explicit-backend errors clear. A discovered device does not guarantee kernel compilation succeeds, so handle construction failures deliberately instead of confusing availability with readiness. +4. Measure cold process startup and warm repeated Python calls separately, using construction/compilation counters in addition to wall time. + +**Acceptance Criteria** +- [ ] Explicit CPU/stabilizer/MPS requests do not compile CUDA/Metal modules. +- [ ] Auto probing does not construct and discard a complete backend; selected backend is initialized once. +- [ ] Device discovery versus compilation failure behavior is documented and tested. +- [ ] Cold/warm public API measurements and counters show removed setup work. + +**Testing Requirements** +- CPU-only builds, absent devices, explicit GPU failure and mocked capability/constructor paths. +- Actual CUDA/Metal smoke tests; verify counters without timing-sensitive CI assertions. + +**References** +- `crates/aleph-cli/src/exec.rs::cuda_available`, `metal_available`, run_circuit. +- `crates/aleph-py/src/run.rs::cuda_available`, run_circuit. +- `crates/aleph-cuda/src/sv/fp32.rs` constructors; `crates/aleph-metal/src/context.rs`. + +----- + +### [P8-09] Coalesce GPU QEC scratch while retaining serial-shot arithmetic + +**GitHub:** [#559](https://github.com/aleph-sim/aleph/issues/559) — closed as of 2026-10-18. + +**Labels:** `area:decoder`, `area:backend-gpu`, `type:optimization`, `priority:medium`, `research` +**Milestone:** Phase 8 +**Estimate:** L +**Depends on:** Q3-01, Q3-02 + +**Description** +Test transposed/tiled per-shot scratch layouts for GPU BP, retaining the existing per-shot arithmetic schedule; evaluate Union-Find only after the regular BP case. + +**Context** +GPU BP and UF use one thread per shot with scratch indexed as shot*stride+element. At a common edge loop iteration, adjacent warp lanes access addresses separated by a full graph stride. The source identifies a plausible coalescing problem; its magnitude has not been measured with hardware counters. + +**Technical Details** +1. Profile the unchanged kernels with representative graph sizes, shot batches and error rates. Record transactions/bandwidth, occupancy, iteration divergence and scratch footprint. +2. Add an edge-major or tiled AoSoA BP scratch variant so adjacent lanes access adjacent shot slots for a common edge, preserving each shot's edge order and FP64 operations. Keep the current layout as A/B reference and update scratch/batch budgeting. +3. Compare layout-only changes before introducing cooperative scheduling or reduced precision. Graph reads shared by lanes and divergent early stops may limit gains; state those regimes explicitly. +4. UF's irregular parent/BFS accesses may behave differently: profile it independently and only retain a layout variant with measured benefit. Preserve tie-breaking and correction identity. + +**Acceptance Criteria** +- [ ] BP layout-only variant matches the reference corrections and convergence behavior on the established oracle campaign. +- [ ] Hardware profiling confirms or rejects the proposed coalescing bottleneck. +- [ ] Time, scratch and batch-size crossover are reported against the same-decoder reference. +- [ ] A measured no-go with reverted candidate code is acceptable; no unsupported speedup claim. + +**Testing Requirements** +- Existing 100k-shot BP oracle, degree-1 infinities, graph extremes and deterministic repeats. +- UF correction/tie-order checks if attempted; explicit real-GPU profiling, not construction-skip success. + +**References** +- `crates/aleph-cuda/src/qec/bp.cu`, `bp.rs`, `uf.cu`, `uf.rs`. +- `docs/perf/qec-q3-gpu-bp.md`, `qec-q3-gpu-uf.md`. +- https://docs.nvidia.com/cuda/cuda-c-best-practices-guide/index.html (coalesced access). + +----- + +### [P8-10] Prototype cooperative GPU BP with explicit accuracy and latency trade-offs + +**GitHub:** [#560](https://github.com/aleph-sim/aleph/issues/560) — closed as of 2026-10-18. + +**Labels:** `area:decoder`, `area:backend-gpu`, `type:optimization`, `priority:low`, `research` +**Milestone:** Phase 8 +**Estimate:** XL +**Depends on:** P8-09 + +**Description** +Prototype warp/block-cooperative BP after layout profiling, and establish whether it improves throughput or single-shot latency at an acceptable accuracy/resource cost. + +**Context** +The current exact-match BP kernel serializes all checks/variables/iterations inside one thread per shot. Layout fixes do not remove that serial critical path. Cooperative execution is a hypothesis, not a demonstrated improvement, and reductions can alter numerical/tie behavior. + +**Technical Details** +1. Pick a bounded graph/degree regime from P8-09 data. Parallelize independent check updates, then variable updates, with a clear message schedule and synchronization contract. +2. Keep FP64 initially. Preserve within-node reduction order where practical; otherwise document numeric tolerance and quantify hard-decision/convergence disagreement, paired logical error and latency. A rate agreement alone does not establish bitwise equivalence. +3. Compare with both original and layout-improved kernels across batch sizes; expose throughput versus single-shot latency, occupancy, scratch and tails. Do not transfer claims from surface codes to qLDPC without a supported fixture. +4. Keep the exact-match reference and do not change defaults until the target operating regime is measured. FP32, relay-BP and OSD integration are separate follow-ups. +5. Use a predeclared go/no-go experiment; if synchronization outweighs parallel work, publish the negative result and stop the rewrite. + +**Acceptance Criteria** +- [ ] Supported cooperative prototype has a documented schedule and oracle/accuracy comparison. +- [ ] Throughput, single-shot/tail latency and memory are measured at matched precision/decoder parameters. +- [ ] At least one supported qLDPC fixture or an explicit surface-only limitation is included. +- [ ] Adoption or rejection follows measured data; negative results can close this research issue. + +**Testing Requirements** +- Tiny Tanner graph reference updates, degree-1 and infinity cases, varied syndrome weight and convergence tails. +- Paired decoder outcomes and real-hardware batch/latency campaigns; report rounding/tie disagreements. + +**References** +- `crates/aleph-cuda/src/qec/bp.cu`, `crates/aleph-qec/src/bp.rs`. +- `docs/perf/qec-q3-gpu-bp.md`; P8-09. + +----- + +### [P8-11] Prove a competitive Metal MPS factorization before another pipeline rewrite + +**GitHub:** [#561](https://github.com/aleph-sim/aleph/issues/561) — open as of 2026-10-18. + +**Labels:** `area:backend-mps`, `area:backend-gpu`, `type:optimization`, `priority:medium`, `research` +**Milestone:** Phase 8 +**Estimate:** XL +**Depends on:** P5.8-06 + +**Description** +Benchmark and prototype multi-threadgroup/tiled MPS SVD/QR factorization, with a go/no-go gate before integrating another large Metal MPS rewrite. + +**Context** +Phase 5.8 removed major host/pipeline costs but the n=16 chi=256 benchmark changed only a few percent; GPU remained 16–22x slower than CPU. gpu_jacobi and move_gpu still dispatch one threadgroup per factorization, including blocks up to 512x512. The numerical guard/f64 fallback is essential and must stay. + +**Technical Details** +1. Isolate representative two-site split and center-move matrices at chi=32/64/128/256/512, including ill-conditioned and truncating cases. Measure pack/factor/sort/install/guard/fallback separately against the current GPU path and faer CPU. +2. Prototype blocked Jacobi or QR stages across multiple threadgroups/dispatches with explicit inter-dispatch dependencies. Do not assume increasing threadgroup size unlocks the whole GPU. +3. Validate reconstruction, orthogonality/canonical form, discarded weight, norm and final observables at the existing tolerance. Quantify fallback frequency and its cost rather than disabling guards. +4. Establish a useful factorization crossover before wiring a complete MPS pipeline. If it exists, integrate only that regime and rerun end-to-end MPS against CPU with actual bond-size distributions and dispatch/sync costs. +5. Small-bond CPU fallback and batching are independent controls; avoid claiming fully resident execution when the chosen path falls back to CPU. + +**Acceptance Criteria** +- [ ] A stage-level matrix/shape benchmark explains the existing wall and fallback frequency. +- [ ] Prototype preserves reconstruction/canonical/truncation correctness on hard fixtures. +- [ ] An integration go/no-go verdict is backed by factorization and projected/measured end-to-end costs. +- [ ] No-go can close the research issue with documented limits; only measured beneficial regimes change production routing. + +**Testing Requirements** +- Existing CPU/FP64 SV MPS oracles and chi-saturating examples. +- Shape/conditioning sweeps, fallback accounting, and idle-Metal before/after timings; preserve current reference implementation. + +**References** +- `crates/aleph-metal/src/mps/gpu_jacobi.rs`, `move_gpu.rs`. +- `crates/aleph-metal/src/shaders/mps_jacobi.metal`, `mps_qr.metal`. +- `docs/perf/phase5.7-audit.md`, `phase5.8.md`. + +----- + +### [P8-12] Validate communication-aware plans on real multi-GPU workloads + +**GitHub:** [#562](https://github.com/aleph-sim/aleph/issues/562) — open as of 2026-10-18. + +**Labels:** `area:backend-dist`, `area:ir`, `area:bench`, `type:optimization`, `priority:medium`, `research` +**Milestone:** Phase 8 +**Estimate:** XL +**Depends on:** P6-05 + +**Description** +Benchmark the current compiler on actual multi-GPU hardware and diagnose compute-pass inflation, representation loss and communication before choosing further scheduling changes. + +**Context** +The earlier AWS n=28 random FP32 run measured 0.87x speedup on two L4s and 1.06x on four. Distributed plans inflated local compute/fusion work before link cost. P6-05 and state-aware cost calibration have since improved plans, but a predicted compiler gain on emulated ranks is not an actual multi-device measurement. + +**Technical Details** +1. Establish current fused single-device, naive distributed, lookahead and compiled baselines with identical circuit/precision/query and reported hardware topology. Record NCCL/P2P/host staging capability rather than assuming it. +2. Decompose wall time into planning, local kernel kinds/passes, exchange bytes/events, synchronization and readout. Detect lost diagonal specialization, extra local swaps, fusion fragmentation and host serialization using the current code, not stale pre-fix reports. +3. Compare useful work and achieved time for both fixed-size strong scaling and fixed-memory reach. QFT's free SWAP relabeling can make superlinear speedup a change in work, so annotate that baseline. +4. Select one measured schedule/representation lever and validate it separately. Do not add hardware, widen fusion or tune a link model when compute inflation is the dominant term. +5. Freeze calibration choices before held-out workload evaluation. Keep predictions and measured results in separate columns and include uncompetitive cases. + +**Acceptance Criteria** +- [ ] Current multi-device measurements compare single-GPU, naive/lookahead and compiled plans on at least 2/4 GPUs when hardware is available. +- [ ] Kernel/communication/work inflation explains scaling limits with attributable measurements. +- [ ] A selected optimization has before/after oracle and end-to-end evidence, or a documented no-go conclusion. +- [ ] Hardware-unavailable results remain labelled prediction-only and do not close measured scaling claims. + +**Testing Requirements** +- Existing distributed equivalence/host-overlap tests, small actual multi-device oracles, held-out plan selection cases. +- Strong/weak scaling with topology, precision and total-work counters; do not disrupt shared services without the repository's authorized isolation procedure. + +**References** +- `docs/perf/p6-multi-gpu.md`, `p6-05-compiler.md`. +- `crates/aleph-ir/src/dist/compile.rs`, `specialize.rs`. +- `crates/aleph-cuda/src/dist/device_sv.rs`, `cost.rs`, `nccl.rs`. + +----- + +### [P8-13] Make public-workflow baselines and experiment gates reproducible + +**GitHub:** [#563](https://github.com/aleph-sim/aleph/issues/563) — closed as of 2026-10-18. + +**Labels:** `area:bench`, `area:infra`, `area:docs`, `type:feature`, `priority:high` +**Milestone:** Phase 8 +**Estimate:** L +**Depends on:** — + +**Description** +Create a common end-to-end benchmark/provenance workflow and require bottleneck/crossover evidence before expensive optimization campaigns. + +**Context** +Historical evidence includes an inert min_amps axis in a 360-cell chunk sweep, Metal pipeline work ahead of competitive factorization, tiny TF32 margins measured best-of, GPU-MWPM/UF algorithm swaps inside speedup headlines, and cost-model contamination from services restarted by a watchdog. Existing reports are valuable but their comparison contracts differ. + +**Technical Details** +1. Record revision, exact public entry point, precision/error budget, optimizer flags, workload seed/model, shots, threads/workers, warmed/cold state, compiler/device identity, topology and raw repeat data in a machine-readable run manifest. +2. Time startup, parse/compile, execution, sample and materialization separately, plus total caller wall time. Run the same public route used by customers; kernel-only results are an additional tier. +3. Use paired/interleaved A/B order and report medians/spread and uncertainty for small wins. Treat best-of as an explicitly labelled throughput ceiling, not sufficient evidence of a stable 2% gain. Verify idleness throughout; obey shared-service isolation authorization and restoration rules. +4. Require a preflight experiment that the swept parameter changes the executed path/counters. Sweep thresholds near the actual transition instead of spending trials on an inert axis. Use a bottleneck upper bound/crossover estimate and a declared stop condition before broad rewrites. +5. Compare same algorithm/precision/core budget first. Keep workflow algorithm changes in separate tables with accuracy data. Re-measure MWPM after Sparse Blossom; archive older dense-MWPM numbers as historical. +6. Add structural/transfer/memory guards through public API paths. Feature compilation or tests that silently skip GPU construction do not establish hardware execution; hardware campaigns must explicitly report executed/skipped status. + +**Acceptance Criteria** +- [ ] Reusable manifests and commands capture total/stage timing, provenance, repeats and resource counters. +- [ ] At least CPU CLI and CUDA Python reference workflows use the harness. +- [ ] Current same-decoder and same-resource baselines are separate from historical/cross-algorithm headlines. +- [ ] Historical no-gain cases are summarized with their causes and experiment stop rules. +- [ ] CI guards exercise intended frontend paths structurally; real-hardware results cannot be recorded as passes when skipped. + +**Testing Requirements** +- Validate manifest schema, subprocess failures, skipped hardware, warm/cold labels and counter capture. +- Reproduce a known fusion/transfer improvement and a small/no-gain case; avoid wall-time thresholds in correctness CI. + +**References** +- `docs/perf/phase2-p2-04.md`, `phase5.8.md`, `p5.11-05-tf32-fused-block.md`. +- `docs/perf/qec-q3-montecarlo.md`, `q1-03b-sparse-blossom.md`, `p6-05-compiler.md`. +- `.github/workflows/ci.yml`; `docs/benchmarking.md`. + +----- + +### [P8-14] Evaluate combined tile residency and overlapped paging + +**GitHub:** [#564](https://github.com/aleph-sim/aleph/issues/564) — closed as of 2026-10-18. + +**Labels:** `area:backend-gpu`, `type:optimization`, `priority:low`, `research` +**Milestone:** Phase 8 +**Estimate:** L +**Depends on:** P5.11-02, P5.11-03, P8-13 + +**Description** +Evaluate a paging executor that combines existing low-qubit multi-gate residency with copy overlap, using the measured duplex ceiling and real locality distribution to bound expectations. + +**Context** +run_paged_batched amortizes transfers over low-only gate runs and achieved up to 9.93x on a locality-rich fixture. run_paged_overlapped overlaps per-gate copies and measured about 1.10x with a 1.24x duplex ceiling. They are separate entry points; the report's conceptual composability does not establish a combined production path or multiplicative end-to-end gains. + +**Technical Details** +1. Baseline both existing executors and inspect actual low-run lengths/high-qubit interruptions on held-out workloads. Retain synchronous residency as the main comparison. +2. Prototype an event-ordered ring of resident tile groups: apply a complete safe low-only run per load, overlap independent groups, and enforce prior scatter completion before host-memory reuse and high-gate transitions. +3. Budget all device buffers and pinned host memory; avoid losing tile size/headroom to an oversized overlap ring. Handle shallow pipelines with the proven synchronous fallback. +4. Measure copies, bytes, residency, fill/drain and compute separately. Large locality gains are fixture-dependent; overlap benefits cannot exceed the measured link/pipeline opportunity. +5. Start FP64, mirror FP32 only if worthwhile. Qubit relabelling/locality-aware scheduling is a later measured lever, not necessary to prove safe composition. + +**Acceptance Criteria** +- [ ] Prototype preserves paging order and matches the synchronous reference at declared precision. +- [ ] Memory budgeting and high-qubit flush/fallback behavior are explicit. +- [ ] Held-out workload A/B measures traffic, wall time and regimes with no gain. +- [ ] Adopt only a measured useful configuration; a documented no-go result can close the research issue. + +**Testing Requirements** +- Forced small tiles with mixed controls/targets and alternating low/high gates; compare complete amplitudes against CPU/reference paging. +- Real-device event-order tests and interleaved idle-machine timing; zero hidden synchronous copies. + +**References** +- `crates/aleph-cuda/src/sv/paged.rs`, `paged_f32.rs`. +- `docs/perf/p5.11-02-overlapped-paging.md`, `p5.11-03-multi-gate-residency.md`. + +----- + +### [P8-15] Bound DEM experiment batches and validate decoder result contracts + +**GitHub:** [#565](https://github.com/aleph-sim/aleph/issues/565) — closed as of 2026-10-18. + +**Labels:** `area:qec`, `area:decoder`, `type:bug`, `priority:medium` +**Milestone:** Phase 8 +**Estimate:** L +**Depends on:** Q0-04 + +**Description** +Process logical-error experiments in bounded batches and reject incomplete decoder outputs before scoring instead of materializing all shots and silently truncating zip iteration. + +**Context** +run_dem_experiment samples all syndromes/truths into vectors, decodes the full batch, then zips predictions with truths. Memory grows with total shots. If a custom decoder returns fewer predictions, zip skips unreturned shots but the reported denominator is still the requested shot count; missing observable bits are also treated as false by mispredicted. + +**Technical Details** +1. Introduce bounded batch size/memory policy and sample by absolute shot index so chunking preserves the existing deterministic stream. Keep sample_shots as an explicit artifact API for callers needing every shot. +2. Validate decode_batch output length and each correction's observable schema before scoring. Return a clear decoder-contract error for incomplete results instead of counting only the returned prefix. +3. Accumulate exact counts across batches with checked arithmetic. Expose processed/requested counts for any intentional partial/budgeted experiment; never silently use an unprocessed denominator. +4. Consider packed syndromes/truth bits and reusable per-worker sampling scratch after the batch boundary is correct. Benchmark small fast decoders where allocation overhead matters and large campaigns where memory dominates. +5. Preserve external PyMatching batch amortization and existing decoder thread-local caches. + +**Acceptance Criteria** +- [ ] Total experiment memory is bounded by configured batch size, not total shots. +- [ ] Short/long prediction vectors and incompatible observable lengths fail explicitly. +- [ ] Fixed-seed exact tallies match the previous valid full-batch path across batch sizes/workers. +- [ ] Sampling, decode, scoring time and peak memory are reported for representative campaigns. + +**Testing Requirements** +- Custom intentionally malformed decoders, zero/partial final batches, multiple observables and deterministic mechanisms. +- Existing CPU/oracle experiments and fixed-seed chunk invariance; packed conversion round trips if introduced. + +**References** +- `crates/aleph-qec/src/experiment.rs::run_dem_experiment`, sample_shots, mispredicted. +- `crates/aleph-qec/src/decoder.rs::Decoder`, `syndrome.rs`. +- Phase 7 P7-01/P7-02 rely on trustworthy experiment scoring. + +----- + +### [P8-16] Use valid logical-error intervals for zero and rare observed failures + +**GitHub:** [#566](https://github.com/aleph-sim/aleph/issues/566) — closed as of 2026-10-18. + +**Labels:** `area:qec`, `area:bench`, `type:bug`, `priority:high` +**Milestone:** Phase 8 +**Estimate:** M +**Depends on:** Q0-04 + +**Description** +Replace or supplement the current Wald half-width with a documented interval that remains meaningful for zero/all failures and small counts. + +**Context** +LogicalErrorResult::new computes 1.96*sqrt(p*(1-p)/n). Zero observed failures yields rate=0 and ci95=0, which cannot establish zero logical risk. The same boundary problem occurs at all failures. Phase 7 rare-event estimators need their own interval method; the existing direct Monte Carlo baseline also needs valid uncertainty. + +**Technical Details** +1. Choose and document an interval (Wilson or exact binomial are starting points), its confidence level, two/one-sided interpretation and zero-shot behavior. Represent lower/upper bounds rather than assuming symmetry around the sample proportion. +2. Inventory Rust/Python/CLI/report consumers of ci95 and define a compatibility/migration policy. Do not silently reinterpret a public half-width as an asymmetric interval. +3. Update comparison/report helpers to use the declared bounds correctly; overlapping intervals are not a proof of decoder equivalence. Preserve paired comparisons where matching shot streams are available. +4. Keep ordinary Bernoulli experiment intervals separate from stratified/weighted/correlated estimators in P7-02/P7-04. +5. Validate coverage offline for small/rare cases and document zero-event upper bounds; zero-shot experiments have no evidence, not a measured zero failure probability. + +**Acceptance Criteria** +- [ ] Zero/all-failure and small-count experiments have valid documented bounds; zero-shot semantics are explicit. +- [ ] Public result/report consumers follow a compatible, tested migration policy. +- [ ] Reference binomial cases and offline coverage checks support the stated method. +- [ ] Weighted/correlated estimators cannot accidentally reuse the ordinary direct-sampling interval. + +**Testing Requirements** +- Exact boundary/reference cases, large n, monotonic bounds and invalid count inputs. +- Offline repeated-sampling coverage within reported simulation uncertainty; small deterministic cases in CI. + +**References** +- `crates/aleph-qec/src/decoder.rs::LogicalErrorResult::new`. +- `crates/aleph-qec/src/experiment.rs`. +- `BACKLOG.md`, P7-02 and P7-04 (distinct estimator contracts). + +----- + # Appendix: Counts and Order - **Phase 0**: 12 issues @@ -4877,6 +5574,7 @@ engine, not a universal replacement for state-vector amplitudes or sampling. **Original Phases 0–6 baseline: 60 issues.** Later sub-phase additions are specified above. - **Phase 7**: 7 issues (one integration/report tracker + six capability issues). +- **Phase 8**: 17 issues (one integration/report tracker + sixteen capability issues). Recommended ordering for first weeks: diff --git a/ROADMAP.md b/ROADMAP.md index 069ad4a8..70cb9ce5 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -119,6 +119,7 @@ Each backend implements a common `Backend` trait. The frontend, IR, parser, and |5 |GPU backend (single-GPU) |2–3 months |cuQuantum integration + custom CUDA where it adds value | |6 |Multi-GPU and distributed |2–3 months |NCCL intra-node + MPI inter-node | |7 |Research instruments |Milestone-driven |Failure diagnosis, rare-event QEC, observable queries, robustness and deadline studies, near-Clifford expectations | +|8 |Performance audit and recovery |Milestone-driven |Public frontend parity, operation-aware memory, reproducible gains and measured research verdicts | Estimates are aggressive solo full-time. Realistic part-time: roughly double. @@ -155,6 +156,27 @@ efficiency and failure regimes, and include at least one end-to-end QEC diagnosi one observable-query study. Research hypotheses are measured, not promised speedups; modelled deadline outcomes must be distinguished from measured hardware capability. +### Phase 8 — Performance Audit and Recovery + +Bring existing optimized execution into public frontends, remove unnecessary setup +and output work, and bound memory by the requested operations. Detailed specifications: +`BACKLOG.md`, P8-00 through P8-16; tracker [#550](https://github.com/aleph-sim/aleph/issues/550). + +As of 2026-10-18, **13 of 16 capability issues are closed**. The +[integration report](docs/perf/p8-performance-recovery.md) links public API timings, +kernel/operation measurements, allocation evidence, adverse cases and raw provenance. +Phase 8 remains in progress: optimized CPU/Metal CLI execution (#551, draft #569), +competitive Metal MPS factorization (#561), and real multi-GPU plan validation +(#562) remain open. Explicit BP/paging opt-ins are completed research outcomes, +not default-routing or universal performance claims. + +**Exit metric:** every capability has a verified repair or its explicitly allowed, +measured research verdict. The report must cover CPU CLI, CUDA Python counts-only, +large-state sampling and noisy trajectories; distinguish modelled reach from actual +allocations and kernel time from public caller time; smoke-run documented commands; +and preserve negative results. The tracker stays open until the three remaining +gates are resolved. This phase does not imply Phase 6 scaling or v1.0 completion. + ----- ## 6. Algorithm Coverage (Benchmark Suite) diff --git a/docs/benchmarking.md b/docs/benchmarking.md index 9c96d69a..53cd8617 100644 --- a/docs/benchmarking.md +++ b/docs/benchmarking.md @@ -110,6 +110,11 @@ Don't upload local numbers to the project default testbed (`self-hosted-linux-x6 ## Public-route experiment gates +The [Phase 8 recovery report](perf/p8-performance-recovery.md) provides the current +capability/adoption ledger, timing and memory boundaries, historical evidence and +remaining hardware gates. Phase 8 is still in progress; its separate speedup ratios +must not be multiplied into an aggregate claim. + Use the [P8-13 public workflow harness](../scripts/benchmarks/README.md) for customer-visible CLI/Python A/B campaigns, in addition to Criterion's kernel-tier measurements. It records provenance, exact workloads/argv, interleaved raw pairs, diff --git a/docs/perf/data/p8-00/README.md b/docs/perf/data/p8-00/README.md new file mode 100644 index 00000000..6d28f4df --- /dev/null +++ b/docs/perf/data/p8-00/README.md @@ -0,0 +1,31 @@ +# Phase 8 integration-report receipts + +Collected 2026-10-18 from main `6114dab7e18c4deac0fda3039060e00f7661b653`. +The report changes documentation only; no benchmark or production source changed. +Command logs normalize trailing blank lines; test output content is otherwise retained. + +- [status.json](status.json): GitHub issue/PR state snapshot; 13/16 capabilities + closed, #551/#561/#562 open. #550 stays open. PR heads and merge commits identify + integration history, not the historical binaries used by child campaigns. +- [validation.json](validation.json): current local platform/toolchain, freshly + built release CPU CLI SHA256, command argv and exits. RAYON_NUM_THREADS=2 bounds + local correctness checks. These are not accepted timing observations. +- [check-1.log](check-1.log): 13 public-workflow contracts including a real CPU CLI + call. Its test mocks resource admission to check output/receipt semantics on CI + or a busy Mac; it does not establish idle-machine performance. +- [check-2.log](check-2.log): 6 pure memory-policy tests. +- [check-3.log](check-3.log): 11 sampler/noise integration and stored oracle tests. + The Aer fixtures are stored references, not a new live Aer campaign. +- [check-4.log](check-4.log): 7 binomial + 8 bounded-experiment tests. One existing + external PyMatching test remains ignored; it is not counted as a pass. +- [check-5.log](check-5.log): 3 threshold-plot schema tests. +- [saved-receipts.json](saved-receipts.json): all six existing P8-13 server campaign + directories independently passed their saved-record consistency check. Four + have `executed` status and exit 0. The failed/contaminated records correctly + exit 2 and contain no successful summary. `passed` here means **revalidation of + the saved classification**, not successful original hardware execution. +- [cli-smoke.txt](cli-smoke.txt): documented Bell counts command on this CLI. + +No GPU, Metal, remote-server or fresh performance campaign was run for this report. +Historical numerical results are linked from the report to their child evidence; +their timestamps/source hashes must not be replaced by this validation identity. diff --git a/docs/perf/data/p8-00/SHA256SUMS b/docs/perf/data/p8-00/SHA256SUMS new file mode 100644 index 00000000..96f6d55b --- /dev/null +++ b/docs/perf/data/p8-00/SHA256SUMS @@ -0,0 +1,10 @@ +578b39c39aa057e42445f0ec8ac40f3cfad9b73c86d633b092231d7cb3adaaa9 README.md +b068e1be267fca2b994124eba39f02d0661067767f19adba029134a91c2a8cd5 check-1.log +2524aade8f370f1d04c6c4f23cf9d79721147b11474f7a745ab0f040b5935340 check-2.log +9d5736c2be9b5c3b1fea5d7b403ef8f6d38703072b9c8c2b30d2465cc8787c21 check-3.log +236d037c66f7b8cc5cfeccd0cbf9d802cf885a4af29c5562c589e13e3e5ecae2 check-4.log +dd73ae35517cf417c4448f3b0cc4b1de21bdc18a0d5cd10bcd351667d49f7f73 check-5.log +89efdf5502fda9fc49b0b0edfa277e06d9b06d038552af12c0fc7da11bb04def cli-smoke.txt +fcd2ef8151ee8546eae3443d67acac652314f2024a37e7630e5c34878877ed1b saved-receipts.json +9f0c8fb7344f7aa10488878936c24409a2fa73b4974c160174f125d8fb753a57 status.json +32246c29b3778a25b387707cac66e32303700b188b0d312022827fc6d2c362b4 validation.json diff --git a/docs/perf/data/p8-00/check-1.log b/docs/perf/data/p8-00/check-1.log new file mode 100644 index 00000000..ccf1460b --- /dev/null +++ b/docs/perf/data/p8-00/check-1.log @@ -0,0 +1,18 @@ +test_changed_executable_rejected_before_launch (test_public_workflow.Contracts) ... ok +test_contamination_aborts_without_launching (test_public_workflow.Contracts) ... ok +test_deterministic_workload_and_summary (test_public_workflow.Contracts) ... ok +test_gpu_settling_is_fixed_and_does_not_retry_until_quiet (test_public_workflow.Contracts) ... ok +test_missing_process_observer_cannot_claim_idle (test_public_workflow.Contracts) ... ok +test_paired_order_and_inert_parameter_stops_before_measurement (test_public_workflow.Contracts) ... ok +test_preserves_venv_executable_symlink (test_public_workflow.Contracts) ... ok +test_real_cpu_cli_entrypoint (test_public_workflow.Contracts) ... ok +test_receipt_missing_hardware_or_counters_is_failure (test_public_workflow.Contracts) ... ok +test_schema_and_amdahl_gate (test_public_workflow.Contracts) ... ok +test_skipped_hardware_never_passes (test_public_workflow.Contracts) ... ok +test_subprocess_failure_and_bad_receipt (test_public_workflow.Contracts) ... ok +test_successful_subprocess_with_wrong_receipt_fails (test_public_workflow.Contracts) ... ok + +---------------------------------------------------------------------- +Ran 13 tests in 1.140s + +OK diff --git a/docs/perf/data/p8-00/check-2.log b/docs/perf/data/p8-00/check-2.log new file mode 100644 index 00000000..4101458d --- /dev/null +++ b/docs/perf/data/p8-00/check-2.log @@ -0,0 +1,17 @@ + Compiling aleph-ir v0.3.0 (/Users/ex/GitHub/aleph/crates/aleph-ir) + Compiling aleph-backend v0.3.0 (/Users/ex/GitHub/aleph/crates/aleph-backend) + Compiling aleph-parser v0.3.0 (/Users/ex/GitHub/aleph/crates/aleph-parser) + Compiling aleph-test v0.3.0 (/Users/ex/GitHub/aleph/crates/aleph-test) + Compiling aleph-sv v0.3.0 (/Users/ex/GitHub/aleph/crates/aleph-sv) + Finished `test` profile [unoptimized + debuginfo] target(s) in 5.87s + Running unittests src/lib.rs (target/debug/deps/aleph_backend-a2fadda868659f85) + +running 6 tests +test memory::tests::overflow_and_invalid_layouts_are_errors ... ok +test memory::tests::requested_views_scratch_and_host_limits_are_accounted ... ok +test memory::tests::executor_workspace_override_includes_shot_buffers_and_checks_overflow ... ok +test memory::tests::runtime_sizes_and_free_memory_drive_fit ... ok +test memory::tests::legacy_cdf_counts_do_not_fit_reference_card_despite_state_fitting ... ok +test memory::tests::counts_reach_scales_with_capacity_and_precision ... ok + +test result: ok. 6 passed; 0 failed; 0 ignored; 0 measured; 36 filtered out; finished in 0.00s diff --git a/docs/perf/data/p8-00/check-3.log b/docs/perf/data/p8-00/check-3.log new file mode 100644 index 00000000..699b349a --- /dev/null +++ b/docs/perf/data/p8-00/check-3.log @@ -0,0 +1,30 @@ + Compiling aleph-sv v0.3.0 (/Users/ex/GitHub/aleph/crates/aleph-sv) + Finished `test` profile [unoptimized + debuginfo] target(s) in 4.09s + Running tests/noise_driver.rs (target/debug/deps/noise_driver-f298de4eadb158cd) + +running 3 tests +test empty_model_reproduces_noiseless_distribution ... ok +test deterministic_same_seed_same_counts ... ok +test noise_actually_fires_under_correct_gate_name ... ok + +test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 3.77s + + Running tests/noise_oracle.rs (target/debug/deps/noise_oracle-526d05c514191d8c) + +running 6 tests +test oracle_readout ... ok +test oracle_phase_damp ... ok +test oracle_combined_ghz3 ... ok +test oracle_amp_damp ... ok +test oracle_depol_cx ... ok +test oracle_depol_h ... ok + +test result: ok. 6 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 9.01s + + Running tests/single_shot.rs (target/debug/deps/single_shot-0cb2b9914ec5be40) + +running 2 tests +test noise_with_pauli_kraus_and_readout_is_worker_invariant ... ok +test single_and_many_draws_match_analytic_distribution ... ok + +test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.54s diff --git a/docs/perf/data/p8-00/check-4.log b/docs/perf/data/p8-00/check-4.log new file mode 100644 index 00000000..e43ece15 --- /dev/null +++ b/docs/perf/data/p8-00/check-4.log @@ -0,0 +1,29 @@ + Compiling aleph-qec v0.3.0 (/Users/ex/GitHub/aleph/crates/aleph-qec) + Finished `test` profile [unoptimized + debuginfo] target(s) in 3.19s + Running tests/binomial_interval.rs (target/debug/deps/binomial_interval-1df52106e512bb6d) + +running 7 tests +test boundaries_no_evidence_and_invalid_tallies ... ok +test conditional_paired_comparison_does_not_assume_independent_rates ... ok +test large_counts_and_overlap_semantics ... ok +test reference_values ... ok +test exact_small_sample_coverage ... ok +test random_direct_counts_produce_ordered_finite_bounds ... ok +test ordered_monotone_symmetric_bounds ... ok + +test result: ok. 7 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.29s + + Running tests/bounded_dem_experiment.rs (target/debug/deps/bounded_dem_experiment-98b2a3e0c79ee9be) + +running 9 tests +test pymatching_batch_boundaries_and_zero_observables ... ignored, requires numpy + stim + pymatching; set PYMATCHING_PYTHON +test zero_batch_size_is_rejected_even_for_zero_shots ... ok +test huge_requested_count_does_not_materialize_all_shots ... ok +test zero_observables_require_exact_empty_corrections ... ok +test deterministic_mechanisms_produce_exact_tallies ... ok +test batch_bound_tail_and_absolute_stream_are_preserved ... ok +test short_long_empty_or_wrong_width_predictions_are_rejected_in_later_batches ... ok +test default_limit_splits_campaign_and_reports_late_schema_index ... ok +test fixed_seed_matches_full_artifact_path_across_batches_and_workers ... ok + +test result: ok. 8 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out; finished in 0.08s diff --git a/docs/perf/data/p8-00/check-5.log b/docs/perf/data/p8-00/check-5.log new file mode 100644 index 00000000..51d388a3 --- /dev/null +++ b/docs/perf/data/p8-00/check-5.log @@ -0,0 +1,8 @@ +test_asymmetric_bounds_and_zero_trials (scripts.python.test_qec_threshold_plot.TestThresholdBounds) ... ok +test_bad_counts_and_bounds (scripts.python.test_qec_threshold_plot.TestThresholdBounds) ... ok +test_legacy_half_width_is_not_reinterpreted (scripts.python.test_qec_threshold_plot.TestThresholdBounds) ... ok + +---------------------------------------------------------------------- +Ran 3 tests in 0.003s + +OK diff --git a/docs/perf/data/p8-00/cli-smoke.txt b/docs/perf/data/p8-00/cli-smoke.txt new file mode 100644 index 00000000..55cf3126 --- /dev/null +++ b/docs/perf/data/p8-00/cli-smoke.txt @@ -0,0 +1,3 @@ +counts (32 shots, seed=7): + |00⟩ 18 (0.5625) + |11⟩ 14 (0.4375) diff --git a/docs/perf/data/p8-00/saved-receipts.json b/docs/perf/data/p8-00/saved-receipts.json new file mode 100644 index 00000000..03f084b1 --- /dev/null +++ b/docs/perf/data/p8-00/saved-receipts.json @@ -0,0 +1,44 @@ +[ + { + "directory": "docs/perf/data/p8-13/server/cpu-control-run", + "status": "executed", + "exit_code": 0, + "expected_exit_code": 0, + "passed": true + }, + { + "directory": "docs/perf/data/p8-13/server/cuda-disabled-run", + "status": "failed", + "exit_code": 2, + "expected_exit_code": 2, + "passed": true + }, + { + "directory": "docs/perf/data/p8-13/server/cuda-f32-run", + "status": "executed", + "exit_code": 0, + "expected_exit_code": 0, + "passed": true + }, + { + "directory": "docs/perf/data/p8-13/server/cuda-fusion-run", + "status": "contaminated", + "exit_code": 2, + "expected_exit_code": 2, + "passed": true + }, + { + "directory": "docs/perf/data/p8-13/server/cuda-fusion-settled", + "status": "executed", + "exit_code": 0, + "expected_exit_code": 0, + "passed": true + }, + { + "directory": "docs/perf/data/p8-13/server/cuda-small-run", + "status": "executed", + "exit_code": 0, + "expected_exit_code": 0, + "passed": true + } +] diff --git a/docs/perf/data/p8-00/status.json b/docs/perf/data/p8-00/status.json new file mode 100644 index 00000000..d54aefd5 --- /dev/null +++ b/docs/perf/data/p8-00/status.json @@ -0,0 +1,234 @@ +{ + "as_of": "2026-10-18", + "main_revision": "6114dab7e18c4deac0fda3039060e00f7661b653", + "closed_capabilities": 13, + "total_capabilities": 16, + "issues": [ + { + "number": 550, + "title": "[P8-00] Performance audit integration and end-to-end recovery report", + "state": "OPEN", + "url": "https://github.com/aleph-sim/aleph/issues/550" + }, + { + "number": 551, + "title": "[P8-01] Use optimized CPU and Metal execution in the CLI", + "state": "OPEN", + "url": "https://github.com/aleph-sim/aleph/issues/551", + "pull_request": { + "number": 569, + "state": "OPEN", + "url": "https://github.com/aleph-sim/aleph/pull/569", + "headRefOid": "a32d4d2e574cd77b2a3828a130072a573e381945", + "mergeCommit": null + } + }, + { + "number": 552, + "title": "[P8-02] Integrate CUDA fusion and layered dispatch into public frontends", + "state": "CLOSED", + "url": "https://github.com/aleph-sim/aleph/issues/552", + "pull_request": { + "number": 583, + "state": "MERGED", + "url": "https://github.com/aleph-sim/aleph/pull/583", + "headRefOid": "94cf457034d7e57241ca7bb846e258ea6d873593", + "mergeCommit": { + "oid": "07a71f98e1ca4758d6f1e0a62c10fe76b2865ce3" + } + } + }, + { + "number": 553, + "title": "[P8-03] Materialize Python GPU amplitudes lazily", + "state": "CLOSED", + "url": "https://github.com/aleph-sim/aleph/issues/553", + "pull_request": { + "number": 568, + "state": "MERGED", + "url": "https://github.com/aleph-sim/aleph/pull/568", + "headRefOid": "24daa1456b3b54a6a206f5925f84083ded2bed4b", + "mergeCommit": { + "oid": "bfd939da86dd2cc4b71d83ee476024335fe3caba" + } + } + }, + { + "number": 554, + "title": "[P8-04] Plan backend reach from requested operations and peak memory", + "state": "CLOSED", + "url": "https://github.com/aleph-sim/aleph/issues/554", + "pull_request": { + "number": 581, + "state": "MERGED", + "url": "https://github.com/aleph-sim/aleph/pull/581", + "headRefOid": "99b7da643d5c41ff3a3e65ddf56765c07976e501", + "mergeCommit": { + "oid": "8e6ddecec9ca947b73765ddc49a8dfd9fef0685f" + } + } + }, + { + "number": 555, + "title": "[P8-05] Replace full-state multi-pass CUDA CDF sampling", + "state": "CLOSED", + "url": "https://github.com/aleph-sim/aleph/issues/555", + "pull_request": { + "number": 582, + "state": "MERGED", + "url": "https://github.com/aleph-sim/aleph/pull/582", + "headRefOid": "6b219610459626bfa6c724ebcced12fafd03843e", + "mergeCommit": { + "oid": "928bdb7eec894d2f4c51fee03bf45f60a6ce2267" + } + } + }, + { + "number": 556, + "title": "[P8-06] Avoid building CPU alias tables for one or few samples", + "state": "CLOSED", + "url": "https://github.com/aleph-sim/aleph/issues/556", + "pull_request": { + "number": 571, + "state": "MERGED", + "url": "https://github.com/aleph-sim/aleph/pull/571", + "headRefOid": "792cb4d661f18473aafea2ab4010aef8a50bd3ff", + "mergeCommit": { + "oid": "e51ba789abc21295c79e1326548099cbb9d3a7d6" + } + } + }, + { + "number": 557, + "title": "[P8-07] Bound noisy trajectory memory and reuse per-worker state", + "state": "CLOSED", + "url": "https://github.com/aleph-sim/aleph/issues/557", + "pull_request": { + "number": 572, + "state": "MERGED", + "url": "https://github.com/aleph-sim/aleph/pull/572", + "headRefOid": "b3a0482ac62b309c5f21bf8af8381cf0a9eef964", + "mergeCommit": { + "oid": "c283272747877863fa4a2723421c8d710a8550b3" + } + } + }, + { + "number": 558, + "title": "[P8-08] Probe GPU availability without compiling discarded backends", + "state": "CLOSED", + "url": "https://github.com/aleph-sim/aleph/issues/558", + "pull_request": { + "number": 567, + "state": "MERGED", + "url": "https://github.com/aleph-sim/aleph/pull/567", + "headRefOid": "e839c0a0188fd5b1d4b021083d4ff654c29db78d", + "mergeCommit": { + "oid": "dce007348272c0e8f00be8d43f3f8b3a218a0b4c" + } + } + }, + { + "number": 559, + "title": "[P8-09] Coalesce GPU QEC scratch while retaining serial-shot arithmetic", + "state": "CLOSED", + "url": "https://github.com/aleph-sim/aleph/issues/559", + "pull_request": { + "number": 592, + "state": "MERGED", + "url": "https://github.com/aleph-sim/aleph/pull/592", + "headRefOid": "8981aa4143b110a06af06e7f75a3a1f7c693a011", + "mergeCommit": { + "oid": "e9f61b6923cef985d9ff7178a2f2a6f59e978e27" + } + } + }, + { + "number": 560, + "title": "[P8-10] Prototype cooperative GPU BP with explicit accuracy and latency trade-offs", + "state": "CLOSED", + "url": "https://github.com/aleph-sim/aleph/issues/560", + "pull_request": { + "number": 593, + "state": "MERGED", + "url": "https://github.com/aleph-sim/aleph/pull/593", + "headRefOid": "eeff5ea5a4ba416fa13e1e749d3baabc3c222826", + "mergeCommit": { + "oid": "ab3aed8027132f42f9df4e9ddc614403da5a3147" + } + } + }, + { + "number": 561, + "title": "[P8-11] Prove a competitive Metal MPS factorization before another pipeline rewrite", + "state": "OPEN", + "url": "https://github.com/aleph-sim/aleph/issues/561" + }, + { + "number": 562, + "title": "[P8-12] Validate communication-aware plans on real multi-GPU workloads", + "state": "OPEN", + "url": "https://github.com/aleph-sim/aleph/issues/562" + }, + { + "number": 563, + "title": "[P8-13] Make public-workflow baselines and experiment gates reproducible", + "state": "CLOSED", + "url": "https://github.com/aleph-sim/aleph/issues/563", + "pull_request": { + "number": 584, + "state": "MERGED", + "url": "https://github.com/aleph-sim/aleph/pull/584", + "headRefOid": "b59e33760759d4cd5a8bf12b94712ddbe19010b2", + "mergeCommit": { + "oid": "40e625cec3bf43ad193fdddf9ac43232ae7deb3a" + } + } + }, + { + "number": 564, + "title": "[P8-14] Evaluate combined tile residency and overlapped paging", + "state": "CLOSED", + "url": "https://github.com/aleph-sim/aleph/issues/564", + "pull_request": { + "number": 594, + "state": "MERGED", + "url": "https://github.com/aleph-sim/aleph/pull/594", + "headRefOid": "58de2a1f34bcdeab4ca89ae5b7cca4d34c7ac3fa", + "mergeCommit": { + "oid": "e079665cf9610b305984347d197805c172f4b2e3" + } + } + }, + { + "number": 565, + "title": "[P8-15] Bound DEM experiment batches and validate decoder result contracts", + "state": "CLOSED", + "url": "https://github.com/aleph-sim/aleph/issues/565", + "pull_request": { + "number": 575, + "state": "MERGED", + "url": "https://github.com/aleph-sim/aleph/pull/575", + "headRefOid": "b9f6696dfdc6a83822a89016e8aafbb800d408fd", + "mergeCommit": { + "oid": "809acd3e4a05ec39b8013790666773ee4f6f39f6" + } + } + }, + { + "number": 566, + "title": "[P8-16] Use valid logical-error intervals for zero and rare observed failures", + "state": "CLOSED", + "url": "https://github.com/aleph-sim/aleph/issues/566", + "pull_request": { + "number": 570, + "state": "MERGED", + "url": "https://github.com/aleph-sim/aleph/pull/570", + "headRefOid": "d8a74140604c93c1e2699d6241a57c4b1893c0c8", + "mergeCommit": { + "oid": "8c9223f2d95998e9cd26c484cb35690816d47080" + } + } + } + ] +} diff --git a/docs/perf/data/p8-00/validation.json b/docs/perf/data/p8-00/validation.json new file mode 100644 index 00000000..6811751d --- /dev/null +++ b/docs/perf/data/p8-00/validation.json @@ -0,0 +1,84 @@ +{ + "date": "2026-10-18", + "purpose": "CPU correctness/documentation smoke; no performance acceptance and no fresh GPU execution", + "source_base": "6114dab7e18c4deac0fda3039060e00f7661b653", + "platform": "macOS-27.0.1-arm64-arm-64bit", + "python": "3.9.6", + "rustc": "rustc 1.98.1 (48a229cea 2026-09-01)", + "cli_sha256": "1b9c9396d9b904e3c7e1e8a82a6b541f8f9f7d7f3d372303f798b94e7b34b8f0", + "environment": { + "RAYON_NUM_THREADS": "2", + "ALEPH_BENCH_TEST_CLI": "/Users/ex/GitHub/aleph/target/release/aleph" + }, + "commands": [ + { + "command": [ + "python3", + "-m", + "unittest", + "discover", + "-s", + "scripts/benchmarks", + "-v" + ], + "exit_code": 0, + "log": "check-1.log" + }, + { + "command": [ + "cargo", + "test", + "--locked", + "-p", + "aleph-backend", + "--lib", + "memory::tests" + ], + "exit_code": 0, + "log": "check-2.log" + }, + { + "command": [ + "cargo", + "test", + "--locked", + "-p", + "aleph-sv", + "--test", + "single_shot", + "--test", + "noise_driver", + "--test", + "noise_oracle" + ], + "exit_code": 0, + "log": "check-3.log" + }, + { + "command": [ + "cargo", + "test", + "--locked", + "-p", + "aleph-qec", + "--test", + "bounded_dem_experiment", + "--test", + "binomial_interval" + ], + "exit_code": 0, + "log": "check-4.log" + }, + { + "command": [ + "python3", + "-m", + "unittest", + "scripts/python/test_qec_threshold_plot.py", + "-v" + ], + "exit_code": 0, + "log": "check-5.log" + } + ] +} diff --git a/docs/perf/p8-13-public-workflow-gates.md b/docs/perf/p8-13-public-workflow-gates.md index 88ce4901..80481b87 100644 --- a/docs/perf/p8-13-public-workflow-gates.md +++ b/docs/perf/p8-13-public-workflow-gates.md @@ -34,7 +34,7 @@ ordinary CPU `aleph run` and CUDA Python `aleph.run`, not a substitute kernel. | [Sparse Blossom](q1-03b-sparse-blossom.md) | Current mainline decoder **implementation**, historical timing snapshot; sparse replaces the former dense matching default | Recalibrate the **sparse** path and PyMatching on identical DEM/syndromes, shots, precision and one core before a new speed claim | | [GPU QEC Monte Carlo](qec-q3-montecarlo.md), GPU UF vs CPU UF | Historical **same-decoder** snapshot, 4.2x aggregate in that report; not a current CPU baseline | UF on both sides; equal noise model and correction/rate acceptance; report resource difference explicitly | | Same report, GPU UF vs CPU dense MWPM | Historical **cross-algorithm** workflow comparison (11.8x to 107x headline); dense MWPM is archived as a baseline for current Sparse Blossom | New MWPM claims must use sparse CPU matching; retain UF/MWPM accuracy separately; never pool with same-decoder table | -| [Q1-03c PR #578](https://github.com/aleph-sim/aleph/pull/578) | Unmerged allocation candidate, not a mainline baseline | Keep candidate and main revisions separate; do not silently attribute its results to main | +| [Q1-03c PR #578](https://github.com/aleph-sim/aleph/pull/578), followed by [#595](https://github.com/aleph-sim/aleph/pull/595) | Both merged by 2026-10-18, after this reference campaign; [allocation](q1-03c-reuse-allocations.md) and [event-scan](q1-03c-event-scans.md) evidence retain their own measured revisions | Use the current sparse path for new comparisons; do not relabel this earlier campaign as a post-merge measurement or claim the original 1.5× target was met | The new run manifests keep comparison classes separate. This issue does not reclassify historical measurements as newly executed calibrations or substitute diff --git a/docs/perf/p8-performance-recovery.md b/docs/perf/p8-performance-recovery.md new file mode 100644 index 00000000..d60b279a --- /dev/null +++ b/docs/perf/p8-performance-recovery.md @@ -0,0 +1,281 @@ +# Phase 8: performance recovery and remaining gates + +Tracker: [#550](https://github.com/aleph-sim/aleph/issues/550). Status snapshot: +**2026-10-18**, main `6114dab7e18c4deac0fda3039060e00f7661b653`. +**13 of 16 capability issues are closed; Phase 8 is not complete.** P8-01, +P8-11 and P8-12 remain open. This report integrates existing measurements; it is +not a new all-backend benchmark campaign or a v1.0 release qualification. + +## What changed for callers + +The audit exposed useful optimized implementations that public calls did not +reach, unnecessary GPU initialization and amplitude downloads, and memory policies +that counted the state but missed the requested output. The largest confirmed +repairs remove work: discarded backend construction, full-state exports, scan +passes and per-shot buffers. Kernel improvements alone do not remove setup, +sampling, result conversion or decoder accuracy limits. + +| Workflow | Measured result | Scope and qualification | +|---|---|---| +| CPU CLI cold process | 194.561 → 1.271 ms on the 2-qubit explicit statevector request | P8-08 removed discarded CUDA construction on the CUDA-enabled Linux CLI; full-process timing, not isolated startup or CPU gate throughput. Three fresh processes per revision on a shared host. | +| CUDA Python counts-only | FP64 n=26 public run/counts 927.076 → 272.642 ms; process peak RSS 2208.562 → 160.457 MiB; run DTOH 1,073,742,096 → 272 B | P8-03 removes eager amplitudes. First explicit state read pays deferred work; read-workflow peak RSS and sampled GPU peak did not improve. | +| CUDA public fusion | n=22 FP64 brickwall Python caller 348.510 → 258.380 ms; median paired ratio 1.342, bootstrap 95% interval 1.319–1.404 | P8-13 rechecks P8-02 with the newer sampler. Includes parse/run/first state read/result release; five pairs on one shared GPU, not a universal crossover. | +| CUDA large-state sampling | n=22 /1024 shots FP64 7.1560 → 0.47761 ms (14.98×), FP32 6.8450 → 0.44612 ms (15.34×) | P8-05 sampling boundary only; preparation/JIT excluded. n=22 is the largest state measured in that campaign. | +| CPU one-shot sampling | FP64 AoS n=20 10.366 → 0.820 ms; additional sampler heap peak 36 MiB → 8 B | P8-06 prepared-state sampler, one worker. No Python/CLI whole-call ratio is inferred. | +| Noisy trajectories | n=20 /8 mixed shots, one worker: allocated bytes 142,614,028 → 25,167,408 | P8-07 reduces allocation traffic and bounds owned contexts. Same-concurrency large-state peak is essentially unchanged; wider-state time ratios 0.983–1.044× do not establish a gain. | +| DEM experiments | 1M shots: additional live requested heap 102,672,376 → 6,741,056 B (15.23× lower) | P8-15 default batches, fixture with 128 detectors/3 observables; exact tallies match. No material throughput gain established. | + +The ratios above have different baselines, timing scopes, hardware and requested +outputs. **Do not multiply them into a Phase 8 speedup.** All quoted timings are +historical child-campaign observations, not timings rerun on this report's main +revision. Later source integration checks establish compatibility, not identical +performance on every subsequent revision. + +## Capability and adoption ledger + +“Closed” describes GitHub task state; adoption is separate. Research prototypes +can meet their criteria while remaining explicit opt-ins. The linked child reports +retain precise oracles, provenance, raw observations and adverse cases. + +| ID / issue | State / PR | Mainline behavior and evidence | +|---|---|---| +| P8-01 / [#551](https://github.com/aleph-sim/aleph/issues/551) | Open; draft [#569](https://github.com/aleph-sim/aleph/pull/569) | Optimized CPU/Metal CLI candidate awaits idle-Metal timing. Main CPU CLI still uses the raw driver. Draft CPU measurements are not mainline results. | +| P8-02 / [#552](https://github.com/aleph-sim/aleph/issues/552) | Closed; [#583](https://github.com/aleph-sim/aleph/pull/583) | Default strict FP64/FP32 width-3 CUDA plans in CLI/Python; raw control retained. [Report](p8-02-public-cuda-execution.md), [data](data/p8-02/summary.json). | +| P8-03 / [#553](https://github.com/aleph-sim/aleph/issues/553) | Closed; [#568](https://github.com/aleph-sim/aleph/pull/568) | Lazy Python GPU amplitudes, copy-once host cache; results retain device amplitudes until materialized/dropped. [Report](p8-03-lazy-gpu-amplitudes.md), [manifest](data/p8-03-manifest.json). | +| P8-04 / [#554](https://github.com/aleph-sim/aleph/issues/554) | Closed; [#581](https://github.com/aleph-sim/aleph/pull/581) | Operation-aware device/host preflight; no counts-to-norm substitution. [Report](p8-04-memory-planning.md), [allocation audit](data/p8-04-allocation-audit.jsonl). | +| P8-05 / [#555](https://github.com/aleph-sim/aleph/issues/555) | Closed; [#582](https://github.com/aleph-sim/aleph/pull/582) | Default deterministic hierarchy/local-CDF policy; old scan selectable from Rust. [Report](p8-05-hierarchical-sampling.md), [raw estimates](data/p8-05/criterion.json). | +| P8-06 / [#556](https://github.com/aleph-sim/aleph/issues/556) | Closed; [#571](https://github.com/aleph-sim/aleph/pull/571) | Zero/one-shot CPU path; alias remains at ≥2 shots. [Report](p8-06-single-shot-sampling.md), [manifest](data/p8-06-manifest.json). | +| P8-07 / [#557](https://github.com/aleph-sim/aleph/issues/557) | Closed; [#572](https://github.com/aleph-sim/aleph/pull/572) | Bounded reusable trajectory contexts; sparse CLI/Python counts, dense Rust compatibility. [Report](p8-07-bounded-noisy-trajectories.md), [manifest](data/p8-07-manifest.json). | +| P8-08 / [#558](https://github.com/aleph-sim/aleph/issues/558) | Closed; [#567](https://github.com/aleph-sim/aleph/pull/567) | Lazy discovery; selected backend still initializes per call. [Report](p8-08-cheap-gpu-probes.md), [cold Python data](data/p8-08-cuda-cold-python.json). | +| P8-09 / [#559](https://github.com/aleph-sim/aleph/issues/559) | Closed; [#592](https://github.com/aleph-sim/aleph/pull/592) | Explicit edge-major/warp-tiled BP scratch; shot-major default retained. [Report](p8-09-bp-scratch.md), [raw evidence](data/p8-09-bp-scratch/README.md). | +| P8-10 / [#560](https://github.com/aleph-sim/aleph/issues/560) | Closed; [#593](https://github.com/aleph-sim/aleph/pull/593) | Explicit Rust cooperative FP64 BP; serial default retained. [Report](p8-10-cooperative-bp.md), [raw evidence](data/p8-10-cooperative-bp/README.md). | +| P8-11 / [#561](https://github.com/aleph-sim/aleph/issues/561) | Open | Competitive multi-threadgroup Metal MPS factorization unproven. [Phase 5.8](phase5.8.md) is historical negative evidence, not a new prototype verdict. | +| P8-12 / [#562](https://github.com/aleph-sim/aleph/issues/562) | Open | Current compiler needs actual multi-GPU measurements. [Compiler model](p6-05-compiler.md) gains do not establish measured scaling. | +| P8-13 / [#563](https://github.com/aleph-sim/aleph/issues/563) | Closed; [#584](https://github.com/aleph-sim/aleph/pull/584) | Public-route receipts, counter admission and contamination gates. [Report](p8-13-public-workflow-gates.md), [harness](../../scripts/benchmarks/README.md). | +| P8-14 / [#564](https://github.com/aleph-sim/aleph/issues/564) | Closed; [#594](https://github.com/aleph-sim/aleph/pull/594) | Explicit FP64 residency + overlap; defaults unchanged. [Report](p8-14-resident-overlap.md), [raw evidence](data/p8-14-resident-overlap/README.md). | +| P8-15 / [#565](https://github.com/aleph-sim/aleph/issues/565) | Closed; [#575](https://github.com/aleph-sim/aleph/pull/575) | Default 65,536-shot batches and exact decoder-result schema checks. [Report](p8-15-bounded-dem-experiments.md), [manifest](data/p8-15-manifest.json). | +| P8-16 / [#566](https://github.com/aleph-sim/aleph/issues/566) | Closed; [#570](https://github.com/aleph-sim/aleph/pull/570) | Explicit direct-binomial bounds; legacy Wald fields remain labeled. [Contract](../qec/binomial-intervals.md), [coverage](data/p8-16-binomial-coverage.json). | + +Follow-up [#573 / PR #574](https://github.com/aleph-sim/aleph/pull/574) +validated manual CPU fused IR and reduced repeated validation overhead. Its +[report](p8-573-manual-fused-ir.md) preserves cold first-use regressions and +AVX512 measurements. Sparse Blossom [#578](https://github.com/aleph-sim/aleph/pull/578) +and [#595](https://github.com/aleph-sim/aleph/pull/595) are now merged too; their +[allocation](q1-03c-reuse-allocations.md) and [event-scan](q1-03c-event-scans.md) +results are related QEC work, not extra P8 capability completions. #511 was closed +by user acceptance at 1.468× the original d=11 baseline, below its 1.5× target. + +## Timing, launches and output costs + +### CPU CLI and discovery + +The P8-08 Linux startup comparison used separate locked release binaries/wheels, +three interleaved fresh processes per revision, Bell n=2 /32 shots /seed 7. +CLI cold time includes startup, execution/output and exit. Python cold time also +includes interpreter/import; warm time is the median of calls 2–20, not a warm +retained backend. Explicit Python SV cold wall fell 212.764 → 24.286 ms; selected +CUDA-FP32 cold wall fell 373.970 → 217.536 ms. Candidate construction counters are +zero for CPU/Clifford-auto and one per explicit GPU call. No module cache was added. + +P8-13's current CPU route is a **no-optimization control**: GHZ n=8 /16 shots, +identical executable, 1.542 vs 1.522 ms; paired bootstrap interval 0.396–1.714. +It establishes a working receipt boundary, not a CPU fusion win. CPU startup +cannot be isolated by subtracting overlapping native stages and stays `null`. +The draft #569 has CPU wins and QFT/tiny-input regressions, but its Metal gate and +integration review are pending. Its `--raw`/`--execution-stats` switches must not +be documented as available on this main revision. + +### CUDA Python: execution is only part of caller time + +P8-02's n=22 /32 shots /seed 552 same-artifact raw/optimized campaign spans five +families and both precisions in CLI/Python. Public caller improvements span +1.07–4.11× in its 20 cells, with three pairs per cell. It predates P8-05; the +sampling algorithm is held fixed within that experiment. FP64 decomposed-QFT +Python execution falls 480.90 → 39.94 ms and gate launches 1177 → 43, while fusion +planning grows 0.034 → 0.545 ms. Caller time falls 581.09 → 141.50 ms, a smaller +ratio. FP64 backend/JIT initialization is about 20 ms and context/preflight +typically 64–79 ms in that campaign. These native stage observations exclude +parts of final teardown; external caller time remains authoritative. Native-CP +FP32 QFT instead regresses (see below). + +P8-13 subsequently measures the combined mainline planner/sampler through public +Python, using five pairs and one preflight pair per campaign, one thread/worker, +and a full same-process warmup with a new backend each call. At n=22 brickwall, +launches fall 327 → 37 and execution 159.754 → 49.039 ms. Sampling is about +0.59–0.62 ms; first state materialization is about 94–107 ms. The 1.342 paired +caller ratio includes parse/run/read/result release, while total subprocess wall +also includes imports/warmups/preflight. These clocks are kept separate. + +For counts-only, P8-03's n=26 FP64 run transfers only 272 B (norm/sampling scalars +plus 32 outcomes). An explicit first state read downloads 1,073,741,824 B and +creates a complex128 host cache; repeated reads copy independent NumPy arrays +without another download. The first-read median grows 255.782 → 897.405 ms as +work moves out of run. Read-workflow peak RSS stays about 2208.6 MiB. A counts-only +result still owns 1 GiB of amplitudes until read or destroyed: retaining many such +results can exhaust GPU memory. Logical ownership counters are not allocator peaks. + +### Sampling, allocation and reach are different claims + +P8-05's n=22 /1024-shot sampler changes construction from 23 launches to two +hierarchy kernels (three with the local CDF). Two norm launches and one search +launch remain. Hierarchical scratch is 132,112 B instead of 67,108,864 B for +the reference's two full CDF buffers. Measured live pool peaks become 67,519,520 B +FP64 and 33,965,088 B FP32 versus 134,496,272 /100,941,840 B. These exclude driver, +module and reserved pool pages. The timed boundary includes norm, guards, +allocation, RNG/upload, search and download, but excludes preparation and JIT. + +The planner's admission of low-shot n=31 FP32 under a synthetic 20 GiB budget is +**modelled reach**, not an actual n=31 sampling run. P8-04 real allocation audits +use n=18; P8-05 maximum measured state is n=22. Device/host headroom and concurrent +allocation races remain; passing preflight reserves no memory. Counts never +silently become a norm. State, execution scratch, sampling and export must be +budgeted together using the actual plan and selected sampler. + +P8-06 one-shot CPU draws remove the alias table; ≥2 shots keep it. P8-07 then +reuses one dense state/Kraus scratch per bounded worker context. These sequential +repairs have different baselines. In P8-07, n=20 with 1/2/4 contexts measures +16/32/64 MiB additional heap, below conservative bounds. Lower allocated bytes +does not mean lower same-concurrency peak or fewer full-state reset/channel passes. +The default 1 GiB incremental buffer policy is not an RSS cap; outcomes remain +O(shots), and Python dictionaries/CLI formatting are outside that Rust budget. + +## Why some approaches stay opt-in or did not pay + +| Finding | Retained evidence | Consequence | +|---|---|---| +| Fewer launches need not improve small public calls | P8-13 n=8 FP64 and n=18 FP32 reduce execution/launches but paired caller intervals include 1 | Stop kernel-only tuning when setup/output dominates; retain public clocks. | +| Native CP FP32 fusion can lose | P8-02 n=22 caller 144.32 → 153.60 ms; reusable-plan execution 13.982 → 19.838 ms | Raw control remains; a precision/size crossover needs calibration before new routing. | +| TF32 width 4 violates the strict public contract | P8-02 coherent-state amplitude error 1.4084e-4 vs CPU FP64 | Public plans keep ordinary arithmetic and width 3; tolerance was not relaxed. | +| Small/high-shot sampling has little scan work to remove | P8-05 n=14 /1M FP32 samples 7.47 → 8.13 ms | Auto is an empirical deterministic policy, not universally fastest. | +| Reuse removes allocations, not trajectory arithmetic | P8-07 wide-state ratios near 1; tiny four-worker Kraus aggregate 6.9% slower including the retained outlier | Accept memory/context control, not a universal speed claim. Final narrow replication improves 1.053× but does not erase prior variability. | +| Better coalescing can worsen locality/divergence | P8-09 warp-tiled d11 /4096 /.03 improves 1.44×; at .001 takes 1.58× longer | Keep shot-major default and explicit alternatives; no batch-size-only auto rule. | +| Cooperative BP shortens the per-shot serial path | P8-10 all 22 tested cells improve 1.78–35.97× against the faster serial reference | Explicit Rust opt-in on one measured GPU; no automatic policy or accuracy gain. | +| Residency and overlap interact | P8-14 held-out locality n=26 improves 1.36× vs residency, 1.42× vs equal-payload larger synchronous tile; shallow case 0.997×, alternating 0.999× vs faster existing path | Explicit FP64 combined executor; do not multiply historical gains or infer FP32 benefit. | +| Bounds improve statistical correctness, not execution | P8-16 0/1000 failures gives [0, 0.003682], not zero uncertainty | Direct fixed-iid binomial contract only; stratification/drift need their own estimators. | + +BP timing is upload + reused decode + mask download/synchronization, excluding +packing, model/JIT/oracle and warm allocation. Identical CPU/GPU corrections do +not make BP a better decoder: surface11 /.03 /4096 still has 1809 logical failures +and 4093 CPU nonconvergences in P8-10. Its varied-input tails are empirical fixed +mixtures, not population latency guarantees. + +Paging timing includes pinned/ring allocation, initialization, execution and drain, +but excludes backend/JIT and norm/export. Forced n=24/26 states fit in-core; this +isolates paging schedule effects and proves no new reach. Matched timelines show +residency and combined locality each move 2 GiB in each direction, while per-gate +overlap moves 40 GiB. Combining overlap saves wall time without further reducing +residency's bytes; copy/kernel busy durations overlap and cannot be summed as wall. + +## Provenance and experiment quality + +| Campaigns | Machine / software | Comparison discipline | +|---|---|---| +| P8-02/03/04/05/08/09/10/13/14 | Shared Linux RTX 4000 SFF Ada, ~20 GiB, driver 580.178.04, CUDA/NVRTC 13, Rust 1.96.0. Early public Python: 3.12.3, NumPy 2.5.0. Exact builds/seeds in child manifests. | Services/clocks unchanged. Early campaigns use three pairs and boundary utilization; later campaigns keep fixed mirrored blocks, continuous telemetry and separate profiles. None proves exclusive GPU use. | +| P8-06 | i5-13500, Linux 6.8, Rust 1.96, CPU0, one Rayon worker, seed 556 | Independent Cargo targets, A/B/B/A; 10 Criterion samples/repeat. Invalid identical-binary pilot discarded. | +| P8-07 | Dual Xeon Silver 4114, Linux 7.0, Rust 1.96, CPUs 3/7/9/11 with SMT siblings monitored, 1/4 workers, seed 557 | Per-case A/B/B/A, external-overlap admission; policy changes, rejected attempts, outlier and final follow-up all retained. | +| P8-15 | Same Xeon host, one worker on CPU3, monitor CPU39, native release flags, seed 565 | A/B/B/A with CPU3/23 guard; common-stage and total boundaries are separate. | +| P8-03 Metal | macOS arm64, Python 3.12, NumPy 2.3.5 | Retained earlier eager-path wheel, not rebuilt CUDA baseline revision; unified-memory RSS is not PCIe traffic. No new Metal timing campaign here. | + +Child manifests pin source/binary hashes, flags, exact inputs and raw samples. +Use those identities to reproduce a historical result; use current main with +new hashes to establish a new baseline. Several campaigns found stale shared-target +artifacts or CI interference. These failed experiments remain documented and are +excluded from accepted summaries. A feature compile, missing-device skip, or an +unchanged counter cannot stand in for actual hardware execution. + +P8-13's saved receipts can be revalidated without CUDA. Retained `failed` and +`contaminated` campaigns stay non-successful; they must not be dropped merely +because executed campaigns are easier to summarize. Hardware oracles, allocation +counters, sampled RSS/VRAM and analytical budgets answer different questions. +Unavailable stage/peak observations remain unavailable, rather than being filled +with zero or guessed from allocation formulas. + +## Reproduce and check the current integration + +Run from the repository root. These bounded CPU commands check public contracts +and saved evidence on a busy machine; **their timings are not performance data**. +The noisy contract tests cover the mixed Pauli/Kraus/readout path and worker/seed +invariance without a new expensive trajectory campaign. + +For this report update, the release CPU CLI built successfully and the commands +below passed **32 Rust + 16 Python tests**, including the actual public CLI guard. +One existing external PyMatching test remains ignored; no new PyMatching or GPU +run is claimed. All six saved P8-13 server campaigns revalidated (four executed, +one failed, one contaminated), preserving their original classifications. The +Bell smoke returned 18 `00` and 14 `11` outcomes. [Validation/provenance](data/p8-00/README.md) +records the exact base, CLI hash, commands and logs. This is scoped verification, +not a new full-workspace suite or hardware performance campaign. + +```sh +cargo build --locked --release -p aleph-cli +ALEPH_BENCH_TEST_CLI="$PWD/target/release/aleph" \ + python3 -m unittest discover -s scripts/benchmarks -v +cargo test --locked -p aleph-backend --lib memory::tests +cargo test --locked -p aleph-sv --test single_shot --test noise_driver --test noise_oracle +cargo test --locked -p aleph-qec --test bounded_dem_experiment --test binomial_interval +python3 -m unittest scripts/python/test_qec_threshold_plot.py -v +python3 scripts/benchmarks/public_workflow.py --validate-result \ + docs/perf/data/p8-13/server/cuda-fusion-settled +``` + +For an ordinary current CPU CLI smoke, create a small input and check counts: + +```sh +printf 'OPENQASM 3.0; include "stdgates.inc"; qubit[2] q; h q[0]; cx q[0],q[1];\n' > /tmp/p8-bell.qasm +target/release/aleph run /tmp/p8-bell.qasm --backend sv --shots 32 --seed 7 +``` + +On a prepared Linux CUDA installation, use a release CLI built with `--features +cuda` and a separately extracted `python,cuda` wheel on PYTHONPATH. GPU tests +below must actually execute; missing hardware is not a pass. These are optional +hardware reproduction commands, not tests run for this documentation update: + +```sh +target/release/aleph run /tmp/p8-bell.qasm --backend cuda-f32 --shots 32 --cuda-diagnostics +target/release/aleph run /tmp/p8-bell.qasm --backend cuda-f32 --shots 32 --cuda-raw --cuda-diagnostics +ALEPH_TEST_PUBLIC_CUDA=1 ALEPH_CUDA_CLI="$PWD/target/release/aleph" \ + python3 scripts/python/test_public_cuda_execution.py -v +ALEPH_TEST_GPU_BACKENDS=cuda,cuda-f32 \ + python3 -m unittest discover -s scripts/python -p test_gpu_results.py -v +ALEPH_REQUIRE_CUDA=1 cargo test --release -p aleph-cuda --features cuda \ + --test qec_bp_oracle -- --nocapture +``` + +```python +import aleph +c = aleph.Circuit.from_qasm('OPENQASM 3.0; include "stdgates.inc"; qubit[2] q; h q[0]; cx q[0],q[1];') +r = aleph.run(c, backend="cuda-f32", shots=32, seed=7) +print(r.counts()) # counts-only; device amplitudes remain owned +print(r.execution_stats()) +state = r.statevector() # explicit download/widen/cache + independent array +del r # release cache/device ownership when no longer needed +``` + +Use the child report commands for sampling Criterion, memory audits, noisy budget +examples and paging/BP raw-data checkers. Public A/B campaigns use the +[P8-13 harness](../../scripts/benchmarks/README.md) with edited absolute artifact +paths/full revisions and a new output directory. Do not reuse placeholder +revisions, shared Cargo targets or a busy Mac for performance acceptance. + +## Remaining exit gates and next decisions + +1. **P8-01 / #551:** integrate current main into draft #569, review semantics and + rerun CPU/Metal oracles. Finish paired public Metal timings on a free Mac, + including small planning overhead and losing workloads. Until then CPU/Metal + CLI optimizer integration remains pending; CUDA integration is already done. +2. **P8-11 / #561:** benchmark representative SVD/QR shapes and guards against + faer/current Metal before another full pipeline rewrite. Accept a competitive + measured regime or a documented no-go. Historical 16–22× MPS slowdowns do not + count as having run this experiment. +3. **P8-12 / #562:** obtain actual 2/4-GPU measurements with topology, work/pass + inflation and communication attribution. Compiler predictions and emulated + ranks do not close measured scaling. MPI/v1.0 work remains separate Phase 6. +4. **P8-00 / #550:** update this report and tracker as those three land; smoke-run + final documented routes and preserve adverse results. Only then evaluate the + phase exit criteria and close the tracker. This report PR uses **Refs #550**. + +Measured follow-up opportunities are native-CP FP32 routing, per-call context/JIT +costs, high-shot host sampling work, and noisy tail variability. Each needs a +separate profile, fixed output/precision, admission rule and focused issue before +implementation. The phase does not authorize another broad rewrite merely because +an individual kernel became faster.