Skip to content

perf(lindblad): faster, lower-memory CTPP steps with a shortlist-and-score default - #276

Open
MSRudolph wants to merge 48 commits into
mainfrom
ctpp/memory
Open

MSRudolph wants to merge 48 commits into
mainfrom
ctpp/memory

Conversation

@MSRudolph

@MSRudolph MSRudolph commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

Continuous-time Pauli propagation (CTPP) steps reuse their storage, and candidate selection for the support is modularized. With the default streaming proposer, the measured ladder cases run 18–32% faster and use 61–65% less peak memory than the original main baseline. On the dephased ring, peak memory falls by 67%, with no measurable runtime improvement. The optional shortlist proposer reduces runtime by 28–46% and peak memory by 57–76% across these cases.

Streaming remains the default. Shortlist is faster on the larger cases, but uses more memory than streaming on the ladder and changes the retained support. Both use magnitude-based retention.

Design

  • Single main storage: candidates are collected in the over-allocated slots of the main string and coefficient arrays. Coefficients are updated in place.
  • Modularized proposers: CandidateProposer is the interface for selecting which candidate strings to admit. Callers can supply a strategy through the existing step methods.
    • Alternative: ShortlistScoreProposer shortlists candidates by individual Hamiltonian contributions, then ranks them by full signed leakage.
    • Default: StreamingProposer remains the default. General dissipators use it automatically.
  • Automatic reuse: LindbladSpec reuses candidate lookup tables, source-order storage, generator connections, and exponential work arrays across step calls. Callers can own this storage through PcStepWorkspace.
  • General-coupling connection cache: connections identify the parent string and Hamiltonian term. Coefficients are read from the model, so unequal couplings need no special case.
  • Paired connection cache: without general dissipators, the Hamiltonian part of the generator is antisymmetric. If string i reaches j through term k, then j reaches i through k with the opposite sign. The cache stores each connected pair once, using 6-byte entries when Hamiltonian term IDs fit in 16 bits and 8-byte entries for larger term lists. The product applies each pair in both directions.
  • Compact Pauli strings: ppvm-lindblad's Word stores only the X and Z bit planes, 32 bytes at 128 qubits instead of 48. The number of qubits comes from the model. Hashes are computed when needed and match the general PauliWord's hashes and ordering, so hash-table order is unchanged. ppvm_pauli_sum symmetry routines still take the general type, exposed as GeneralWord, with conversions at that boundary.

Benchmarks vs the original main baseline

Measured on 2026-11-09 after removing forward-looking retention: median of three fresh, interleaved runs per implementation, Apple arm64 on AC power, mimalloc (the allocator the Python extension uses), 6 threads, MIMALLOC_PURGE_DELAY=0. Each case uses dt=0.1, retained support limit B and admission cap 3B. Times measure propagation; build time is excluded. Memory is OS-reported peak RSS.

The baseline is the original PR control at 342e3234. Both current proposers use the same binary built from the reverted working tree. Both models evolve a magnetization operator in the Heisenberg picture.

  • Ladder: 2 × 41 sites, open boundaries, no dissipation. H = Σ⟨ij⟩ (XᵢXⱼ + YᵢYⱼ) over nearest-neighbour bonds on a ladder topology. Observable is the average of Z on the two sites in the middle.
  • Ring: 61 sites with all-to-all couplings. H = Σᵢ<ⱼ Jᵢⱼ (XᵢXⱼ + YᵢYⱼ), with Jᵢⱼ ∝ rᵢⱼ^−1.1 (ring distance), normalized so that Σⱼ Jᵢⱼ² = 1. Each site also has Z dephasing at rate γ = 8. Observable is Z on the center site.

Default: streaming

Values show baseline → current.

Model String limit B Propagation time Peak RSS
82-spin ladder, Jt=0→5 2^17 13.42 → 9.17 s (−32%) 300 → 105 MiB (−65%)
82-spin ladder, Jt=0→5 2^19 78.24 → 64.32 s (−18%) 940 → 367 MiB (−61%)
61-spin dephased ring, Jt=0→2 2^19 276.90 → 277.20 s (unchanged) 1,885 → 629 MiB (−67%)

Optional: shortlist-and-score

Values show the same baseline → current with shortlist selected.

Model String limit B Propagation time Peak RSS
82-spin ladder, Jt=0→5 2^17 13.42 → 9.73 s (−28%) 300 → 130 MiB (−57%)
82-spin ladder, Jt=0→5 2^19 78.24 → 50.61 s (−35%) 940 → 392 MiB (−58%)
61-spin dephased ring, Jt=0→2 2^19 276.90 → 149.14 s (−46%) 1,885 → 453 MiB (−76%)

For comparison across string representations, S = 56·B bytes: 7 MiB at B=2^17 and 28 MiB at B=2^19. Differences below about 5% are not treated as resolved performance gains.

Numerical agreement

These comparisons measure agreement with the original baseline, rather than accuracy against an exact reference. Neither current proposer is bit-identical to that baseline. All three repeats reproduce identical numerical trajectories within each implementation and case.

  • Ring: streaming differs by about 2e-11 in final MSD, and shortlist by 9e-6. The largest absolute density-profile differences over all recorded times are 4.7e-12 and 2.1e-9, respectively.
  • Ladder: final MSD at B=2^19 is 35.711 for the baseline, 35.692 for streaming and 35.276 for shortlist. At B=2^17 it is 35.213, 35.267 and 35.130, respectively. Changing the retained support changes the approximation; runtime and memory gains alone do not establish improved accuracy.
  • Revert check: the current streaming ladder trajectories match the previous streaming benchmarks with forecasting disabled bit for bit, including state hashes.

@MSRudolph
MSRudolph requested review from AlexSchuckert and Roger-luo and a balanced review from Copilot November 3, 2026 10:40

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

@github-actions

github-actions Bot commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor
PR Preview Action v1.8.2

QR code for preview link

🚀 View preview at
https://QuEraComputing.github.io/ppvm/pr-preview/pr-276/

Built to branch gh-pages at 2026-11-09 16:35 UTC.
Preview will be ready when the GitHub Pages deployment is complete.

MSRudolph and others added 5 commits November 3, 2026 17:36
The candidate position table and source ordering are rebuilt by every
proposal and unused by selection, so holding them through the
exponential only raised its peak allocation.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Final truncation allocated a fresh magnitude array alongside the
retained Taylor scratch. The real predictor-corrector step lends that
scratch to the rank cap instead.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The crate's word type was ppvm_pauli_word::PauliWord, which also stores
the register size and a cached hash, making a 128-qubit word 48 bytes.
Word now holds only the X and Z planes (32 bytes); the register size
comes from the model and hashes are computed on demand with the same
digest and ordering, so hash-table iteration order is unchanged.
ppvm_pauli_sum symmetry routines still take the general type, exposed
as GeneralWord, with conversions at that boundary.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With words no longer caching their hash, each action term would be
hashed again for sorting and again for every basis-index probe. The
action plan now emits each output word with its hash, and the sort and
index lookups (get_hashed) reuse it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ries

With only Hermitian Pauli jumps, the off-diagonal generator is real
antisymmetric term by term, so the unit cache keeps each connected pair
once and the product applies it in both directions with opposite signs.
Pairs are grouped into block-to-block tiles with 16-bit block-local
positions and a 16-bit term code (32-bit when the model has more terms),
making an entry 6 bytes instead of 8 for every directed connection.
Entry storage is sized to the block's count with 1/16 headroom rather
than doubled, and the exponential's shift and norm bound are reduced
while streaming rows instead of from a per-row array.

Products agree with the general generator to rounding, not bit for bit,
because each row sums its parents in a different order.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Copilot AI balanced review requested due to automatic review settings November 4, 2026 08:26

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

Copilot AI balanced review requested due to automatic review settings November 5, 2026 15:27

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

@Roger-luo

Copy link
Copy Markdown
Collaborator

I think this is better to be merged after the main refactor gets in - the new trait system gives more flexibility in choosing different PauliWord data structures

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants