perf: zero-heap-allocation Poseidon hash path

[ananas-block]

Removes every heap allocation from the Poseidon hash path and cuts per-call cost. Hash outputs are unchanged.

Why

light-poseidon backs the sol_poseidon syscall through solana-poseidon, and compiles natively into the validator rather than into BPF programs (its dependency is gated behind cfg(not(any(target_os = "solana", target_arch = "bpf")))). solana_poseidon::hashv constructs a fresh hasher on every call, so parameter construction sits on the hot path alongside hashing.

Three sources of allocation, measured with a counting global allocator:

Source	width 2	width 13
PoseidonParameters as Vec<F> + Vec<Vec<F>>, rebuilt per construction	5	16
apply_mds rebuilding state with collect() once per round	64	73
Byte path: a BigUint per input, a Vec<u8> for the output	~6	~43
Total per syscall call	75	132

Results

Interleaved A/B against origin/main, three runs each, release, native arm64, for the shape hashv uses (construct, then hash_bytes_be):

Width	main	this branch	Time	Allocations
2	7.69 us	5.65 us	-26.5%	75 -> 0
3	11.72 us	9.05 us	-22.8%	80 -> 0
13	132.08 us	122.23 us	-7.5%	132 -> 0

The gain is largest at small widths, where fixed per-call overhead dominates. At width 13 the O(width^2) MDS multiply dominates and is untouched here; that is what the sparse-MDS follow-up targets.

How

1. Parameters are &'static and flat. ark and mds are &'static [F], with mds flattened row-major. The generated constants are static arrays built with Fr::new, a const fn that performs the Montgomery reduction at compile time, where From<BigInt> called from_bigint at run time on every construction.
2. The permutation runs on the stack. Poseidon no longer holds a state field at all; state lives in a fixed-size stack array for the duration of a hash.
3. Byte inputs convert straight into that state. There is no intermediate input buffer, so there is no second capacity that could overflow.

Poseidon::new is now fallible and rejects a width outside 2..=MAX_X5_LEN up front, which is what makes the fixed-capacity state provably safe. PoseidonParameters::new_unchecked skips dimension validation for the generated sets, whose dimensions are fixed at generation time; PoseidonParameters::new validates, for parameters from anywhere else.

Compatibility with Agave

solana-poseidon 5.0.0 compiles against this branch unedited, verified by building and running it with a path patch, not just by comparing signatures. The wrapper agrees with the library for both endiannesses, input counts 1..=12 succeed, 13 is rejected with InvalidWidthCircom, and a value at the modulus is still rejected with InputLargerThanModulus.

Two things this constrained:

• Poseidon could not become const-generic over width, because new_circom takes a runtime length. A const-generic version measured meaningfully faster in a prototype (4.70 us at width 2, about -39%) because the inner loops unroll. Kept out to avoid forcing a dispatch change on callers.

• No new PoseidonError variants. solana-poseidon-5.0.0/src/lib.rs:227-249 matches the enum exhaustively with no wildcard arm when converting to PoseidonSyscallError, so adding a variant is an E0004 compile error there even with every signature unchanged. (Agave's syscall handler does not match on the variant, but this From impl in the crate does, and it is what breaks.) The out-of-range width and inconsistent-dimensions cases therefore reuse InvalidWidthCircom, whose message is generalised. Caught by the downstream build, not by the test suite.

  The distinction from the other breaking changes above is deliberate rather than inconsistent: PoseidonParameters, Poseidon::new and get_poseidon_parameters are not used by solana-poseidon at all, so breaking them is invisible to Agave. An error variant is not. Tests that would otherwise lose the ability to tell the two failures apart assert the error payload instead of the variant.

new_circom, with_domain_tag_circom, hash_bytes_be, hash_bytes_le, validate_bytes_length, bytes_to_prime_field_element_be and bytes_to_prime_field_element_le all keep their signatures. Byte-API error precedence is unchanged: lengths are validated first, then values converted, then the input count checked. That ordering is load-bearing and is covered by the existing padding_collision tests, which caught an earlier version of this branch getting it wrong.

Proving outputs did not change

This is consensus-relevant code, so that claim needed more than the existing tests.

light-poseidon/tests/differential.rs checks 144 vectors captured from main before any change: widths 2..=13, field API and both byte endiannesses, covering all-zero, all-one, sequential and p-1 inputs plus eight deterministic pseudo-random cases per width. If one fails, the fix is the code, never the fixture.

The parameter file was regenerated from the existing in-crate constants rather than re-derived from the sage script, so the values are identical by construction. It then survived a full read-write round trip through the new API with the vectors still passing.

Independently confirmed from another direction: the parallel sparse-MDS work derived its constants twice, once from the pre-rewrite parameter file and once from the regenerated one, and got byte-identical output. That derivation multiplies and inverts these constants across 60-odd rounds per width, so a single differing limb anywhere in ark or mds would have diverged.

Tests

51 tests, all passing. The 30 existing integration tests are unmodified.

• differential.rs (5): outputs against the frozen oracle.
• boundaries.rs (10): widths 0, 1, 2, 13, 14 and beyond through both constructors; input counts 0, width-1, width, width+1, 12, 13, 14, 32, 200 through the field and both byte APIs; a valid call after a rejected one; and a truncated MDS reaching the permutation, which errors rather than panicking or hashing with missing terms.
• allocations.rs (1): asserts zero allocations for construction plus hashing at every width across all three APIs, so the property cannot silently regress.

The Criterion benchmark previously measured only hash on field elements, leaving the byte parser and serializer unmeasured. It now also covers hash_bytes_be/le and the construct-per-call syscall shape.

cargo clippy --workspace --all-targets -- -D warnings is clean.

Review notes

The 50k-line diff is almost entirely the generated parameters/bn254_x5.rs (43,056 deletions, 7,237 insertions; it shrinks because each constant collapses from six lines to one). The reviewable diff is roughly 630 lines: lib.rs, the xtask generator, and the new tests. The generator is the thing to check, not its output.

xtask/src/generate_parameters.rs is updated to emit the new format but could not be executed here: it needs SageMath and a clone of hadeshash, and target/params is not in the repo. It is untested by construction.

num-bigint is dropped as a direct dependency, since the byte path was its only user in this crate. It is still compiled: ark-ff, ark-ec, ark-serialize and ark-ff-macros all depend on it (cargo tree -i num-bigint). This is not a build-time or dependency-tree win, only one fewer crate this code calls into directly.

Two behaviour changes worth knowing

Oversized parameter tables are now rejected. permute validates ark.len() and mds.len() against the declared dimensions exactly, once, before the round loop. The old code indexed rather than iterated, so an over-long table was silently tolerated. Anyone sharing one constant table across widths now gets an error instead of a hash. The bundled sets match exactly, so the syscall path is unaffected.

That check is load-bearing rather than defensive: chunks_exact and zip both stop at the shorter side, so without it a truncated mds would hash successfully with missing terms — the same silent-truncation failure mode as PR #60. A width of 0 is rejected for the same reason, since chunks_exact(0) panics.

MSRV is now 1.88, declared in Cargo.toml. slice::as_chunks/as_chunks_mut are stable since 1.88.0; they were adopted to satisfy a clippy lint that CI's newer toolchain raised.

Out of scope

Sparse MDS for partial rounds, which is where the remaining width-13 cost lives, is a separate change and is being worked on in parallel. PRs #60 and #61 overlap this work and are deliberately left independent.

The version is deliberately not bumped. These changes are breaking (PoseidonParameters field types, Poseidon::new and PoseidonParameters::new now fallible, get_poseidon_parameters specialised to Fr) and will need a bump whenever a release is cut.

🤖 Generated with Claude Code