perf(crypto): stream field-element bytes into hashers and transcript#773
perf(crypto): stream field-element bytes into hashers and transcript#773Oppen wants to merge 8 commits into
Conversation
…it elf identity Replaces the build-time-embedded-target scheme (reverted in the prior three commits) with a uniform verifier: decode_commitment/page_commitments for the inner program are supplied via private input instead of recomputed in-VM (~45x fewer cycles), and ProofOptions is fixed per build (`min`/`blowup8` Cargo features) so a malicious private input can't downgrade security level. On success the guest commits elf_digest(inner_elf) || decode_commitment || page_commitments, attesting exactly what it verified without needing to recompute or re-embed the inner ELF at every recursion level.
…ram_id The verifier guest supplies DECODE/page preprocessed roots via private input and previously committed a bare elf_digest, which a custom prover could decouple from the constrained program (a proof of Y committed as X — see the new recursion_soundness_gap_poc). Commit program_id(inner_elf, decode, pages) || inner_public_output instead: folding the roots into the committed identity makes a supplied-root substitution diverge from an honest native recompute, which the top-level host checks. verify_with_options itself does not bind the supplied roots to the ELF; that binding is external and its docstring (and VmAirs::new's) is corrected to say so instead of overstating root validation. Also: commit the inner proof's public_output so the attestation covers the result, not just identity; serialize the recursion-elf presets (.NOTPARALLEL) to avoid the parallel-build clobber; build recursion ELFs in test-fast/ test-prover and un-ignore the artifact-gated recursion tests.
crypto/stark: StarkProof, MultiProof, FriDecommitment, BusPublicInputs, Table, PolynomialOpenings, and DeepPolynomialOpening dual-derive rkyv alongside serde. The verifier operates over a StarkProofView (owned or an rkyv-archived buffer read in place); multi_verify and multi_verify_archived both build the matching view and share one verification implementation, so neither the owned nor the archived path pays a serialization cost. prover: the recursion guest verifies its inner proof straight from the mmapped private-input region via a 12-byte aligning magic/version prefix, with no deserialization pass. Continuation and CLI verification go through the same proof-view path. executor/syscalls: get_private_input_slice() borrows the private input in place instead of copying it into a Vec. bin/cli: proof file persistence uses rkyv instead of bincode. tooling/ethrex-tests: ethrex's guest tests move into their own detached workspace, since ethrex pulls in rkyv's "unaligned" feature which cannot coexist with the root workspace's "aligned" feature in one resolved dependency graph.
Oppen
force-pushed
the
perf/rkyv-serialization
branch
from
ea14eeb to
2e68711
Compare
Eliminates the per-element heap allocation in Merkle leaf hashing and Fiat-Shamir transcript appends. AsBytes gains a stream_bytes sink method, overridden zero-alloc for Goldilocks base and degree-3 extension; the Merkle backends and DefaultTranscript use it instead of as_bytes()/ to_bytes_be(). Adds hash_data_parts + verify_merkle_path_from_hash so row-pair openings hash two slices directly instead of concatenating them into a throwaway Vec first; verify_composition_poly_opening and FieldElementVectorBackend::hash_data now delegate to their row-pair/parts siblings instead of duplicating the same hashing loop. Recursion guest: single-query 89.7M -> 73.7M cycles, multi-query 2.21B -> 1.82B cycles.
stream_bytes was calling the base-field override three times, one per limb, each landing as its own Digest::update on the guest — three BlockBuffer::digest_blocks + memcpy calls instead of one. Disassembly of the guest ELF showed dyn dispatch itself was fully devirtualized by the #[inline(always)] chain, so the actual cost was the call count, not indirection. Reuse the existing write_bytes_be override (already zero-alloc, byte-identical layout) into a stack buffer and sink once. Recursion guest: single-query 73.9M -> 71.1M cycles, multi-query 1.82B -> 1.78B cycles.
Oppen
force-pushed
the
perf/fixed-fe-bytes
branch
from
aa8d48c to
ddb97be
Compare
|
/ai-review |
|
/bench |
Codex Code ReviewNo findings. I reviewed only the PR diff and did not identify safety/security, correctness, significant performance, or dead-code issues in the changed code. Tests were not run, per the static-review instructions. |
|
Review: perf(crypto): stream field-element bytes into hashers and transcript Reviewed the diff focusing on the byte-identity invariant, since the whole refactor rests on
Leaf-hash equivalence in the verifier also checks out:
No memory-safety, panic, or unbounded-allocation concerns — the changes remove allocations rather than add them, and the stack buffers are fixed-size and fully written before use. Minor (Low / cosmetic):
Nothing blocking. Clean, byte-preserving perf refactor. |
Benchmark — ethrex 20 transfers (median of 3)Table parallelism: auto (cores / 3)
Commit: ddb97be · Baseline: cached · Runner: self-hosted bench |
AI ReviewPR #773 · 8 changed files Findings
Status column reflects the verdict from the verifier: deepseek-verifier (openrouter/deepseek/deepseek-v4-pro). AI-001: Auxiliary `&` reborrow in `verify_composition_poly_opening` call
Claim The new body of Evidence In Suggested fix Replace Reviewer Lanes
Verification Lanes
Native Codex and Claude reviews run separately and post their own comments. They are not included in this structured provenance report. Raw lane outputs, candidates, final issues, and model metrics are uploaded as workflow artifacts. |
Oppen
force-pushed
the
perf/rkyv-serialization
branch
2 times, most recently
from
89468a5 to
39abfad
Compare
|
superseded by already merged PR |
1 participant
Summary
AsBytesgains astream_bytessink method, overridden zero-alloc for the Goldilocks base field and its degree-3 extension, replacingas_bytes()/to_bytes_be()returning a freshVec<u8>per element per hash.FieldElementVectorBackend::hash_data_parts+verify_merkle_path_from_hashso row-pair openings (verify_opening_pair,verify_composition_poly_opening,verify_fri_layer_openings) hash two slices directly instead of concatenating them into a throwawayVecfirst;verify_composition_poly_openingandhash_datanow delegate to their row-pair/parts siblings instead of duplicating the hashing loop.stream_byteswas calling the base-field override three times (one per limb), landing as three separateDigest::updatecalls on the guest instead of one. Disassembly confirmed thedyn FnMutsink itself was fully devirtualized (no indirect-call cost), so the actual waste was call count. Now reuses the existing zero-allocwrite_bytes_beoverride into a stack buffer and sinks once.Test plan
cargo test --workspace --exclude math-cuda(492 prover + 137 stark tests, all green) after each commitmake test-ethrexcargo test -p lambda-vm-prover --lib test_recursion_execute_1query -- --ignored --nocapture(in-VM verify accepts, guest commitsvk_digest ‖ output, byte-identical across every change)make test-profile-recursion-single/-multicycle counts confirmed against baseline after each commitllvm-objdump) confirmingstream_bytesdevirtualizes and the extension-field batching removes redundantBlockBuffer::digest_blocks/memcpycalls