Skip to content

fix: type-stable multi-batch Jacobian and Hessian assembly for static arrays - #1080

Open
devmotion wants to merge 6 commits into
mainfrom
dmw/static-multibatch-assembly
Open

devmotion wants to merge 6 commits into
mainfrom
dmw/static-multibatch-assembly

Conversation

@devmotion

@devmotion devmotion commented Oct 12, 2026 •

Copy link
Copy Markdown
Member

Stacked on #1078.

Replaces mapreduce(hcat/vcat, ...) in the multi-batch Jacobian and Hessian paths with new internal hooks stack_batches_col/row(block, batch_size_settings, seed_example), which take the function computing the block of each batch:

  • The generic methods collect the blocks with a comprehension and concatenate them with a single reduce(hcat/vcat, ...). A comprehension rather than map, since map fails to infer when the block function calls map itself.
  • The StaticArrays extension overloads them to concatenate an ntuple of blocks, with the number of batches deduced from the seed type. It also overloads the new vec_block with similar_type, so that flattened blocks of e.g. an MMatrix are MVectors and the result is an MMatrix instead of a SizedMatrix backed by a Matrix.

Multi-batch Jacobians and Hessians of static arrays are now inferred and return static matrices (turning #1078's @test_broken into passing tests). Tests cover all batch methods (B = 1, aligned, unaligned, single batch) for pushforward and pullback Jacobians (out-of-place and in-place, real and complex) and Hessians, plus JET checks over the static DIT scenarios. The in-place Hessian with AutoReverseFromPrimitive already fails JET on main and is marked broken.

Why reduce instead of mapreduce

reduce(hcat, ::AbstractVector{<:AbstractVecOrMat}) (and vcat) has a specialized Base method that allocates the result once. mapreduce(f, hcat, v) has no such method and falls back to pairwise hcat, re-copying the partial result at every level.

Concatenating A blocks of size m × B (Chairmarks):

m × (A × B) mapreduce(f, hcat, v) reduce(hcat, map(f, v))
10 × (3 × 4) 89 ns, 3.0 KB 93 ns, 2.4 KB
100 × (9 × 12) 12.4 µs, 573 KB 3.5 µs, 191 KB
1000 × (84 × 12) 78 ms, 358 MB 0.36 ms, 16 MB

End-to-end, prepared jacobian(identity, prep, backend, x) with dense x and chunksize = 12 (Julia 1.11):

backend, length(x) before after
AutoSimpleFiniteDiff, 100 31.9 µs, 1.16 MB 29.3 µs, 0.87 MB
AutoSimpleFiniteDiff, 1000 12.2 ms, 402 MB 2.7 ms, 74 MB
AutoReverseFromPrimitive(AutoSimpleFiniteDiff), 1000 721 ms, 5.23 GB 419 ms, 4.90 GB

(stack was also considered: equivalent for columns, but rows need an extra permutedims copy.)

🤖 Generated with Claude Code

devmotion and others added 2 commits October 12, 2026 11:37
…ic arrays

Follow-up to #1076: lock in that `pick_batchsize` and `prepare_jacobian`
infer the batch size from the type for `SVector`, `SMatrix` and `MVector`,
with adaptive and fixed batch sizes. Mark multi-batch `jacobian` inference
as broken, since `mapreduce(hcat, ...)` changes the static size at every step.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… arrays

Compute all batch blocks with `map` and concatenate them once with
`reduce(hcat/vcat, ...)` via new internal hooks `stack_batches_col` and
`stack_batches_row`, instead of `mapreduce(hcat/vcat, ...)` with an
in-loop slice of the last batch. The StaticArrays extension overloads
the hooks to concatenate a statically sized tuple of blocks, so that
multi-batch Jacobians and Hessians of static arrays are inferred and
return static matrices.

For dense arrays this avoids the repeated reallocation of pairwise
`hcat`/`vcat` (e.g. 10x faster, 5x less memory for a 1000x1000
Jacobian with chunksize 12).

Also remove the unreachable unaligned branch in the single-batch
Jacobian methods: a single batch is always aligned.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@devmotion
devmotion requested a review from gdalle as a code owner October 12, 2026 14:51
@codecov

codecov Bot commented Oct 12, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 98.87640% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 98.27%. Comparing base (e64b3a4) to head (f460474).

Files with missing lines Patch % Lines
...iationInterface/test/Core/SimpleFiniteDiff/test.jl 93.33% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1080      +/-   ##
==========================================
+ Coverage   98.21%   98.27%   +0.05%     
==========================================
  Files         143      143              
  Lines        8246     8274      +28     
==========================================
+ Hits         8099     8131      +32     
+ Misses        147      143       -4     
Flag Coverage Δ
DI 98.12% <98.87%> (+0.07%) ⬆️
DIT 98.67% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Base automatically changed from dmw/static-batchsize-inference-tests to main October 13, 2026 13:23
…-assembly

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@gdalle gdalle left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, this looks like a worthwhile improvement! A few things to iron out though

- split `aligned ? M : M[...]` into if/else blocks for coverage
- drop the unused `B_last` field from `BatchSizeSettings`
- route the B=1 Jacobian methods through `stack_batches_col/row` too,
  so they also return static matrices for static arrays

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@gdalle gdalle left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! I'm good with the changes to the source code, a few more tests and this will be good to go!

devmotion and others added 2 commits October 30, 2026 17:04
Pass a block function to `stack_batches_col/row` instead of a vector of
blocks. The generic method collects the blocks with a comprehension,
since `map` fails to infer when the block function calls `map` itself
(e.g. pullbacks via pushforwards). The StaticArrays methods build the
blocks with `ntuple` and allocate no intermediate vector.

Flatten blocks with `vec_block`, which uses `similar_type` for static
arrays: `vec` of an `MMatrix` is a `SizedVector` view, and concatenating
those allocates a `SizedMatrix` backed by a `Matrix`.

Tests cover all batch methods (B = 1, aligned, unaligned, single batch)
for pushforward and pullback Jacobians (out-of-place and in-place, real
and complex) and Hessians, with inference checks, a ForwardDiff Jacobian
through `AutoForward/ReverseFromPrimitive`, and JET checks over the
static DIT scenarios. The in-place Hessian with a pullback-based inner
gradient is marked broken, as it already failed to infer before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@gdalle-bot gdalle-bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Important

This review was generated by a coding agent on behalf of @gdalle.

Thanks for the follow-up! I can reproduce the dense speedup: prepared jacobian(identity, ...) with length(x) = 1000 and AutoSimpleFiniteDiff(; chunksize = 10) goes from 51 ms / 460 MB to 7.4 ms / 64 MB. Static multi-batch results are now inferred too.

Two blockers, detailed inline:

  1. Compile time on the B=1 path for static arrays: up to 386 s for an SVector{100}, from 2 s on main.
  2. Extra allocation in the unaligned case: the final slice copies the whole Jacobian a second time. Cropping only the last block, as on main, avoids it.

The other comments are non-blocking. Housekeeping: the description still says "Stacked on #1078", which is merged, and the branch is behind main.

Measured on Julia 1.13.4 against main at 7f042d1. Compile times are single runs in fresh processes.

block::F, ::DI.BatchSizeSettings{B, singlebatch, aligned}, ::NTuple{B, T}
) where {F, B, singlebatch, aligned, T <: StaticArray}
N, A = static_batch_count(T, B)
M = hcat(ntuple(block, Val(A))...)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocker: compile time. Now that the B=1 methods go through this hook, A = N here. ntuple(block, Val(A)) plus the splatted hcat unroll over every seed, and each block(a) is a full pushforward or pullback call. On main, B=1 went through stack and returned a Matrix.

Test case: jacobian(x -> x .^ 2, prep, backend, x::SVector{N}), with AutoSimpleFiniteDiff(; chunksize = 1) (forward) or AutoReverseFromPrimitive of it (reverse):

case main: first call / later calls PR: first call / later calls
N = 32, reverse, B = 1 2.6 s / 3.0 ms 42.8 s / 151 µs
N = 64, forward, B = 1 2.0 s / 65 µs 109 s / 419 µs
N = 100, forward, B = 1 2.0 s / 75 µs 386 s / 790 µs
N = 64, forward, B = 4 77 s / 220 µs 15 s / 74 µs

The last row improves, so the problem is specifically unrolling over a large A. B=1 is the default pick_batchsize, so this hits every backend that relies on the generic Jacobian (Mooncake, Tracker, ChainRules, Diffractor).

Suggestion: don't unroll over batches. Compute the first block, then allocate the result once with its static size (similar(b1, Size(m, N)) gives an MMatrix). Fill in the other blocks with a runtime loop, writing only the first N columns, and convert back with similar_type at the end. The sizes stay static, so the result should still be inferred, and compile time no longer grows with A. The same applies to stack_batches_row below.

if aligned
return M
else
return M[:, 1:(batch_size_settings.N)]

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocker: extra allocation in the unaligned case. reduce builds the full m × (A * B) matrix, then M[:, 1:N] copies the whole Jacobian a second time. On main, only the last block was cropped, which costs m × B_last. The docstring's "allocates the result only once" therefore no longer holds when !aligned.

Standalone benchmark of the assembly alone, with m = N = 1000 and B = 12 (so A = 84, unaligned). Computing the blocks themselves takes 0.7 ms and 8 MB:

strategy total time bytes on top of the blocks
this PR (reduce, then slice) 2.0 ms 16 MB
crop the last block, then reduce (as on main) 1.3 ms 8 MB

Suggestion: crop the last block inside the comprehension to its first N - (A - 1) * B columns, which doesn't need B_last back. Alternatively, preallocate the result from the first block and copy each block into a view. That also avoids keeping all A blocks alive at once. The same applies to stack_batches_row below.

Side note: collecting all blocks before copying also means that, for B=1, vec(dy) must never alias a buffer reused by the next pushforward call (for a Vector, vec(dy) is dy itself). On main, stack copied each block eagerly. Backends in this repo all return fresh outputs, but the contract isn't documented, and preallocating would sidestep it.

)

jac = stack(eachindex(batched_seeds); dims = 2) do a
return stack_batches_col(batch_size_settings, seed_example) do a

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking: routing B=1 through stack_batches_col changes the return type for static inputs from Matrix to SMatrix/MMatrix, e.g. for Mooncake with an SVector. SMatrix is immutable, so downstream code doing J[i, j] = ... would break. Static output is probably the right direction, but it deserves an entry in the Unreleased section of CHANGELOG.md and a mention in the PR description. The same goes for the B=1 pullback method.

# so that concatenating the blocks preserves static sizes.

function static_batch_count(::Type{T}, B::Int) where {T <: StaticArray}
N = length(T)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking: N comes from the seed type here, while the generic method uses batch_size_settings.N. They agree today because seeds and settings come from the same array, but nothing checks it. A cheap N == batch_size_settings.N check in stack_batches_col/row would make the invariant explicit.

Related: dispatch happens on the seed type rather than on the blocks, so with mixed inputs the result type depends on the batch mode:

  • Vector → SVector, forward mode: Array for B=1, aligned and unaligned, but SArray for a single batch.
  • SVector → Vector, reverse mode: Array, except MArray for a single batch.

Main was also inconsistent here, so this isn't a regression, but a test pinning the intended behaviour would help.

@testset "Pushforward, chunksize $chunksize" for chunksize in (1, 3, 4, 6)
backend = DI.AutoForwardFromPrimitive(AutoForwardDiff(; chunksize))
J = @inferred jacobian(f, backend, x)
@test J isa StaticMatrix

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SizedMatrix <: StaticMatrix, so these checks would still pass without the vec_block change, which is meant to avoid SizedMatrix{Matrix} results. Checking Union{SArray, MArray} as in test/Core/Internals/batchsize.jl, or the exact expected type, would guard it. The same applies to the other isa StaticMatrix checks in this file.

return nothing
end
g(x) = sum(abs2, x) * sum(x) # non-diagonal Hessian
has_type(J, x) = x isa StaticArray ? J isa Union{SArray, MArray} : J isa Matrix

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Accepting either SArray or MArray means a change from SArray to MArray would go unnoticed, i.e. a heap-allocated result instead of an isbits one. The expected type is deterministic for each input and mode (e.g. the forward Jacobian of an SVector is an SMatrix), so I'd pin it exactly.

Also, every static test here uses length 6, where the unrolling cost in the extension is invisible. A moderately sized case, e.g. an SVector{32} with chunk size 1, would have exposed the compile-time regression.

end
end
end
@test_throws ArgumentError("Cannot concatenate zero batches") jacobian(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking: this test locks in an error as the expected behaviour for an empty input, whereas the correct result would be an m × 0 Jacobian. The error predates this PR, but I'd rather not cement it in a test. An empty dense input also still fails with an opaque reduce error, so the static and dense paths now disagree.

logging = LOGGING,
)
# multiple batches
forward_backend = AutoSimpleFiniteDiff(; chunksize = 2)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking: with chunk size 2, the JET checks only exercise the aligned multi-batch path. B=1 and unaligned are only covered by @inferred in test/Core/Internals/batchsize.jl. @inferred checks the return type but not runtime dispatch inside the call. Could you add chunk sizes that hit those paths, if the static scenario sizes allow it?

type_stability = :prepared,
logging = LOGGING,
)
@testset "Type stability (broken)" for scen in filter(inplace_hessian, multibatch_scenarios)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there an issue tracking this pre-existing JET failure? Linking it here would make sure the broken marker gets removed once it's fixed.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants