Repository navigation
Conversation
…ic arrays Follow-up to #1076: lock in that `pick_batchsize` and `prepare_jacobian` infer the batch size from the type for `SVector`, `SMatrix` and `MVector`, with adaptive and fixed batch sizes. Mark multi-batch `jacobian` inference as broken, since `mapreduce(hcat, ...)` changes the static size at every step. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… arrays Compute all batch blocks with `map` and concatenate them once with `reduce(hcat/vcat, ...)` via new internal hooks `stack_batches_col` and `stack_batches_row`, instead of `mapreduce(hcat/vcat, ...)` with an in-loop slice of the last batch. The StaticArrays extension overloads the hooks to concatenate a statically sized tuple of blocks, so that multi-batch Jacobians and Hessians of static arrays are inferred and return static matrices. For dense arrays this avoids the repeated reallocation of pairwise `hcat`/`vcat` (e.g. 10x faster, 5x less memory for a 1000x1000 Jacobian with chunksize 12). Also remove the unreachable unaligned branch in the single-batch Jacobian methods: a single batch is always aligned. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #1080 +/- ##
==========================================
+ Coverage 98.21% 98.27% +0.05%
==========================================
Files 143 143
Lines 8246 8274 +28
==========================================
+ Hits 8099 8131 +32
+ Misses 147 143 -4
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
…-assembly Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
gdalle
left a comment
There was a problem hiding this comment.
Thanks, this looks like a worthwhile improvement! A few things to iron out though
- split `aligned ? M : M[...]` into if/else blocks for coverage - drop the unused `B_last` field from `BatchSizeSettings` - route the B=1 Jacobian methods through `stack_batches_col/row` too, so they also return static matrices for static arrays Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
gdalle
left a comment
There was a problem hiding this comment.
Thanks! I'm good with the changes to the source code, a few more tests and this will be good to go!
Pass a block function to `stack_batches_col/row` instead of a vector of blocks. The generic method collects the blocks with a comprehension, since `map` fails to infer when the block function calls `map` itself (e.g. pullbacks via pushforwards). The StaticArrays methods build the blocks with `ntuple` and allocate no intermediate vector. Flatten blocks with `vec_block`, which uses `similar_type` for static arrays: `vec` of an `MMatrix` is a `SizedVector` view, and concatenating those allocates a `SizedMatrix` backed by a `Matrix`. Tests cover all batch methods (B = 1, aligned, unaligned, single batch) for pushforward and pullback Jacobians (out-of-place and in-place, real and complex) and Hessians, with inference checks, a ForwardDiff Jacobian through `AutoForward/ReverseFromPrimitive`, and JET checks over the static DIT scenarios. The in-place Hessian with a pullback-based inner gradient is marked broken, as it already failed to infer before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
gdalle-bot
left a comment
There was a problem hiding this comment.
Important
This review was generated by a coding agent on behalf of @gdalle.
Thanks for the follow-up! I can reproduce the dense speedup: prepared jacobian(identity, ...) with length(x) = 1000 and AutoSimpleFiniteDiff(; chunksize = 10) goes from 51 ms / 460 MB to 7.4 ms / 64 MB. Static multi-batch results are now inferred too.
Two blockers, detailed inline:
- Compile time on the B=1 path for static arrays: up to 386 s for an
SVector{100}, from 2 s on main. - Extra allocation in the unaligned case: the final slice copies the whole Jacobian a second time. Cropping only the last block, as on main, avoids it.
The other comments are non-blocking. Housekeeping: the description still says "Stacked on #1078", which is merged, and the branch is behind main.
Measured on Julia 1.13.4 against main at 7f042d1. Compile times are single runs in fresh processes.
| block::F, ::DI.BatchSizeSettings{B, singlebatch, aligned}, ::NTuple{B, T} | ||
| ) where {F, B, singlebatch, aligned, T <: StaticArray} | ||
| N, A = static_batch_count(T, B) | ||
| M = hcat(ntuple(block, Val(A))...) |
There was a problem hiding this comment.
Blocker: compile time. Now that the B=1 methods go through this hook, A = N here. ntuple(block, Val(A)) plus the splatted hcat unroll over every seed, and each block(a) is a full pushforward or pullback call. On main, B=1 went through stack and returned a Matrix.
Test case: jacobian(x -> x .^ 2, prep, backend, x::SVector{N}), with AutoSimpleFiniteDiff(; chunksize = 1) (forward) or AutoReverseFromPrimitive of it (reverse):
| case | main: first call / later calls | PR: first call / later calls |
|---|---|---|
| N = 32, reverse, B = 1 | 2.6 s / 3.0 ms | 42.8 s / 151 µs |
| N = 64, forward, B = 1 | 2.0 s / 65 µs | 109 s / 419 µs |
| N = 100, forward, B = 1 | 2.0 s / 75 µs | 386 s / 790 µs |
| N = 64, forward, B = 4 | 77 s / 220 µs | 15 s / 74 µs |
The last row improves, so the problem is specifically unrolling over a large A. B=1 is the default pick_batchsize, so this hits every backend that relies on the generic Jacobian (Mooncake, Tracker, ChainRules, Diffractor).
Suggestion: don't unroll over batches. Compute the first block, then allocate the result once with its static size (similar(b1, Size(m, N)) gives an MMatrix). Fill in the other blocks with a runtime loop, writing only the first N columns, and convert back with similar_type at the end. The sizes stay static, so the result should still be inferred, and compile time no longer grows with A. The same applies to stack_batches_row below.
| if aligned | ||
| return M | ||
| else | ||
| return M[:, 1:(batch_size_settings.N)] |
There was a problem hiding this comment.
Blocker: extra allocation in the unaligned case. reduce builds the full m × (A * B) matrix, then M[:, 1:N] copies the whole Jacobian a second time. On main, only the last block was cropped, which costs m × B_last. The docstring's "allocates the result only once" therefore no longer holds when !aligned.
Standalone benchmark of the assembly alone, with m = N = 1000 and B = 12 (so A = 84, unaligned). Computing the blocks themselves takes 0.7 ms and 8 MB:
| strategy | total time | bytes on top of the blocks |
|---|---|---|
this PR (reduce, then slice) |
2.0 ms | 16 MB |
crop the last block, then reduce (as on main) |
1.3 ms | 8 MB |
Suggestion: crop the last block inside the comprehension to its first N - (A - 1) * B columns, which doesn't need B_last back. Alternatively, preallocate the result from the first block and copy each block into a view. That also avoids keeping all A blocks alive at once. The same applies to stack_batches_row below.
Side note: collecting all blocks before copying also means that, for B=1, vec(dy) must never alias a buffer reused by the next pushforward call (for a Vector, vec(dy) is dy itself). On main, stack copied each block eagerly. Backends in this repo all return fresh outputs, but the contract isn't documented, and preallocating would sidestep it.
| ) | ||
|
|
||
| jac = stack(eachindex(batched_seeds); dims = 2) do a | ||
| return stack_batches_col(batch_size_settings, seed_example) do a |
There was a problem hiding this comment.
Non-blocking: routing B=1 through stack_batches_col changes the return type for static inputs from Matrix to SMatrix/MMatrix, e.g. for Mooncake with an SVector. SMatrix is immutable, so downstream code doing J[i, j] = ... would break. Static output is probably the right direction, but it deserves an entry in the Unreleased section of CHANGELOG.md and a mention in the PR description. The same goes for the B=1 pullback method.
| # so that concatenating the blocks preserves static sizes. | ||
|
|
||
| function static_batch_count(::Type{T}, B::Int) where {T <: StaticArray} | ||
| N = length(T) |
There was a problem hiding this comment.
Non-blocking: N comes from the seed type here, while the generic method uses batch_size_settings.N. They agree today because seeds and settings come from the same array, but nothing checks it. A cheap N == batch_size_settings.N check in stack_batches_col/row would make the invariant explicit.
Related: dispatch happens on the seed type rather than on the blocks, so with mixed inputs the result type depends on the batch mode:
Vector → SVector, forward mode:Arrayfor B=1, aligned and unaligned, butSArrayfor a single batch.SVector → Vector, reverse mode:Array, exceptMArrayfor a single batch.
Main was also inconsistent here, so this isn't a regression, but a test pinning the intended behaviour would help.
| @testset "Pushforward, chunksize $chunksize" for chunksize in (1, 3, 4, 6) | ||
| backend = DI.AutoForwardFromPrimitive(AutoForwardDiff(; chunksize)) | ||
| J = @inferred jacobian(f, backend, x) | ||
| @test J isa StaticMatrix |
There was a problem hiding this comment.
SizedMatrix <: StaticMatrix, so these checks would still pass without the vec_block change, which is meant to avoid SizedMatrix{Matrix} results. Checking Union{SArray, MArray} as in test/Core/Internals/batchsize.jl, or the exact expected type, would guard it. The same applies to the other isa StaticMatrix checks in this file.
| return nothing | ||
| end | ||
| g(x) = sum(abs2, x) * sum(x) # non-diagonal Hessian | ||
| has_type(J, x) = x isa StaticArray ? J isa Union{SArray, MArray} : J isa Matrix |
There was a problem hiding this comment.
Accepting either SArray or MArray means a change from SArray to MArray would go unnoticed, i.e. a heap-allocated result instead of an isbits one. The expected type is deterministic for each input and mode (e.g. the forward Jacobian of an SVector is an SMatrix), so I'd pin it exactly.
Also, every static test here uses length 6, where the unrolling cost in the extension is invisible. A moderately sized case, e.g. an SVector{32} with chunk size 1, would have exposed the compile-time regression.
| end | ||
| end | ||
| end | ||
| @test_throws ArgumentError("Cannot concatenate zero batches") jacobian( |
There was a problem hiding this comment.
Non-blocking: this test locks in an error as the expected behaviour for an empty input, whereas the correct result would be an m × 0 Jacobian. The error predates this PR, but I'd rather not cement it in a test. An empty dense input also still fails with an opaque reduce error, so the static and dense paths now disagree.
| logging = LOGGING, | ||
| ) | ||
| # multiple batches | ||
| forward_backend = AutoSimpleFiniteDiff(; chunksize = 2) |
There was a problem hiding this comment.
Non-blocking: with chunk size 2, the JET checks only exercise the aligned multi-batch path. B=1 and unaligned are only covered by @inferred in test/Core/Internals/batchsize.jl. @inferred checks the return type but not runtime dispatch inside the call. Could you add chunk sizes that hit those paths, if the static scenario sizes allow it?
| type_stability = :prepared, | ||
| logging = LOGGING, | ||
| ) | ||
| @testset "Type stability (broken)" for scen in filter(inplace_hessian, multibatch_scenarios) |
There was a problem hiding this comment.
Is there an issue tracking this pre-existing JET failure? Linking it here would make sure the broken marker gets removed once it's fixed.
Stacked on #1078.
Replaces
mapreduce(hcat/vcat, ...)in the multi-batch Jacobian and Hessian paths with new internal hooksstack_batches_col/row(block, batch_size_settings, seed_example), which take the function computing the block of each batch:reduce(hcat/vcat, ...). A comprehension rather thanmap, sincemapfails to infer when the block function callsmapitself.ntupleof blocks, with the number of batches deduced from the seed type. It also overloads the newvec_blockwithsimilar_type, so that flattened blocks of e.g. anMMatrixareMVectors and the result is anMMatrixinstead of aSizedMatrixbacked by aMatrix.Multi-batch Jacobians and Hessians of static arrays are now inferred and return static matrices (turning #1078's
@test_brokeninto passing tests). Tests cover all batch methods (B = 1, aligned, unaligned, single batch) for pushforward and pullback Jacobians (out-of-place and in-place, real and complex) and Hessians, plus JET checks over the static DIT scenarios. The in-place Hessian withAutoReverseFromPrimitivealready fails JET onmainand is marked broken.Why
reduceinstead ofmapreducereduce(hcat, ::AbstractVector{<:AbstractVecOrMat})(andvcat) has a specialized Base method that allocates the result once.mapreduce(f, hcat, v)has no such method and falls back to pairwisehcat, re-copying the partial result at every level.Concatenating
Ablocks of sizem × B(Chairmarks):m× (A×B)mapreduce(f, hcat, v)reduce(hcat, map(f, v))End-to-end, prepared
jacobian(identity, prep, backend, x)with densexandchunksize = 12(Julia 1.11):length(x)AutoSimpleFiniteDiff, 100AutoSimpleFiniteDiff, 1000AutoReverseFromPrimitive(AutoSimpleFiniteDiff), 1000(
stackwas also considered: equivalent for columns, but rows need an extrapermutedimscopy.)🤖 Generated with Claude Code