Repository navigation
fix: type-stable multi-batch Jacobian and Hessian assembly for static arrays #1080
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
71df520
01e1274
2e87322
3209372
dcf6496
f460474
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change | |||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
|
@@ -2,14 +2,52 @@ module DifferentiationInterfaceStaticArraysExt | ||||||||||||||||
|
|
|||||||||||||||||
| using ADTypes: AutoForwardDiff, AutoEnzyme | |||||||||||||||||
| import DifferentiationInterface as DI | |||||||||||||||||
| using StaticArrays: SArray, StaticArray | |||||||||||||||||
| using StaticArrays: SArray, Size, SOneTo, StaticArray, similar_type | |||||||||||||||||
|
|
|||||||||||||||||
| # a flat vector of the same kind (e.g. `MVector` for an `MMatrix`) keeps concatenated blocks `SArray`s or `MArray`s | |||||||||||||||||
| DI.vec_block(a::StaticArray) = similar_type(a, Size(length(a)))(a) | |||||||||||||||||
|
|
|||||||||||||||||
| function DI.stack_vec_col(t::NTuple{B, <:StaticArray}) where {B} | |||||||||||||||||
| return hcat(map(vec, t)...) | |||||||||||||||||
| return hcat(map(DI.vec_block, t)...) | |||||||||||||||||
| end | |||||||||||||||||
|
|
|||||||||||||||||
| function DI.stack_vec_row(t::NTuple{B, <:StaticArray}) where {B} | |||||||||||||||||
| return vcat(transpose.(map(vec, t))...) | |||||||||||||||||
| return vcat(transpose.(map(DI.vec_block, t))...) | |||||||||||||||||
| end | |||||||||||||||||
|
|
|||||||||||||||||
| # The number of seeds `N` and of batches `A` are deduced from the seed type, | |||||||||||||||||
| # so that concatenating the blocks preserves static sizes. | |||||||||||||||||
|
|
|||||||||||||||||
| function static_batch_count(::Type{T}, B::Int) where {T <: StaticArray} | |||||||||||||||||
| N = length(T) | |||||||||||||||||
| A = cld(N, B) | |||||||||||||||||
| # otherwise `hcat()` would silently return an empty `Vector{Any}` | |||||||||||||||||
| A > 0 || throw(ArgumentError("Cannot concatenate zero batches")) | |||||||||||||||||
| return N, A | |||||||||||||||||
| end | |||||||||||||||||
|
|
|||||||||||||||||
| function DI.stack_batches_col( | |||||||||||||||||
| block::F, ::DI.BatchSizeSettings{B, singlebatch, aligned}, ::NTuple{B, T} | |||||||||||||||||
| ) where {F, B, singlebatch, aligned, T <: StaticArray} | |||||||||||||||||
| N, A = static_batch_count(T, B) | |||||||||||||||||
| M = hcat(ntuple(block, Val(A))...) | |||||||||||||||||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Blocker: compile time. Now that the B=1 methods go through this hook, Test case:
The last row improves, so the problem is specifically unrolling over a large Suggestion: don't unroll over batches. Compute the first block, then allocate the result once with its static size ( |
|||||||||||||||||
| if aligned | |||||||||||||||||
| return M | |||||||||||||||||
| else | |||||||||||||||||
| return M[:, SOneTo(N)] | |||||||||||||||||
| end | |||||||||||||||||
| end | |||||||||||||||||
|
|
|||||||||||||||||
| function DI.stack_batches_row( | |||||||||||||||||
| block::F, ::DI.BatchSizeSettings{B, singlebatch, aligned}, ::NTuple{B, T} | |||||||||||||||||
| ) where {F, B, singlebatch, aligned, T <: StaticArray} | |||||||||||||||||
| N, A = static_batch_count(T, B) | |||||||||||||||||
| M = vcat(ntuple(block, Val(A))...) | |||||||||||||||||
| if aligned | |||||||||||||||||
| return M | |||||||||||||||||
| else | |||||||||||||||||
| return M[SOneTo(N), :] | |||||||||||||||||
| end | |||||||||||||||||
| end | |||||||||||||||||
|
|
|||||||||||||||||
| DI.ismutable_array(::Type{<:SArray}) = false | |||||||||||||||||
|
|
|||||||||||||||||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -354,22 +354,16 @@ end | |
|
|
||
| function _jacobian_aux( | ||
| f_or_f!y::FY, | ||
| prep::PushforwardJacobianPrep{SIG, <:BatchSizeSettings{B, true, aligned}}, | ||
| prep::PushforwardJacobianPrep{SIG, <:BatchSizeSettings{B, true}}, | ||
| backend::AbstractADType, | ||
| x, | ||
| contexts::Vararg{Context, C}, | ||
| ) where {FY, SIG, B, aligned, C} | ||
| (; batch_size_settings, batched_seeds, pushforward_prep) = prep | ||
| (; B_last) = batch_size_settings | ||
|
gdalle marked this conversation as resolved.
Resolved
|
||
| ) where {FY, SIG, B, C} | ||
| (; batched_seeds, pushforward_prep) = prep | ||
| dy_batch = pushforward( | ||
| f_or_f!y..., pushforward_prep, backend, x, only(batched_seeds), contexts... | ||
| ) | ||
| block = stack_vec_col(dy_batch) | ||
| if aligned | ||
| return block | ||
| else | ||
| return block[:, 1:B_last] | ||
| end | ||
|
gdalle marked this conversation as resolved.
Resolved
|
||
| return stack_vec_col(dy_batch) | ||
| end | ||
|
|
||
| function _jacobian_aux( | ||
|
|
@@ -379,13 +373,13 @@ function _jacobian_aux( | |
| x, | ||
| contexts::Vararg{Context, C}, | ||
| ) where {FY, SIG, C} | ||
| (; batched_seeds, seed_example, pushforward_prep) = prep | ||
| (; batch_size_settings, batched_seeds, seed_example, pushforward_prep) = prep | ||
|
|
||
| pushforward_prep_same = prepare_pushforward_same_point( | ||
| f_or_f!y..., pushforward_prep, backend, x, seed_example, contexts... | ||
| ) | ||
|
|
||
| jac = stack(eachindex(batched_seeds); dims = 2) do a | ||
| return stack_batches_col(batch_size_settings, seed_example) do a | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Non-blocking: routing B=1 through |
||
| dy = only( | ||
| pushforward( | ||
| f_or_f!y..., | ||
|
|
@@ -396,26 +390,24 @@ function _jacobian_aux( | |
| contexts..., | ||
| ) | ||
| ) | ||
| return vec(dy) | ||
| return vec_block(dy) | ||
| end | ||
| return jac | ||
| end | ||
|
|
||
| function _jacobian_aux( | ||
| f_or_f!y::FY, | ||
| prep::PushforwardJacobianPrep{SIG, <:BatchSizeSettings{B, false, aligned}}, | ||
| prep::PushforwardJacobianPrep{SIG, <:BatchSizeSettings{B, false}}, | ||
| backend::AbstractADType, | ||
| x, | ||
| contexts::Vararg{Context, C}, | ||
| ) where {FY, SIG, B, aligned, C} | ||
| ) where {FY, SIG, B, C} | ||
| (; batch_size_settings, batched_seeds, seed_example, pushforward_prep) = prep | ||
| (; A, B_last) = batch_size_settings | ||
|
|
||
| pushforward_prep_same = prepare_pushforward_same_point( | ||
| f_or_f!y..., pushforward_prep, backend, x, seed_example, contexts... | ||
| ) | ||
|
|
||
| jac = mapreduce(hcat, eachindex(batched_seeds)) do a | ||
| return stack_batches_col(batch_size_settings, seed_example) do a | ||
| dy_batch = pushforward( | ||
| f_or_f!y..., | ||
| pushforward_prep_same, | ||
|
|
@@ -424,37 +416,25 @@ function _jacobian_aux( | |
| batched_seeds[a], | ||
| contexts..., | ||
| ) | ||
| block = stack_vec_col(dy_batch) | ||
| if !aligned && a == A | ||
| return block[:, 1:B_last] | ||
| else | ||
| return block | ||
| end | ||
| return stack_vec_col(dy_batch) | ||
| end | ||
| return jac | ||
| end | ||
|
|
||
| function _jacobian_aux( | ||
| f_or_f!y::FY, | ||
| prep::PullbackJacobianPrep{SIG, <:BatchSizeSettings{B, true, aligned}}, | ||
| prep::PullbackJacobianPrep{SIG, <:BatchSizeSettings{B, true}}, | ||
| backend::AbstractADType, | ||
| x, | ||
| contexts::Vararg{Context, C}, | ||
| ) where {FY, SIG, B, aligned, C} | ||
| (; batch_size_settings, batched_seeds, pullback_prep) = prep | ||
| (; B_last) = batch_size_settings | ||
| ) where {FY, SIG, B, C} | ||
| (; batched_seeds, pullback_prep) = prep | ||
| dx_batch = pullback( | ||
| f_or_f!y..., pullback_prep, backend, x, only(batched_seeds), contexts... | ||
| ) | ||
| if eltype(x) <: Complex | ||
| dx_batch = map(conj, dx_batch) | ||
| end | ||
| block = stack_vec_row(dx_batch) | ||
| if aligned | ||
| return block | ||
| else | ||
| return block[1:B_last, :] | ||
| end | ||
| return stack_vec_row(dx_batch) | ||
| end | ||
|
|
||
| function _jacobian_aux( | ||
|
|
@@ -464,56 +444,49 @@ function _jacobian_aux( | |
| x, | ||
| contexts::Vararg{Context, C}, | ||
| ) where {FY, SIG, C} | ||
| (; batched_seeds, seed_example, pullback_prep) = prep | ||
| (; batch_size_settings, batched_seeds, seed_example, pullback_prep) = prep | ||
|
|
||
| pullback_prep_same = prepare_pullback_same_point( | ||
| f_or_f!y..., pullback_prep, backend, x, seed_example, contexts... | ||
| ) | ||
|
|
||
| jac = stack(eachindex(batched_seeds); dims = 1) do a | ||
| return stack_batches_row(batch_size_settings, seed_example) do a | ||
| dx = only( | ||
| pullback( | ||
| f_or_f!y..., pullback_prep_same, backend, x, batched_seeds[a], contexts... | ||
| ) | ||
| ) | ||
| # each block is a row of the Jacobian | ||
| if eltype(x) <: Complex | ||
| return map(conj, vec(dx)) | ||
| return transpose(map(conj, vec_block(dx))) | ||
| else | ||
| return vec(dx) | ||
| return transpose(vec_block(dx)) | ||
| end | ||
| end | ||
| return jac | ||
| end | ||
|
|
||
| function _jacobian_aux( | ||
| f_or_f!y::FY, | ||
| prep::PullbackJacobianPrep{SIG, <:BatchSizeSettings{B, false, aligned}}, | ||
| prep::PullbackJacobianPrep{SIG, <:BatchSizeSettings{B, false}}, | ||
| backend::AbstractADType, | ||
| x, | ||
| contexts::Vararg{Context, C}, | ||
| ) where {FY, SIG, B, aligned, C} | ||
| ) where {FY, SIG, B, C} | ||
| (; batch_size_settings, batched_seeds, seed_example, pullback_prep) = prep | ||
| (; A, B_last) = batch_size_settings | ||
|
|
||
| pullback_prep_same = prepare_pullback_same_point( | ||
| f_or_f!y..., pullback_prep, backend, x, seed_example, contexts... | ||
| ) | ||
|
|
||
| jac = mapreduce(vcat, eachindex(batched_seeds)) do a | ||
| return stack_batches_row(batch_size_settings, seed_example) do a | ||
| dx_batch = pullback( | ||
| f_or_f!y..., pullback_prep_same, backend, x, batched_seeds[a], contexts... | ||
| ) | ||
| if eltype(x) <: Complex | ||
| dx_batch = map(conj, dx_batch) | ||
| end | ||
| block = stack_vec_row(dx_batch) | ||
| if !aligned && a == A | ||
| return block[1:B_last, :] | ||
| else | ||
| return block | ||
| end | ||
| return stack_vec_row(dx_batch) | ||
| end | ||
| return jac | ||
| end | ||
|
|
||
| function _jacobian_aux!( | ||
|
|
||
| Original file line number | Diff line number | Diff line change | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| @@ -1,6 +1,46 @@ | |||||||||||
| """ | |||||||||||
| vec_block(a) | |||||||||||
|
|
|||||||||||
| Flatten `a` into a vector suitable for concatenation into a Jacobian or Hessian block. | |||||||||||
| """ | |||||||||||
| vec_block(a) = vec(a) | |||||||||||
|
|
|||||||||||
| stack_vec_col(t::NTuple) = stack(vec, t; dims = 2) | |||||||||||
| stack_vec_row(t::NTuple) = stack(vec, t; dims = 1) | |||||||||||
|
|
|||||||||||
| """ | |||||||||||
| stack_batches_col(block, batch_size_settings, seed_example) | |||||||||||
| stack_batches_row(block, batch_size_settings, seed_example) | |||||||||||
|
|
|||||||||||
| Concatenate the blocks `block(a)` computed from all batches `a` along columns (resp. rows), and drop the trailing entries of the last batch if it was padded by wrapping around the seeds. | |||||||||||
|
|
|||||||||||
| The `seed_example` is only used for dispatch, e.g. to deduce the number of seeds statically. | |||||||||||
|
|
|||||||||||
| Unlike `mapreduce(hcat, ...)`, `reduce(hcat, ...)` allocates the result only once. | |||||||||||
| """ | |||||||||||
| function stack_batches_col( | |||||||||||
| block::F, batch_size_settings::BatchSizeSettings{B, singlebatch, aligned}, _ | |||||||||||
| ) where {F, B, singlebatch, aligned} | |||||||||||
| # a comprehension, since `map` fails to infer when `block` calls `map` itself | |||||||||||
| M = reduce(hcat, [block(a) for a in 1:(batch_size_settings.A)]) | |||||||||||
| if aligned | |||||||||||
| return M | |||||||||||
| else | |||||||||||
| return M[:, 1:(batch_size_settings.N)] | |||||||||||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Blocker: extra allocation in the unaligned case. Standalone benchmark of the assembly alone, with
Suggestion: crop the last block inside the comprehension to its first Side note: collecting all blocks before copying also means that, for B=1, |
|||||||||||
| end | |||||||||||
| end | |||||||||||
|
|
|||||||||||
| function stack_batches_row( | |||||||||||
| block::F, batch_size_settings::BatchSizeSettings{B, singlebatch, aligned}, _ | |||||||||||
| ) where {F, B, singlebatch, aligned} | |||||||||||
| M = reduce(vcat, [block(a) for a in 1:(batch_size_settings.A)]) | |||||||||||
| if aligned | |||||||||||
| return M | |||||||||||
| else | |||||||||||
| return M[1:(batch_size_settings.N), :] | |||||||||||
| end | |||||||||||
| end | |||||||||||
|
|
|||||||||||
| """ | |||||||||||
| ismutable_array(x) | |||||||||||
|
|
|||||||||||
|
|
|||||||||||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Non-blocking:
Ncomes from the seed type here, while the generic method usesbatch_size_settings.N. They agree today because seeds and settings come from the same array, but nothing checks it. A cheapN == batch_size_settings.Ncheck instack_batches_col/rowwould make the invariant explicit.Related: dispatch happens on the seed type rather than on the blocks, so with mixed inputs the result type depends on the batch mode:
Vector → SVector, forward mode:Arrayfor B=1, aligned and unaligned, butSArrayfor a single batch.SVector → Vector, reverse mode:Array, exceptMArrayfor a single batch.Main was also inconsistent here, so this isn't a regression, but a test pinning the intended behaviour would help.