Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -2,14 +2,52 @@ module DifferentiationInterfaceStaticArraysExt

using ADTypes: AutoForwardDiff, AutoEnzyme
import DifferentiationInterface as DI
using StaticArrays: SArray, StaticArray
using StaticArrays: SArray, Size, SOneTo, StaticArray, similar_type

# a flat vector of the same kind (e.g. `MVector` for an `MMatrix`) keeps concatenated blocks `SArray`s or `MArray`s
DI.vec_block(a::StaticArray) = similar_type(a, Size(length(a)))(a)

function DI.stack_vec_col(t::NTuple{B, <:StaticArray}) where {B}
return hcat(map(vec, t)...)
return hcat(map(DI.vec_block, t)...)
end

function DI.stack_vec_row(t::NTuple{B, <:StaticArray}) where {B}
return vcat(transpose.(map(vec, t))...)
return vcat(transpose.(map(DI.vec_block, t))...)
end

# The number of seeds `N` and of batches `A` are deduced from the seed type,
# so that concatenating the blocks preserves static sizes.

function static_batch_count(::Type{T}, B::Int) where {T <: StaticArray}
N = length(T)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking: N comes from the seed type here, while the generic method uses batch_size_settings.N. They agree today because seeds and settings come from the same array, but nothing checks it. A cheap N == batch_size_settings.N check in stack_batches_col/row would make the invariant explicit.

Related: dispatch happens on the seed type rather than on the blocks, so with mixed inputs the result type depends on the batch mode:

  • Vector → SVector, forward mode: Array for B=1, aligned and unaligned, but SArray for a single batch.
  • SVector → Vector, reverse mode: Array, except MArray for a single batch.

Main was also inconsistent here, so this isn't a regression, but a test pinning the intended behaviour would help.

A = cld(N, B)
# otherwise `hcat()` would silently return an empty `Vector{Any}`
A > 0 || throw(ArgumentError("Cannot concatenate zero batches"))
return N, A
end

function DI.stack_batches_col(
block::F, ::DI.BatchSizeSettings{B, singlebatch, aligned}, ::NTuple{B, T}
) where {F, B, singlebatch, aligned, T <: StaticArray}
N, A = static_batch_count(T, B)
M = hcat(ntuple(block, Val(A))...)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocker: compile time. Now that the B=1 methods go through this hook, A = N here. ntuple(block, Val(A)) plus the splatted hcat unroll over every seed, and each block(a) is a full pushforward or pullback call. On main, B=1 went through stack and returned a Matrix.

Test case: jacobian(x -> x .^ 2, prep, backend, x::SVector{N}), with AutoSimpleFiniteDiff(; chunksize = 1) (forward) or AutoReverseFromPrimitive of it (reverse):

case main: first call / later calls PR: first call / later calls
N = 32, reverse, B = 1 2.6 s / 3.0 ms 42.8 s / 151 µs
N = 64, forward, B = 1 2.0 s / 65 µs 109 s / 419 µs
N = 100, forward, B = 1 2.0 s / 75 µs 386 s / 790 µs
N = 64, forward, B = 4 77 s / 220 µs 15 s / 74 µs

The last row improves, so the problem is specifically unrolling over a large A. B=1 is the default pick_batchsize, so this hits every backend that relies on the generic Jacobian (Mooncake, Tracker, ChainRules, Diffractor).

Suggestion: don't unroll over batches. Compute the first block, then allocate the result once with its static size (similar(b1, Size(m, N)) gives an MMatrix). Fill in the other blocks with a runtime loop, writing only the first N columns, and convert back with similar_type at the end. The sizes stay static, so the result should still be inferred, and compile time no longer grows with A. The same applies to stack_batches_row below.

if aligned
return M
else
return M[:, SOneTo(N)]
end
end

function DI.stack_batches_row(
block::F, ::DI.BatchSizeSettings{B, singlebatch, aligned}, ::NTuple{B, T}
) where {F, B, singlebatch, aligned, T <: StaticArray}
N, A = static_batch_count(T, B)
M = vcat(ntuple(block, Val(A))...)
if aligned
return M
else
return M[SOneTo(N), :]
end
end

DI.ismutable_array(::Type{<:SArray}) = false
Expand Down
75 changes: 24 additions & 51 deletions DifferentiationInterface/src/first_order/jacobian.jl
Original file line number Diff line number Diff line change
Expand Up @@ -354,22 +354,16 @@ end

function _jacobian_aux(
f_or_f!y::FY,
prep::PushforwardJacobianPrep{SIG, <:BatchSizeSettings{B, true, aligned}},
prep::PushforwardJacobianPrep{SIG, <:BatchSizeSettings{B, true}},
backend::AbstractADType,
x,
contexts::Vararg{Context, C},
) where {FY, SIG, B, aligned, C}
(; batch_size_settings, batched_seeds, pushforward_prep) = prep
(; B_last) = batch_size_settings
Comment thread
gdalle marked this conversation as resolved.
Resolved
) where {FY, SIG, B, C}
(; batched_seeds, pushforward_prep) = prep
dy_batch = pushforward(
f_or_f!y..., pushforward_prep, backend, x, only(batched_seeds), contexts...
)
block = stack_vec_col(dy_batch)
if aligned
return block
else
return block[:, 1:B_last]
end
Comment thread
gdalle marked this conversation as resolved.
Resolved
return stack_vec_col(dy_batch)
end

function _jacobian_aux(
Expand All @@ -379,13 +373,13 @@ function _jacobian_aux(
x,
contexts::Vararg{Context, C},
) where {FY, SIG, C}
(; batched_seeds, seed_example, pushforward_prep) = prep
(; batch_size_settings, batched_seeds, seed_example, pushforward_prep) = prep

pushforward_prep_same = prepare_pushforward_same_point(
f_or_f!y..., pushforward_prep, backend, x, seed_example, contexts...
)

jac = stack(eachindex(batched_seeds); dims = 2) do a
return stack_batches_col(batch_size_settings, seed_example) do a

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking: routing B=1 through stack_batches_col changes the return type for static inputs from Matrix to SMatrix/MMatrix, e.g. for Mooncake with an SVector. SMatrix is immutable, so downstream code doing J[i, j] = ... would break. Static output is probably the right direction, but it deserves an entry in the Unreleased section of CHANGELOG.md and a mention in the PR description. The same goes for the B=1 pullback method.

dy = only(
pushforward(
f_or_f!y...,
Expand All @@ -396,26 +390,24 @@ function _jacobian_aux(
contexts...,
)
)
return vec(dy)
return vec_block(dy)
end
return jac
end

function _jacobian_aux(
f_or_f!y::FY,
prep::PushforwardJacobianPrep{SIG, <:BatchSizeSettings{B, false, aligned}},
prep::PushforwardJacobianPrep{SIG, <:BatchSizeSettings{B, false}},
backend::AbstractADType,
x,
contexts::Vararg{Context, C},
) where {FY, SIG, B, aligned, C}
) where {FY, SIG, B, C}
(; batch_size_settings, batched_seeds, seed_example, pushforward_prep) = prep
(; A, B_last) = batch_size_settings

pushforward_prep_same = prepare_pushforward_same_point(
f_or_f!y..., pushforward_prep, backend, x, seed_example, contexts...
)

jac = mapreduce(hcat, eachindex(batched_seeds)) do a
return stack_batches_col(batch_size_settings, seed_example) do a
dy_batch = pushforward(
f_or_f!y...,
pushforward_prep_same,
Expand All @@ -424,37 +416,25 @@ function _jacobian_aux(
batched_seeds[a],
contexts...,
)
block = stack_vec_col(dy_batch)
if !aligned && a == A
return block[:, 1:B_last]
else
return block
end
return stack_vec_col(dy_batch)
end
return jac
end

function _jacobian_aux(
f_or_f!y::FY,
prep::PullbackJacobianPrep{SIG, <:BatchSizeSettings{B, true, aligned}},
prep::PullbackJacobianPrep{SIG, <:BatchSizeSettings{B, true}},
backend::AbstractADType,
x,
contexts::Vararg{Context, C},
) where {FY, SIG, B, aligned, C}
(; batch_size_settings, batched_seeds, pullback_prep) = prep
(; B_last) = batch_size_settings
) where {FY, SIG, B, C}
(; batched_seeds, pullback_prep) = prep
dx_batch = pullback(
f_or_f!y..., pullback_prep, backend, x, only(batched_seeds), contexts...
)
if eltype(x) <: Complex
dx_batch = map(conj, dx_batch)
end
block = stack_vec_row(dx_batch)
if aligned
return block
else
return block[1:B_last, :]
end
return stack_vec_row(dx_batch)
end

function _jacobian_aux(
Expand All @@ -464,56 +444,49 @@ function _jacobian_aux(
x,
contexts::Vararg{Context, C},
) where {FY, SIG, C}
(; batched_seeds, seed_example, pullback_prep) = prep
(; batch_size_settings, batched_seeds, seed_example, pullback_prep) = prep

pullback_prep_same = prepare_pullback_same_point(
f_or_f!y..., pullback_prep, backend, x, seed_example, contexts...
)

jac = stack(eachindex(batched_seeds); dims = 1) do a
return stack_batches_row(batch_size_settings, seed_example) do a
dx = only(
pullback(
f_or_f!y..., pullback_prep_same, backend, x, batched_seeds[a], contexts...
)
)
# each block is a row of the Jacobian
if eltype(x) <: Complex
return map(conj, vec(dx))
return transpose(map(conj, vec_block(dx)))
else
return vec(dx)
return transpose(vec_block(dx))
end
end
return jac
end

function _jacobian_aux(
f_or_f!y::FY,
prep::PullbackJacobianPrep{SIG, <:BatchSizeSettings{B, false, aligned}},
prep::PullbackJacobianPrep{SIG, <:BatchSizeSettings{B, false}},
backend::AbstractADType,
x,
contexts::Vararg{Context, C},
) where {FY, SIG, B, aligned, C}
) where {FY, SIG, B, C}
(; batch_size_settings, batched_seeds, seed_example, pullback_prep) = prep
(; A, B_last) = batch_size_settings

pullback_prep_same = prepare_pullback_same_point(
f_or_f!y..., pullback_prep, backend, x, seed_example, contexts...
)

jac = mapreduce(vcat, eachindex(batched_seeds)) do a
return stack_batches_row(batch_size_settings, seed_example) do a
dx_batch = pullback(
f_or_f!y..., pullback_prep_same, backend, x, batched_seeds[a], contexts...
)
if eltype(x) <: Complex
dx_batch = map(conj, dx_batch)
end
block = stack_vec_row(dx_batch)
if !aligned && a == A
return block[1:B_last, :]
else
return block
end
return stack_vec_row(dx_batch)
end
return jac
end

function _jacobian_aux!(
Expand Down
22 changes: 7 additions & 15 deletions DifferentiationInterface/src/second_order/hessian.jl
Original file line number Diff line number Diff line change
Expand Up @@ -159,44 +159,36 @@ function hessian(
contexts::Vararg{Context, C},
) where {F, SIG, C}
check_prep(f, prep, backend, x, contexts...)
(; batched_seeds, seed_example, hvp_prep) = prep
(; batch_size_settings, batched_seeds, seed_example, hvp_prep) = prep

hvp_prep_same = prepare_hvp_same_point(
f, hvp_prep, backend, x, seed_example, contexts...
)

hess = mapreduce(hcat, eachindex(batched_seeds)) do a
return stack_batches_col(batch_size_settings, seed_example) do a
dg = only(hvp(f, hvp_prep_same, backend, x, batched_seeds[a], contexts...))
return vec(dg)
return vec_block(dg)
end
return hess
end

function hessian(
f::F,
prep::HVPGradientHessianPrep{SIG, <:BatchSizeSettings{B, false, aligned}},
prep::HVPGradientHessianPrep{SIG, <:BatchSizeSettings{B, false}},
backend::AbstractADType,
x,
contexts::Vararg{Context, C},
) where {F, SIG, B, aligned, C}
) where {F, SIG, B, C}
check_prep(f, prep, backend, x, contexts...)
(; batch_size_settings, batched_seeds, seed_example, hvp_prep) = prep
(; A, B_last) = batch_size_settings

hvp_prep_same = prepare_hvp_same_point(
f, hvp_prep, backend, x, seed_example, contexts...
)

hess = mapreduce(hcat, eachindex(batched_seeds)) do a
return stack_batches_col(batch_size_settings, seed_example) do a
dg_batch = hvp(f, hvp_prep_same, backend, x, batched_seeds[a], contexts...)
block = stack_vec_col(dg_batch)
if !aligned && a == A
return block[:, 1:B_last]
else
return block
end
return stack_vec_col(dg_batch)
end
return hess
end

function hessian!(
Expand Down
11 changes: 2 additions & 9 deletions DifferentiationInterface/src/utils/batchsize.jl
Original file line number Diff line number Diff line change
Expand Up @@ -13,23 +13,16 @@ Configuration for the batch size deduced from a backend and a sample array of le

- `N::Int`: array length
- `A::Int`: number of batches `A = div(N, B, RoundUp)`
- `B_last::Int`: size of the last batch (if `aligned` is `false`)
"""
struct BatchSizeSettings{B, singlebatch, aligned}
N::Int
A::Int
B_last::Int
end

function BatchSizeSettings{B, singlebatch, aligned}(N::Integer) where {B, singlebatch, aligned}
B > N > 0 && throw(ArgumentError("Batch size $B larger than input size $N"))
if B == N == 0
A = B_last = 0
else
A = div(N, B, RoundUp)
B_last = N % B
end
return BatchSizeSettings{B, singlebatch, aligned}(N, A, B_last)
A = B == N == 0 ? 0 : div(N, B, RoundUp)
return BatchSizeSettings{B, singlebatch, aligned}(N, A)
end

function BatchSizeSettings(B::Int, N::Int)
Expand Down
40 changes: 40 additions & 0 deletions DifferentiationInterface/src/utils/linalg.jl
Original file line number Diff line number Diff line change
@@ -1,6 +1,46 @@
"""
vec_block(a)

Flatten `a` into a vector suitable for concatenation into a Jacobian or Hessian block.
"""
vec_block(a) = vec(a)

stack_vec_col(t::NTuple) = stack(vec, t; dims = 2)
stack_vec_row(t::NTuple) = stack(vec, t; dims = 1)

"""
stack_batches_col(block, batch_size_settings, seed_example)
stack_batches_row(block, batch_size_settings, seed_example)

Concatenate the blocks `block(a)` computed from all batches `a` along columns (resp. rows), and drop the trailing entries of the last batch if it was padded by wrapping around the seeds.

The `seed_example` is only used for dispatch, e.g. to deduce the number of seeds statically.

Unlike `mapreduce(hcat, ...)`, `reduce(hcat, ...)` allocates the result only once.
"""
function stack_batches_col(
block::F, batch_size_settings::BatchSizeSettings{B, singlebatch, aligned}, _
) where {F, B, singlebatch, aligned}
# a comprehension, since `map` fails to infer when `block` calls `map` itself
M = reduce(hcat, [block(a) for a in 1:(batch_size_settings.A)])
if aligned
return M
else
return M[:, 1:(batch_size_settings.N)]

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocker: extra allocation in the unaligned case. reduce builds the full m × (A * B) matrix, then M[:, 1:N] copies the whole Jacobian a second time. On main, only the last block was cropped, which costs m × B_last. The docstring's "allocates the result only once" therefore no longer holds when !aligned.

Standalone benchmark of the assembly alone, with m = N = 1000 and B = 12 (so A = 84, unaligned). Computing the blocks themselves takes 0.7 ms and 8 MB:

strategy total time bytes on top of the blocks
this PR (reduce, then slice) 2.0 ms 16 MB
crop the last block, then reduce (as on main) 1.3 ms 8 MB

Suggestion: crop the last block inside the comprehension to its first N - (A - 1) * B columns, which doesn't need B_last back. Alternatively, preallocate the result from the first block and copy each block into a view. That also avoids keeping all A blocks alive at once. The same applies to stack_batches_row below.

Side note: collecting all blocks before copying also means that, for B=1, vec(dy) must never alias a buffer reused by the next pushforward call (for a Vector, vec(dy) is dy itself). On main, stack copied each block eagerly. Backends in this repo all return fresh outputs, but the contract isn't documented, and preallocating would sidestep it.

end
end

function stack_batches_row(
block::F, batch_size_settings::BatchSizeSettings{B, singlebatch, aligned}, _
) where {F, B, singlebatch, aligned}
M = reduce(vcat, [block(a) for a in 1:(batch_size_settings.A)])
if aligned
return M
else
return M[1:(batch_size_settings.N), :]
end
end

"""
ismutable_array(x)

Expand Down
Loading
Loading