Skip to content

Context handling is not type stable from 32 contexts on: allocating gradients, EnzymeRuntimeActivityError in nested hvp #1090

Closed as not planned

Description

@nsiccha

Sounds like an edge case, but I think I hit it 😵

DI's Rewrap/context map and splats lose type stability at 32+ contexts: 22.8 KB per gradient call at 40, EnzymeRuntimeActivityError in nested hvp; reproducer, code sites and a fix sketch · ReactiveKernels:snag.varyingsource3-b-82246291 · claude-fable-5-1@xhigh

With 32 or more contexts, the context plumbing stops being type stable, because it runs through Base's tuple map and through splats, both of which Julia specializes only up to 32 elements. At first order every call then allocates (22.8 KB per gradient call at 40 contexts in the example below, against 272 B at 8). At second order through Enzyme, Rewrap is part of the function the outer pass differentiates, and Enzyme rejects its unspecialized path with EnzymeRuntimeActivityError, under reverse over forward and under forward over reverse alike.

32 is reached sooner than it looks: ReverseOverForward prepends four contexts of its own (FunctionContext, the inner backend, the tangent, the Rewrap), so a user function with 28 contexts hits it in hvp. Generated code that hands a model's data to the function as one Constant per array gets there routinely.

Where it comes from

All on main at 7f042d1 (identical in the v0.7.22 release):

Base's map(f, t::Tuple) is unrolled for up to 31 elements; from 32 (Any32) it fills a Vector{Any} and splats it, so its result is inferred as Tuple (base/tuple.jl, map(f, t::Any32)). A splat of more than 32 elements is left as Core._apply_iterate by the optimizer (max_tuple_splat = 32), so each such call site dispatches at run time.

Reproducer

Julia 1.10.12, DifferentiationInterface 0.7.22, Enzyme 0.13.219 (the registered releases), Linux x86_64.

using DifferentiationInterface
import DifferentiationInterface as DI
import Enzyme

# 1. DifferentiationInterface alone: context handling is inferred only below 32 contexts.
for C in (31, 32)
    cs = ntuple(i -> Constant(Float64(i)), C)
    rewrap = DI.Rewrap(cs...)
    unwrapped = map(DI.unwrap, cs)
    rewrap(unwrapped...)
    println("Rewrap with $C contexts: returns ", only(Base.return_types(rewrap, typeof(unwrapped))),
            ", allocates ", @allocated(rewrap(unwrapped...)), " B per call")
end

# 2. Through Enzyme: a function of K Constant contexts and one Cache, at first and second order.
reverse = AutoEnzyme(; mode = Enzyme.Reverse)
forward = AutoEnzyme(; mode = Enzyme.Forward)
for K in (8, 40)
    ws = [Symbol(:w, j) for j in 1:K]
    name = Symbol(:f, K)
    @eval function $name(x, $(ws...), box)
        v = (x[1]^2 * x[2]) * $(foldl((a, b) -> :($a + $b), [:(sum($w)) for w in ws]))
        box[] = v
        return v
    end
    f = getfield(Main, name)
    x = [0.3, 0.7]
    contexts = (ntuple(j -> Constant(fill(Float64(j), 3)), K)..., Cache(Ref(0.0)))
    total = sum(3j for j in 1:K)
    prep = prepare_gradient(f, reverse, x, contexts...)
    gradient(f, prep, reverse, x, contexts...)
    println("K=$K: warm gradient allocates ", @allocated(gradient(f, prep, reverse, x, contexts...)), " B per call")
    for (label, backend) in (("reverse over forward", SecondOrder(reverse, forward)),
                             ("forward over reverse", SecondOrder(forward, reverse)))
        try
            hv = only(hvp(f, backend, x, ([1.0, 0.0],), contexts...))
            println("K=$K: $label hvp, max error ", maximum(abs, hv .- total .* [2x[2], 2x[1]]))
        catch err
            println("K=$K: $label hvp throws ", nameof(typeof(err)))
        end
    end
end

Output (the function at K = 8 is the same shape with the same Cache, so the only difference is the number of contexts):

Rewrap with 31 contexts: returns NTuple{31, Constant{Float64}}, allocates 752 B per call
Rewrap with 32 contexts: returns Tuple, allocates 1600 B per call
K=8: warm gradient allocates 272 B per call
K=8: reverse over forward hvp, max error 0.0
K=8: forward over reverse hvp, max error 0.0
K=40: warm gradient allocates 22816 B per call
K=40: reverse over forward hvp throws EnzymeRuntimeActivityError
K=40: forward over reverse hvp throws EnzymeRuntimeActivityError

Both errors read EnzymeRuntimeActivityError: Detected potential need for runtime activity. Constant memory is stored (or returned) to a differentiable variable and correctness cannot be guaranteed with static activity analysis. […] Failure within method: shuffled_single_pushforward(::Vector{Float64}, ::typeof(f40), ::AutoEnzyme{…}, …) for reverse over forward (raised from src/first_order/pushforward.jl:554), and the same text with shuffled_gradient(…) for forward over reverse (src/first_order/gradient.jl:160). Those are the functions whose bodies call the Rewrap.

Possible fix

Build the context tuples in a way that stays inferred at any length, for instance ntuple(i -> maker(contexts[i]), Val(C)) (its generated branch unrolls at every C) or a @generated body for Rewrap and the two Enzyme helpers. The splats into the backend's entry points are a separate cost that a generated call could remove in the same way; Enzyme's own entry points re-splat their arguments too, which is outside this issue.

Activity

  1. gdalle-bot commented on Nov 18, 2026

    @gdalle-bot
    Contributor

    Note

    This answer was drafted by a coding agent (Claude Code) at @gdalle's request.

    Thanks for the detailed report! The diagnosis is right, but the suggested fix is not enough on its own.

    Can max_tuple_splat be changed?

    Not by an individual user. It is a field of the native compiler's InferenceParams / OptimizationParams, hardcoded to 32, with no command-line flag, environment variable or per-module macro (unlike Base.Experimental.@max_methods). Changing it requires rebuilding Julia or a custom AbstractInterpreter, which normal execution doesn't use. And even then it wouldn't fix map(f, t::Any32) in base/tuple.jl, which is an ordinary Base method that goes through a Vector{Any} regardless of compiler parameters.

    What I measured

    Julia 1.13.4, Enzyme 0.13.221, DI main, your reproducer at K = 40 (K = 8 gives 80 B and correct hvp in every variant):

    variant Rewrap inferred at 32+ warm gradient reverse-over-forward hvp forward-over-reverse hvp
    baseline no 23 360 B EnzymeRuntimeActivityError AssertionError
    ntuple(..., Val(C)) in Rewrap, make_context_shadows, translate_prepared! yes 7 200 B EnzymeRuntimeActivityError AssertionError
    all arrays packed into a single Constant, plus the Cache n/a (2 contexts) 80 B correct correct

    So switching to ntuple fixes Rewrap and cuts allocations by about 3×, but the nested hvp still fails. The remaining cost comes from splatting more than 32 arguments, which leaves a run-time Core._apply_iterate at each call site: f(x, contexts...), FixTail, autodiff(..., contexts...), and Enzyme's own entry points, which re-splat internally. Removing all of these would take @generated call helpers across every backend extension, and still wouldn't cover Enzyme's internals.

    Recommended workaround

    Pack the data into one Constant holding a tuple, NamedTuple or struct, and unpack it inside the function. For generated code this should be a small change:

    @eval function g40(x, data, box)
        ($(ws...),) = data
        v = (x[1]^2 * x[2]) * $(foldl((a, b) -> :($a + $b), [:(sum($w)) for w in ws]))
        box[] = v
        return v
    end
    contexts = (Constant(ntuple(j -> fill(Float64(j), 3), 40)), Cache(Ref(0.0)))

    If you also have many caches, they can be grouped the same way, but note that only some backends accept a Cache holding (named) tuples of arrays (see the Cache docstring).

    On the DI side

    I plan to document the 32-context limit and the packing advice in the Context docs. The ntuple change is cheap and safe, so it may go in as well, but only as a partial improvement, not a fix for this issue.

  2. deleted a comment from gdalle-bot on Nov 18, 2026
  3. gdalle commented on Nov 18, 2026

    @gdalle
    Member

    Hey there! I had my trusted clanker investigate your issue, and I'm not sure DI can fully fix it. We can improve some of it, but the best fix will always be grouping contexts until you have fewer than the number at which dynamic dispatch kicks in. Is it possible in your case?

  4. nsiccha commented on Nov 18, 2026

    @nsiccha
    Author

    I'd think so, yes!

  5. gdalle commented on Nov 18, 2026

    @gdalle
    Member

    Let me know how it goes!

  6. nsiccha commented on Nov 18, 2026

    @nsiccha
    Author

    Ah, I forgot, "I" had already fixed/worked around that locally! :)

  7. gdalle commented on Nov 18, 2026

    @gdalle
    Member

    Alright, think I'm gonna close this as "unplanned" then!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions