Skip to content

Differentiate the gamma tail functions under ForwardDiff - #231

Open
ParadaCarleton wants to merge 6 commits into
JuliaStats:masterfrom
ParadaCarleton:forwarddiff-gamma-tail
Open

ParadaCarleton wants to merge 6 commits into
JuliaStats:masterfrom
ParadaCarleton:forwarddiff-gamma-tail

Conversation

@ParadaCarleton

@ParadaCarleton ParadaCarleton commented Sep 21, 2026 •

Copy link
Copy Markdown

Closes #161.

SpecialFunctions.gamma_inc, which every gamma tail bottoms out in, is defined for Float16/Float32/Float64 only, so gammacdf, gammaccdf, gammalogcdf and gammalogccdf throw on Duals.

This adds a ForwardDiffExt extension that defines these four functions on Dual arguments via ForwardDiff.@define_ternary_dual_op: the value is computed on the primals and the θ and x partials are attached analytically (|∂P/∂x| = f_k(x), |∂P/∂θ| = k f_{k+1}(x), in log space, so they are 0 outside the support and the log forms stay accurate in the tails). A shape tangent throws an ArgumentError, because ∂/∂k has no elementary closed form. A Dual shape with zero partials (e.g. promoted by Distributions.Gamma) is accepted.

Tests compare against the Erlang closed forms and finite differences, including tails, the support boundary, and all Dual/tag combinations.

test/qa.jl ignores value, partials, dual_definition_retval, @define_ternary_dual_op and ≺, which ForwardDiff does not declare public.

gamma_inc is defined for Float16, Float32 and Float64, so gammacdf and its
siblings threw on a Dual. The extension evaluates the tail on the primal and
attaches the tangent analytically: writing the tail as P(k, x/theta) gives
d/dtheta = -(x/theta) d/dx, so one derivative carries both tangents. A nonzero
shape tangent throws, since d/dk has no elementary closed form.

Closes JuliaStats#161
@codecov

codecov Bot commented Sep 21, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 76.53%. Comparing base (7ed3a86) to head (8a136fe).

Additional details and impacted files
@@            Coverage Diff             @@
##           master     #231      +/-   ##
==========================================
+ Coverage   75.91%   76.53%   +0.61%     
==========================================
  Files          23       24       +1     
  Lines        1146     1176      +30     
==========================================
+ Hits          870      900      +30     
  Misses        276      276              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Comment thread ext/StatsFunsForwardDiffExt.jl Resolved
Comment thread ext/StatsFunsForwardDiffExt.jl Resolved
Comment thread ext/StatsFunsForwardDiffExt.jl Resolved
Comment thread ext/StatsFunsForwardDiffExt.jl Resolved
Comment thread ext/StatsFunsForwardDiffExt.jl Resolved
Comment thread test/qa.jl Resolved
ParadaCarleton and others added 2 commits September 21, 2026 17:57
@ParadaCarleton

ParadaCarleton commented Sep 22, 2026 •

Copy link
Copy Markdown
Author

Thanks. All six addressed in a42e542.

  • Module/file naming. Fixed.
  • Generality / tags / NaN-safety. Rewritten on ForwardDiff.@define_ternary_dual_op, so every argument/tag combination is generated, value/partials are called tag-qualified, and the tangents go through dual_definition_retval, i.e. _mul_partials and the NaN-safe path. Each branch only forms the partials whose argument is actually dual.
  • No differentiation wrt k. The methods now attach to the public gammalogcdf/gammalogccdf rather than the _-prefixed internals: gammalogcdf(k, θ, x) = _gammalogcdf(map(float, promote(k, θ, x))...) promotes k to a Dual before the internal is reached, so a rule on the internal can never see a non-dual k. Pre-promotion, a dual k means the caller really is differentiating wrt the shape, and that throws.
  • test/qa.jl. Reverted, except for four names on check_all_qualified_accesses_are_public: value, partials, dual_definition_retval, @define_ternary_dual_op, since ForwardDiff declares nothing public.

@devmotion devmotion left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the update! A few remaining issues (details inline):

  • A Dual shape with zero partials throws. This happens, for example, when the shape comes from Distributions.Gamma(α, θ), which promotes α, so cdf(Gamma(2.0, θ), x) can't be differentiated wrt θ.
  • The tangents are NaN at the boundary of the support. The θ partial can also be computed more robustly in log space.
  • The tests mostly compare against the same formulas the implementation uses. They don't cover the branches where θ and x are both duals (same or different tags), and they don't reach the tails.
  • Runic fails on ext/ForwardDiffExt.jl and test/forwarddiff.jl.
  • The formatting changes in test/qa.jl are still there.
  • The PR description is outdated: it mentions finite differences and the old set of test/qa.jl ignores.

Comment thread ext/ForwardDiffExt.jl Resolved
Comment thread ext/ForwardDiffExt.jl Resolved
Comment thread ext/ForwardDiffExt.jl Resolved
Comment thread ext/ForwardDiffExt.jl Resolved
Comment thread ext/ForwardDiffExt.jl Resolved
Comment thread test/forwarddiff.jl Resolved
Comment thread test/forwarddiff.jl Resolved
Comment thread test/qa.jl Resolved
Comment thread test/qa.jl Resolved
Comment thread ext/ForwardDiffExt.jl Resolved
…ests

- Compute the θ partial as k f_{k+1}(x) and both partials in log space, so they
  are zero outside the support and at x = 0 instead of NaN.
- A Dual shape with zero partials (e.g. promoted by Distributions.Gamma) is
  replaced by its value; only a nonzero shape tangent throws.
- Collapse the four generated _dual_tail methods into one function.
- Test against the Erlang closed forms and finite differences, in the tails,
  at the boundary of the support, and for every Dual/tag combination.
- Revert the formatting changes in test/qa.jl; keep the ForwardDiff ignores,
  which ExplicitImports needs because it also checks the extension.
@ParadaCarleton
ParadaCarleton force-pushed the forwarddiff-gamma-tail branch from e87fa64 to 263a00e Compare October 5, 2026 00:07
Comment thread ext/ForwardDiffExt.jl Outdated
Comment on lines +6 to +8
# `SpecialFunctions.gamma_inc`, which every gamma tail bottoms out in, is defined
# for `Float16`, `Float32` and `Float64` only, so for `Dual` arguments we evaluate
# the function on the primal values and attach the partials analytically.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

gamma_inc also has BigFloat methods. Maybe just say it's not defined for Duals?

Comment thread ext/ForwardDiffExt.jl Outdated
# the function on the primal values and attach the partials analytically.

# With the tail written as `P(k, x/θ)`, `|∂P/∂x| = f_k(x)` and `|∂P/∂θ| = k f_{k+1}(x)`,
# both in log space so they are zero outside the support and at `x = 0`. The log forms

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not at x = 0 for k ≤ 1 (1/θ and Inf):

Suggested change
# both in log space so they are zero outside the support and at `x = 0`. The log forms
# both in log space so they are zero outside the support. The log forms

Comment thread ext/ForwardDiffExt.jl Outdated
# With the tail written as `P(k, x/θ)`, `|∂P/∂x| = f_k(x)` and `|∂P/∂θ| = k f_{k+1}(x)`,
# both in log space so they are zero outside the support and at `x = 0`. The log forms
# divide them by the tail probability `exp(primal)`.
function _dual_tail(f, complementary::Bool, logarithmic::Bool, k::Real, θ::Real, x::Real)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

log(k) widens an integer k to Float64, e.g. gammacdf(2, Dual(1f0, 1f0), 1.5f0) returns a Dual{Nothing, Float64}. Promote first (and add a Float32 test)?

Suggested change
function _dual_tail(f, complementary::Bool, logarithmic::Bool, k::Real, θ::Real, x::Real)
function _dual_tail(f, complementary::Bool, logarithmic::Bool, k::Real, θ::Real, x::Real)
k, θ, x = promote(k, θ, x)

Comment thread ext/ForwardDiffExt.jl Outdated
Comment on lines +15 to +16
log_partial_x = StatsFuns.gammalogpdf(k, θ, x)
log_partial_θ = log(k) + StatsFuns.gammalogpdf(k + 1, θ, x)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The y- and z-only bodies always compute both partials (one unused gammalogpdf). Worth avoiding?

Comment thread test/forwarddiff.jl Outdated
for x in (0.5, 1.0, 4.0, 12.0)
]
@testset "Erlang closed form: $f($k, $θ, $x)" for (f, erlang, k, θ, x) in erlang_cases
@test ForwardDiff.value(f(k, θ, Dual(x, 1.0))) ≈ f(k, θ, x)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

True by construction. Compare with the closed form?

Suggested change
@test ForwardDiff.value(f(k, θ, Dual(x, 1.0))) ≈ f(k, θ, x)
@test ForwardDiff.value(f(k, θ, Dual(x, 1.0))) ≈ erlang(k, θ, x)

Comment thread test/forwarddiff.jl Outdated
Comment on lines +86 to +87
# The Poisson tails promote `λ` and the shape to `Dual`s, which `_gammalogcdf` and
# `_gammalogccdf` do not accept (JuliaStats/StatsFuns.jl#161).

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The Poisson tails don't promote; the gamma tails do:

Suggested change
# The Poisson tails promote `λ` and the shape to `Dual`s, which `_gammalogcdf` and
# `_gammalogccdf` do not accept (JuliaStats/StatsFuns.jl#161).
# The Poisson tails call the gamma tails with a `Dual` `x = λ`, which failed before
# (JuliaStats/StatsFuns.jl#161).

Comment thread test/forwarddiff.jl
dx = ForwardDiff.derivative(y -> erlang(2, 1.0, y), 1.5)
@test ForwardDiff.derivative(t -> f(oftype(t, 2), t, 1.5), 1.0) ≈ dθ
@test ForwardDiff.derivative(y -> f(oftype(y, 2), 1.0, y), 1.5) ≈ dx
@test ForwardDiff.gradient(v -> f(2 + 0 * v[1], v[1], v[2]), [1.0, 1.5]) ≈ [dθ, dx]

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also a hessian with a promoted shape?

Suggested change
@test ForwardDiff.gradient(v -> f(2 + 0 * v[1], v[1], v[2]), [1.0, 1.5]) ≈ [dθ, dx]
@test ForwardDiff.gradient(v -> f(2 + 0 * v[1], v[1], v[2]), [1.0, 1.5]) ≈ [dθ, dx]
@test ForwardDiff.hessian(v -> f(2 + 0 * v[1], v[1], v[2]), [1.0, 1.5]) ≈
ForwardDiff.hessian(v -> erlang(2, v[1], v[2]), [1.0, 1.5])

Comment thread test/forwarddiff.jl Outdated
Comment on lines +117 to +126
@test ForwardDiff.derivative(
t -> ForwardDiff.derivative(s -> f(2 + 0 * s, t, 1.5), 1.0),
1.0,
) ==
0
@test ForwardDiff.derivative(
s -> ForwardDiff.derivative(t -> f(2 + 0 * s, t, 1.5), 1.0),
1.0,
) ==
0

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These are 0 whatever the θ partial is. Maybe check that it survives?

Suggested change
@test ForwardDiff.derivative(
t -> ForwardDiff.derivative(s -> f(2 + 0 * s, t, 1.5), 1.0),
1.0,
) ==
0
@test ForwardDiff.derivative(
s -> ForwardDiff.derivative(t -> f(2 + 0 * s, t, 1.5), 1.0),
1.0,
) ==
0
@test ForwardDiff.derivative(
t -> ForwardDiff.derivative(s -> s * f(2 + 0 * s, t, 1.5), 1.0),
1.0,
) ≈ dθ
@test ForwardDiff.derivative(
s -> s * ForwardDiff.derivative(t -> f(2 + 0 * s, t, 1.5), 1.0),
1.0,
) ≈ dθ

@devmotion

Copy link
Copy Markdown
Member

Two PRs that are related to this one:

Neither is merged or released yet. Once both are released, gammacdf and gammaccdf would differentiate in θ and x without any rule here. The first argument is promoted to a Dual with zero partials, and the derivative then propagates through max(0, x) / θ. Locally, with both branches (and a stand-in for JuliaDiff/ForwardDiff.jl#855, which nested derivatives need), against BigFloat:

julia> ForwardDiff.derivative(y -> gammacdf(2.5, 1.5, y), 3.0)  # relative error 1e-16
0.19196788093577977

With the current releases, all four functions still need the rules from this PR. With the new versions, only the log versions would, since _gammalogcdf and _gammalogccdf are defined only for Float16/Float32/Float64.

On accuracy: exp(gammalogpdf(k, θ, x)) loses precision for large k, because of the cancellation between xlogy(k - 1, x/θ), x/θ and loggamma(k). Relative error of ∂x gammacdf(k, 1, x) at x = k, against BigFloat:

k exp(gammalogpdf) gamma_inc_dx (#871)
Float64 1e4 1.1e-13 8e-17
Float64 1e6 6.8e-10 3e-17
Float64 1e10 9.8e-6 9e-17
Float32 1e3 1.5e-4 1.5e-8
Float32 1e4 6.8e-4 1.8e-8
Float32 1e6 16% 1.7e-8

The θ partial, log(k) + gammalogpdf(k + 1, θ, x), has the same issue. Where the density doesn't underflow, ∂P/∂x = gamma_inc_dx(k, x/θ) / θ and ∂P/∂θ = -(x/θ) ∂P/∂x avoid the cancellation. In the deep tails of the log versions, the log-space form is still needed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Relax _gammalogccdf argument types not to be restricted to Float64

2 participants