Repository navigation
Differentiate the gamma tail functions under ForwardDiff - #231
ParadaCarleton wants to merge 6 commits into
Conversation
gamma_inc is defined for Float16, Float32 and Float64, so gammacdf and its siblings threw on a Dual. The extension evaluates the tail on the primal and attaches the tangent analytically: writing the tail as P(k, x/theta) gives d/dtheta = -(x/theta) d/dx, so one derivative carries both tangents. A nonzero shape tangent throws, since d/dk has no elementary closed form. Closes JuliaStats#161
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #231 +/- ##
==========================================
+ Coverage 75.91% 76.53% +0.61%
==========================================
Files 23 24 +1
Lines 1146 1176 +30
==========================================
+ Hits 870 900 +30
Misses 276 276 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Co-authored-by: David Müller-Widmann <devmotion@users.noreply.github.com>
|
Thanks. All six addressed in a42e542.
|
devmotion
left a comment
There was a problem hiding this comment.
Thanks for the update! A few remaining issues (details inline):
- A
Dualshape with zero partials throws. This happens, for example, when the shape comes fromDistributions.Gamma(α, θ), which promotesα, socdf(Gamma(2.0, θ), x)can't be differentiated wrtθ. - The tangents are
NaNat the boundary of the support. The θ partial can also be computed more robustly in log space. - The tests mostly compare against the same formulas the implementation uses. They don't cover the branches where θ and x are both duals (same or different tags), and they don't reach the tails.
- Runic fails on
ext/ForwardDiffExt.jlandtest/forwarddiff.jl. - The formatting changes in
test/qa.jlare still there. - The PR description is outdated: it mentions finite differences and the old set of
test/qa.jlignores.
…ests
- Compute the θ partial as k f_{k+1}(x) and both partials in log space, so they
are zero outside the support and at x = 0 instead of NaN.
- A Dual shape with zero partials (e.g. promoted by Distributions.Gamma) is
replaced by its value; only a nonzero shape tangent throws.
- Collapse the four generated _dual_tail methods into one function.
- Test against the Erlang closed forms and finite differences, in the tails,
at the boundary of the support, and for every Dual/tag combination.
- Revert the formatting changes in test/qa.jl; keep the ForwardDiff ignores,
which ExplicitImports needs because it also checks the extension.
e87fa64 to
263a00e
Compare
| # `SpecialFunctions.gamma_inc`, which every gamma tail bottoms out in, is defined | ||
| # for `Float16`, `Float32` and `Float64` only, so for `Dual` arguments we evaluate | ||
| # the function on the primal values and attach the partials analytically. |
There was a problem hiding this comment.
gamma_inc also has BigFloat methods. Maybe just say it's not defined for Duals?
| # the function on the primal values and attach the partials analytically. | ||
|
|
||
| # With the tail written as `P(k, x/θ)`, `|∂P/∂x| = f_k(x)` and `|∂P/∂θ| = k f_{k+1}(x)`, | ||
| # both in log space so they are zero outside the support and at `x = 0`. The log forms |
There was a problem hiding this comment.
Not at x = 0 for k ≤ 1 (1/θ and Inf):
| # both in log space so they are zero outside the support and at `x = 0`. The log forms | |
| # both in log space so they are zero outside the support. The log forms |
| # With the tail written as `P(k, x/θ)`, `|∂P/∂x| = f_k(x)` and `|∂P/∂θ| = k f_{k+1}(x)`, | ||
| # both in log space so they are zero outside the support and at `x = 0`. The log forms | ||
| # divide them by the tail probability `exp(primal)`. | ||
| function _dual_tail(f, complementary::Bool, logarithmic::Bool, k::Real, θ::Real, x::Real) |
There was a problem hiding this comment.
log(k) widens an integer k to Float64, e.g. gammacdf(2, Dual(1f0, 1f0), 1.5f0) returns a Dual{Nothing, Float64}. Promote first (and add a Float32 test)?
| function _dual_tail(f, complementary::Bool, logarithmic::Bool, k::Real, θ::Real, x::Real) | |
| function _dual_tail(f, complementary::Bool, logarithmic::Bool, k::Real, θ::Real, x::Real) | |
| k, θ, x = promote(k, θ, x) |
| log_partial_x = StatsFuns.gammalogpdf(k, θ, x) | ||
| log_partial_θ = log(k) + StatsFuns.gammalogpdf(k + 1, θ, x) |
There was a problem hiding this comment.
The y- and z-only bodies always compute both partials (one unused gammalogpdf). Worth avoiding?
| for x in (0.5, 1.0, 4.0, 12.0) | ||
| ] | ||
| @testset "Erlang closed form: $f($k, $θ, $x)" for (f, erlang, k, θ, x) in erlang_cases | ||
| @test ForwardDiff.value(f(k, θ, Dual(x, 1.0))) ≈ f(k, θ, x) |
There was a problem hiding this comment.
True by construction. Compare with the closed form?
| @test ForwardDiff.value(f(k, θ, Dual(x, 1.0))) ≈ f(k, θ, x) | |
| @test ForwardDiff.value(f(k, θ, Dual(x, 1.0))) ≈ erlang(k, θ, x) |
| # The Poisson tails promote `λ` and the shape to `Dual`s, which `_gammalogcdf` and | ||
| # `_gammalogccdf` do not accept (JuliaStats/StatsFuns.jl#161). |
There was a problem hiding this comment.
The Poisson tails don't promote; the gamma tails do:
| # The Poisson tails promote `λ` and the shape to `Dual`s, which `_gammalogcdf` and | |
| # `_gammalogccdf` do not accept (JuliaStats/StatsFuns.jl#161). | |
| # The Poisson tails call the gamma tails with a `Dual` `x = λ`, which failed before | |
| # (JuliaStats/StatsFuns.jl#161). |
| dx = ForwardDiff.derivative(y -> erlang(2, 1.0, y), 1.5) | ||
| @test ForwardDiff.derivative(t -> f(oftype(t, 2), t, 1.5), 1.0) ≈ dθ | ||
| @test ForwardDiff.derivative(y -> f(oftype(y, 2), 1.0, y), 1.5) ≈ dx | ||
| @test ForwardDiff.gradient(v -> f(2 + 0 * v[1], v[1], v[2]), [1.0, 1.5]) ≈ [dθ, dx] |
There was a problem hiding this comment.
Also a hessian with a promoted shape?
| @test ForwardDiff.gradient(v -> f(2 + 0 * v[1], v[1], v[2]), [1.0, 1.5]) ≈ [dθ, dx] | |
| @test ForwardDiff.gradient(v -> f(2 + 0 * v[1], v[1], v[2]), [1.0, 1.5]) ≈ [dθ, dx] | |
| @test ForwardDiff.hessian(v -> f(2 + 0 * v[1], v[1], v[2]), [1.0, 1.5]) ≈ | |
| ForwardDiff.hessian(v -> erlang(2, v[1], v[2]), [1.0, 1.5]) |
| @test ForwardDiff.derivative( | ||
| t -> ForwardDiff.derivative(s -> f(2 + 0 * s, t, 1.5), 1.0), | ||
| 1.0, | ||
| ) == | ||
| 0 | ||
| @test ForwardDiff.derivative( | ||
| s -> ForwardDiff.derivative(t -> f(2 + 0 * s, t, 1.5), 1.0), | ||
| 1.0, | ||
| ) == | ||
| 0 |
There was a problem hiding this comment.
These are 0 whatever the θ partial is. Maybe check that it survives?
| @test ForwardDiff.derivative( | |
| t -> ForwardDiff.derivative(s -> f(2 + 0 * s, t, 1.5), 1.0), | |
| 1.0, | |
| ) == | |
| 0 | |
| @test ForwardDiff.derivative( | |
| s -> ForwardDiff.derivative(t -> f(2 + 0 * s, t, 1.5), 1.0), | |
| 1.0, | |
| ) == | |
| 0 | |
| @test ForwardDiff.derivative( | |
| t -> ForwardDiff.derivative(s -> s * f(2 + 0 * s, t, 1.5), 1.0), | |
| 1.0, | |
| ) ≈ dθ | |
| @test ForwardDiff.derivative( | |
| s -> s * ForwardDiff.derivative(t -> f(2 + 0 * s, t, 1.5), 1.0), | |
| 1.0, | |
| ) ≈ dθ |
…nto forwarddiff-gamma-tail
…losed forms and Float32
|
Two PRs that are related to this one:
Neither is merged or released yet. Once both are released, julia> ForwardDiff.derivative(y -> gammacdf(2.5, 1.5, y), 3.0) # relative error 1e-16
0.19196788093577977With the current releases, all four functions still need the rules from this PR. With the new versions, only the log versions would, since On accuracy:
The |
Closes #161.
SpecialFunctions.gamma_inc, which every gamma tail bottoms out in, is defined forFloat16/Float32/Float64only, sogammacdf,gammaccdf,gammalogcdfandgammalogccdfthrow onDuals.This adds a
ForwardDiffExtextension that defines these four functions onDualarguments viaForwardDiff.@define_ternary_dual_op: the value is computed on the primals and the θ and x partials are attached analytically (|∂P/∂x| = f_k(x),|∂P/∂θ| = k f_{k+1}(x), in log space, so they are 0 outside the support and the log forms stay accurate in the tails). A shape tangent throws anArgumentError, because∂/∂khas no elementary closed form. ADualshape with zero partials (e.g. promoted byDistributions.Gamma) is accepted.Tests compare against the Erlang closed forms and finite differences, including tails, the support boundary, and all Dual/tag combinations.
test/qa.jlignoresvalue,partials,dual_definition_retval,@define_ternary_dual_opand≺, which ForwardDiff does not declare public.