Skip to content

Initialize Dirichlet fitting without sample variable - #2077

Open
andreasnoack wants to merge 4 commits into
masterfrom
an/issue602
Open

andreasnoack wants to merge 4 commits into
masterfrom
an/issue602

Conversation

@andreasnoack

Copy link
Copy Markdown
Member

This avoids dividing by zero. The new initialization is based on the paper by Thomas Minka linked in the comment. This is an alternative to #610 based on some of the ideas in the paper. It also avoids the extra allocation. Of course, at the cost of some special function evaluations.

Fixes #602

@andreasnoack
andreasnoack requested a review from devmotion June 30, 2026 21:35
@andreasnoack
andreasnoack force-pushed the an/issue602 branch 2 times, most recently from e85d66b to 4b5191e Compare July 1, 2026 07:37
@codecov

codecov Bot commented Jul 1, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 85.18519% with 4 lines in your changes missing coverage. Please review.
✅ Project coverage is 87.75%. Comparing base (264883f) to head (6a16ec4).

Files with missing lines Patch % Lines
src/multivariate/dirichlet.jl 85.18% 4 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #2077      +/-   ##
==========================================
+ Coverage   87.35%   87.75%   +0.40%     
==========================================
  Files         149      149              
  Lines        8936     8911      -25     
==========================================
+ Hits         7806     7820      +14     
+ Misses       1130     1091      -39     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@devmotion

devmotion commented Jul 2, 2026 •

Copy link
Copy Markdown
Member

AFAICT the PR mixes two approaches presented by Minka, the EM-like fixed-point iteration and the Newton algorithm. It's not immediately clear to me why one would want to combine those two approaches and switch after a seemingly arbitrary amount of 5 fixed-point iterations to the Newton algorithm. IMO the crucial and currently in a few cases problematic part is just the initialization of the algorithms. As Minka mentions as well, there's many different possible approaches for initializing the algorithms, e.g., based on method of moments (obviously, different choices of moments will lead to different initializations, Minka already discusses two different approaches) or approximations of the likelihood function, e.g., using Stirling's approximation.

Exploiting the moment equations $E\left[p_k\right] = \alpha_k / \alpha_0$ for $k = 1, \ldots, K$ (as currently done) seems natural. However, since one of these $K$ equations is redundant, it leaves the question what additional equation/constraint to include to obtain an equation system that can be solved for the $K$ parameters $\alpha_1, \ldots, \alpha_K$. Minka presents different approaches in eq. 21, 23 and 42. The approaches in eq. 21 and 23 break down if the sample variance of $p_1$ (eq 21) or any $p_k$ (eq 23) is zero. Eq. 42 doesn't suffer from this problem, so as suggested in this PR it would be a more stable approach. Alternatively, one could consider other moments such as

$$E\left[\sum_k p_k^2\right] = \frac{1 + \alpha_0 \sum_k {(\alpha_k / \alpha_0)}^2}{1 + \alpha_0}$$

which AFAICT would give moment estimates

$$\alpha_k = \hat{E}\left[p_k\right] \left(\frac{1 - \sum_{\tilde{k}} \hat{E}\left[p_{\tilde{k}}^2\right]}{\sum_{\tilde{k}} \hat{E}\left[p_{\tilde{k}}^2\right] - \sum_{\tilde{k}} {\hat{E}\left[p_{\tilde{k}}\right]}^2} \right) = \hat{E}[p_k] \frac{\sum_{\tilde{k}} \hat{E}\left[p_{\tilde{k}}(1-p_{\tilde{k}})\right]}{\sum_{\tilde{k}} \hat{var}\left[p_{\tilde{k}}\right]} $$

(seemingly a symmetrized version of eq. 21?). This would only break down if the sample variance of all $p_k$ is zero, ie., all samples are identical, in which case the estimation breaks down anyway.

So, taken together, my impression is: Using eq. 42 would give a less brittle initialization but one could also consider other approaches, e.g., based on different moments; but it's not clear to me why one would want to combine the EM-like fixed-point iteration with the Newton algorithm, using a different more stable initialization for the Newton algorithm alone (or the EM-like fixed-point iteration, if one prefers that approach) seems sufficient.

@andreasnoack

Copy link
Copy Markdown
Member Author

The problem is that the parameters must be positive, so the initialization has to be sufficiently good to ensure that the Newton step is feasible. The fixed-point iterations will always produce positive values, but it has linear convergence. I also tried the initialization suggested in https://www.sciencedirect.com/science/article/abs/pii/S0167947307002848, but then I can trigger a maxiter error by making the repeated value small enough.

@devmotion

devmotion commented Jul 3, 2026 •

Copy link
Copy Markdown
Member

To me it seems there's two separate problems you're facing and trying to address: the problematic initialization (the original issue reported in #602) and infeasible iterates of the Newton algorithm. Seemingly the first problem could be fixed by an initialization that doesn't break down if the sample variance of some $p_k$ is zero, e.g., by making use of eq. 42 in Minka's report, the estimate for $\alpha_0$ in the linked paper or possibly by considering different moments as outlined above. The second problem seems merely a consequence of applying (unconstrained) Newton method on a constrained space, and could maybe be approached by e.g. applying Newton method on the log space of $\log \alpha_k$ (AFAICT the efficient structure of the Hessian would be preserved) or using a penalty method.

@andreasnoack

Copy link
Copy Markdown
Member Author

The two issues are closely related. In theory, we could just initialize all the values at one, except for the risk of stepping to an infeasible argument. Hence, we need to find a sufficiently good initialization before switching to Newton's method and thereby ensure fast convergence through the concave objective function.

Seemingly the first problem could be fixed by an initialization that doesn't break down if the sample variance of some $p_k$ is zero, e.g., by making use of eq. 42 in Minka's report, the estimate for $α_0$ in the linked paper or possibly by considering different moments as outlined above.

Yes, but as mentioned above, I could trigger maxiter issues with these initializers.

...e.g. applying Newton method on the log space of $log(α_k)$ (AFAICT the efficient structure of the Hessian would be preserved)

The global concavity of the objective function is not preserved under this reparametrization. We could try a constrained optimizer, but that would be a larger change than the current one.

@devmotion

Copy link
Copy Markdown
Member

To avoid the concavity issue, we could use natural gradient descent in log space instead of of Newton's method. In the standard parameter space, Newton's method is exactly the natural gradient method with step size 1 (the Hessian is constant with respect to the data and hence identical to the FIM). AFAICT this would correspond to iterations $\alpha \to \alpha \odot \exp.{\left(\gamma s(\alpha) ./ \alpha\right)}$ with step size $\gamma$ and $s(\alpha)$ being the step direction of the corresponding Newton iteration in non-log space $\alpha \to \alpha + s(\alpha)$.

@andreasnoack

Copy link
Copy Markdown
Member Author

I'm not sure I follow. If you change the parametrization, then the objective function is no longer globally concave. It might be possible to optimize it anyway with Fisher scoring aka natural gradient descent, but you'd lose the guarantees that the concavity provides.

@devmotion

Copy link
Copy Markdown
Member

I didn't remember the details and all my thinking after the break, so used Claude to get up on speed again. Its conclusion was

On the substance, you're structurally right and Andreas is practically right

😄

@devmotion

Copy link
Copy Markdown
Member

I tried to reproduce the maxiter failures you mentioned to get a better understanding of what's going on, and AFAICT the fixed-point iterations do help but not for the reason you gave.

With

ε = 1e-10
X = [ε ε; 0.5(1 - ε) 0.3(1 - ε); 0.5(1 - ε) 0.7(1 - ε)]

eq. 42 alone stops at maxiter = 25 with gnorm ≈ 5.4 whereas eq. 42 followed by the five fixed-point iterations converges after 8 iterations, so the motivation is clear. However, in the failing run the clamping α[k] = max(α[k], 1e-12) in fit_dirichlet! is never triggered, and just increasing maxiter is sufficient, the iteration converges after 30 steps. This does not seem specific to the example: in ~1500 random problems ($K$ between 2 and 8, $\alpha_k$ spanning $10^{-3}$ to $10^3$) the Newton iteration initialized with moment estimates converged in every case I checked if maxiter was large enough (0 out of 275 problems with $n$ between 50 and 200 failed with maxiter = 1000), but the required number of iterations has a long right tail (median 6, 95% quantile 41, max 57), whereas with the initialization in this PR it is median 5, 95% quantile 7, max 8. On the other hand, the clamping is still triggered in about 4% of the random problems with the initialization in this PR.

So AFAICT what the fixed-point iterations buy is a smaller number of Newton iterations and not feasibility of the Newton steps, ie., they move the initial value into the region where the quadratic convergence of the Newton iteration kicks in, and that's what makes maxiter = 25 sufficient. That's a reasonable thing to want, so I withdraw my objection against combining the two algorithms. But then I think it would be good to make it explicit in the code, the returned value is not a moment estimate anymore, so possibly the function and the comments should state that it computes a starting value by running a few iterations of the fixed-point algorithm. Hardcoding the 5 seems fine to me, fit_mle accepts init anyway if someone wants to control the starting value, but since Minka gives no guidance it would be good if the comment recorded what motivates the choice, eg. that a few iterations are sufficient to reach the region where the Newton iteration converges quadratically. Independently of this PR, it seems to me that maxiter = 25 together with tol = 1e-12 for the absolute sup-norm of the gradient, and the clamping without any line search or check of the objective function, are the actually fragile parts here.

One thing that has to be fixed in any case: the second argument of _dirichlet_mle_init2 changed from $\gamma = \hat{E}[p^2]$ to $\log\mu = \hat{E}[\log p]$, but dirichlet_mle_init(P, w) still computes $\gamma$ and passes it. Both are Vector{Float64}, so there's no MethodError, only wrong results. E.g., for

X = [0.1 0.2 0.3; 0.5 0.3 0.4; 0.4 0.5 0.3]
w = ones(3)

eq. 42 returns a negative $\alpha_0 \approx -0.835$ (digamma of a negative non-integer is finite, so nothing errors), the fixed-point iterations run away, and the starting value ends up at $\approx [23.4, 26.4, 26.4]$ instead of $\approx [5.4, 11.1, 11.1]$. The weighted fit_mle then throws ErrorException("No convergence after 25 (maxiter) iterations.") even though it works on master. This is not caught by CI because the only test of the weighted fit_mle is commented out in test/multivariate/dirichlet.jl:121. Could we uncomment it here? A test with w = fill(2.0, n) would have caught it as well.

Some minor comments. The title and description say that the PR avoids dividing by zero, but the degenerate cases are still there, $n = 1$ and identical samples still divide by zero and the result only changes from Inf to NaN. What's fixed is the case where only some of the $p_k$ are constant, so possibly the description could be adjusted, and arguably these degenerate cases should throw a more informative error. Moreover, fit_mle now computes sum(log, P, dims=2) twice, once in dirichlet_mle_init and once in suffstats. If suffstats is computed first and elogp is passed to the initialization, then eq. 42 is basically free and only the invdigamma evaluations are added. The hard-coded values in the new test are fine IMO, I obtain the same numbers with an independent implementation.

Regarding the natural gradient suggestion, I don't want to block the PR on it, but I think we talked past each other. In what I suggested the step direction is still computed in the $\alpha$ parametrization, where the objective function is concave and hence $\nabla f \cdot s &gt; 0$, and the log parametrization only affects the step itself ($\alpha \odot \exp{(\gamma s ./ \alpha)}$ coincides with $\alpha + \gamma s$ to first order), so the concavity of the objective function in $\alpha$ is not given up. Relatedly, AFAICT the Hessian is data-independent and equal to the negative FIM, so Newton's method, Fisher scoring and natural gradient with unit step size coincide in the $\alpha$ parametrization anyway. That said, I implemented it with an Armijo line search and it does not perform better than the plain Newton iteration, it stalls before reaching gnorm < 1e-12 in ~20% of my random problems and has a worse tail, which I think just moves the problem to the stopping criterion mentioned above. So this should be a separate PR, if we want to pursue it at all.

And just for the record regarding the symmetrized moment estimator I mentioned above: AFAICT the reason why it is more stable is that $\sum_k (\hat{E}[p_k] - \hat{E}[p_k^2]) / \sum_k \hat{var}[p_k]$ is the variance-weighted mean of the per-component estimates of $\alpha_0$ on master (since $\hat{E}[p_k] - \hat{E}[p_k^2] = \hat{var}[p_k] \hat{\alpha}_0^{(k)}$) whereas master uses the unweighted mean, so components with small sample variance are downweighted instead of blowing up the estimate. But since it shows the same long tail of Newton iterations it doesn't address the point you care about, so eq. 42 seems fine to me.

@andreasnoack

Copy link
Copy Markdown
Member Author

Thanks for the thorough review! I've pushed a commit (and rebased on master) that should address your points:

  • Fixed the weighted dirichlet_mle_init, which still passed E[p^2] where the initialization now expects E[log(p)], and re-enabled the commented-out weighted fit_mle test. Also added a test that uniform weights reproduce the unweighted fit.
  • Renamed _dirichlet_mle_init2 to _dirichlet_mle_init and updated the comments to state that it returns a starting value for the Newton iterations rather than a moment estimate, and that the five fixed point iterations are there to reach the region of quadratic convergence.
  • fit_mle now computes suffstats before the initialization and reuses the log-mean, so eq 42 only adds the invdigamma evaluations.
  • When all samples are identical (including n = 1), the denominator in eq 42 is zero, so the initialization now throws an informative ArgumentError instead of producing NaNs. Added tests for both degenerate cases.

I've left maxiter, tol, and the clamping unchanged and agree that any changes there, as well as the natural gradient idea, belong in a separate PR.

@andreasnoack

Copy link
Copy Markdown
Member Author

I've also been speculating whether it is better to be structurally or practically right. I'm not sure.

@andreasnoack

Copy link
Copy Markdown
Member Author

Codecov flagged that the weighted dirichlet_mle_init was no longer covered: after moving the suffstats computation ahead of the initialization, fit_mle calls _dirichlet_mle_init directly and the dirichlet_mle_init(P[, w]) wrappers had become dead code only referenced by a test. I've removed them and adjusted the test to use _dirichlet_mle_init instead.

@devmotion

Copy link
Copy Markdown
Member

AFAICT the new s > 0 guard for the degenerate case doesn't hold up to roundoff. For identical samples $s$ is mathematically zero, but it is computed as the difference between the log of a summed mean and the mean of logs, so its sign is arbitrary:

# denominator is non-negative, and it is zero only if all samples are identical,
# in which case the MLE doesn't exist.
s = sum(μₖ * (log(μₖ) - logμₖ) for (μₖ, logμₖ) in zip(μ, logμ))
s > 0 || throw(ArgumentError(
"the samples are all identical so the MLE of the Dirichlet distribution doesn't exist"))
α₀ = (K - 1) / 2 / s
# Refine with a few iterations of the fixed point iteration in Minka eq 9. The

With P = repeat([0.1, 0.5, 0.4], 1, n) the ArgumentError is thrown for $n \in {1,2,3,4,6,7,8,9,11,12}$, but $n = 5$ returns $\alpha \approx [1.04 \cdot 10^{16}, 5.22 \cdot 10^{16}, 4.18 \cdot 10^{16}]$ and $n = 10$ returns $[\infty, \infty, \infty]$, in both cases reported as a successful fit. In ~4000 random problems with exactly identical samples ($K$ between 2 and 8, $n$ between 2 and 30) the guard is bypassed in 41%, and the result is then either $\alpha \sim 10^{14} \ldots 10^{16}$ or every component clamped to 1e-12. Which $n$ happen to work isn't predictable from the input, and the two new tests use $n = 2$ and $n = 1$, which do:

# The MLE doesn't exist when all samples are identical
@test_throws ArgumentError fit_mle(Dirichlet, X[:, [1, 1]])
@test_throws ArgumentError fit_mle(Dirichlet, X[:, 1:1])
end

This is not a regression, master returns [NaN, NaN, NaN] for all of these, but IMO we shouldn't claim the guarantee in the comment and the PR description if it holds for ~59% of the inputs. Could we loop over n in 1:12 in the test and assert that fit_mle either throws or returns a finite positive $\alpha$?

The reason these come back as converged rather than as an error is that fit_dirichlet! drops NaN gradients, since abs(NaN) > gnorm is false:

agk = abs(gk)
if agk > gnorm
gnorm = agk
end
end

so gnorm stays 0.0 and gnorm < tol succeeds. gnorm = max(gnorm, abs(gk)) is sufficient, since max propagates NaN in Julia (max(0.0, NaN) === NaN) and the existing ErrorException then fires. I said above that fit_dirichlet! should be a separate PR and I still think so for maxiter, tol and the clamping, but this one seems an unconditional bug to me, and it is what turns the cases above into silently wrong results, so I'd include it here.

Relatedly, a zero entry in P now returns NaN instead of throwing. For P = [0.0 0.2 0.3; 0.5 0.3 0.3; 0.5 0.5 0.4] master throws No convergence after 25 (maxiter) iterations whereas the PR returns Dirichlet([NaN, NaN, NaN]): $\log\mu_k = -\infty$ gives $s = \infty$, which passes s > 0, so $\alpha_0 = 0$, digamma(0) = -Inf and invdigamma(-Inf) = NaN. The input is invalid either way, but isfinite(s) && s > 0 would keep it an error.

Some minor comments. digamma(α₀) is loop-invariant in the inner loop:

α = μ
for _ in 1:5
for k in 1:K
α[k] = invdigamma(digamma(α₀) + logμ[k])
end
α₀ = sum(α)
end

digamma is neither inlined nor marked pure, so AFAICT it is not hoisted; pulling it out of the loop over $k$ gives bit-identical results and saves 7% ($K = 3$) to 17% ($K = 10$) of the initialization, and it also makes visible that eq 9 updates $\alpha_0$ only once per sweep.

The second argument is named logμ and the comment calls it the sample log-mean, but it is the mean of the logs, whereas log(μₖ) one character away on the same line is the log of the mean. Both call sites already call it elogp, consistent with mean_logp and with fit_dirichlet!. Since the bug in the previous round was exactly this kind of mixup, could we use elogp here as well? In the same vein, α = μ aliases and overwrites the caller's vector, which the inline comment explains but the name doesn't, so either _dirichlet_mle_init! or α = similar(μ) seems preferable to me. The extra allocation should be unmeasurable next to $5K$ invdigamma calls, each of which runs up to 25 digamma and trigamma evaluations itself.

sum(w) here duplicates the tw that suffstats has already accumulated, in the same spirit as the sum(log, P) de-duplication:

elogp = mean_logp(suffstats(Dirichlet, P, w))
α = isempty(init) ? _dirichlet_mle_init(P * vec(w) ./ sum(w), elogp) : init
fit_dirichlet!(elogp, α; maxiter=maxiter, tol=tol, debug=debug)
end

(K - 1) / (2 * s) reads better than (K - 1) / 2 / s. And the error message says the samples are all identical, but it also fires for samples that are merely close, e.g. [0.1 0.1+1e-9; 0.5 0.5; 0.4 0.4-1e-9].

For the record, on the well-posed problems the initialization is a clear improvement: in 600 random problems ($K$ between 2 and 8, $\alpha_k$ spanning $10^{-3}$ to $10^3$, $n$ between 50 and 200) I get 0 failures with this PR against 18 on master, the number of Newton iterations is median 5, 95% quantile 6, max 7 against 6, 8 and 12 on master, and where both converge the estimates agree to a relative difference of $4 \cdot 10^{-14}$.

@andreasnoack

Copy link
Copy Markdown
Member Author

Thanks for another careful round — all points should now be addressed: gnorm is accumulated with max so NaN gradients propagate, the guard is isfinite(s) && s > 0, the comment/error no longer claim reliable detection of identical samples, the test loops over n in 1:12 (plus a case with zeros), and the minor points (elogp naming, _dirichlet_mle_init!, hoisted digamma(α₀), reusing tw, (K - 1) / (2 * s)) are in.

One additional finding: the NaN fix turned out to be necessary but not sufficient. For repeat([0.1, 0.5, 0.4], 1, 10) the fit still returned [Inf, Inf, Inf] as converged without any NaN entering gnorm: the loop applied a Newton update before checking convergence, and from an already-converged iterate with α ~ 1e16 that extra step overflows because the denominator of b — mathematically approaching (K - 1)/2 — is wiped out by cancellation at that scale. The loop now checks gnorm < tol before updating, so the returned iterate is the one that satisfied the criterion. Healthy fits just skip one redundant final step; the hardcoded test values change only within the gradient tolerance.

This avoids dividing by zero. The new initialization is based on the
paper by Thomas Minka linked in the comment.
- Fix the weighted dirichlet_mle_init which still passed E[p^2] where the
  initialization now expects E[log(p)], and re-enable the weighted fit_mle test
  that would have caught it
- Document that the initialization returns a starting value for the Newton
  iterations rather than a moment estimate, and why five fixed point iterations
  are used. Rename _dirichlet_mle_init2 to _dirichlet_mle_init
- Compute suffstats before the initialization in fit_mle so the log-mean is
  only computed once
- Throw an informative ArgumentError when all samples are identical since the
  MLE doesn't exist in that case, and add tests for the degenerate cases
fit_mle now computes the starting value via _dirichlet_mle_init directly, so
the dirichlet_mle_init methods were only used in a test and the weighted
method wasn't covered at all. Adjust the test to use _dirichlet_mle_init.
- Check convergence in fit_dirichlet! before updating α so that the returned
  iterate is the one that satisfied the convergence criterion. Previously, an
  extra Newton update was applied after convergence which could produce
  non-finite values for degenerate samples since the denominator of b is
  dominated by cancellation errors when the elements of α are very large
- Accumulate the gradient norm with max in fit_dirichlet! so NaN gradients
  propagate to the convergence check and trigger the maxiter error instead of
  being silently dropped by the comparison
- Guard the initialization with isfinite(s) so samples with zero entries throw
  instead of returning NaN estimates, and soften the comment and error message
  since roundoff prevents reliable detection of (nearly) identical samples
- Rename _dirichlet_mle_init to _dirichlet_mle_init! since it overwrites its
  first argument, and rename the second argument to elogp since it is the mean
  of the logs rather than the log of the mean
- Hoist the loop invariant digamma(α₀) out of the inner fixed point loop and
  reuse the total weight from suffstats instead of recomputing sum(w)
- Test that fit_mle either throws or returns a finite positive estimate for
  identical samples with n = 1:12 and that samples with zeros throw

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Dirichlet initialization for fit diverges when one of the pk is constant across samples

2 participants