Repository navigation
Initialize Dirichlet fitting without sample variable - #2077
andreasnoack wants to merge 4 commits into
Conversation
e85d66b to
4b5191e
Compare
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #2077 +/- ##
==========================================
+ Coverage 87.35% 87.75% +0.40%
==========================================
Files 149 149
Lines 8936 8911 -25
==========================================
+ Hits 7806 7820 +14
+ Misses 1130 1091 -39 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
AFAICT the PR mixes two approaches presented by Minka, the EM-like fixed-point iteration and the Newton algorithm. It's not immediately clear to me why one would want to combine those two approaches and switch after a seemingly arbitrary amount of 5 fixed-point iterations to the Newton algorithm. IMO the crucial and currently in a few cases problematic part is just the initialization of the algorithms. As Minka mentions as well, there's many different possible approaches for initializing the algorithms, e.g., based on method of moments (obviously, different choices of moments will lead to different initializations, Minka already discusses two different approaches) or approximations of the likelihood function, e.g., using Stirling's approximation. Exploiting the moment equations which AFAICT would give moment estimates (seemingly a symmetrized version of eq. 21?). This would only break down if the sample variance of all So, taken together, my impression is: Using eq. 42 would give a less brittle initialization but one could also consider other approaches, e.g., based on different moments; but it's not clear to me why one would want to combine the EM-like fixed-point iteration with the Newton algorithm, using a different more stable initialization for the Newton algorithm alone (or the EM-like fixed-point iteration, if one prefers that approach) seems sufficient. |
|
The problem is that the parameters must be positive, so the initialization has to be sufficiently good to ensure that the Newton step is feasible. The fixed-point iterations will always produce positive values, but it has linear convergence. I also tried the initialization suggested in https://www.sciencedirect.com/science/article/abs/pii/S0167947307002848, but then I can trigger a maxiter error by making the repeated value small enough. |
|
To me it seems there's two separate problems you're facing and trying to address: the problematic initialization (the original issue reported in #602) and infeasible iterates of the Newton algorithm. Seemingly the first problem could be fixed by an initialization that doesn't break down if the sample variance of some |
|
The two issues are closely related. In theory, we could just initialize all the values at one, except for the risk of stepping to an infeasible argument. Hence, we need to find a sufficiently good initialization before switching to Newton's method and thereby ensure fast convergence through the concave objective function.
Yes, but as mentioned above, I could trigger maxiter issues with these initializers.
The global concavity of the objective function is not preserved under this reparametrization. We could try a constrained optimizer, but that would be a larger change than the current one. |
|
To avoid the concavity issue, we could use natural gradient descent in log space instead of of Newton's method. In the standard parameter space, Newton's method is exactly the natural gradient method with step size 1 (the Hessian is constant with respect to the data and hence identical to the FIM). AFAICT this would correspond to iterations |
|
I'm not sure I follow. If you change the parametrization, then the objective function is no longer globally concave. It might be possible to optimize it anyway with Fisher scoring aka natural gradient descent, but you'd lose the guarantees that the concavity provides. |
|
I didn't remember the details and all my thinking after the break, so used Claude to get up on speed again. Its conclusion was
😄 |
|
I tried to reproduce the With ε = 1e-10
X = [ε ε; 0.5(1 - ε) 0.3(1 - ε); 0.5(1 - ε) 0.7(1 - ε)]eq. 42 alone stops at So AFAICT what the fixed-point iterations buy is a smaller number of Newton iterations and not feasibility of the Newton steps, ie., they move the initial value into the region where the quadratic convergence of the Newton iteration kicks in, and that's what makes One thing that has to be fixed in any case: the second argument of X = [0.1 0.2 0.3; 0.5 0.3 0.4; 0.4 0.5 0.3]
w = ones(3)eq. 42 returns a negative Some minor comments. The title and description say that the PR avoids dividing by zero, but the degenerate cases are still there, Regarding the natural gradient suggestion, I don't want to block the PR on it, but I think we talked past each other. In what I suggested the step direction is still computed in the And just for the record regarding the symmetrized moment estimator I mentioned above: AFAICT the reason why it is more stable is that |
6d5a961 to
f15eb0a
Compare
|
Thanks for the thorough review! I've pushed a commit (and rebased on master) that should address your points:
I've left |
|
I've also been speculating whether it is better to be structurally or practically right. I'm not sure. |
|
Codecov flagged that the weighted |
|
AFAICT the new Distributions.jl/src/multivariate/dirichlet.jl Lines 236 to 243 in 2651285 With Distributions.jl/test/multivariate/dirichlet.jl Lines 180 to 183 in 2651285 This is not a regression, master returns The reason these come back as converged rather than as an error is that Distributions.jl/src/multivariate/dirichlet.jl Lines 298 to 303 in 2651285 so Relatedly, a zero entry in Some minor comments. Distributions.jl/src/multivariate/dirichlet.jl Lines 249 to 256 in 2651285
The second argument is named
Distributions.jl/src/multivariate/dirichlet.jl Lines 352 to 356 in 2651285
For the record, on the well-posed problems the initialization is a clear improvement: in 600 random problems ( |
|
Thanks for another careful round — all points should now be addressed: One additional finding: the NaN fix turned out to be necessary but not sufficient. For |
This avoids dividing by zero. The new initialization is based on the paper by Thomas Minka linked in the comment.
- Fix the weighted dirichlet_mle_init which still passed E[p^2] where the initialization now expects E[log(p)], and re-enable the weighted fit_mle test that would have caught it - Document that the initialization returns a starting value for the Newton iterations rather than a moment estimate, and why five fixed point iterations are used. Rename _dirichlet_mle_init2 to _dirichlet_mle_init - Compute suffstats before the initialization in fit_mle so the log-mean is only computed once - Throw an informative ArgumentError when all samples are identical since the MLE doesn't exist in that case, and add tests for the degenerate cases
fit_mle now computes the starting value via _dirichlet_mle_init directly, so the dirichlet_mle_init methods were only used in a test and the weighted method wasn't covered at all. Adjust the test to use _dirichlet_mle_init.
- Check convergence in fit_dirichlet! before updating α so that the returned iterate is the one that satisfied the convergence criterion. Previously, an extra Newton update was applied after convergence which could produce non-finite values for degenerate samples since the denominator of b is dominated by cancellation errors when the elements of α are very large - Accumulate the gradient norm with max in fit_dirichlet! so NaN gradients propagate to the convergence check and trigger the maxiter error instead of being silently dropped by the comparison - Guard the initialization with isfinite(s) so samples with zero entries throw instead of returning NaN estimates, and soften the comment and error message since roundoff prevents reliable detection of (nearly) identical samples - Rename _dirichlet_mle_init to _dirichlet_mle_init! since it overwrites its first argument, and rename the second argument to elogp since it is the mean of the logs rather than the log of the mean - Hoist the loop invariant digamma(α₀) out of the inner fixed point loop and reuse the total weight from suffstats instead of recomputing sum(w) - Test that fit_mle either throws or returns a finite positive estimate for identical samples with n = 1:12 and that samples with zeros throw
ed57f0d to
6a16ec4
Compare
This avoids dividing by zero. The new initialization is based on the paper by Thomas Minka linked in the comment. This is an alternative to #610 based on some of the ideas in the paper. It also avoids the extra allocation. Of course, at the cost of some special function evaluations.
Fixes #602