Repository navigation
Jacobian term shouldn't be included when transforming loglikelihoods #1705
Description
When working in a transformed parameter space to perform maximum likelihood estimation, we don't need to include the Jacobian term provided we are not also transforming the data.
This follows from the "invariance property of the ML estimator": https://en.wikipedia.org/wiki/Maximum_likelihood_estimation
Metadata
Metadata
Assignees
Labels
No labels
Activity
MichaelClerx commentedon Feb 20, 2025
I think this means we need to update the
Transformationclass to have atransform_logpdfand atransform_loglikelihood.To do that, we'll need to start distinguishing between unnormalised log pdfs (which take a random variable as input) and loglikelihoods (which take a non-random variable as input, plus data sampled from some distribution (but fixed)). Might have some far-reaching consequences software wise...
See #457
The use case for this is performing MLP using the Transform class and an optimiser: at the moment that uses the transform_logpdf method even if you pass it a likelihood
MichaelClerx commentedon Feb 20, 2025
Thoughts @ben18785 , @martinjrobins , @chonlei ?
martinjrobins commentedon Feb 20, 2025
is this just for MLP? How about if the likelihood is part of a log posterior, do you transform its output then?
MichaelClerx commentedon Feb 20, 2025
No but you'd have[changed prior] * [unchanged likelihood] = [changed pdf]. So the likelihood itself would be treated the same as in MLE, but the posterior would still change due to the prior changing.MichaelClerx commentedon Feb 20, 2025
See https://theoryandpractice.org/stats-ds-book/distributions/invariance-of-likelihood-to-reparameterizaton.html
MichaelClerx commentedon Feb 20, 2025
And https://en.wikipedia.org/wiki/Maximum_a_posteriori_estimation#Limitations
martinjrobins commentedon Feb 21, 2025
so the likelihood is multiplied by a factor (lets say A where A = dp(sigma)/dsigma is the jacobian of the pdf) due to the change of variables, but MLE is unaffected by this since A is constant wrt the parameters sigma. The prior has an A that might be dependent on the parameters, so this needs to be transformed. More broadly, any logpdf that has a jacobian that is constant wrt the parameters sigma does not need to be transformed. So perhaps a pints logpdf can have a function that returns a bool indicating if its jacobian is constant wrt the parameters? Then the Transformation class can just check this?
mirams commentedon Jun 17, 2025
I wrote a little explainer for the team here:
ParameterTransformIssue.pptx
DavidGavaghan commentedon Jun 17, 2025
Thanks Gary - I think your first solution is probably the best one - the second one requires more understanding and people might still get it wrong.
MichaelClerx commentedon Jan 26, 2026
So:
transform_logpdfand atransform_loglikelihoodOptimisationControllerSound OK, @mirams @frankiepe ?
MichaelClerx commentedon Feb 5, 2026
@mirams am I right in thinking that we don't have to do anything special for the LogPosterior?
I.e. do we want to return
calculated log_pdf + jacobian term, which is the same as(calculated log_prior + jacobian term) + calculated log_likelihood?mirams commentedon Feb 5, 2026
If you mean
return JacTransform(log_prior) + log_likelihoodthen yes, that should be fine I think. But make sure there's a couple of tests with an analytic answer, like the uniform-log example in the powerpoint (should be simple enough with a flat likelihood and uniform prior that gets transformed)MichaelClerx commentedon Feb 5, 2026
Slight change of plan:
TransformedLogLikelihoodtransform_logpdfreturn the appropriate type, based on whether it's a prior, likelihood, or otherMichaelClerx commentedon Mar 6, 2026
Fixed in #1730