Repository navigation
Comparison matrix of models versus optimisers versus inference methods #229
Description
I'm writing a repo that simply takes all the toy models in pints, and all the methods (optimisers and inference), and then tests every method against every model. This will take a while, so its doing it all on arcus-b (there is a lot of machine-specific stuff in there, so it's not suitable to put into Pints itself)
When I'm comparing optimisers, I compare using the following criteria:
- final score
- time taken to reach the final score (this is the time taken using all the cores of a whole node on arcus-b)
I might also average these results over multiple runs of the optimiser, since some of them will be stochastic.
I'm less sure how to compare the inference methods, perhaps:
- Effective sample size (in a given time limit?)
- Rhat
- anything else?
What other criteria do you all think are necessary @MichaelClerx @ben18785 @sanmitraghosh @mirams @chonlei ? I'm hoping this will give a bunch of heat maps comparing the performance of all of our methods, and will go into the first paper
Activity
mirams commentedon Feb 12, 2018
A direct count of the number of forward solves involved in getting to an optimum/converged posterior is nice to have.
MichaelClerx commentedon Feb 12, 2018
Good point! Relates to #203
martinjrobins commentedon Feb 21, 2018
Here are a couple of heat maps I did using the optimisers, just one run each, and 1% noise:
score_with_noise_0.pdf
time_with_noise_0.pdf
MichaelClerx commentedon Feb 21, 2018
Just discussing if, for optimisers, we want to show the mean score of multiple runs, or the best score.
@DavidGavaghan ?
mirams commentedon Feb 21, 2018
I think you want to see the distribution of optimiser results really, will have a big impact on use if they always get the same answer versus have a wide distribution of results.
martinjrobins commentedon Feb 21, 2018
MichaelClerx commentedon Jun 18, 2024
@martinjrobins can this one be closed?