Repository navigation
Add suite of diagnostic plots to support FIMS #475
Description
My goal is to get these all merged into main by the end of the calendar year. I have done an initial division of plots into Tier 1 / Tier 2. Tier 1 are those that will be relatively easy, and where there's obvious support for multiple models. Tier 2 are those that will be a little more time consuming and represent plots that might not be applicable to all models
Some of these diagnostics / etc have already been built by @iantaylor-NOAA and @MOshima-PIFSC in FIMSdiags -- so stockplotr might not be the best place for some of this stuff
| # | Stage | Diagnostic | What it shows | Based on | Builds on / needs |
|---|---|---|---|---|---|
| 1 | Merged (#465) | Add FIMS to convert_output() |
stockplotr's standard format: observed/predicted rows, units, -999 as NA, lognormal error as CV | — | main |
| 2 | Merged (#468) | plot_index(interval, log) |
Add lognormal intervals around the observed index with log option | r4ss::SSplotIndices(), JABBA::jbplot_cpuefits(), JABBA::jbplot_logfits(), wham:::plot.index.4.panel(), stockassessment::fitplot() |
1 |
| 3 | Merged (#469) | plot_index(relative = TRUE) |
Show index divided by its mean, all fleets on one panel | JABBA::jbplot_indices() |
2 |
| 4 | Open (#473) | plot_runs_test() |
Plot log residuals and a runs test | ss3diags::SSplotRunstest(), JABBA::jbrunstest(), JABBA::jbplot_runstest() |
1 |
| 5 | Open (#481) | plot_joint_residuals() |
Plot index residuals from all fleets on one panel, with a smoothed trend and the overall RMSE | ss3diags::SSplotJABBAres(), JABBA::jbplot_residuals(), wham:::plot.all.stdresids.fn() |
4 (reuses index_residuals()) |
| 6 | Open (#482) | plot_data_availability() |
Show data types each fleet has in each year | r4ss::SSplotData(), stockassessment::dataplot(), spict::plotspict.data() |
1 |
| 7 | Tier 1 | Landings observed vs. expected | Observed landings with intervals, plus expected landings; can be a new function like plot_landings() options |
r4ss::SSplotCatch(), r4ss::SSplotDiscard(), JABBA::jbplot_catcherror(), wham:::plot.catch.4.panel(), stockassessment::fitplot(), spict::plotspict.catch() |
1 (and the interval code from 2) |
| 8 | Tier 1 | Residual diagnostics panel | Per fleet: residuals vs. year, vs. predicted value, useful plots include a histogram, a QQ plot and an ACF plot | wham:::plot.osa.residuals(), spict::plotspict.diagnostic(), JABBA::jbplot_stdresiduals() |
4 |
| 9 | Tier 2 | Residual test summary | p-values by fleet: bias, variance of 1, lag-1 autocorrelation, equal variance above/below the median, normality | stockassessment::residplot(), spict::sumspict.diagnostics() |
One fit; log residuals from 4, OSA in FIMS |
| 10 | Tier 2 | Likelihood components | Negative log-likelihood by data type and fleet | wham:::plot.ll.table.fn() |
One fit; per-observation likelihoods |
| 11 | Tier 2 | Parameter diagnostics | Are estimates running into bounds?, show CVs, maybe maximum gradient, Hessian check; prior vs. posterior | wham::check_convergence(), wham::check_estimability(), stockassessment::corplot(), stockassessment::partable(), spict::plotspict.priors(), JABBA::jbplot_ppdist() |
Bounds, priors and gradients in convert_output() |
| 12 | Tier 2 | OSA residual versions of 8 and 9 | OSA residuals, needed for compositions | wham::make_osa_residuals(), stockassessment::residplot(), spict::calc.osa.resid() |
FIMS to compute them; convert_output() |
| 13 | Tier 2 | Process deviations | Deviations in random-effect processes (abundance, F, biomass) | stockassessment::procres(), spict::plotspict.diagnostic.process(), JABBA::jbplot_procdev(), wham:::plot.NAA.4.panel() |
FIMS random effects beyond recruitment deviations |
| 14 | Tier 2 | Retrospective plots with Mohn's rho | SSB, F and recruitment from each peel, with Mohn's rho | r4ss::retro(), ss3diags::SSplotRetro(), JABBA::jbplot_retro(), wham::mohns_rho(), wham:::plot.retro(), stockassessment::mohn(), spict::plotspict.retro() |
Retrospective peels |
| 15 | Tier 2 | Retrospective recruitment ("squid") | How each cohort's recruitment estimate changes as years are added | r4ss::SSplotRetroRecruits() |
Same peels as 14 |
| 16 | Tier 2 | Hindcast cross-validation (MASE) | How well each index predicts left-out years | ss3diags::SSplotHCxval(), ss3diags::SSmase(), JABBA::jbhcxval(), JABBA::jbmase(), spict::calc.mase(), spict::plotspict.hindcast() |
Hindcast runs |
| 17 | Tier 2 | Likelihood profile | Total and component likelihoods across fixed values of a key parameter, such as R0 | r4ss::profile(), r4ss::SSplotProfile(), spict::likprof.spict(), spict::plotspict.likprof() |
Profile runs |
| 18 | Tier 2 | Leave-one-fleet-out | SSB and F from refits without each fleet | stockassessment::leaveout(), stockassessment::runwithout() |
One refit per fleet |
| 19 | Tier 2 | Jitter | Likelihoods and estimates from refits with random starting values | r4ss::jitter(), wham::jitter_wham(), stockassessment::jit(), spict::check.ini() |
Jitter runs |
| 20 | Tier 2 | Observation influence | Change in key parameters when each observation is left out | spict::calc.influence(), spict::plotspict.infl() |
One refit per observation |
Activity
ericward-noaa commentedon Nov 8, 2026
Assigning myself @ericward-noaa
[-]Add suite of diagnostic plots[/-][+]Add suite of diagnostic plots to support FIMS[/+]JZahner1 commentedon Nov 9, 2026
@ericward-noaa Im happy to help with some of these. Retrospectives (both peels and squid plots) should be easy to implement in the near future, once multi-model processing in
convert_outputis merged in.ericward-noaa commentedon Nov 9, 2026
Sounds great @JZahner1 , happy to have you take on as many of these as you're interested in!
kellijohnson-NOAA commentedon Nov 9, 2026
I am not sure what would be needed for a "retrospective peel" plot, how would that figure be any different than just plotting spawning biomass (or any other metric of interest) across a suite of models?
JZahner1 commentedon Nov 9, 2026
@kellijohnson-NOAA thats actually exactly what it is. I think there's some ease-of-use data processing that could potentially be done to ensure that users get back the plot they're expecting (e.g., removing projection years from retrospective peels), but otherwise you are correct.
iantaylor-NOAA commentedon Nov 9, 2026
The discussion of the retrospective plots raises a question for me of how much ease of use is the right amount, and what, if anything, should be up to regional preferences. The slide below shows what I thought the ideal was in 2022 (the rest of the four days full of great talks on model diagnostics are available at https://www.capamresearch.org/content/diagnostics-workshop-presentations).
Thinking in terms of national standardization, I don't think it makes sense to have a region-specific function for diagnostics, and maybe not even an agency-specific function, since an easy-to-use function would be useful to all users of stockplotr. But where should a "run all the diagnostics I need" function live? In model-related packages, like FIMS, since it requires running models, in stockplotr, or some blend? It can also evolve and iterate over time.
ericward-noaa commentedon Nov 10, 2026
I think we'd want the wrapper function to exist in stockplotr. There'd have to be some general agreement about plots to flip vs not, but would be pretty easy. If the wrapper didn't meet the needs of specific regions, it seems like those regions could also write custom wrappers to blend some stockplotr plots with custom things
JZahner1 commentedon Nov 12, 2026
@iantaylor-NOAA I think there has to be a distinction between a "run_diagnostics" function, and a "plot_diagnostics" function. I think keeping the plotting functionality in
stockplotris perfectly fine, but I don't thinkstockplotrshould be doing any actual diagnostic calculations. I was thinking about this earlier in the context of adding mohn's who values to a retrospective peels plot. I'm not sure calcualting that statistic is the responsibility ofstockplotr. But maybe, for very easily calculated diagnostics like that, its fine for the plotting function to calculate it directly.