Ship precomputed response normalization constants - #24
Merged
Conversation
Computing a model's response normalization constant requires simulating ~30 minutes of naturalistic stimuli, which has to be done for every model of an ensemble before figures that normalize responses can be drawn. That is infeasible on a laptop and made the paper figures effectively unreproducible outside a cluster, even though the constants themselves are only a handful of floats per model. Store them instead: - Constants for the released ensemble ship with the package in flyvis/data/responses_norm.h5 (57 kB) and are loaded silently. - Constants computed for any other ensemble are written to <ensemble_dir>/responses_norm.h5, so a custom ensemble pays the cost once. - Ensemble.responses_norm resolves both before falling back to simulating. Constants are keyed by model name and validated against the SHA256 of the checkpoint their responses were computed from, so the order of an ensemble is irrelevant and a checkpoint retrained in place -- or an unrelated ensemble trained into a directory of the same name -- is recomputed rather than silently normalized with foreign constants. The values are bit-identical to what the previous inline implementation computed. Add `flyvis responses-norm` to compute and store the constants of an ensemble. Also add examples/figure_04_top_models.py, which reproduces figure 4a,b for the models with the lowest task error rather than the task-optimal cluster, plus the plotting support it needs: `angular_tuning` returns the unnormalized tuning, `plot_angular_tuning` gains `model_reduction` to reduce across models with e.g. the median and `normalize_by` to put several curves on a shared scale. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #24 +/- ##
==========================================
+ Coverage 38.07% 38.68% +0.61%
==========================================
Files 74 75 +1
Lines 9574 9738 +164
==========================================
+ Hits 3645 3767 +122
- Misses 5929 5971 +42
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Computing a model's response normalization constant requires simulating ~30 minutes of naturalistic stimuli, which has to be done for every model of an ensemble before figures that normalize responses can be drawn. That is infeasible on a laptop and made the paper figures effectively unreproducible outside a cluster, even though the constants themselves are only a handful of floats per model.
Store them instead:
Constants are keyed by model name and validated against the SHA256 of the checkpoint their responses were computed from, so the order of an ensemble is irrelevant and a checkpoint retrained in place -- or an unrelated ensemble trained into a directory of the same name -- is recomputed rather than silently normalized with foreign constants. The values are bit-identical to what the previous inline implementation computed.
Add
flyvis responses-normto compute and store the constants of an ensemble.Also add examples/figure_04_top_models.py, which reproduces figure 4a,b for the models with the lowest task error rather than the task-optimal cluster, plus the plotting support it needs:
angular_tuningreturns the unnormalized tuning,plot_angular_tuninggainsmodel_reductionto reduce across models with e.g. the median andnormalize_byto put several curves on a shared scale.