Repository navigation
append_state(batch=k): append a batch of states in one write per parameter - #96
Closed
ConnorStoneAstro wants to merge 1 commit into
Closed
ConnorStoneAstro wants to merge 1 commit into
ConnorStoneAstro wants to merge 1 commit into
Conversation
…meter After set_values with a batched array of k parameter vectors, every parameter value carries a leading axis of k states; append_state(path, batch=k) now appends them all at once (one resize and one write per parameter instead of one file open per state), repeating a parameter whose value has no state axis (static, or a pointer to static ones) for every state. A value that fits neither shape raises. The single-state path reads the value once (it was evaluated twice per parameter). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
Append failures can leave stale flags and partially extended datasets, causing retries to omit parameters and saved state counts to diverge, while single-state reshaping can silently accept incompatible shapes.
3 open issues
What changed in this PR
Adds batched state appends to HDF5 persistence, allowing multiple parameter states to be saved with one resize and write per parameter.
Changes:
- Adds and documents
append_state(batch=k), forwarding the batch size through the graph. - Handles batched values, repeats values without a state axis, and rejects incompatible batch shapes.
- Adds coverage for batched writes, static-value repetition, indexed loading, and shape errors.
| File | Description |
|---|---|
tests/test_save.py |
Tests batched appends, saved values, loading, and invalid shapes. |
src/caskade/param.py |
Implements batched dataset writes and reuses the evaluated parameter value. |
src/caskade/base.py |
Exposes and documents the batch argument and forwards it through traversal. |
Review effort: Balanced
Comment on lines
+697
to
+700
| raise ValueError( | ||
| f"Param {self.name}: value of shape {value.shape} is neither one state of shape " | ||
| f"{state_shape} nor a batch of {batch} states of that shape" | ||
| ) |
Comment on lines
+702
to
+703
| dataset.resize((n + rows.shape[0],) + state_shape) | ||
| dataset[n:] = rows |
| dataset = h5group["value"] | ||
| state_shape = dataset.shape[1:] | ||
| if batch is None: | ||
| rows = value.reshape((1,) + state_shape) |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #96 +/- ##
=========================================
Coverage 100.00% 100.00%
=========================================
Files 13 13
Lines 1712 1723 +11
=========================================
+ Hits 1712 1723 +11
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.


After set_values with a batched array of k parameter vectors, every parameter value carries a leading axis of k states; append_state(path, batch=k) now appends them all at once (one resize and one write per parameter instead of one file open per state), repeating a parameter whose value has no state axis (static, or a pointer to static ones) for every state. A value that fits neither shape raises. The single-state path reads the value once (it was evaluated twice per parameter).