Skip to content

append_state(batch=k): append a batch of states in one write per parameter - #96

Closed
ConnorStoneAstro wants to merge 1 commit into
mainfrom
batched-append-state
Closed

ConnorStoneAstro wants to merge 1 commit into
mainfrom
batched-append-state

Conversation

@ConnorStoneAstro

Copy link
Copy Markdown
Owner

After set_values with a batched array of k parameter vectors, every parameter value carries a leading axis of k states; append_state(path, batch=k) now appends them all at once (one resize and one write per parameter instead of one file open per state), repeating a parameter whose value has no state axis (static, or a pointer to static ones) for every state. A value that fits neither shape raises. The single-state path reads the value once (it was evaluated twice per parameter).

…meter

After set_values with a batched array of k parameter vectors, every
parameter value carries a leading axis of k states; append_state(path,
batch=k) now appends them all at once (one resize and one write per
parameter instead of one file open per state), repeating a parameter
whose value has no state axis (static, or a pointer to static ones) for
every state. A value that fits neither shape raises. The single-state
path reads the value once (it was evaluated twice per parameter).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Copilot AI balanced review requested due to automatic review settings November 8, 2026 20:23

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Append failures can leave stale flags and partially extended datasets, causing retries to omit parameters and saved state counts to diverge, while single-state reshaping can silently accept incompatible shapes.

3 open issues
What changed in this PR

Adds batched state appends to HDF5 persistence, allowing multiple parameter states to be saved with one resize and write per parameter.

Changes:

  • Adds and documents append_state(batch=k), forwarding the batch size through the graph.
  • Handles batched values, repeats values without a state axis, and rejects incompatible batch shapes.
  • Adds coverage for batched writes, static-value repetition, indexed loading, and shape errors.
File Description
tests/​test_save.py Tests batched appends, saved values, loading, and invalid shapes.
src/​caskade/​param.py Implements batched dataset writes and reuses the evaluated parameter value.
src/​caskade/​base.py Exposes and documents the batch argument and forwards it through traversal.

Review effort: Balanced

Comment thread src/caskade/param.py
Comment on lines +697 to +700
raise ValueError(
f"Param {self.name}: value of shape {value.shape} is neither one state of shape "
f"{state_shape} nor a batch of {batch} states of that shape"
)
Comment thread src/caskade/param.py
Comment on lines +702 to +703
dataset.resize((n + rows.shape[0],) + state_shape)
dataset[n:] = rows
Comment thread src/caskade/param.py
dataset = h5group["value"]
state_shape = dataset.shape[1:]
if batch is None:
rows = value.reshape((1,) + state_shape)
@codecov

codecov Bot commented Nov 8, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 100.00%. Comparing base (a88413f) to head (65a6c89).

Additional details and impacted files
@@            Coverage Diff            @@
##              main       #96   +/-   ##
=========================================
  Coverage   100.00%   100.00%           
=========================================
  Files           13        13           
  Lines         1712      1723   +11     
=========================================
+ Hits          1712      1723   +11     
Flag Coverage Δ
unittests 100.00% <100.00%> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@ConnorStoneAstro
ConnorStoneAstro deleted the batched-append-state branch November 8, 2026 20:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants