Skip to content

Reduce memory footprint of analysisLoop and extractStats - #29

Merged
eacharles merged 1 commit into
mainfrom
eac/reduce-memory-footprint
Nov 16, 2026
Merged

eacharles merged 1 commit into
mainfrom
eac/reduce-memory-footprint

Conversation

@eacharles

Copy link
Copy Markdown
Contributor
  • analysisLoop: add write_dir option to serialize per-cell output tables as parquet fragments and evict CellData immediately after each cell, avoiding accumulation of all CellData in memory across the full run
  • analysisLoop: add evict_cells option to drop CellData from cell_dict after each cell without writing to disk
  • extractStats: add read_dir option to reassemble results from parquet fragments written by analysisLoop(write_dir=...)
  • extractStats: add batch_size option (default 500) to concat in rolling batches rather than a single end-of-loop concat, bounding live fragment count at any point

Generated with AI

Co-Authored-By: SLAC AI

- analysisLoop: add write_dir option to serialize per-cell output tables
  as parquet fragments and evict CellData immediately after each cell,
  avoiding accumulation of all CellData in memory across the full run
- analysisLoop: add evict_cells option to drop CellData from cell_dict
  after each cell without writing to disk
- extractStats: add read_dir option to reassemble results from parquet
  fragments written by analysisLoop(write_dir=...)
- extractStats: add batch_size option (default 500) to concat in rolling
  batches rather than a single end-of-loop concat, bounding live fragment
  count at any point

Generated with AI

Co-Authored-By: SLAC AI
@eacharles
eacharles merged commit 9947b99 into main Nov 16, 2026
4 checks passed
@eacharles
eacharles deleted the eac/reduce-memory-footprint branch November 16, 2026 18:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant