Skip to content

A collection's group membership nodes are reused with its rows - #101

Merged
matthewjones372 merged 3 commits into
mainfrom
claude/stoic-pascal-v0q6pz
Nov 16, 2026
Merged

matthewjones372 merged 3 commits into
mainfrom
claude/stoic-pascal-v0q6pz

Conversation

@matthewjones372

Copy link
Copy Markdown
Collaborator

Fixes the graph-memory leak in groupings. A collection whose rows come and go, like a stream's window, used to grow without bound.

The leak

A grouping puts a row in a group through a bool membership node, present && key == "that group". Every (row slot, group) pair ever seen kept its node, and its member in that group's total, for good. A removed row's nodes are reused by the next row, but each reuse under a different key made a new membership node. So the graph grew towards rows × groups.

The same totals, before and after (100,000 edits through a 1,000-row window over 300 groups):

before after
nodes at the end 88,308, still growing 4,306, flat from the first few thousand edits
peak memory 58 MB 13 MB
time 4.7 s 1.6 s

The fix

When a row is removed, and the graph keeps no history, its membership nodes become spare. When its slot is reused, the new row takes one:

  • A spare for its own group is simply kept.
  • Otherwise, a spare for another group is taken over:
    • Its member in the old group's total is emptied: value and presence become an always-false node, (no member), and its leaf becomes the identity, recombined at once.
    • The node is recompiled in place to compare with the new key. It reads the same nodes, so no links change and there's no relink.
    • It's renamed for its new group.
    • It joins the new group's total, filling an emptied member if there is one.

Things that deliberately don't change:

  • Within one row's life, every membership node stays. A key can come back to a group, and a scenario plan that varies the key needs a node for every group the row can be in.
  • With history on, nothing is taken. An earlier update's group totals are still explained by the members they had.
  • A group's tree holds no more members than it had rows at once. Emptied places are refilled before it grows.

Supporting changes:

  • Aggregate: a member can be emptied (vacate) and refilled. A node that leaves stays linked (LINKED in by_node), so rejoining needs no new link, and its changes are ignored meanwhile.
  • Names: a stored-name node can be renamed.
  • Graph::remove: without history, it also lets go of a text or list value the node held. A removed node's slot, some tens of bytes, still stays, because ids are never reissued.

Tests

  • New: a 20,000-edit window over 50 groups:
    • its node count is the same from 3,000 edits on;
    • every group adds up its rows;
    • no group holds more than 500 members.
  • New: with history on, membership nodes are kept.
  • New (watershed-gpu): after 3,000 edits of churn, a scenario plan of group totals and counts gives the graph's results bit for bit. The test asserts the trees really contain emptied members.
  • Existing property test: collections_match_a_plain_map_added_up_from_scratch covers removals, reuse, groups by a field and by a calculated column, parallel mode, and a consistency check after every update. It passes. It fails, at step 6, when the old member isn't emptied, and when the node isn't recompiled. That was checked by hand and recorded in docs/testing.md.
  • Workspace: cargo test --workspace: 241 pass. Clippy, including --features cuda, and cargo doc are clean.
  • Browser: the 48 browser tests pass against the rebuilt WebAssembly, including the cookbook's SSE recipe, which streams edits grouped by wiki.

Docs

  • watershed-core README: grouping and removal.
  • The grid README.
  • docs/limitations.md.
  • docs/testing.md.

Not done here

Each totalled column of a grouping still has its own membership node per row. The grid's group_sum and group_count over the same key each make one. Sharing them would cut grouped-table memory by about the number of totals; that's a possible follow-up.


Generated by Claude Code

…expressions

Totals in pieces. A total was one node of a plan: on a GPU each scenario
added its members in one thread, and every member's value kept a slot until
the total read it (8 MB a scenario at a million rows). Now a large total is
cut into whole subtrees of the same number of members (a power of two, as
many as keep a piece under about 512 instructions, at most 256), each a
node, and those again; the total combines the top ones. Each piece is a
whole subtree of the graph's tree, so the order of combining, and the
result, are the graph's bit for bit. The pieces of a level are one shape and
short enough for a kernel each, so a GPU works them out side by side.

Members worked out in their piece. A node read only by totals (a column
read by its sum, min and max, say), and at most 24 instructions, gets no
slot: each total works it out inside its piece. On a 1,000,000-row
collection, a plan of the sum and the maximum now compiles in 0.8 s (0.9 s
before), needs 254 KB a scenario (8 MB before) and runs a scenario in 4.6 ms
on the CPU on four cores (6 ms before), with the same results.

Counts. A count adds 1 for each member present, exact in an f64 up to 2^53,
and comes out as the f64 of the same number. A whole number can be compared
and the smaller or larger of two taken; arithmetic on whole numbers is still
refused (the graph's does not round, and can overflow). An output may now be
a whole number.

Choices. choose(index, options) is a chain of conditionals on index == k.
The index must be the same in every scenario and in range, or the plan
refuses the node by name (the graph would hold an error); then the choice
folds to the option it picks.

Constant folding. A part of a node's program that reads nothing varied or
shared is worked out when the plan is made, by running it, so it gives what
running it would; a constant condition keeps only the branch taken. Nodes
with nothing to fold are passed through without being copied.

Deep expressions. A program that needs a stack deeper than 32 was refused
(TooDeep). Now the operand that needs most is made a node of its own until
every piece fits, so a lookup written as hundreds of nested conditionals, as
the spreadsheet example writes VLOOKUP, runs. TooDeep is no longer returned.

No new instructions: kernels and the plan file format are unchanged.

Tests: random totals now include counts, and passed with every total cut
into pieces of two members (checked by hand). New: a 100,000-row total in
pieces with no slot a row, read back from a file; counts of groups whose rows
change with the varied input, also in clusters; choices of each option, out
of range, and by a number that varies; a 300-row lookup split into nodes;
on the GPU, the large total (also in clusters), the counts and the lookup,
replacing the test of a total too long for its own kernel; unit tests that
every piece is short and few shapes, and of constant folding. Every example
in watershed-gpu, at a small size, gives the same output as before.

Docs: scenario-plans.md (compilation, totals, pieces), limitations.md, the
README's limits (which still said totals were incremental only), the crate
README and lib.rs, testing.md.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AdzBXcxXfE7iKtPMVoTTfr
A grouping puts a row in a group through a bool membership node, present &&
key == that group. Every (row, group) pair a row's nodes had ever been in
kept its node, and its member in that group's total, for good. A collection
whose rows come and go (a stream's window) reuses a removed row's nodes for
the next row, but each reuse under another key made a new membership node,
so the graph grew towards rows x groups: 100,000 edits through a 1,000-row
window over 300 groups ended with 88,308 nodes and climbing.

Now, when a row is removed (and the graph keeps no history), its membership
nodes are spare. When its nodes are reused, the new row takes one: a spare
for its own group is simply kept; otherwise one for another group is taken
over. Its member in the old group's total is emptied (its value and presence
become an always-false node, "(no member)", and its leaf the identity), the
node is compiled to compare with the new key in place of the old (same
reads, so no link changes and no relink), renamed for its new group, and
joins the new group's total, filling an emptied member if there is one. So
the same 100,000 edits end with 4,306 nodes, flat from the first few
thousand, in 13 MB rather than 58 MB, and 1.6 s rather than 4.7 s, with the
same totals.

Within one row's life its membership nodes all stay: its key can come back,
and a scenario plan that varies the key needs one for every group the row
can be in. With history on nothing is taken, so an earlier update's totals
are still explained by the members they had.

Aggregates: a member can be emptied and refilled. A node that leaves stays
linked (marked LINKED in by_node), so it needs no new link if it joins again
and its changes are ignored meanwhile. Names: a stored-name node can be
renamed. Graph::remove, without history, also lets go of a text or list
value the node held.

Tests: a 20,000-edit window over 50 groups stays the same size after the
first few thousand edits, every group adds up its rows, and no group holds
more members than it had rows at once; with history on, membership nodes are
kept; a scenario plan of group totals with emptied members gives the
graph's totals bit for bit. collections_match_a_plain_map_added_up_from_scratch
(removals, reuse, groups by a field and by a calculated column, parallel)
passes, and fails when the old member is not emptied or the node not
recompiled. All workspace tests and the 48 browser tests pass.

Docs: watershed-core's README (grouping, removal), the grid README,
limitations.md, testing.md.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AdzBXcxXfE7iKtPMVoTTfr
@matthewjones372
matthewjones372 merged commit 8ca2220 into main Nov 16, 2026
9 checks passed
@matthewjones372
matthewjones372 deleted the claude/stoic-pascal-v0q6pz branch November 16, 2026 19:04
matthewjones372 added a commit that referenced this pull request Nov 16, 2026
…#102)

* A collection's groupings by the same key share their membership nodes

A grouping puts a row in a group through a membership node, present && key
== that group, which does not depend on what the grouping adds up. Yet each
grouping had its own: a sum and a count of each group made two a row, and
the grid, which makes a grouping per totalled column, six for six totals.

Now a collection's groupings by the same field or column share them (a
Pool per key): the first grouping makes a row's membership node, the others
use it, each adding it to its own group's total. When a removed row's node
is taken for another group (#101), the pool compiles and renames it once,
and each grouping, as it follows the row, empties its member in the old
group's total and joins the new one.

The bookkeeping is smaller too. Group keys are numbered once a pool, so a
grouping holds a row's group and its counts as numbers, not strings, and a
row's membership entries (nearly always one) are held in place, with a list
only for rows that have more.

Fifty thousand rows, six totals grouped by one column of 100 groups:

  core collection: 300,600 nodes, 226 MB, 1.44 s -> 50,600 nodes, 54 MB,
  0.42 s; moving 1,000 rows between groups 47 ms -> 16 ms.
  grid: 256 MB, 1.21 s -> 74 MB, 0.40 s.

The grid's budget for keeping group totals in the graph rises from 100,000
row-and-column pairs to 300,000, about the memory the old budget allowed.

Tests: groupings by one key make a membership node a row between them, and
a row that moves leaves all their old group's totals.
collections_match_a_plain_map_added_up_from_scratch (a sum and a count by a
field and by a calculated column) passes, and fails when a grouping does not
empty its old group's member on a shared node taken for another group. All
workspace tests and the 48 browser tests pass.

Docs: watershed-core's README, the grid README, limitations.md,
benchmarks.md (the grid's grouping figure is measured again), testing.md.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AdzBXcxXfE7iKtPMVoTTfr

* The paging retry test waits for the failure to settle

The React cookbook's "a page that fails to load is asked for again" timed
out in CI waiting to click Try again: the button had been visible, then
went before the click. Showing the failure changes the bar above the grid
(a longer message, and the button), which can move the grid and give it one
more scroll event near the end; the grid then asks for the page again, as
it does on any scroll near the end. If that request came after the test had
stopped failing the page, the page loaded and the button rightly went.

The test meant to wait until the failure had settled ("with nothing asked
for since") but only waited for the button. It now waits until half a
second passes with no request for the page and the button showing, then
stops failing it and clicks. It passed 40 runs of its own here, and the
suite passes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AdzBXcxXfE7iKtPMVoTTfr

---------

Co-authored-by: Claude <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants