Repository navigation
A collection's group membership nodes are reused with its rows - #101
Merged
Merged
Conversation
…expressions Totals in pieces. A total was one node of a plan: on a GPU each scenario added its members in one thread, and every member's value kept a slot until the total read it (8 MB a scenario at a million rows). Now a large total is cut into whole subtrees of the same number of members (a power of two, as many as keep a piece under about 512 instructions, at most 256), each a node, and those again; the total combines the top ones. Each piece is a whole subtree of the graph's tree, so the order of combining, and the result, are the graph's bit for bit. The pieces of a level are one shape and short enough for a kernel each, so a GPU works them out side by side. Members worked out in their piece. A node read only by totals (a column read by its sum, min and max, say), and at most 24 instructions, gets no slot: each total works it out inside its piece. On a 1,000,000-row collection, a plan of the sum and the maximum now compiles in 0.8 s (0.9 s before), needs 254 KB a scenario (8 MB before) and runs a scenario in 4.6 ms on the CPU on four cores (6 ms before), with the same results. Counts. A count adds 1 for each member present, exact in an f64 up to 2^53, and comes out as the f64 of the same number. A whole number can be compared and the smaller or larger of two taken; arithmetic on whole numbers is still refused (the graph's does not round, and can overflow). An output may now be a whole number. Choices. choose(index, options) is a chain of conditionals on index == k. The index must be the same in every scenario and in range, or the plan refuses the node by name (the graph would hold an error); then the choice folds to the option it picks. Constant folding. A part of a node's program that reads nothing varied or shared is worked out when the plan is made, by running it, so it gives what running it would; a constant condition keeps only the branch taken. Nodes with nothing to fold are passed through without being copied. Deep expressions. A program that needs a stack deeper than 32 was refused (TooDeep). Now the operand that needs most is made a node of its own until every piece fits, so a lookup written as hundreds of nested conditionals, as the spreadsheet example writes VLOOKUP, runs. TooDeep is no longer returned. No new instructions: kernels and the plan file format are unchanged. Tests: random totals now include counts, and passed with every total cut into pieces of two members (checked by hand). New: a 100,000-row total in pieces with no slot a row, read back from a file; counts of groups whose rows change with the varied input, also in clusters; choices of each option, out of range, and by a number that varies; a 300-row lookup split into nodes; on the GPU, the large total (also in clusters), the counts and the lookup, replacing the test of a total too long for its own kernel; unit tests that every piece is short and few shapes, and of constant folding. Every example in watershed-gpu, at a small size, gives the same output as before. Docs: scenario-plans.md (compilation, totals, pieces), limitations.md, the README's limits (which still said totals were incremental only), the crate README and lib.rs, testing.md. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AdzBXcxXfE7iKtPMVoTTfr
A grouping puts a row in a group through a bool membership node, present && key == that group. Every (row, group) pair a row's nodes had ever been in kept its node, and its member in that group's total, for good. A collection whose rows come and go (a stream's window) reuses a removed row's nodes for the next row, but each reuse under another key made a new membership node, so the graph grew towards rows x groups: 100,000 edits through a 1,000-row window over 300 groups ended with 88,308 nodes and climbing. Now, when a row is removed (and the graph keeps no history), its membership nodes are spare. When its nodes are reused, the new row takes one: a spare for its own group is simply kept; otherwise one for another group is taken over. Its member in the old group's total is emptied (its value and presence become an always-false node, "(no member)", and its leaf the identity), the node is compiled to compare with the new key in place of the old (same reads, so no link changes and no relink), renamed for its new group, and joins the new group's total, filling an emptied member if there is one. So the same 100,000 edits end with 4,306 nodes, flat from the first few thousand, in 13 MB rather than 58 MB, and 1.6 s rather than 4.7 s, with the same totals. Within one row's life its membership nodes all stay: its key can come back, and a scenario plan that varies the key needs one for every group the row can be in. With history on nothing is taken, so an earlier update's totals are still explained by the members they had. Aggregates: a member can be emptied and refilled. A node that leaves stays linked (marked LINKED in by_node), so it needs no new link if it joins again and its changes are ignored meanwhile. Names: a stored-name node can be renamed. Graph::remove, without history, also lets go of a text or list value the node held. Tests: a 20,000-edit window over 50 groups stays the same size after the first few thousand edits, every group adds up its rows, and no group holds more members than it had rows at once; with history on, membership nodes are kept; a scenario plan of group totals with emptied members gives the graph's totals bit for bit. collections_match_a_plain_map_added_up_from_scratch (removals, reuse, groups by a field and by a calculated column, parallel) passes, and fails when the old member is not emptied or the node not recompiled. All workspace tests and the 48 browser tests pass. Docs: watershed-core's README (grouping, removal), the grid README, limitations.md, testing.md. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AdzBXcxXfE7iKtPMVoTTfr
matthewjones372
added a commit
that referenced
this pull request
Nov 16, 2026
…#102) * A collection's groupings by the same key share their membership nodes A grouping puts a row in a group through a membership node, present && key == that group, which does not depend on what the grouping adds up. Yet each grouping had its own: a sum and a count of each group made two a row, and the grid, which makes a grouping per totalled column, six for six totals. Now a collection's groupings by the same field or column share them (a Pool per key): the first grouping makes a row's membership node, the others use it, each adding it to its own group's total. When a removed row's node is taken for another group (#101), the pool compiles and renames it once, and each grouping, as it follows the row, empties its member in the old group's total and joins the new one. The bookkeeping is smaller too. Group keys are numbered once a pool, so a grouping holds a row's group and its counts as numbers, not strings, and a row's membership entries (nearly always one) are held in place, with a list only for rows that have more. Fifty thousand rows, six totals grouped by one column of 100 groups: core collection: 300,600 nodes, 226 MB, 1.44 s -> 50,600 nodes, 54 MB, 0.42 s; moving 1,000 rows between groups 47 ms -> 16 ms. grid: 256 MB, 1.21 s -> 74 MB, 0.40 s. The grid's budget for keeping group totals in the graph rises from 100,000 row-and-column pairs to 300,000, about the memory the old budget allowed. Tests: groupings by one key make a membership node a row between them, and a row that moves leaves all their old group's totals. collections_match_a_plain_map_added_up_from_scratch (a sum and a count by a field and by a calculated column) passes, and fails when a grouping does not empty its old group's member on a shared node taken for another group. All workspace tests and the 48 browser tests pass. Docs: watershed-core's README, the grid README, limitations.md, benchmarks.md (the grid's grouping figure is measured again), testing.md. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AdzBXcxXfE7iKtPMVoTTfr * The paging retry test waits for the failure to settle The React cookbook's "a page that fails to load is asked for again" timed out in CI waiting to click Try again: the button had been visible, then went before the click. Showing the failure changes the bar above the grid (a longer message, and the button), which can move the grid and give it one more scroll event near the end; the grid then asks for the page again, as it does on any scroll near the end. If that request came after the test had stopped failing the page, the page loaded and the button rightly went. The test meant to wait until the failure had settled ("with nothing asked for since") but only waited for the button. It now waits until half a second passes with no request for the page and the button showing, then stops failing it and clicks. It passed 40 runs of its own here, and the suite passes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AdzBXcxXfE7iKtPMVoTTfr --------- Co-authored-by: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes the graph-memory leak in groupings. A collection whose rows come and go, like a stream's window, used to grow without bound.
The leak
A grouping puts a row in a group through a
boolmembership node,present && key == "that group". Every (row slot, group) pair ever seen kept its node, and its member in that group's total, for good. A removed row's nodes are reused by the next row, but each reuse under a different key made a new membership node. So the graph grew towards rows × groups.The same totals, before and after (100,000 edits through a 1,000-row window over 300 groups):
The fix
When a row is removed, and the graph keeps no history, its membership nodes become spare. When its slot is reused, the new row takes one:
(no member), and its leaf becomes the identity, recombined at once.Things that deliberately don't change:
Supporting changes:
Aggregate: a member can be emptied (vacate) and refilled. A node that leaves stays linked (LINKEDinby_node), so rejoining needs no new link, and its changes are ignored meanwhile.Names: a stored-name node can be renamed.Graph::remove: without history, it also lets go of a text or list value the node held. A removed node's slot, some tens of bytes, still stays, because ids are never reissued.Tests
collections_match_a_plain_map_added_up_from_scratchcovers removals, reuse, groups by a field and by a calculated column, parallel mode, and a consistency check after every update. It passes. It fails, at step 6, when the old member isn't emptied, and when the node isn't recompiled. That was checked by hand and recorded indocs/testing.md.cargo test --workspace: 241 pass. Clippy, including--features cuda, andcargo docare clean.Docs
docs/limitations.md.docs/testing.md.Not done here
Each totalled column of a grouping still has its own membership node per row. The grid's
group_sumandgroup_countover the same key each make one. Sharing them would cut grouped-table memory by about the number of totals; that's a possible follow-up.Generated by Claude Code