[Adam-Vandervorst]

I purpose we use this paradigm under the cata interface.

[Adam-Vandervorst]

@luketpeterson any problem merging this?

[luketpeterson]

@luketpeterson any problem merging this?

In concept no. There is a (small) bit of work to make the API is consistent with the cata that's there already.

[adamv-symbolica]

Giving Fable some time with it:

Blocking

B1 — goat_val_count double-counts the root value

trie_map.rs:509–521. PathMap::recursive_cata already ends with
collapse_f(self.root_val(), Some(w), &[]), so the closure counts the root value; the caller
then adds root_val again.

let mut map = PathMap::new();
map.insert(b"", ());
map.insert(b"a", ());
assert_eq!(map.val_count(), 2);      // ok
assert_eq!(map.goat_val_count(), 2); // FAILS: returns 3

Fix: drop the + root_val (and the now-pointless match self.root()), or make the collapse
closure ignore the final root-val invocation.

B2 — LineListNode "Case 10" (Val, Child with different first bytes) violates the branch contract

line_list_node.rs:2952–2977. The PR's own tests establish the contract:
at a branch, branch_f is called once per branch in ascending mask-bit order with the branch
mask, so the algebra can attribute each W to its byte (mask.indexed_bit(acc.idx)). Case 10
breaks it twice:

• the first branch_f call passes &ByteMask::new() (empty) instead of &mask
  (line 2960 — contrast with Cases 4/5/8 which pass &mask on every call);
• it processes the child slot first, but slots are stored sorted by first byte, so for
  (Val@a, Child@b) the calls arrive in descending byte order.

// keys {"a", "b1", "b2"} -> root pair node (Val@'a', Child@'b')
// Reconstructing paths via the documented mask contract yields:
//   [[98], [238,49], [238,50]]   (0xEE = sentinel for "mask had no bit at idx")
// instead of [[97], [98,49], [98,50]]

The value gets attributed to byte b, and the child subtrie to an empty mask. Any
mask-sensitive algebra (including the PR's own recursive_cata_jumping_total_len bench closure,
which would panic on .unwrap()) is wrong or crashes on tries that contain this node shape.
Fix: pass &mask on both calls and emit the value (lower byte) before the child.

B3 — Zipper recursive_cata silently returns "empty" for a mid-node focus

morphisms.rs:513–521. The blanket impl uses get_focus().0.borrow(), which
returns Some only for the BorrowedRc/OwnedRc variants of AbstractNodeRef. A zipper
focused part-way into a node (BorrowedTiny/BorrowedDyn — exactly what TinyRefNode exists
for) gets None and is treated as an empty trie:

let mut map = PathMap::new();
map.insert(b"abc1", ());
map.insert(b"abc2", ());
let mut rz = map.read_zipper();
rz.descend_to(b"ab");                       // path exists
rz.recursive_cata::<..>(count_vals ...)     // returns 0, expected 2

Fix: handle the remaining variants — e.g. go through as_tagged()/try_as_tagged() and add a
TaggedNodeRef-level entry point, or fall back to into_option() (accepting the clone), or
panic loudly rather than returning a wrong answer.

B4 — Branch-byte convention differs between node types, so results depend on physical layout

DenseByteNode includes the branch byte in the collapse_f prefix
(dense_byte_node.rs:415, core::slice::from_ref(&key_byte)) and
represents it in the mask given to branch_f. LineListNode strips the byte from the prefix
(&key0[1..]) and represents it only in the mask. An algebra therefore cannot know whether
prefix[0] is the branch byte or a distinct following byte:

// Path-reconstruction algebra (byte taken from mask, prefix appended):
//   correct on LineListNode pair shapes,
//   duplicates the first byte on DenseByteNode: [1,7,13] -> [1,1,7,13]

The same logical trie yields different W depending on which physical nodes back it. The PR's
own sum-digits tests only pass because the algebra threads a bool flag ("value was at empty
prefix") through W to compensate — a workaround that no external user will discover from the
docs. This is the "API consistency with the existing cata" work the reviewer already flagged on
the PR. Fix: pick one convention (LLN's byte-in-mask-only matches the existing
into_cata_jumping_* sub_path semantics best) and align ByteNode::node_recursive_cata;
then document it on Summarization.

B5 — Unbounded recursion: stack overflow on deep tries; PR's own test aborts the all_dense_nodes suite

recursive_cata_cached recurses once per physical node. The PR's
recursive_cata_stack_overflow_smoke (morphisms.rs:2232) documents overflow
between 8–10 KB of path depth on default features — and under --features all_dense_nodes
(1 byte per node) the very same test overflows and SIGABRTs the whole test binary at
PATH_LEN = 8_000. As a public API this is a panic-free-abort footgun on adversarial/deep data,
and as merged it leaves a red test config. Minimum: gate or shrink the smoke test per feature
and document the depth limit prominently; proper fix: explicit work-stack or segmented stacks
(e.g. stacker::maybe_grow) in recursive_cata_cached.

Non-blocking cleanups

• Dead code: node_goat_val_count (trait method + 6 impls) lost its only consumer when
  traverse_physical was removed; either delete the chain or keep goat_val_count on it for
  the comparison's sake — not both.
• Commented-out blocks left in line_list_node.rs (old generic implementation, old
  node_goat_val_count) and dense_byte_node.rs.
• unreachable_unchecked on header patterns (line_list_node.rs:2980):
  header values 1–7 (slot1 used, slot0 free) are assumed impossible; if that invariant is ever
  violated this is UB rather than a panic. A debug_assert!/unreachable! in debug builds
  would be cheap insurance.
• Option<Acc> dance in ByteNode::node_recursive_cata (Some(Acc::default()) +
  unwrap_unchecked) — ws is always Some; a plain Acc binding works.
• pub(crate) values on ByteNode is only applied to the non-nightly field variant and no
  code outside the module reads it — revert the visibility change.
• Docs: Summarization docs still carry GOAT/dev-branch placeholders;
  recursive_cata_stepping links to [Catamorphism::recursive_cata] (wrong trait); the
  COMPUTE_PATH=false caveat ("no reliable child_masks") deserves a loud, user-facing warning
  since it silently changes what the closures receive.
• Bench honesty: recursive_cata_jumping_total_len asserts only the count, not the length —
  the two implementations being compared do not agree on total length today (a consequence of B4).
• slim_dispatch's TaggedNodeRef::node_val_count now takes the gxhash map while the node impls
  take std — moot while the feature is bitrotted on master, but worth aligning if it's revived;
  same for the missing BridgeNode arm in recursive_cata_dispatch under bridge_nodes.

[adamv-symbolica]

Path byte should not be represented in prefix.

[luketpeterson]

Wow that took a lot more work than expected to get the API into shape. But it should be ready to merge now.

[adamv-symbolica]

Fable 5.1

1. Recursive engine misplaces the value in LineListNode Case 9 with a ≥3-byte child key

Blocker. src/line_list_node.rs ~line 2915 (summarize!(Some(val), Some(child_w), &key1[1..])).

map = {a, abcd, abce}          // plain inserts
recursive rebuild -> {ab, abcd, abce}   // value "a" lands at "ab"

summarize_run positions a value before the last byte of the prefix it is given, so a value at the
start of a run followed by a ≥2-byte valueless tail is pushed one level too deep. Case 7 had the same
construction error and was fixed (lines 2876–2884); Case 9 was not. Count-only algebras are unaffected,
but hash() disagrees between the engines on this map and any path-sensitive algebra is wrong. The
iterative engine is correct. The randomized tests miss the shape because it needs exactly one value
above a ≥3-byte valueless run.

Fix: mirror the Case 7 fix — summarize the child under key1[2..], fold it under mask{key1[1]},
finalize with the value, then wrap with key0. Add {a, abcd, abce} to the equivalence suite.

2. Iterative engine aborts (debug) / commits UB (release) on dangling branches

Blocker for relying on the iterative engine. src/zipper.rs ~line 2497, ZipperConcrete::is_shared.

map = {aa, ab, ba, bb, b}
map.remove_val_at(b"ba", false);   // prune = false
map.remove_val_at(b"bb", false);
CatamorphismCached::val_count(&map)          // 3 — recursive engine guards the empty node
CatamorphismCachedIterative::val_count(&map) // SIGABRT: non-unwinding UB-check panic in refcount()

A no-prune removal leaves a valueless branch that is logical structure (path_exists() is true, it is
in child_mask). The iterative cache-key lookup calls is_shared(), which does
node_get_child(..).unwrap() and then refcount() on the child with no empty-sentinel check, reading
through the sentinel pointer. Inherited from master (into_cata_cached aborts identically at
d967f0c), but the PR routes CatamorphismCachedIterative, ACT val_count, and CatamorphismDebug
through it and presents it as the stack-safe engine.

Fix: guard with !node.is_empty() && node.refcount() > 1, as recursive_cata_cached already does.
Add the dangling-branch map to the equivalence suite.

3. PrefixZipper under the blanket recursive impl drops the prefix

src/morphisms.rs ~line 647 (blanket CatamorphismCached for Z: ZipperInfallibleSubtries),
src/prefix_zipper.rs ~line 702.

pz = PrefixZipper::new(b"xy", {ab, ac}.read_zipper())
pz.make_map()        -> {xyab, xyac}
recursive rebuild    -> {ab, ac}        // iterative rebuild is correct

PrefixZipper::get_focus() delegates to the source while its make_map() includes the prefix, so the
blanket impl (which rebuilds from get_focus()) disagrees with the zipper's own subtrie semantics.
Any wrapper zipper whose logical position differs from its focus node has the same problem.

Fix (decision needed): exclude wrappers from the blanket impl (marker trait or explicit impls), or
give PrefixZipper a CatamorphismCached impl that delegates to the iterative body.

[adamv-symbolica]

One more

Iterative engine hangs on a LineListNode shape left behind by join_k_path_into (found by a randomized edit-program run against 3839f31)

join_k_path_into shortens two value keys in place: (Val@"aaa", Val@"bab") → (Val@"aa", Val@"ab"). factor_prefix accepts that (rule A: "slot 0 contains a value"), but insertion never builds it — a fresh {aa, ab} is Child@"a" -> {a, b} — and nothing that indexes children can walk it: descend_indexed_byte fails (master's side-effecting cata trips its debug_assert at morphisms.rs:~844), the recursive engine trips Case 7's debug_assert_eq!(key0.len(), 1), and CatamorphismCachedIterative loops forever (summarize/fold on byte a, never advancing to b). Plain iter() works, which is why it went unnoticed. The shape is pre-existing master behavior; the hang is new with the iterative engine.

Standalone test (drop into tests/); the assertions are ordered so it fails before it can hang:

use pathmap::PathMap;
use pathmap::zipper::*;
use pathmap::morphisms::{CatamorphismCached, CatamorphismCachedIterative};

#[test]
fn drop_head_shared_first_byte_shape() {
    let mut m = PathMap::<()>::new();
    m.set_val_at(b"aaa", ());
    m.set_val_at(b"bab", ());
    assert!(m.write_zipper().join_k_path_into(1, false));
    assert_eq!(m.iter().map(|(k, _)| k).collect::<Vec<_>>(), vec![b"aa".to_vec(), b"ab".to_vec()]);

    let mut rz = m.read_zipper();
    rz.descend_to_byte(b'a');
    assert_eq!(rz.child_count(), 2);
    assert!(rz.descend_indexed_byte(1).is_some(), "indexed descent cannot reach the second child"); // fails here today

    assert_eq!(CatamorphismCached::val_count(&m), 2);           // debug build: Case 7 assert
    assert_eq!(CatamorphismCachedIterative::val_count(&m), 2);  // never returns
}

Fix that makes this pass (and keeps both full suites green): in LineListNode::factor_prefix, rule A should also require a 1-byte slot-0 key —

let legal_overlap = overlap == 1 && (
    (!self.is_child_ptr::<0>() && key0.len() == 1) ||
    (!self.is_child_ptr::<1>() && key0.len()==1 && key1.len()==1 ));

— so the shape gets factored into the canonical Child@"a" -> {a, b}. Alternatively the engines could learn to traverse it, but since indexed descent itself can't, canonicalizing at the producer seems right. Worth adding to the recursive-vs-iterative equivalence suite either way.

[luketpeterson]

ACT should get a fast recursive implementation of cata, modeled on the ACT val_count in master, and val_count benchmarks that can easily be back-ported to master.

[github-actions]

Bench A/B vs base: done

job log · 2026-11-02 19:07:13 UTC

base 9598a1a → head 9135c2f, 3 round(s), median of each run averaged; negative is faster

bench	cases	geomean	largest gain	largest loss	>5% faster	>5% slower
shakespeare	7	-8.2%	-58.8% shakespeare/shakespeare_sentences_val_count_act	+36.8% shakespeare/shakespeare_words_val_count	3	2
cities	5	-2.1%	-27.1% cities/cities_val_count_act	+13.4% cities/cities_get_act	1	2
sparse_keys	102	-0.7%	-11.5% binary_drop_head/100	+4.3% sparse_subtract_after_join/16000	3	0
binary_keys	83	+0.0%	-9.5% binary_descend_until/2000	+19.1% binary_meet/800	6	6
superdense_keys	102	+0.7%	-12.6% superdense_k_path_iter/500	+13.5% superdense_descend_until/32000	9	9
act_paths	46	-0.9%	-8.7% shape_map_to_act_cata/random_long	+3.4% size_dense_paths_to_act/400000	1	0
zipper_head_owned	16	+1.1%	-4.9% zipper_head_owned/shared_parent_reader_val	+4.4% zipper_head_owned/borrowed_head_read_value_in_shared_parent	0	0
product_zipper	4	-2.0%	-2.9% product_zipper/generic_act_act	-0.5% product_zipper/generic_pathmap_pathmap	0	0

42 case(s) moved more than 5%

bench	case	base	head	change
shakespeare	shakespeare/shakespeare_sentences_val_count_act	38.10 ms	15.70 ms	-58.8%
shakespeare	shakespeare/shakespeare_words_val_count	324.37 µs	443.83 µs	+36.8%
cities	cities/cities_val_count_act	53.83 ms	39.27 ms	-27.1%
shakespeare	shakespeare/shakespeare_sentences_val_count	2.77 ms	3.38 ms	+22.0%
binary_keys	binary_meet/800	78.62 µs	93.63 µs	+19.1%
binary_keys	binary_meet/200	20.40 µs	24.18 µs	+18.5%
binary_keys	binary_meet/1600	154.57 µs	182.93 µs	+18.4%
binary_keys	binary_meet/400	38.71 µs	45.35 µs	+17.2%
binary_keys	binary_meet/50	5.21 µs	6.09 µs	+16.9%
binary_keys	binary_meet/100	10.35 µs	12.02 µs	+16.1%
shakespeare	shakespeare/shakespeare_sentences_get	53.86 ms	46.41 ms	-13.8%
superdense_keys	superdense_descend_until/32000	230.70 µs	261.83 µs	+13.5%
cities	cities/cities_get_act	658.13 ms	746.43 ms	+13.4%
superdense_keys	superdense_descend_until/4000	27.47 µs	31.11 µs	+13.3%
superdense_keys	superdense_descend_until/16000	113.87 µs	128.93 µs	+13.2%
superdense_keys	superdense_k_path_iter/500	3.62 µs	3.16 µs	-12.6%
superdense_keys	superdense_descend_until/8000	56.51 µs	63.50 µs	+12.4%
sparse_keys	binary_drop_head/100	8.18 µs	7.24 µs	-11.5%
superdense_keys	superdense_descend_until/2000	13.45 µs	14.94 µs	+11.1%
superdense_keys	superdense_k_path_iter/1000	7.05 µs	6.29 µs	-10.8%
superdense_keys	superdense_k_path_iter/2000	14.07 µs	12.56 µs	-10.7%
superdense_keys	superdense_k_path_iter/8000	56.70 µs	50.81 µs	-10.4%
superdense_keys	superdense_k_path_iter/4000	28.18 µs	25.31 µs	-10.2%
superdense_keys	superdense_meet_low_overlap/32000	23.66 µs	25.98 µs	+9.8%
binary_keys	binary_descend_until/2000	193.97 µs	175.63 µs	-9.5%
superdense_keys	superdense_descend_until/1000	6.33 µs	6.93 µs	+9.4%
superdense_keys	superdense_k_path_iter/16000	113.50 µs	103.07 µs	-9.2%
act_paths	shape_map_to_act_cata/random_long	43.20 ms	39.45 ms	-8.7%
superdense_keys	superdense_insert/1600	41.99 µs	38.59 µs	-8.1%
binary_keys	binary_insert/1600	183.40 µs	169.93 µs	-7.3%
cities	cities/cities_get	77.04 ms	82.29 ms	+6.8%
binary_keys	binary_zipper_prev_iter/200	177.30 µs	165.23 µs	-6.8%
binary_keys	binary_zipper_prev_iter/100	90.80 µs	84.70 µs	-6.7%
binary_keys	binary_zipper_prev_iter/50	46.71 µs	43.62 µs	-6.6%
sparse_keys	sparse_zipper_prev_step_iter/100	22.83 µs	21.33 µs	-6.6%
superdense_keys	superdense_drop_bench/100	53 ns	50 ns	-6.3%
binary_keys	binary_zipper_prev_iter/400	342.93 µs	322.50 µs	-6.0%
superdense_keys	superdense_zipper_prev_step_iter/1600	11.39 µs	12.06 µs	+5.9%
shakespeare	shakespeare/shakespeare_sentences_insert	55.51 ms	52.31 ms	-5.8%
superdense_keys	superdense_zipper_step_iter/400	4.23 µs	3.99 µs	-5.7%
superdense_keys	superdense_zipper_prev_step_iter/400	3.90 µs	4.09 µs	+5.1%
sparse_keys	sparse_descend_until/500	13.07 µs	12.40 µs	-5.1%

Full tables per bench are in the job log and the bench-out artifact.

[luketpeterson]

I think this PR is finally ready to go. However, the one last piece of validation I want to do is see if the bespoke val_counts that are getting replaces got any slower by moving the implementation over to the recursive catamorphism.

[luketpeterson]

Here are the complete benchmark results.
recursive-cata-benchmark-results.zip

Bottom line, recursive cata is massively faster than iterative cata, as discussed above.

Since val_count is now subsumed by recursive cata, the val_count benchmark results are a bit more nuanced. There is an essential tradeoff in the implementation whether to count the value first or to visit the downstream first. Recursive cata made a different tradeoff than the hard-coded val_count resulting in better performance for catamorphism workloads, and about a ~10-20% uplift on "real" tries (binary trie shape), roughly neutral on the superdense benchmark, but a similar penalty on the "sparse" tries (a bunch of non-branching paths).

On ACT, there was no recursive cata at all. Implementing one gave a 3-8x speedup over the iterative cata but a 5-40x perf improvement in val_count vs the implementation on the ACTZipper!