eval/optimize/dryrun: nested aux+PAO batching, perf-first objective, dry-run cost-profile backend, rank-general CSV moments - #568
Open
evaleev wants to merge 79 commits into
Open
Conversation
…vel >= 2) Add a predict_hook to CacheManager, mirroring shaped_product_hook but observe-only: consulted at each binary Product node before materialization (self-gated on the eval trace level) to emit a "Predict" line naming the op and its predicted result footprint. This names an intermediate that exhausts memory even when it dies materializing (the post-hoc Eval line never prints for such an op). The hook is propagated into the batched scratch cache; batched products carry no shaped-product hook, so they are reported as unshaped(batched). The TA backend factory make_predict_hook() builds the hook from a TAEvalContext, reusing the shaped hook's result-outer-trange computation and a ToT average-inner-extent estimate to predict the footprint.
Add Logger::eval.heap_stats (std::function<std::string()>); when set, its (already rank-reduced) return is appended to each Eval line after rss=, and omitted entirely when unset. Lets a backend report allocator-level memory (e.g. glibc all-arena in-use vs system) right at an RSS jump, to separate live heap from retained-free heap.
Add Logger::eval.release_memory (std::function<void()>) and call it via log::release_after_op() after each freshly evaluated op in evaluate(), regardless of trace level. Lets a backend return retained-free heap to the OS between ops -- e.g. the large per-batch transients freed while building a batched DF leaf -- so they do not linger as allocator-retained free pages and inflate RSS before the next big op allocates. Local/non-collective by contract; empty hook = no-op (default).
evaleev
force-pushed
the
feature/eval-predicted-peak-trace
branch
from
July 3, 2026 02:33
5810a5c to
8e23906
Compare
The eval-time footprint estimate lifted the result's dense outer volume by an operand's average inner density (size_in_bytes / dense outer volume). That borrows the operand's fill, so it underestimates any contraction whose result is denser than its operands -- most severely the free-mu~ giant g(mu~,mu~,K).C, where the free mu~ spreads from the per-pair domain to the full range (~3x low in practice). It is also computed from materialized operand sizes, which are not what the factorizer's analytic cost model uses, so it cannot reflect or validate the tree it chose. Its one unique feature -- naming an op that OOMs before its post-hoc Eval line prints -- does not survive the real failure mode (async SIGBUS / OOM-kill), where the buffered Predict line never flushes. Remove make_predict_hook and its tot_avg_inner_factor / pred_outer_elems helpers, predict_hook_type, set_predict_hook / predict_hook(), the eval-time consultation in evaluate(), and the make_batched_scratch propagation. The post-hoc Eval line (actual sizes) and the shaped-product hook are unchanged.
Regression guard for the C60 PNO-CCSD OOM. The batched CSV-CCk residual
term
g{i_3;i_1;K} C{a_1<i_1,i_2>;mu} g{mu;mu;K} C{mu;a_3<i_2,i_3>}
factorizes, under aux(K)-only batchability, into the double-proto
intermediate I(i_1,i_2,i_3,K; a_3<i_2,i_3>, a_1<i_1,i_2>) -- two 2-occ
PNO composites sharing i_2, ~185 GB materialized -- which is what OOMs on
C60 (1800 AOs, 8 ranks, 768 GB node).
The probe pins the mechanism:
- DenseFLOPs / DensePeakSize pick a g.g-first tree that never forms the
double-proto (max_imed 1.3e11 elems);
- DensePeakSizeBatched with only K sliceable picks the double-proto
(max_imed 1.8e13) -- correctly, since with K the sole batch axis the
double-proto's K-sliced peak beats the avoiders' unsliceable mu-tilde
intermediate;
- making mu-tilde (PAO) batchable TOO flips STO back to the g.g-first
tree (max_imed 1.3e11) -- the double-proto becomes avoidable.
So the OOM root cause is the too-small batchable-index set (aux only),
not a suboptimal STO or a batching-eval bug. Guards any future change to
the batched peak objective or the batchable-index policy.
… to PeakBatchedModel
…ax, min-peak fallback) PeakBatchedModel::reconstruct now selects the root-frontier point by peak_threshold (bytes) instead of the unconditional peak_flops_tolerance band: among points whose peak*numeric_size fits peak_threshold, pick min flops (ties by lower peak); if none fit, fall back to min peak. Default peak_threshold = +infinity makes every point feasible, so min-flops wins, matching the non-batched schedule unless a finite threshold is set. peak_flops_tolerance is kept on PeakBatchedModel for source compatibility but is no longer consulted there; it still gates the unbatched DensePeakSize model. Doc comments in options.hpp/single_term.hpp updated to reflect the split. Adjusted three existing batched tests whose assertions were inherently peak-driven (not flop-driven) demonstrations, so the default +inf threshold no longer masks what they exercise: - "OSV deferral reproducer (tetramer term 3)": the persistent_only-gate probe now sets peak_threshold=1.0 to force the min-peak fallback. - "quadratic bubble: early-K integral vs late-K t.(gC)": choose() and real_config_integral() now set peak_threshold=1.0 for the same reason (this test is gated behind __OPTIMIZE__ and was verified via a separate -O1 recompile of test_optimize.cpp relinked against the Debug libraries). New test: "threshold gates batching: default (+inf) picks min-flops regardless of K_b; near-zero threshold falls back to min-peak", reusing the quadratic-bubble motif to show the divergence directly.
…s.term_batch_axes
Task 3.2 of the PNO-CCSD aux+PAO batching feature: thread the per-node
sliced-Index-sets that PeakBatchedModel::reconstruct_axes (Task 3.1)
computes out of the optimizer, so each optimized residual summand
carries which axes to slice at each of its contraction nodes.
- cost_model.hpp: add run_single_term_opt_axes, a companion to
run_single_term_opt that calls Model::reconstruct_axes instead of
reconstruct, returning {EvalSequence, node_axes} with the same nt==1/
nt==2 shortcuts (empty axes / one empty entry respectively).
- single_term.hpp: thread an optional
container::vector<container::svector<Index>>* out_axes = nullptr
through both the detail and public single_term_opt overloads. The
DensePeakSizeBatched arm calls run_single_term_opt_axes when out_axes
is set; every other objective asserts out_axes is null. Both
overloads clear *out_axes on entry so an early return (e.g. prod
with < 3 factors) never leaves it holding a previous call's data.
- optimize.cpp (opt_pure_product): declare node_axes, pass it through
only on the DensePeakSizeBatched call path, and after building the
result record (*opts.term_batch_axes)[result.get()] = node_axes when
the out-channel is set. optimize_impl/optimize() need no changes:
OptimizeOptions is threaded by reference/value-copy already, so the
shared_ptr out-channel survives unchanged.
- options.hpp: add OptimizeOptions::term_batch_axes (a
shared_ptr<unordered_map<Expr const*, per-node axes>>), default null
(no behavior change). Added the includes/forward-decl (container.hpp,
<unordered_map>, <memory>, class Expr) it needs.
Verified the RPN-to-Product alignment: single_term_opt's Product-
building loop (single_term.hpp) iterates the EvalSequence left to
right and, on each -1, pops rexpr (top of stack, most recently pushed)
then lexpr (pushed before it) to form Product{lexpr, rexpr}. Since the
sequence is emitted in the same left-first post-order in both
reconstruct and reconstruct_axes (identical build() recursion:
recurse into the lp_first-selected "first" operand, then the
"second", then append -1), the j-th -1 encountered by the loop is
exactly the j-th entry pushed into reconstruct_axes's node_axes. No
behavior change for existing callers (term_batch_axes defaults to
null); build + [optimize] unit tests pass (339 assertions, 12 cases).
Task 3.3 of the PNO-CCSD aux+PAO batching feature: consume Task 3.2's
OptimizeOptions::term_batch_axes at binarize time.
- eval_expr.hpp: add a private container::svector<Index> batch_axes_{}
member to EvalExpr with batch_axes()/set_batch_axes() accessors, and
BinarizationOptions::node_batch_axes (per-contraction-node sliced-sets,
RPN/post-order, left-first). Default-empty in both cases, so existing
callers see no behavior change.
- eval_expr.cpp: thread a std::size_t& node_counter through
impl::binarize / binarize(Sum, ...) / binarize(Product, ...). In
binarize(Product, ...)'s make_prod lambda, the tensor*tensor branch is
the genuine DP contraction node (scalar*scalar and scalar*tensor
branches are scaffolding from bare scalar factors or the trailing
scalar wrap, never counted by the optimizer): stamp
opts.node_batch_axes[node_counter] onto it when in range, then always
advance node_counter. The top-level binarize(ExprPtr, ...) entry point
owns the counter and, when opts.node_batch_axes is non-empty, asserts
the final count matches its size -- a mismatch means the optimizer's
and binarize's post-orders diverged.
Verified the alignment: single_term_opt builds every contraction as a
Flatten::No 2-factor Product, so binarize's fold_left_to_node visits
exactly one tensor*tensor node per level, in the same left-first
post-order reconstruct_axes emitted node_axes in.
- test_optimize.cpp: new [optimize][annotate] test round-trips
optimize() (DensePeakSizeBatched, forced batching via a tiny
peak_threshold, an aux Κ index shared between two factors) through
binarize(), and checks a node got annotated with the aux index that
was force-batched.
[annotate] (new test) / [optimize] (13 cases, 344 assertions) /
[EvalExpr] (81 assertions) / [EvalNode] (84 assertions) / [export]
(1955 assertions) all pass.
…h counts Confirms Task 4.2's re-entrant nesting already threads make_scope_guard unchanged into the reinstalled inner evaluator, and the outer scope_guard RAII object stays alive across the per-batch evaluate() calls that trigger that re-entry. So a backend guard that relaxes block-sparse screening scaled by its own level's batch count composes multiplicatively across nesting depth for free: net relaxation = product of batch counts over all alive levels, matching the invariant that a contribution significant over the full product of batch axes must not be screened away in any individual per-level batch cell. Documents this nesting semantics at the make_scope_guard parameter, the no_scope_guard/make_no_scope_guard default-factory doc, and the scope_guard RAII site in make_batched_custom_evaluator. Adds a structural composition test (dense TensorD has no real block-sparse screening to relax, so numeric validation is deferred to the Phase 6 end-to-end MPQC run): a custom ScopeGuardFactory whose RAII guard records construction/destruction against a shared "currently alive" stack, driven over the existing depth-2 nested batching tree. Both guards are alive simultaneously (stack depth 2) once per outer batch, and every such instance records outer_batches * inner_batches == 9, proving the multiplicative composition invariant.
sliced_footprints priced a sliced axis at min(e, batch_target_size(ix)) with no floor, while the TiledArray backend's result materialization floors the realized slice to max(target,1) (result.hpp:374). A target of 0 (a common default, e.g. MPQC's aux_target_size) made the DP model see a zero-byte footprint for that axis and never force a second axis to also slice, even though the runtime slices to a whole tile anyway. Floor the model target to >=1 so it matches the runtime in the degenerate zero-target case.
…not hardwired dense
…LOPs+exec cost, not just size)
…sidual Adds sequant::eval::dryrun::SizeRegime (per-space extents + CSV/OSV power-mean moment tables) and a dedicated unit-test OBJECT-lib group exercising it against the real C60 CSV-CCSD residual fixture (mpqc's csv_eqn_Rs.serialized). Confirms the deserialize path and the DensePeakSizeBatched + term_batch_axes + binarize round-trip mechanics end-to-end (per-summand optimize(), Product::Flatten::Yes re-flattening of the deserializer's literal nesting), plus a self-consistency memsize check. The fixture is pre-transform (no PAO/DF-aux indices), so this is infrastructure and mechanics only -- the actual PAO/K batch-axis go/no-go verdict needs a post-transform fixture and is deferred to a follow-up. See .superpowers/sdd/task-1-report.md for the full history.
…kend, and runtime replay Completes the DryRun eval backend across two fronts: 1. The real (post-transform) PAO(mu~)/DF-aux(K) batch-axis go/no-go verdict, deferred by Task 1: a [dryrun-df] case that deserializes the real post-transform CSV-CCSD doubles residual (fixture tests/unit/data/csv_ccsd_doubles_residual_df.txt, dumped from mpqc at the exact point handed to sequant::optimize()), runs the batched single-term optimizer under a C60-scale regime, and confirms the DP annotates a mu~ batch axis on the free-mu~ giant intermediate (matching the ~1.2 TB cluster-trace anchor almost exactly). 2. The DryRun eval backend itself (Tasks 2-6 of doc/dev/plans/2026-07-04-dryrun-eval-backend.md): CostModel wraps the optimizer's own memsize_counter/flops_counter/roofline_op_cost verbatim, adding only an ExtentOverrides indirection so a Result can report a runtime-realized (sliced) size rather than always the regime's nominal extent. ResultDryRun/ResultDryRunNested are zero-data Result tokens (index set + cost model only, no tensor storage) mirroring ResultTensorTAPP's structure; both concrete types share one op implementation (DryRunOps) and dispatch flat-vs-nested by content via make_dryrun_result, matching how the real engine decides tensor-of-tensor-ness. EvalExprDryRun/DryRunLeafEvaluator complete the plumbing evaluate() needs. The [dryrun-eval] harness replays the giant term's DP-annotated, binarized schedule through the REAL runtime (make_batched_custom_evaluator / evaluate<Trace::On>, with SEQUANT_EVAL_TRACE defined for this translation unit so note_working_set() fires through the batched evaluator's nested per-batch/per-member recursion, not just the outermost interception) against these zero-data tokens, reusing one BatchPolicy object for both optimize() and the runtime evaluator. On the current codebase the replay does NOT reproduce the suspected "K-sliced but mu~-full" bug: the giant's realized peak stays near 1.5 GB (a >99.9% reduction from its 1.21 TB nominal size), and the largest single materialized result (1.556 GB) matches the nominal size divided by the product of the mu~ and K batch counts (18 x 43) to within tile-rounding, i.e. both axes are genuinely nested-sliced by the runtime as currently implemented.
Add a design spec for a perf-first / peak-second single-term optimization
objective. The existing DensePeakSize model is peak-first: peak_threshold is a
hard filter over factorizations and FLOPS only breaks ties among survivors.
With PAO (mu-tilde) sliceable this structurally prefers the fully-sliceable but
FLOPS-catastrophic 4-PAO integral over the correct (gC)(gC) PPL, whose PNO-pair
leg has an irreducible peak floor above the threshold. This is the C60
PNO-CCSD OOM root cause.
The fix swaps the primary/secondary keys in the two frontier selectors
(select_root, pareto_best), keeping peak_threshold as the slice-down-to target
rather than the feasibility gate. Slicing is perf-neutral in the roofline model
(cflops computed once on unsliced sizes), so a roofline-perf primary rejects the
4-PAO and leaves slicing to the peak-second key.
Names encode the lexicographic order (Dense{Primary}{Secondary}, Space=peak,
Time=perf): rename DensePeakSize -> DenseSpaceTime (peak-first, kept as a
deprecated alias), add DenseTimeSpace (perf-first, new).
Rename DensePeakSize{,Batched} -> DenseSpaceTime{,Batched} (old names kept as
deprecated aliases sharing the same enum values, so every existing
`== DensePeakSize` guard keeps working) and add the perf-first / peak-second
duals DenseTimeSpace{,Batched}.
The peak-first model treats peak_threshold as a hard filter over
factorizations, so with PAO (mu-tilde) sliceable it structurally prefers the
fully-sliceable but FLOPS-catastrophic 4-PAO integral over the correct (gC)(gC)
particle-ladder. The perf-first model adds a `perf_first` flag to PeakModel /
PeakBatchedModel; when set, the root-frontier selector sorts by (flops, then
peak) instead of (peak, then flops). Because pareto_insert keeps one min-peak
point per distinct flops value, this both picks the cheapest factorization and
takes its fully-sliced realization, and peak_threshold is no longer consulted
as a feasibility gate (it can no longer force the 4-PAO). The Pareto frontier,
relax(), roofline cost, and the pareto_best test helper are unchanged.
single_term_opt routes DenseTimeSpace{,Batched} to the peak models with
perf_first set; optimize() gains dispatch arms for both new values.
Tests: DenseTimeSpaceBatched reaches the flop-optimal (gC)(gC) factorization at
both +inf and a near-zero peak_threshold (unlike peak-first strict, which pays
the ladder), and the enum aliases compare equal.
Add the [dryrun-perf] test plus the post-transform DryRun harness apparatus
([dryrun-probe]/[dryrun-df]/[dryrun-eval]) used to diagnose the C60 PNO-CCSD
OOM. [dryrun-perf] optimizes the real C60 giant term (index 38) under both the
peak-first (DenseSpaceTimeBatched) and perf-first (DenseTimeSpaceBatched)
objectives at the faithful config (occ=120, PNO=42, OSV=310, mu-tilde=1800,
aux=4320, pao_ts=256, aux_ts=72, peak_threshold=40 GB) and asserts the
factorization the DP picks.
Result: peak-first forms the fully-sliceable 4-PAO AO integral (4 free
mu-tilde, sliced to 34 GB, under the 40 GB threshold, so it survives the hard
filter despite catastrophic flops) -- the C60 pathology. Perf-first forms no
4-PAO (max 2 free mu-tilde); its largest giant is the correct (gC)(gC)
intermediate {mu-tilde, a<i,i>, K} at 89 GB, far below the 627-769 GB that
OOM'd the real run and below node memory. The fix is validated in-harness.
Extend [dryrun-perf] to also REPLAY the zero-data schedule (both objectives)
through the real eval loop + real CacheManager and report
cache.working_set_hwmark() alongside the largest single transient (result=) and
its full trace line.
Two findings, now documented in the test:
(1) The outer working_set_hwmark is NOT the whole peak under batching. The
batched custom evaluator runs each batch against a SEPARATE scratch cache
(make_batched_scratch, eval.hpp:1386), so the peak-first 4-PAO giant's per-op
hw= reaches ~38.9 GB INSIDE that scratch while the outer accessor reports
only ~0.2 GB. The outer accessor tracks cached (cross-batch) residency, not
batched-inner transients.
(2) The runtime Result sizing is not moment-aware. The perf-first largest
transient = 358.47 GB = 120^2*42^4*8 (occ^2*PNO^4) EXACTLY = the twin-PNO
size_in_bytes() artifact; the DP cost model's moment-aware size (~89 GB, via
inner_pow) is the real number.
Conclusion: the moment-aware DP peak model is the reliable predictor today; the
runtime replay hwmark is not, until scratch hwmarks propagate and the Result
sizing is moment-aware.
… + batch_axes per node)
Two overlapping whole-residual harnesses, neither asserting correctness: [dryrun-trace] replayed the forest through the shared cache and wrote a per-op trace file; [dryrun-df] prints the per-term mu~/K batch-axis escape verdict. The trace replay is already covered by the [dryrun][cost_profile] unit tests and the trace file is a regenerable one-off, so cut [dryrun-trace] entirely. Keep [dryrun-df]'s per-term verdict (uniquely useful C60 diagnostic, with the env sweep knobs) but mark it hidden ([.]) since it is a dev sweep, not a regression test -- select it explicitly with "[dryrun-df]". The standing tests are now [dryrun-objective] (comparative) + [dryrun-eval] (runtime replay) + the SizeRegime/CostModel/Result unit tests + the [peak]/[cost_profile]/[cache] infra tests. Visible dryrun suite green (120 assertions/21 cases).
Sweeps all 55 C60 residual terms under both objectives and reports the summed-flops ratio (perf-first/peak-first) and max modelled peak per objective -- the decision input for whether enabling dense_time_space (perf-first) at scale is worth it. Test-only; no production change. Result: ratio 0.142 (perf-first ~7x fewer flops), max peak 49.5 vs 5862 GB.
Replace the code-recursive descent in evaluate(node, le, cache) -- the only evaluate overload with unbounded recursion (the layout/multi-root overloads just wrap or loop over it) -- with an explicit std::deque work stack. Depth is now bounded by the heap, not the C++ call stack, so a deep tree (a Sum or product chain with many operands) no longer risks a stack overflow in the traversal. The per-node cache key is EvalExpr's stored O(1) hash, so nothing in the walk recurses. The rewrite preserves behavior byte-identically: - Checked cache wrapper: a hit returns the phase-applied cached pointer; a miss on a mapped node schedules a store once computed (the store_after flag replaces the recursive evaluate<..., Unchecked> re-entry). - Custom-evaluator interception is consulted when a frame is first visited and a non-null result short-circuits the subtree (children never pushed) -- the subtree pruning batched eval relies on is unchanged. - leaf / Adjoint / Sum / Product dispatch, the shaped-product hook, de-nesting, canonicalization-phase multiplication, eager per-op release, and every trace log site (MultByPhase, custom-eval, leaf, binary, cache access/store) are carried over unchanged. Verified against the eval oracle ([eval_tapp], [dryrun-eval] incl. the batched custom-evaluator giant-term replay, [dryrun-leaf/result/nested/costmodel], [EvalNode], [EvalExpr]): 247 assertions in 16 cases, identical to the pre-refactor baseline. Note: this makes the *traversal* stack-safe; binarize/EvalExpr construction and FullBinaryNode's (unique_ptr-owned) destructor remain code-recursive and would overflow first on a single very deep tree. Full-pipeline stack-safety for one arbitrarily deep tree would need those addressed too; in current use the residual is a vector of shallow summand trees.
…tive FullBinaryNode owns its children by unique_ptr, so the implicitly-generated destructor recursed to the tree's depth; deep_copy() and the free function transform_node() likewise recursed. On a deep tree (a long contraction chain or a nested Sum) these overflowed the C++ call stack at depth ~thousands, which is why building/copying/destroying such a tree crashed before evaluation even ran. Replace all three with explicit-stack iterative implementations: - ~FullBinaryNode() dismantles the subtree via a work list, detaching each node's children before it is destroyed so every node destruction is O(1). - deep_copy() and transform_node() do an iterative post-order build with an explicit frame stack (std::deque, so a top-frame reference survives push_back). transform_node applies its data map post-order; as a pure map its application order is immaterial, and its only callers (typed binarize<T> and the TA to_ta_node helper) are order-insensitive. The tree fold (fold_left_to_node / the range-accumulate ctor) and the tree visitors (detail::visit, parent-pointer based) were already iterative. size(), operator==, and digraph/tikz remain recursive but are not on the deep-tree build/eval/destroy path. Adds a [FullBinaryNode][stack-safety] test that builds, copies, transform_nodes, and destroys a depth-200000 tree (all O(N)); pre-refactor this overflowed the stack. Full FullBinaryNode + eval suites remain green.
…ort giant under occ-slice
Reconcile the predicted-peak / batching / dry-run work with master's merged #576 (outer-product pruning + build_context skip + fast_flops), #575/#574 (iterative evaluate()/binary_node), and TNv3 high-order aux work. Conflict resolutions: - options.hpp: CostParams and OptimizeOptions carry BOTH the branch's peak_threshold / term_batch_axes and master's prune_outer_products. - single_term.hpp: keep the perf-first objective dispatch (perf_first, DenseTimeSpace/DenseTimeSpaceBatched, peak_threshold, out_axes) and thread master's prune_outer_products into every model. - cost_model.hpp: PeakModel / PeakBatchedModel keep the branch's perf-first members (perf_first, peak_threshold, numeric_size, charge_batch_recompute) and gain master's prune_outer_products; build_context bodies auto-merged so the pruning skip + fast_flops precompute coexist with the perf-first tables. - optimize.cpp: CostParams init passes both peak_threshold and prune_outer_products. - eval.hpp: keep the branch's malloc_trim release_after_op() hook at each op. - test_optimize.cpp: keep both the branch's threshold/perf-first tests and master's pruning + fast-flops-parity tests. The branch's own (superseded) outer-product-pruning commits were dropped before this merge since #576 supersedes them; the predict-hook add/remove experiment nets to no hook, matching master.
The forest-over-external-axis nesting test built its external Hadamard spectator as a bare occupied index shared among three tensor slots (g, h, p) plus the result. Master's TensorNetworkV3::canonicalize_slots deliberately rejects a non-auxiliary index shared among >2 tensor slots (it has no well-defined bra/ket slot type), so eval_node threw once this branch merged master's newer TNv3. Switch the spectator to an auxiliary index (x_9): a high-order aux hyperindex carried into the result IS supported (the same case the eval_with_tiledarray test exercises), the arrays stay flat so the intra-term batched evaluator runs on its supported path, and the forest + nested-batching mechanism under test is axis-agnostic. A second inner-aux predicate keeps the contracted batch axis (x_1) distinct from the spectator. Occupied-external recognition and sizing are covered separately at the DP/cost level.
is_valid recognized Constant, Variable, Tensor, Product and Sum but fell through to a hard assertion for any other expression type. Power (base^ exponent) is a legitimate expression -- it appears, for example, in CC/EOM residual expressions as an inverse-denominator factor -- so a consumer that validates such an expression (e.g. MPQC's process_for_evaluation) aborts in assertion-enabled builds and, with the assertion elided in release, silently treats a Power-bearing expression as valid only by luck. Add a Power case: a Power is valid iff its base is valid (the exponent is a rational and is always well-formed). Power is atomic, so the base is not reached by the children loop and is validated explicitly here.
…l spaces is_pure_occupied/is_pure_unoccupied/contains_occupied/contains_unoccupied reduced an index's quantum numbers to physical-particle (spin) attributes and then looked up the vacuum-(un)occupied space at those qns via a throwing retrieve. For an index whose reduced qns has no registered occupied space this threw instead of answering the occupancy question. Two cases motivated this: - Physical spaces (AO, PAO) span real particle states and are spin-independent, not spin-less; they must carry the convention's spin_any in the physical sector. add_ao_spaces/add_pao_spaces registered them with only their LCAO trait bit (spin bits empty), so physical_particle_attributes() reported no spin and the occupancy lookup queried a spin flavor with no occupied space. Fix: add_ao_spaces/add_pao_spaces now take an explicit spin_any parameter (Spin::null for SpinConvention::Legacy, else Spin::any, matching make_*_spaces) and register the AO/PAO spaces with spin_any | trait. - Non-physical auxiliary spaces (density-fitting, batching) legitimately carry no spin; asking whether they are occupied must answer false, not throw. The four occupancy predicates now route through non-throwing helpers (vacuum_occupied_type_or_null / complete_type_or_null / vacuum_unoccupied_type_or_null) that return a null type when no space is registered at the reduced qns. Byte-identical for physical inputs. Update unit-test callers of add_ao_spaces/add_pao_spaces to pass the spin_any matching each test's registry convention.
Retype EvalExpr::batch_axes_ and every producer/consumer of it (optimize's term_batch_axes/node_batch_axes, PeakBatchedModel::reconstruct_axes, the batched evaluator's candidate_axes lookup) from a bare Index to std::pair<Index, AxisKind>. AxisKind lives in core/eval/fwd.hpp next to DeNest so core/optimize (which has no other dependency on eval_expr.hpp) can see it without pulling in the whole header. Pure refactor: every producer tags entries AxisKind::Contracted, so behavior is unchanged. A later change will start emitting AxisKind::External for never-contracted spectator indices, which the batched evaluator can then handle without a separate forest-level mechanism.
PeakBatchedModel::reconstruct_axes now appends AxisKind::External entries for genuine spectator (never-contracted) axes carried on a node's result, alongside the existing AxisKind::Contracted entries for sliced axes. Gated by a new PeakBatchedModel::batch_spectator_indices member (default false), threaded from CostParams::batch_spectator_indices <- BatchPolicy::batch_spectator_indices. Every other PeakBatchedModel construction is unaffected and stays byte-identical.
…eak-threshold trigger Fold compute_external_batch_axis's conditions (ii) perf-first objective and (iv) unseeded root peak (bytes) over peak_threshold into PeakBatchedModel::reconstruct_axes's own emit_external gate, alongside the existing batch_spectator_indices flag. The gate reuses the SAME selected root (select_root(ctx, st), root_B=0) that seeded_root_peak_batched already reads for condition (iv), so no extra model pass is needed. compute_external_batch_axis / ExternalBatchAxis / OptimizeResult's external_batch_axis field are untouched; they still feed the MPQC forest path (cck.ipp) until that is retired.
…esult A batch axis stamped External (AxisKind::External) on a node is a spectator index that survives free onto the node's result -- a Hadamard/batched-matmul mode present on every operand and the result, contracted at no node. Such an axis must be SCATTERED: the per-block partials are disjoint slices of one result, assembled via write_into_slice into a pre-sized destination, not accumulated via add_inplace (correct only for a contracted axis). make_batched_custom_evaluator now classifies the picked axis by AxisKind. A Contracted axis (or the depth-0 heuristic fallback, which only ever yields a contracted index) takes the existing accumulate path byte-for-byte. An External axis takes a new scatter branch: pre-size the node's result to the axis's full extent (from the carrying leaf's tiling), then for each block slice every carrying leaf, evaluate the node subtree on a per-block scratch, and write_into_slice the block partial into its slice. Inner axes still nest through the same per-block reinstall, so External composes with Contracted. Adds Result::pre_sized_zeros_over_mode (base throws; flat ResultTensorTA builds a zero array with the axis widened to full) and a [batched-external] TA test that evaluates a spectator-carrying tree with external batching on (block < extent) and off and requires the two elementwise-equal, with a scope-guard witness proving the on path actually blocks.
…g signature Extend make_batched_scratch's per-node slicing signature and its seed/consistency predicates to range over the batch's External axes as well as its contracted member axis. A node carrying any batched External (spectator) axis is non-invariant under the external slice, so it is never seeded full from the real cache and never shared across occurrences whose external signature differs; it is recomputed sliced. The External axis set is obtained exactly as the evaluator obtains it (partition batch_axes() by AxisKind). With no External axis in play the list is empty and the contracted-only path is byte-identical. Also add the ToT ResultTensorOfTensorTA::pre_sized_zeros_over_mode override (carried from Task 5's flat-only pre-size): widen the first block partial's outer axis mode to the carrier leaf's full tiling and zero-fill with well-formed (empty-inner) outer tiles, a valid destination the ToT write_into_slice scatter overwrites. CSV/PNO-CCSD residuals use ToT tiles. Tests (tag [eval][batched-external]): batched_scratch_no_seed_external asserts an invariant persistent node carrying a batched External axis is excluded from the seeds (with a no-External control that still seeds it); batched_scratch_tot_presize_scatter drives pre_sized_zeros_over_mode + write_into_slice for a ToT and requires exact reassembly.
The old "forest over a single external batch axis" special case has been replaced by the unified per-node batch_axes_ (AxisKind::External) mechanism introduced in the previous commits. Delete the now-dead code and its tests: - optimize.hpp/.cpp: ExternalBatchAxis struct, OptimizeResult:: external_batch_axis member, and compute_external_batch_axis. - eval.hpp: spectator_outer_mode and evaluate_forest_over_external_axis. - eval_expr.cpp/.hpp: outer_proto_position definition and declaration. - test_eval_expr.cpp: eval_expr_outer_proto_position and eval_expr_outer_proto_position_mixed_node_guard. - test_eval_ta.cpp: the two eval_forest_over_external_occ TEST_CASEs. - test_optimize.cpp: the OLD "optimize emits an external batch axis for the PPL term" TEST_CASE. Reworded dangling comments in cost_model.hpp and the kept [optimize][batch] tests in test_optimize.cpp that referenced the deleted compute_external_batch_axis by name. The new-unified-path tests ([eval][batched-external] in test_eval_ta.cpp and reconstruct_axes_emits_external_per_node in test_optimize.cpp) are untouched.
Add three [eval][batched-external] tests: two occupied External axes nested on one product node (product-of-blocks firing witness), a genuinely Hadamard External axis shared by both operands of a contraction plus the result, and an External axis on an outer node nested with a Contracted axis on an inner node.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Eval/optimize/dry-run infrastructure for costing, sizing, and batching CSV-CCk / PNO-CCSD factorizations, motivated by the C60 PNO-CCSD OOM (the dense free-μ̃
(g·C)(g·C)giant). This branch grew well past its original "observe-only predicted-footprint hook" scope (that hook has since been removed — see below). Paired MPQC PR: ValeevGroup/mpqc4#783.What this delivers
1. Nested multi-axis (aux + PAO + occupied) batching in the evaluator
EvalExpr.batch_axes_:binarizestamps per-node sliced-sets; the batched evaluator prefers the per-node annotated batch axis.optimize()recognizes such occupied protoindices as batchable and emits the choice as a forest-levelExternalBatchAxissignal (domain-neutralBatchPolicyconfig). This lets PNO-CCSD batch over occupieds, not just aux/PAO, to bound the dense free-mu-tilde(g.C)(g.C)giant.OptimizeOptions.term_batch_axessurfaces per-node sliced-sets per summand in RPN order.2. Threshold-gated batched selection in
optimize()PeakBatchedModel/reconstructed_batched_peakoracle prices the batch contribution buffer;peak_threshold(bytes) plumbed throughBatchPolicy/CostParams.accumulation_factorallowed for m≥2 (nested), gated by the DP==oracle identity. Sliced-axis model target floored to ≥1 to match the runtime.3. Perf-first
DenseTimeSpace{,Batched}single-term objective4. Dry-run cost-profile eval backend
Resultbackend +CostModelthat replays a factorized IR and reports memsize + FLOPs + exec cost (a pluggable sizing oracle; dense by default), not just size.CostProfilestruct + reusablecost_profile()entry point;[dryrun-perf]/[dryrun-trace]routed through it. Opt-in scratch-fold peak sink for a faithful batched-replay peak; gated cache with batchable-axis veto and per-term reset.SizeRegime: per-space extents + per-rank CSV moment tables (power means). Post-transform (μ̃ + Κ) C60 residual fixtures and DP-inspection mechanics; power-mean sizing contract locked with a regression test.inner_powdispatches by cluster rank (1→OSV, 2→PNO, ≥3→acsv_moment_by_ranktable if present, else the PNO-table fallback), so CSV-CCSDT triples are sized by their own domain. Pairs with the MPQC rank-generalCSVSizeMoments.5. Index-space registry: occupancy predicates robust to non-physical spaces
Enabling the occupied-spectator scan in
optimize()against a real CSV/PNO registry surfaced a latent registry bug.is_pure_occupied/is_pure_unoccupied/contains_occupied/contains_unoccupiedreduced an index's quantum numbers to physical-particle (spin) attributes and then looked up the vacuum-(un)occupied space via a throwing retrieve, so an index whose reduced qns had no registered occupied space threw instead of answering the occupancy question. Two coordinated fixes:spin_anyin the physical (spin) sector alongside their LCAO trait bit.add_ao_spaces/add_pao_spacespreviously registered them with only the trait bit (spin bits empty), sophysical_particle_attributes()reported no spin and the occupancy lookup queried a spin flavor with no occupied space. These now take an explicitspin_anyparameter (Spin::nullforSpinConvention::Legacy, elseSpin::any, matchingmake_*_spaces) and registerspin_any | trait.vacuum_occupied_type_or_null/complete_type_or_null/vacuum_unoccupied_type_or_null) that return a null type when no space is registered at the reduced qns. Byte-identical for physical inputs.Unit-test callers of
add_ao_spaces/add_pao_spaceswere updated to pass thespin_anymatching each test's registry convention (Default/None →Spin::any, Legacy →Spin::null).6. Fixes / misc
tot_inner_rank.CostToken/DryRunfor clarity; reframe the spec as cost-model replay.Note on history
The original observe-only predicted-footprint (predict) hook on
CacheManagerwas added and then removed within this branch (commitsa19dbfb5f→bc5bd6cb5); it nets to zero. The dry-run cost-profile backend above is its principled replacement — it predicts the whole factorized IR up front rather than per-op during materialization. This PR's title/body have been updated accordingly.Validation
[dryrun]/[dryrun-df]/[dryrun-perf]/[optimize]suites pass; new rank-general dispatch regression test passes. The registry fix keeps the full space/mbpt suite green ([elements],[mbpt],[mbpt/cc],index_space,spin,[algorithms]incl. the Legacy-convention PAO canonicalization test) and[optimize](which exercises the PAO+DFcompute_external_batch_axisemission path); MPQChe10-csv-cck-2-paocorrelation energy is unchanged vs the pre-fix binary to ~1e-13. Branch base ismaster.