From 5da94d430b7c49b0ee025019fa9242953549801e Mon Sep 17 00:00:00 2001 From: Garret Rieger Date: Sat, 17 Oct 2026 01:29:11 +0000 Subject: [PATCH 01/13] Add design doc for all pairs merging exploration. --- docs/all_pairs_merging.md | 136 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 136 insertions(+) create mode 100644 docs/all_pairs_merging.md diff --git a/docs/all_pairs_merging.md b/docs/all_pairs_merging.md new file mode 100644 index 00000000..00b66ff6 --- /dev/null +++ b/docs/all_pairs_merging.md @@ -0,0 +1,136 @@ +# All-Pairs Cost-Based Segment Merging + +Author: Garret Rieger +Date: Oct 16th, 2026 + +## Introduction + +During cost-based segmentation ([segmenter.md](segmenter.md)), `Merger::TryNextMerge()` iteratively merges segments within each merge group to minimize the expected number of bytes transferred: + +$$\text{cost}(S) = \sum_{m=1}^M \sum_{p_i \in S} P_m(c_i) \times (\text{size}(p_i) + k)$$ + +Currently, candidate selection in `Merger::MergeSegmentWithCosts()` uses a **single-base ("leader-follower")** greedy strategy: + +1. Pick the first active segment in `candidate_segments_` (`base_segment_index`, which starts as the highest-probability segment in the merge group). +2. Evaluate merges between `base_segment_index` and all other active segments (plus composite conditions involving `base_segment_index`). +3. Apply the lowest-cost-delta merge found for `base_segment_index`, reprocess changed glyph conditions, and repeat with the **same** `base_segment_index` until no further merges reduce cost (and `min_group_size` is met), at which point `base_segment_index` is marked finished and removed from `candidate_segments_`. + +While restricting each pass to a single `base_segment_index` avoids maintaining global pairwise state, it has a structural quality limitation: `base_segment_index` ($s_0$) will greedily absorb a lower-probability segment $s_j$ even when $\Delta \text{cost}(s_0, s_j)$ is only marginally negative (e.g. $-5$ bytes) while $s_j$ has a much stronger merge partner $s_k$ (e.g. $\Delta \text{cost}(s_j, s_k) = -150$ bytes). Once $s_0$ absorbs $s_j$, $s_j$'s glyphs are loaded at $s_0$'s higher probability, permanently preventing the $(s_j, s_k)$ merge. + +This document describes a plan to prototype **All-Pairs Agglomerative Merging** in `Merger`, gated behind `--experimental_all_pairs_merging`, while keeping runtime and Brotli compression call counts competitive with (or lower than) the single-base approach. + +--- + +## Performance Challenges of Naive All-Pairs Merging + +Upgrading to all-pairs merging naively introduces two potential performance cliffs: + +1. **Upfront Brotli Compression Cliff**: Evaluating exact cost deltas (`CandidateMerge::ComputeCostDelta`, which compresses candidate glyph sets with Brotli) across all $\binom{N}{2}$ pairs before making the first merge would perform $O(N^2)$ Brotli compressions with a weak pruning threshold (`delta_to_beat = 0.0`). Moreover, every applied merge invalidates up to $2N$ pairs involving the merged segments, discarding most of that upfront Brotli work. +2. **Best-Case Delta (`ComputeCostDelta`) Overhead**: Even without Brotli, calling the general `CandidateMerge::ComputeCostDelta` on all $\binom{N}{2}$ pairs (and $O(N)$ pairs after each merge) is expensive if every evaluation allocates `modified_conditions` and `new_conditions` hash maps, copies `ActivationCondition`s, and computes $O(|s_a| \cdot |s_b|)$ bigram union bounds via `AggregateMergedProbability()`. + +--- + +## Algorithm Design + +### 1. Two-Phase Execution (`Phase 1: Cost Reduction` $\rightarrow$ `Phase 2: Minimum Group Size`) + +In the current single-base merger, a segment that starts below `min_group_size` (e.g. 1 codepoint when `min_group_size = 4`) begins its scan with `delta_to_beat = nullopt`, meaning positive-cost-delta candidates cannot be pruned until a negative-cost-delta candidate is encountered. + +In all-pairs mode, segment merging is split into two phases: + +* **Phase 1 (All-Pairs Cost-Reducing Merges, $\Delta \text{cost} < 0$)**: A global priority queue evaluates and applies merges with strictly negative cost deltas ($\Delta \text{cost} < 0.0$) across all active segments above `optimization_cutoff_segment_`. Because Phase 1 only accepts negative deltas, every candidate evaluation has an immediate upper bound of `delta_to_beat <= 0.0` (tightened further by the current heap minimum), enabling aggressive pruning of positive-delta pairs across the entire merge group. +* **Phase 2 (Minimum Group Size Enforcement)**: Once Phase 1 exhausts all negative-cost-delta merges, any segment that already meets `MinimumGroupSize()` is marked finished in $O(1)$. Remaining undersized segments are merged using `MergeSegmentWithCosts()` in descending probability order to meet `MinimumGroupSize()`. If a forced merge in Phase 2 produces a segment with new negative-cost-delta opportunities, Phase 1 can resume for that segment. +* **Patch Merging**: As before, once all segment merges complete, `TryNextPatchMerge()` runs if `UsePatchMerges()` is enabled. + +### 2. Lazy Two-Stage Priority Queue (`BEST_CASE` $\rightarrow$ `EXACT`) + +Instead of computing exact Brotli-backed cost deltas when inserting candidates into the priority queue, a single min-heap stores both **lower-bound (`is_exact = false`)** and **exact (`is_exact = true`)** entries: + +```cpp +struct QueuedCandidate { + double cost_delta; + bool is_exact; + // Canonical set of all segments participating in the merge (size >= 2). + // The surviving base segment is always *segments.min(). + ift::common::SegmentSet segments; + uint32_t created_epoch; + std::optional evaluated_merge; // Present when is_exact == true. +}; +``` + +#### Deterministic Total Ordering +To guarantee run-to-run determinism, heap entries are ordered by: +1. `cost_delta` ascending (most negative delta first). +2. `is_exact` (`true` before `false`, so an exact candidate tied with a lower bound is popped and applied immediately). +3. `segments` lexicographically ascending (`SegmentSet::operator<`). + +#### Pop-and-Refine Loop +When searching for the next Phase 1 merge: +1. Pop the top `QueuedCandidate` from the min-heap. +2. **Staleness check**: If any $s \in \text{candidate.segments}$ has `segment_epoch_[s] > candidate.created_epoch` (meaning $s$ was removed or had its conditions/glyphs modified after this entry was computed), discard the entry and continue. +3. **If `is_exact == false`**: + * Determine `delta_to_beat`: `0.0` (since Phase 1 requires $\Delta \text{cost} < 0.0$). Note: we cannot use the next heap entry's `cost_delta` as `delta_to_beat` if that entry is still `is_exact == false`, because its exact delta may rise when refined; however, we *can* track the best `is_exact == true` delta currently in the heap as `best_exact_delta_in_heap` and pass `std::min(0.0, best_exact_delta_in_heap)` as `delta_to_beat`. + * Evaluate `CandidateMerge::AssessSegmentMerge(*this, *segments.min(), segments, baseline_with_delta_to_beat)`. + * If a valid `CandidateMerge` is returned with `CostDelta() < 0.0`, push `(merge.CostDelta(), is_exact = true, segments, current_epoch_, merge)` back into the heap. +4. **If `is_exact == true`**: + * Because its exact cost delta is $\le$ the `best_case` lower bound of every other candidate in the heap, it is the globally best merge. Apply `candidate.evaluated_merge->Apply(*this)` and return its `InvalidationSet`. + +**Why this reduces Brotli calls**: In single-base mode, if `s_0` only has a weak best merge (e.g. $\Delta = -3$), `delta_to_beat` is $-3$, so `s_0` runs Brotli on every candidate with `best_case_delta < -3`. In the lazy global heap, the strongest merges across the entire group (e.g. $\Delta = -200$) are popped first; pairs with weaker `best_case_delta` values never run Brotli until the global best delta rises to meet them, by which time many of their constituent segments have already been merged elsewhere. + +### 3. Probability-Ordered Row Pruning & Inert Fast-Path + +In single-base merging, only the lowest-indexed segment ever merges, so iterating `candidate_segments_` in index order automatically iterates in non-increasing probability order and satisfies `P(base) >= P(other)`. Under all-pairs merging, any segment $s$ can merge and increase its probability. + +To preserve `BestCaseInertProbabilityThreshold` pruning and avoid $O(N^2)$ calls to `AggregateMergedProbability()`: + +1. **Per-Segment Cached Metadata**: `Merger` maintains flat vectors indexed by `segment_index_t`: + * `cached_agg_prob_[s]`: `AggregateProbability(s)` + * `cached_exclusive_size_[s]`: `patch_size_cache->GetPatchSize(ExclusiveGlyphs(s))` (or `0` if empty) +2. **Probability-Ordered Active Set**: Active segments below `optimization_cutoff_segment_` with non-empty exclusive glyphs are maintained in an ordered structure sorted by `(cached_agg_prob_[s] descending, s ascending)`. +3. **Row Pruning via `BestCaseInertProbabilityThreshold`**: + * Any exclusive pair $\{u, v\}$ is oriented during candidate generation so that $u$ is higher-ranked in probability order ($P_{\text{agg}}(u) \ge P_{\text{agg}}(v)$). + * When scanning lower-probability partners $v$ for segment $u$ in descending order of $P_{\text{agg}}(v)$, once $u$ is inert and $P_{\text{agg}}(v) \le \text{BestCaseInertProbabilityThreshold}(\text{size}_u, P_{\text{agg}}(u), 0.0)$, all subsequent inert segments $v'$ also have $P_{\text{agg}}(v') \le P_{\text{agg}}(v)$ and cannot achieve a negative cost delta with $u$, allowing the scan to `break` early (or skip remaining inert segments). +4. **$O(1)$ Pre-Check & Inert Fast-Path in `ComputeCostDelta`**: + * When both segments $u$ and $v$ are inert, their only modified conditions are their two unitary exclusive conditions `(u)` and `(v)`, which collapse into a single unitary condition `(min(u, v))`. + * Under `best_case_size_reduction_fraction` ($r_{\text{best}}$), the best-case merged patch size is: + $$S_{\text{best}}(u, v) = \max(S_u, S_v) + \max\left(\lfloor (\min(I_u, I_v)) \cdot r_{\text{best}} \rfloor, \text{BEST\_CASE\_MERGE\_SIZE\_DELTA}\right)$$ + where $I_s = \max(0, S_s - \text{base\_overhead})$. + * Since $P_{\text{agg}}(u \cup v) \ge \max(P_{\text{agg}}(u), P_{\text{agg}}(v)) = P_{\text{agg}}(u)$, we can check in ~5 arithmetic operations—**before** calling `AggregateMergedProbability()` or allocating any hash maps—whether: + $$P_{\text{agg}}(u) \cdot (S_{\text{best}}(u, v) - S_u) - P_{\text{agg}}(v) \cdot (S_v + k_{\text{net}}) \ge \text{delta\_to\_beat}$$ + If this holds, the pair cannot possibly pass `ComputeCostDelta` and is rejected in nanoseconds. + * If it passes the pre-check, the inert fast-path computes `AggregateMergedProbability({u, v})` and evaluates the 3-term cost delta directly without constructing `modified_conditions` or `new_conditions` maps. + +### 4. Composite Condition Candidates + +In single-base mode, `CollectCompositeCandidateMerges(base_segment_index)` scans `OrderedConditions()` for every base segment, potentially evaluating a 3-segment condition $\{s_1, s_2, s_3\}$ multiple times. + +In all-pairs mode (when `target_mode != UNICODE_RANGE`): +* Each composite condition $c$ with triggering segments $S = c.\text{TriggeringSegments()}$ satisfying $|S| > 2$, $*S.\text{min}() < \text{optimization\_cutoff\_segment\_}$, $S \cap \text{candidate\_segments\_} \neq \emptyset$, and $S \subseteq \text{inscope\_segments\_}$ defines a canonical candidate merge set $S$ (with `base = *S.min()`). +* Unique composite sets $S$ are evaluated with `ComputeCostDelta(*this, S, std::nullopt, 0.0)` and, if $< 0.0$, pushed into the same priority queue alongside pairwise candidates. + +### 5. Epoch-Based Invalidation Across `TryNextMerge()` Calls + +In `ClosureGlyphSegmenter::CodepointToGlyphSegments()`, `context.ReprocessChanged(*maybe_modified)` is called *after* `merger.TryNextMerge()` returns. Consequently, updated glyph conditions and groupings for a merge applied in call $k$ are only finalized at the entry of call $k + 1$. + +To handle this cleanly without changing `TryNextMerge()`'s external contract: + +1. **Epoch Tracking**: + * `Merger` maintains a monotonic counter `uint32_t current_epoch_ = 0` and a vector `std::vector segment_epoch_` initialized to `0`. + * Removed segments (`to_merge` or `MarkFinished`) have `segment_epoch_[s]` set to `UINT32_MAX`. +2. **Dirty Segment Recording on Merge Application (Call $k$)**: + * When a merge $M$ into `base` is applied: + * For an **inert merge** (`input_segments_are_inert_ && new_segment_is_inert`), `CandidateMerge::Apply()` updates `glyph_condition_set` and `glyph_groupings` in-place. Only `base` is added to `dirty_segments_`. + * For a **non-inert merge**, before `Apply()` invalidates conditions, `Merger` records in `dirty_segments_`: `base`, every segment in $c.\text{TriggeringSegments()}$ for $c \in \bigcup_{s \in M} \text{TriggeringSegmentToConditions}(s)$, and every segment in the pre-merge condition of `invalidated_glyphs_`. It also saves `pending_non_inert_base_ = base` and `pending_invalidated_glyphs_ = invalidated_glyphs_`. +3. **Refreshing Dirty Segments at Next `TryNextMerge()` Entry (Call $k + 1$)**: + * If `pending_non_inert_base_` is set (meaning `context.ReprocessChanged` just ran), also add to `dirty_segments_` all segments in `TriggeringSegmentToConditions(*pending_non_inert_base_)` and in the post-reprocessing conditions of `pending_invalidated_glyphs_` (intersected with `candidate_segments_`). + * Increment `++current_epoch_`. For each $s \in \text{dirty\_segments\_}$, set `segment_epoch_[s] = current_epoch_` and refresh `cached_agg_prob_[s]`, `cached_exclusive_size_[s]`, and $s$'s position in the probability-ordered active set. + * Generate new pairwise candidates involving `dirty_segments_` (avoiding duplicate `{u, v}` evaluations when both $u, v \in \text{dirty\_segments\_}$) and any composite condition candidates intersecting `dirty_segments_`, then clear `dirty_segments_`. + +--- + +## Implementation Plan / TODO + +- [ ] **Commit 1: Add `--experimental_all_pairs_merging` flag, per-segment cost metadata cache, and inert 2-segment fast-path.** Declare `ABSL_DECLARE_FLAG(bool, experimental_all_pairs_merging)` in `ift/encoder/merger.h` (default `false`), add cached per-segment aggregate probability and exclusive patch size vectors on `Merger`, and add an $O(1)$ pre-check + fast-path for 2-segment inert cost delta evaluations in `CandidateMerge::ComputeCostDelta` (with unit tests in `ift/encoder/candidate_merge_test.cc`). +- [ ] **Commit 2: Implement the lazy two-stage all-pairs priority queue and probability-ordered candidate generation in `Merger`.** Add `QueuedCandidate`, probability-ordered active segment tracking with `BestCaseInertProbabilityThreshold` row pruning, composite condition candidate queuing, and the lazy `BEST_CASE` $\rightarrow$ `EXACT` pop-and-refine loop for Phase 1 cost-reducing merges. +- [ ] **Commit 3: Implement epoch-based invalidation and incremental `dirty_segments_` refresh across `TryNextMerge()` calls.** Track affected segments for both inert and non-inert merges, refresh `dirty_segments_` at the start of `TryNextMerge()` after `SegmentationContext::ReprocessChanged()`, and reset all-pairs state in `Merger::ReassignInitSubset()`. +- [ ] **Commit 4: Integrate Phase 2 (`min_group_size` enforcement) and add unit and determinism tests.** Transition to Phase 2 for any undersized segments remaining after Phase 1 completes, and add tests in `ift/encoder/candidate_merge_test.cc` and `ift/encoder/closure_glyph_segmenter_test.cc` verifying that `--experimental_all_pairs_merging=true` avoids single-base greedy traps, handles composite conditions / `min_group_size` / `UNICODE_RANGE` mode, and is deterministic. From b6c2f85524703b522acc4f53a6d9908f8c56d9ef Mon Sep 17 00:00:00 2001 From: Garret Rieger Date: Mon, 19 Oct 2026 22:47:00 +0000 Subject: [PATCH 02/13] Add fast path for computing cost deltas with two inert segments. --- docs/all_pairs_merging.md | 2 +- ift/encoder/BUILD | 1 + ift/encoder/candidate_merge.cc | 154 +++++++++++++++++++++++++--- ift/encoder/candidate_merge_test.cc | 148 ++++++++++++++++++++++++++ ift/encoder/merger.cc | 113 +++++++++++++++++--- ift/encoder/merger.h | 76 +++++++++++++- ift/encoder/mock_patch_size_cache.h | 5 + 7 files changed, 470 insertions(+), 29 deletions(-) diff --git a/docs/all_pairs_merging.md b/docs/all_pairs_merging.md index 00b66ff6..25551222 100644 --- a/docs/all_pairs_merging.md +++ b/docs/all_pairs_merging.md @@ -130,7 +130,7 @@ To handle this cleanly without changing `TryNextMerge()`'s external contract: ## Implementation Plan / TODO -- [ ] **Commit 1: Add `--experimental_all_pairs_merging` flag, per-segment cost metadata cache, and inert 2-segment fast-path.** Declare `ABSL_DECLARE_FLAG(bool, experimental_all_pairs_merging)` in `ift/encoder/merger.h` (default `false`), add cached per-segment aggregate probability and exclusive patch size vectors on `Merger`, and add an $O(1)$ pre-check + fast-path for 2-segment inert cost delta evaluations in `CandidateMerge::ComputeCostDelta` (with unit tests in `ift/encoder/candidate_merge_test.cc`). +- [x] **Commit 1: Add `--experimental_all_pairs_merging` flag, per-segment cost metadata cache, and inert 2-segment fast-path.** Declare `ABSL_DECLARE_FLAG(bool, experimental_all_pairs_merging)` in `ift/encoder/merger.h` (default `false`), add cached per-segment aggregate probability and exclusive patch size vectors on `Merger`, and add an $O(1)$ pre-check + fast-path for 2-segment inert cost delta evaluations in `CandidateMerge::ComputeCostDelta` (with unit tests in `ift/encoder/candidate_merge_test.cc`). - [ ] **Commit 2: Implement the lazy two-stage all-pairs priority queue and probability-ordered candidate generation in `Merger`.** Add `QueuedCandidate`, probability-ordered active segment tracking with `BestCaseInertProbabilityThreshold` row pruning, composite condition candidate queuing, and the lazy `BEST_CASE` $\rightarrow$ `EXACT` pop-and-refine loop for Phase 1 cost-reducing merges. - [ ] **Commit 3: Implement epoch-based invalidation and incremental `dirty_segments_` refresh across `TryNextMerge()` calls.** Track affected segments for both inert and non-inert merges, refresh `dirty_segments_` at the start of `TryNextMerge()` after `SegmentationContext::ReprocessChanged()`, and reset all-pairs state in `Merger::ReassignInitSubset()`. - [ ] **Commit 4: Integrate Phase 2 (`min_group_size` enforcement) and add unit and determinism tests.** Transition to Phase 2 for any undersized segments remaining after Phase 1 completes, and add tests in `ift/encoder/candidate_merge_test.cc` and `ift/encoder/closure_glyph_segmenter_test.cc` verifying that `--experimental_all_pairs_merging=true` avoids single-base greedy traps, handles composite conditions / `min_group_size` / `UNICODE_RANGE` mode, and is deterministic. diff --git a/ift/encoder/BUILD b/ift/encoder/BUILD index ee2d58e4..79822a93 100644 --- a/ift/encoder/BUILD +++ b/ift/encoder/BUILD @@ -131,6 +131,7 @@ cc_library( "@abseil-cpp//absl/status:statusor", "@abseil-cpp//absl/strings", "@abseil-cpp//absl/strings:str_format", + "@abseil-cpp//absl/types:span", "@harfbuzz", ], ) diff --git a/ift/encoder/candidate_merge.cc b/ift/encoder/candidate_merge.cc index 70118b61..a29e47f6 100644 --- a/ift/encoder/candidate_merge.cc +++ b/ift/encoder/candidate_merge.cc @@ -1,6 +1,7 @@ #include "ift/encoder/candidate_merge.h" #include +#include #include #include #include @@ -11,6 +12,7 @@ #include "absl/log/log.h" #include "absl/status/status.h" #include "absl/status/statusor.h" +#include "absl/types/span.h" #include "ift/common/compat_id.h" #include "ift/common/font_data.h" #include "ift/common/font_helper.h" @@ -43,7 +45,7 @@ using ift::common::GlyphSet; using ift::common::SegmentSet; using ift::common::Woff2; -static double SumDeltas(std::vector& deltas) { +static double SumDeltas(absl::Span deltas) { // Delta lists often come from hash containers which do not have stable // ordering. Since floating point addition is not associative we need // to first sort into a determnistic order or risk the sum result being @@ -67,6 +69,21 @@ static double SumDeltas(std::vector& deltas) { return total; } +static double SumDeltas(std::vector& deltas) { + return SumDeltas(absl::MakeSpan(deltas)); +} + +// Bounds the maximum possible floating-point rounding discrepancy between an +// unsorted running sum (current_delta) of up to max_terms elements and the +// final sorted sum produced by SumDeltas(). +static double MaxSumError(size_t max_terms, double initial_magnitude_sum, + double initial_delta, double current_delta) { + double magnitude_sum = + initial_magnitude_sum + (current_delta - initial_delta); + return 2.0 * max_terms * std::numeric_limits::epsilon() * + magnitude_sum; +} + StatusOr CandidateMerge::IsPatchTooSmall( Merger& merger, segment_index_t base_segment_index, const GlyphSet& glyphs) { @@ -176,6 +193,7 @@ Status CandidateMerge::ApplyPatchMerge(Merger& merger) { // CombinePatches() will do invalidation as needed, so nothing else needs to // be done to apply this merge. + merger.InvalidateExclusivePatchSizes(); return merger.Context().glyph_groupings.CombinePatches(*patch_merge_glyphs_, {}); } @@ -618,6 +636,104 @@ StatusOr CandidateMerge::ComputeBestCaseInitFontCostDelta( return cost_delta; } +template +static StatusOr ComputeInertPairCostDelta( + Merger& merger, const SegmentSet& merged_segments, segment_index_t u, + segment_index_t v, uint32_t size_u, uint32_t size_v, + const std::optional& exclusive_gids, + std::optional delta_to_beat) { + const auto& context = merger.Context(); + const auto& patch_size_cache = context.patch_size_cache; + const uint32_t per_request_overhead = merger.Strategy().NetworkOverheadCost(); + const uint32_t base_overhead = TRY(merger.BasePatchOverhead()); + + double prob_u = TRY(merger.AggregateProbability(u)); + double prob_v = TRY(merger.AggregateProbability(v)); + double d_u = prob_u * (size_u + per_request_overhead); + double d_v = prob_v * (size_v + per_request_overhead); + + uint32_t inc_u = (size_u > base_overhead) ? (size_u - base_overhead) : 0; + uint32_t inc_v = (size_v > base_overhead) ? (size_v - base_overhead) : 0; + uint32_t largest_patch_size = std::max(size_u, size_v); + uint32_t extra_raw = std::min(inc_u, inc_v); + + uint32_t size = 0; + if (best_case) { + double best_case_reduction_fraction = + merger.Strategy().BestCaseSizeReductionFraction(); + uint32_t extra = + std::max((uint32_t)(extra_raw * best_case_reduction_fraction), + Merger::BEST_CASE_MERGE_SIZE_DELTA); + size = largest_patch_size + extra; + } else { + const GlyphSet* glyphs_for_size = nullptr; + GlyphSet combined_glyphs; + const auto& glyphs_u = context.glyph_groupings.ExclusiveGlyphs(u); + const auto& glyphs_v = context.glyph_groupings.ExclusiveGlyphs(v); + if (exclusive_gids.has_value() && glyphs_u.is_subset_of(*exclusive_gids) && + glyphs_v.is_subset_of(*exclusive_gids)) { + glyphs_for_size = &*exclusive_gids; + } else { + combined_glyphs = glyphs_u; + combined_glyphs.union_set(glyphs_v); + if (exclusive_gids.has_value()) { + combined_glyphs.union_set(*exclusive_gids); + } + glyphs_for_size = &combined_glyphs; + } + if (glyphs_for_size->empty()) { + std::array deltas = {-d_u, -d_v}; + return SumDeltas(absl::MakeSpan(deltas)); + } + size = TRY(patch_size_cache->GetPatchSize(*glyphs_for_size)); + if (merger.ShouldRecordMergedSizeReductions() && extra_raw != 0) { + int32_t extra_actual = ((int32_t)size) - largest_patch_size; + merger.RecordMergedSizeReduction((double)extra_actual / + (double)extra_raw); + } + } + + double s = (size + per_request_overhead); + double initial_delta = -d_u - d_v; + double initial_magnitude_sum = -initial_delta; + + if (best_case && delta_to_beat.has_value()) { + // We can cheaply compute a lower bound to the post merge probability. + // Since delta = added - removed, delta will be most negative + // with a lower probability for the merged patch. Thus if + // if the lower bound cannot beat the threshold then this case + // can be rejected. + double min_prob_u = TRY(merger.MinAggregateProbability(u)); + double min_prob_v = TRY(merger.MinAggregateProbability(v)); + double min_merged_prob = std::max(min_prob_u, min_prob_v); + double min_d_merged = min_merged_prob * s; + double min_current_delta = initial_delta + min_d_merged; + if (min_current_delta > *delta_to_beat) { + double max_error = MaxSumError(3, initial_magnitude_sum, initial_delta, + min_current_delta); + if (min_current_delta - max_error > *delta_to_beat) { + return absl::CancelledError(); + } + } + } + + double p = TRY(merger.AggregateMergedProbability(merged_segments)); + double d_merged = p * s; + double current_delta = initial_delta + d_merged; + if (delta_to_beat.has_value() && current_delta > *delta_to_beat) { + double max_error = + MaxSumError(3, initial_magnitude_sum, initial_delta, current_delta); + if (current_delta - max_error > *delta_to_beat) { + return absl::CancelledError(); + } + } + + std::array deltas = {-d_u, -d_v, d_merged}; + double cost_delta = SumDeltas(absl::MakeSpan(deltas)); + VLOG(1) << " = " << cost_delta; + return cost_delta; +} + template StatusOr CandidateMerge::ComputeCostDelta( Merger& merger, const SegmentSet& merged_segments, @@ -628,6 +744,28 @@ StatusOr CandidateMerge::ComputeCostDelta( VLOG(1) << "cost delta for merge of " << merged_segments.ToString() << " = "; + const auto& context = merger.Context(); + const bool unicode_range_mode = + (context.GetTargetMode() == config::UNICODE_RANGE); + + if (merged_segments.size() == 2) { + segment_index_t u = *merged_segments.min(); + segment_index_t v = *merged_segments.max(); + if (context.InertSegments().contains(u) && + context.InertSegments().contains(v)) { + uint32_t size_u = TRY(merger.ExclusivePatchSize(u)); + uint32_t size_v = TRY(merger.ExclusivePatchSize(v)); + if (size_u > 0 && size_v > 0 && + (best_case || unicode_range_mode || !exclusive_gids.has_value() || + !context.glyph_groupings.UnmappedGlyphs().intersects( + *exclusive_gids))) { + return ComputeInertPairCostDelta( + merger, merged_segments, u, v, size_u, size_v, exclusive_gids, + delta_to_beat); + } + } + } + // These are conditions which will be modified by applying the merge. These // are conditions that intersect merged_segments. // @@ -653,12 +791,8 @@ StatusOr CandidateMerge::ComputeCostDelta( new_conditions.reserve(modified_conditions.size()); segment_index_t base = *merged_segments.min(); - const auto& context = merger.Context(); - const bool unicode_range_mode = - (context.GetTargetMode() == config::UNICODE_RANGE); const auto& patch_size_cache = context.patch_size_cache; - uint32_t base_overhead = - unicode_range_mode ? TRY(patch_size_cache->GetPatchSize({})) : 0; + uint32_t base_overhead = TRY(merger.BasePatchOverhead()); // Cost delta contributions are collected here and summed at the end rather // than accumulated as we go. modified_conditions and new_conditions are both @@ -795,12 +929,8 @@ StatusOr CandidateMerge::ComputeCostDelta( current_delta += d; if (delta_to_beat.has_value() && current_delta > *delta_to_beat) { - // Bound the maximum possible FP rounding discrepancy between the unsorted - // running sum (current_delta) and the final sorted sum (SumDeltas). - double magnitude_sum = - initial_magnitude_sum + (current_delta - initial_delta); - double max_error = 2.0 * deltas.capacity() * - std::numeric_limits::epsilon() * magnitude_sum; + double max_error = MaxSumError(deltas.capacity(), initial_magnitude_sum, + initial_delta, current_delta); if (current_delta - max_error > *delta_to_beat) { return absl::CancelledError(); } diff --git a/ift/encoder/candidate_merge_test.cc b/ift/encoder/candidate_merge_test.cc index 40b1f8bb..66b8a027 100644 --- a/ift/encoder/candidate_merge_test.cc +++ b/ift/encoder/candidate_merge_test.cc @@ -1628,4 +1628,152 @@ TEST_F(CandidateMergeTest, g_grave); } +TEST_F(CandidateMergeTest, ComputeCostDelta_InertPairFastPathAndPreCheck) { + // s0: 'a' -> {69}, s1: 'b' -> {70}, s2: 'c' -> {71} (all inert in Roboto). + std::vector segments = { + {{'a'}}, + {{'b'}}, + {{'c'}}, + }; + freq::UnicodeFrequencies frequencies{ + {{' ', ' '}, 100}, {{'a', 'a'}, 80}, {{'b', 'b'}, 5}, {{'c', 'c'}, 75}}; + + ClosureGlyphSegmenter segmenter(8, 8, PATCH, CLOSURE_ONLY, resolver); + auto context = SegmentationContext::InitializeSegmentationContext( + roboto.get(), {}, segments, segmenter.unmapped_glyph_handling(), + segmenter.condition_analysis_mode(), segmenter.brotli_quality(), + segmenter.init_font_merging_brotli_quality(), resolver); + ASSERT_TRUE(context.ok()) << context.status(); + ASSERT_TRUE(context->InertSegments().contains(0)); + ASSERT_TRUE(context->InertSegments().contains(1)); + ASSERT_TRUE(context->InertSegments().contains(2)); + + Merger merger = *Merger::New( + *context, + MergeStrategy::CostBased( + *ProbabilityProfile::Unigram(std::move(frequencies)), 75, 4), + all, all); + + MockPatchSizeCache* size_cache = new MockPatchSizeCache(); + size_cache->SetPatchSize({69}, 200); + size_cache->SetPatchSize({70}, 300); + size_cache->SetPatchSize({71}, 240); + size_cache->SetPatchSize({69, 71}, 320); + context->patch_size_cache.reset(size_cache); + + // First evaluation warms the per-segment exclusive patch size cache for s0 + // and s2. + size_cache->ResetCallCount(); + auto best_case_0_2 = CandidateMerge::ComputeCostDelta( + merger, {0, 2}, std::nullopt, std::nullopt); + ASSERT_TRUE(best_case_0_2.ok()) << best_case_0_2.status(); + EXPECT_EQ(size_cache->GetPatchSizeCallCount(), 2); + + // Expected best-case merged size for {0, 2}: max(200, 240) + min(200, 240)/2 + // = 240 + 100 = 340. + double p_0_or_2 = 0.80 + 0.75 - 0.80 * 0.75; + double expected_best_0_2 = + -0.80 * (200 + 75) - 0.75 * (240 + 75) + p_0_or_2 * (340 + 75); + EXPECT_NEAR(*best_case_0_2, expected_best_0_2, 1e-9); + + // Subsequent best-case evaluation of {0, 2} hits the per-segment size cache + // with zero calls to PatchSizeCache::GetPatchSize. + size_cache->ResetCallCount(); + auto best_case_0_2_cached = CandidateMerge::ComputeCostDelta( + merger, {0, 2}, std::nullopt, std::nullopt); + ASSERT_TRUE(best_case_0_2_cached.ok()) << best_case_0_2_cached.status(); + EXPECT_EQ(*best_case_0_2_cached, *best_case_0_2); + EXPECT_EQ(size_cache->GetPatchSizeCallCount(), 0); + + // Evaluating {0, 1} with delta_to_beat = 0.0: s0 (p=0.80, size=200) and + // s1 (p=0.05, size=300) has best-case merged size 300 + 100 = 400. Even at + // the lower-bound probability max(0.80, 0.05) = 0.80, the delta is + // 0.80*(400+75) - 0.80*(200+75) - 0.05*(300+75) = 160 - 18.75 = +141.25 > 0, + // so the O(1) pre-check cancels immediately. + auto cancelled_precheck = + CandidateMerge::ComputeCostDelta(merger, {0, 1}, std::nullopt, 0.0); + EXPECT_TRUE(absl::IsCancelled(cancelled_precheck.status())); + + // Exact evaluation of {0, 2} via the inert fast-path. + GlyphSet exclusive_0_2 = {69, 71}; + auto exact_0_2 = CandidateMerge::ComputeCostDelta( + merger, {0, 2}, exclusive_0_2, std::nullopt); + ASSERT_TRUE(exact_0_2.ok()) << exact_0_2.status(); + double expected_exact_0_2 = + -0.80 * (200 + 75) - 0.75 * (240 + 75) + p_0_or_2 * (320 + 75); + EXPECT_NEAR(*exact_0_2, expected_exact_0_2, 1e-9); + + // Boundary check: 1 ULP below exact_0_2 does not cancel early. + auto boundary_0_2 = CandidateMerge::ComputeCostDelta( + merger, {0, 2}, exclusive_0_2, std::nextafter(*exact_0_2, -INFINITY)); + ASSERT_TRUE(boundary_0_2.ok()) << boundary_0_2.status(); + EXPECT_EQ(*boundary_0_2, *exact_0_2); + + // Applying the inert merge of {0, 2} invalidates s0's cached metadata while + // keeping s1's cached exclusive patch size intact. + auto candidate = + CandidateMerge::AssessSegmentMerge(merger, 0, {2}, std::nullopt); + ASSERT_TRUE(candidate.ok()) << candidate.status(); + ASSERT_TRUE(candidate->has_value()); + ASSERT_TRUE((*candidate)->Apply(merger).ok()); + + size_cache->ResetCallCount(); + EXPECT_EQ(*merger.ExclusivePatchSize(1), 300u); + EXPECT_EQ(size_cache->GetPatchSizeCallCount(), 0); + EXPECT_EQ(*merger.ExclusivePatchSize(0), 320u); + EXPECT_EQ(size_cache->GetPatchSizeCallCount(), 1); + EXPECT_NEAR(*merger.AggregateProbability(0), p_0_or_2, 1e-9); +} + +TEST_F(CandidateMergeTest, + ComputeCostDelta_InertPairFastPath_UnicodeRangeMode) { + std::vector segments = { + {{'a'}}, // gid 69 + {{'c'}}, // gid 71 + }; + freq::UnicodeFrequencies frequencies{ + {{' ', ' '}, 100}, {{'a', 'a'}, 80}, {{'c', 'c'}, 75}}; + + ClosureGlyphSegmenter segmenter(8, 8, PATCH, CLOSURE_ONLY, resolver, + ift::config::UNICODE_RANGE); + auto context = SegmentationContext::InitializeSegmentationContext( + roboto.get(), {}, segments, segmenter.unmapped_glyph_handling(), + segmenter.condition_analysis_mode(), segmenter.brotli_quality(), + segmenter.init_font_merging_brotli_quality(), resolver, + ift::config::UNICODE_RANGE); + ASSERT_TRUE(context.ok()) << context.status(); + + Merger merger = *Merger::New( + *context, + MergeStrategy::CostBased( + *ProbabilityProfile::Unigram(std::move(frequencies)), 75, 4), + all, all); + + MockPatchSizeCache* size_cache = new MockPatchSizeCache(); + size_cache->SetPatchSize({}, 100); + size_cache->SetPatchSize({69}, 200); + size_cache->SetPatchSize({71}, 240); + size_cache->SetPatchSize({69, 71}, 320); + context->patch_size_cache.reset(size_cache); + + // In UNICODE_RANGE mode, base_overhead = 100 is subtracted before computing + // incremental sizes: inc_0 = 200 - 100 = 100, inc_1 = 240 - 100 = 140, + // extra = min(100, 140) * 0.5 = 50, so best-case merged size = 240 + 50 = 290. + double p_0_or_1 = 0.80 + 0.75 - 0.80 * 0.75; + double expected_best = + -0.80 * (200 + 75) - 0.75 * (240 + 75) + p_0_or_1 * (290 + 75); + auto best_delta = CandidateMerge::ComputeCostDelta( + merger, {0, 1}, std::nullopt, std::nullopt); + ASSERT_TRUE(best_delta.ok()) << best_delta.status(); + EXPECT_NEAR(*best_delta, expected_best, 1e-9); + + GlyphSet exclusive_gids = {69, 71}; + double expected_exact = + -0.80 * (200 + 75) - 0.75 * (240 + 75) + p_0_or_1 * (320 + 75); + auto exact_delta = CandidateMerge::ComputeCostDelta( + merger, {0, 1}, exclusive_gids, std::nullopt); + ASSERT_TRUE(exact_delta.ok()) << exact_delta.status(); + EXPECT_NEAR(*exact_delta, expected_exact, 1e-9); +} + } // namespace ift::encoder diff --git a/ift/encoder/merger.cc b/ift/encoder/merger.cc index f4fd96c1..918c7f44 100644 --- a/ift/encoder/merger.cc +++ b/ift/encoder/merger.cc @@ -25,23 +25,80 @@ ABSL_FLAG(bool, record_merged_size_reductions, false, "When enabled the merger will record the percent size reductions of " "each assessed merge."); +ABSL_FLAG(bool, experimental_all_pairs_merging, false, + "When enabled the cost-based merger considers all segment pairs " + "globally instead of only pairs with the current highest probability " + "base segment."); + namespace ift::encoder { bool Merger::ShouldRecordMergedSizeReductions() const { return absl::GetFlag(FLAGS_record_merged_size_reductions); } -StatusOr Merger::AggregateProbability( +bool Merger::UseAllPairsMerging() const { + return strategy_.UseCosts() && + absl::GetFlag(FLAGS_experimental_all_pairs_merging); +} + +StatusOr Merger::ComputeAndCacheAggregateProbability( segment_index_t segment_index) const { double total = 0.0; + double min_total = 0.0; + const auto& segments = Context().SegmentationInfo().Segments(); for (const auto& profile : strategy_.ProbabilityProfiles()) { - total += TRY(profile.Calculator()) - ->ComputeProbability(Context().SegmentationInfo().Segments(), segment_index) - .Value(); + auto bound = + TRY(profile.Calculator())->ComputeProbability(segments, segment_index); + total += bound.Value(); + min_total += bound.Min(); + } + if (segment_index < cached_agg_prob_.size()) { + cached_agg_prob_[segment_index] = total; + cached_min_agg_prob_[segment_index] = min_total; } return total; } +void Merger::EnsurePatchSizeCacheValid() const { + if (exclusive_sizes_dirty_ || + cached_patch_size_cache_ != context_->patch_size_cache.get()) { + size_t num_segments = Context().SegmentationInfo().Segments().size(); + cached_exclusive_size_.assign(num_segments, UINT32_MAX); + cached_base_overhead_ = std::nullopt; + cached_patch_size_cache_ = context_->patch_size_cache.get(); + exclusive_sizes_dirty_ = false; + } +} + +StatusOr Merger::ComputeAndCacheExclusivePatchSize( + segment_index_t segment_index) const { + EnsurePatchSizeCacheValid(); + if (segment_index < cached_exclusive_size_.size() && + cached_exclusive_size_[segment_index] != UINT32_MAX) { + return cached_exclusive_size_[segment_index]; + } + + const auto& glyphs = context_->glyph_groupings.ExclusiveGlyphs(segment_index); + uint32_t size = 0; + if (!glyphs.empty()) { + size = TRY(context_->patch_size_cache->GetPatchSize(glyphs)); + } + if (segment_index < cached_exclusive_size_.size()) { + cached_exclusive_size_[segment_index] = size; + } + return size; +} + +StatusOr Merger::ComputeAndCacheBasePatchOverhead() const { + EnsurePatchSizeCacheValid(); + if (cached_base_overhead_.has_value()) { + return *cached_base_overhead_; + } + uint32_t overhead = TRY(context_->patch_size_cache->GetPatchSize({})); + cached_base_overhead_ = overhead; + return overhead; +} + StatusOr Merger::AggregateProbability( const ActivationCondition& condition) const { const auto& segments = Context().SegmentationInfo().Segments(); @@ -52,6 +109,18 @@ StatusOr Merger::AggregateProbability( return total; } +StatusOr Merger::AggregateMergedProbability( + const SegmentSet& merged_segments) const { + const auto& segments = Context().SegmentationInfo().Segments(); + double total = 0.0; + for (const auto& profile : strategy_.ProbabilityProfiles()) { + total += TRY(profile.Calculator()) + ->ComputeMergedProbability(segments, merged_segments) + .Value(); + } + return total; +} + StatusOr Merger::AggregateMergedProbability( const ActivationCondition& condition, segment_index_t merged_segment_index, @@ -336,6 +405,12 @@ Status Merger::MoveSegmentsToInitFont(size_t profile_index) { Status Merger::ResetSegmentProbabilities() const { size_t num_segments = Context().SegmentationInfo().Segments().size(); + cached_agg_prob_.assign(num_segments, -1.0); + cached_min_agg_prob_.assign(num_segments, -1.0); + cached_exclusive_size_.assign(num_segments, UINT32_MAX); + cached_base_overhead_ = std::nullopt; + cached_patch_size_cache_ = context_->patch_size_cache.get(); + exclusive_sizes_dirty_ = false; return strategy_.ResetSegmentProbabilities(num_segments); } @@ -360,6 +435,18 @@ uint32_t Merger::AssignMergedSegment(segment_index_t base, (*calc)->InvalidateSegmentProbabilities(invalidated); } } + for (segment_index_t s : invalidated) { + if (s < cached_agg_prob_.size()) { + cached_agg_prob_[s] = -1.0; + cached_min_agg_prob_[s] = -1.0; + } + if (s < cached_exclusive_size_.size()) { + cached_exclusive_size_[s] = UINT32_MAX; + } + } + if (!is_inert) { + exclusive_sizes_dirty_ = true; + } candidate_segments_.subtract(to_merge); candidate_segments_.insert(base); return Context().AssignMergedSegment(base, to_merge, merged_segment, @@ -425,12 +512,12 @@ StatusOr Merger::ComputeSegmentCutoff() const { double total_cost = 0.0; double overhead = strategy_.NetworkOverheadCost(); for (segment_index_t s : candidate_segments_) { - auto segment_glyphs = context_->glyph_groupings.ExclusiveGlyphs(s); - if (segment_glyphs.empty()) { + uint32_t patch_size = TRY(ExclusivePatchSize(s)); + if (patch_size == 0) { continue; } - double size = TRY(context_->patch_size_cache->GetPatchSize(segment_glyphs)); + double size = patch_size; double probability = TRY(AggregateProbability(s)); total_cost += probability * (size + overhead); } @@ -440,12 +527,12 @@ StatusOr Merger::ComputeSegmentCutoff() const { for (auto it = candidate_segments_.rbegin(); it != candidate_segments_.rend(); it++) { segment_index_t s = *it; - auto segment_glyphs = context_->glyph_groupings.ExclusiveGlyphs(s); - if (segment_glyphs.empty()) { + uint32_t patch_size = TRY(ExclusivePatchSize(s)); + if (patch_size == 0) { continue; } - double size = TRY(context_->patch_size_cache->GetPatchSize(segment_glyphs)); + double size = patch_size; double probability = TRY(AggregateProbability(s)); cutoff_tail_cost -= probability * (size + overhead); if (cutoff_tail_cost < 0.0) { @@ -632,11 +719,7 @@ double Merger::BestCaseInertProbabilityThreshold( Status Merger::CollectExclusiveCandidateMerges( uint32_t base_segment_index, std::optional& smallest_candidate_merge) { - auto base_glyphs = - context_->glyph_groupings.ExclusiveGlyphs(base_segment_index); - uint32_t base_size = - TRY(Context().patch_size_cache->GetPatchSize(base_glyphs)); - + uint32_t base_size = TRY(ExclusivePatchSize(base_segment_index)); double base_probability = TRY(AggregateProbability(base_segment_index)); double inert_threshold = -1.0; diff --git a/ift/encoder/merger.h b/ift/encoder/merger.h index 1c5b9e17..e0f3b9f6 100644 --- a/ift/encoder/merger.h +++ b/ift/encoder/merger.h @@ -2,7 +2,10 @@ #define IFT_ENCODER_MERGER_H_ #include +#include +#include +#include "absl/flags/declare.h" #include "ift/common/int_set.h" #include "ift/encoder/activation_condition.h" #include "ift/encoder/candidate_merge.h" @@ -11,6 +14,8 @@ #include "ift/encoder/segmentation_context.h" #include "ift/encoder/types.h" +ABSL_DECLARE_FLAG(bool, experimental_all_pairs_merging); + namespace ift::encoder { // Finds and merge segments/patches from a in progress segmentation. @@ -94,6 +99,8 @@ class Merger { bool ShouldRecordMergedSizeReductions() const; + bool UseAllPairsMerging() const; + void LogMergedSizeHistogram() const; // The total cost of a segmentation is defined as the sum of the costs @@ -118,12 +125,64 @@ class Merger { // Computes the aggregate probability for the segment at segment_index, using // cached segment probabilities when available. absl::StatusOr AggregateProbability( - segment_index_t segment_index) const; + segment_index_t segment_index) const { + if (segment_index < cached_agg_prob_.size() && + cached_agg_prob_[segment_index] >= 0.0) { + return cached_agg_prob_[segment_index]; + } + return ComputeAndCacheAggregateProbability(segment_index); + } + + // Computes the lower-bound aggregate probability (sum of ProbabilityBound::Min + // across profiles) for the segment at segment_index, using cached segment + // probabilities when available. + absl::StatusOr MinAggregateProbability( + segment_index_t segment_index) const { + if (segment_index < cached_min_agg_prob_.size() && + cached_min_agg_prob_[segment_index] >= 0.0) { + return cached_min_agg_prob_[segment_index]; + } + TRYV(ComputeAndCacheAggregateProbability(segment_index)); + return cached_min_agg_prob_.at(segment_index); + } + + // Returns the cached patch size in bytes of the exclusive glyph set for + // segment_index (or 0 if segment_index has no exclusive glyphs). + absl::StatusOr ExclusivePatchSize( + segment_index_t segment_index) const { + if (!exclusive_sizes_dirty_ && + cached_patch_size_cache_ == context_->patch_size_cache.get() && + segment_index < cached_exclusive_size_.size() && + cached_exclusive_size_[segment_index] != UINT32_MAX) { + return cached_exclusive_size_[segment_index]; + } + return ComputeAndCacheExclusivePatchSize(segment_index); + } + + // Returns the base patch overhead in bytes (0 in IFT mode; the cached empty + // subset WOFF2 size in UNICODE_RANGE mode). + absl::StatusOr BasePatchOverhead() const { + if (context_->GetTargetMode() != config::UNICODE_RANGE) { + return 0; + } + if (!exclusive_sizes_dirty_ && + cached_patch_size_cache_ == context_->patch_size_cache.get() && + cached_base_overhead_.has_value()) { + return *cached_base_overhead_; + } + return ComputeAndCacheBasePatchOverhead(); + } + + void InvalidateExclusivePatchSizes() { exclusive_sizes_dirty_ = true; } // Computes the aggregate probability of condition being activated. absl::StatusOr AggregateProbability( const ActivationCondition& condition) const; + // Computes the aggregate probability of the union of merged_segments. + absl::StatusOr AggregateMergedProbability( + const ift::common::SegmentSet& merged_segments) const; + // Same as AggregateProbability(condition), but computes the probability of // condition as it would be if merged_segment_index has been replaced by // the union of merged_segments. @@ -162,6 +221,12 @@ class Merger { absl::StatusOr> TryNextPatchMerge(); absl::Status ResetSegmentProbabilities() const; + absl::StatusOr ComputeAndCacheAggregateProbability( + segment_index_t segment_index) const; + absl::StatusOr ComputeAndCacheExclusivePatchSize( + segment_index_t segment_index) const; + absl::StatusOr ComputeAndCacheBasePatchOverhead() const; + void EnsurePatchSizeCacheValid() const; absl::Status InitOptimizationCutoff(); absl::StatusOr ComputeSegmentCutoff() const; @@ -273,6 +338,15 @@ class Merger { // Percent reduction of data beyond the single largest input patch. absl::btree_map merged_size_reduction_histogram_; + + // Per-segment cached aggregate probability, lower-bound aggregate + // probability, and exclusive patch size in bytes. + mutable std::vector cached_agg_prob_; + mutable std::vector cached_min_agg_prob_; + mutable std::vector cached_exclusive_size_; + mutable std::optional cached_base_overhead_; + mutable const PatchSizeCache* cached_patch_size_cache_ = nullptr; + mutable bool exclusive_sizes_dirty_ = true; }; } // namespace ift::encoder diff --git a/ift/encoder/mock_patch_size_cache.h b/ift/encoder/mock_patch_size_cache.h index 3bd6db20..4faa35e3 100644 --- a/ift/encoder/mock_patch_size_cache.h +++ b/ift/encoder/mock_patch_size_cache.h @@ -11,6 +11,7 @@ class MockPatchSizeCache : public PatchSizeCache { public: absl::StatusOr GetPatchSize( const ift::common::GlyphSet& gids) override { + call_count_++; auto it = patch_sizes_.find(gids); if (it != patch_sizes_.end()) { VLOG(0) << "GetPatchSize(" << gids.ToString() << ") = " << it->second; @@ -27,8 +28,12 @@ class MockPatchSizeCache : public PatchSizeCache { patch_sizes_[gids] = size; } + uint64_t GetPatchSizeCallCount() const { return call_count_; } + void ResetCallCount() { call_count_ = 0; } + private: absl::flat_hash_map patch_sizes_; + uint64_t call_count_ = 0; }; } // namespace ift::encoder From 49c7310b7602bcd87c2214e0f9f0c98e925da2d9 Mon Sep 17 00:00:00 2001 From: Garret Rieger Date: Tue, 20 Oct 2026 01:14:58 +0000 Subject: [PATCH 03/13] Start implementation of all pairs merging. - Adds new AllPairsMerger which tracks all pairs merging specific state. - Does not yet include invalidation logic. --- docs/all_pairs_merging.md | 2 +- ift/encoder/BUILD | 2 + ift/encoder/all_pairs_merger.cc | 259 ++++++++++++++++++++++++++++++++ ift/encoder/all_pairs_merger.h | 113 ++++++++++++++ ift/encoder/candidate_merge.cc | 7 +- ift/encoder/candidate_merge.h | 6 + ift/encoder/merger.cc | 33 ++-- ift/encoder/merger.h | 55 ++++--- 8 files changed, 436 insertions(+), 41 deletions(-) create mode 100644 ift/encoder/all_pairs_merger.cc create mode 100644 ift/encoder/all_pairs_merger.h diff --git a/docs/all_pairs_merging.md b/docs/all_pairs_merging.md index 25551222..aa7591ea 100644 --- a/docs/all_pairs_merging.md +++ b/docs/all_pairs_merging.md @@ -131,6 +131,6 @@ To handle this cleanly without changing `TryNextMerge()`'s external contract: ## Implementation Plan / TODO - [x] **Commit 1: Add `--experimental_all_pairs_merging` flag, per-segment cost metadata cache, and inert 2-segment fast-path.** Declare `ABSL_DECLARE_FLAG(bool, experimental_all_pairs_merging)` in `ift/encoder/merger.h` (default `false`), add cached per-segment aggregate probability and exclusive patch size vectors on `Merger`, and add an $O(1)$ pre-check + fast-path for 2-segment inert cost delta evaluations in `CandidateMerge::ComputeCostDelta` (with unit tests in `ift/encoder/candidate_merge_test.cc`). -- [ ] **Commit 2: Implement the lazy two-stage all-pairs priority queue and probability-ordered candidate generation in `Merger`.** Add `QueuedCandidate`, probability-ordered active segment tracking with `BestCaseInertProbabilityThreshold` row pruning, composite condition candidate queuing, and the lazy `BEST_CASE` $\rightarrow$ `EXACT` pop-and-refine loop for Phase 1 cost-reducing merges. +- [x] **Commit 2: Implement the lazy two-stage all-pairs priority queue and probability-ordered candidate generation in `AllPairsMerger`.** Add `AllPairsMerger` (`ift/encoder/all_pairs_merger.h`/`.cc`) with `QueuedCandidate`, probability-ordered active segment tracking with `BestCaseInertProbabilityThreshold` row pruning, composite condition candidate queuing, and the lazy `BEST_CASE` $\rightarrow$ `EXACT` pop-and-refine loop for Phase 1 cost-reducing merges. - [ ] **Commit 3: Implement epoch-based invalidation and incremental `dirty_segments_` refresh across `TryNextMerge()` calls.** Track affected segments for both inert and non-inert merges, refresh `dirty_segments_` at the start of `TryNextMerge()` after `SegmentationContext::ReprocessChanged()`, and reset all-pairs state in `Merger::ReassignInitSubset()`. - [ ] **Commit 4: Integrate Phase 2 (`min_group_size` enforcement) and add unit and determinism tests.** Transition to Phase 2 for any undersized segments remaining after Phase 1 completes, and add tests in `ift/encoder/candidate_merge_test.cc` and `ift/encoder/closure_glyph_segmenter_test.cc` verifying that `--experimental_all_pairs_merging=true` avoids single-base greedy traps, handles composite conditions / `min_group_size` / `UNICODE_RANGE` mode, and is deterministic. diff --git a/ift/encoder/BUILD b/ift/encoder/BUILD index 79822a93..4698b15f 100644 --- a/ift/encoder/BUILD +++ b/ift/encoder/BUILD @@ -78,6 +78,7 @@ cc_library( cc_library( name = "segmentation_context", srcs = [ + "all_pairs_merger.cc", "candidate_merge.cc", "complex_condition_finder.cc", "dependency_closure.cc", @@ -89,6 +90,7 @@ cc_library( "woff2_patch_size_cache.cc", ], hdrs = [ + "all_pairs_merger.h", "candidate_merge.h", "complex_condition_finder.h", "condition_to_glyphs_index.h", diff --git a/ift/encoder/all_pairs_merger.cc b/ift/encoder/all_pairs_merger.cc new file mode 100644 index 00000000..ebef8b92 --- /dev/null +++ b/ift/encoder/all_pairs_merger.cc @@ -0,0 +1,259 @@ +#include "ift/encoder/all_pairs_merger.h" + +#include +#include +#include +#include + +#include "absl/container/btree_set.h" +#include "absl/status/status.h" +#include "absl/status/statusor.h" +#include "ift/common/int_set.h" +#include "ift/common/try.h" +#include "ift/encoder/activation_condition.h" +#include "ift/encoder/candidate_merge.h" +#include "ift/encoder/invalidation_set.h" +#include "ift/encoder/merger.h" +#include "ift/encoder/types.h" + +using absl::btree_set; +using absl::Status; +using absl::StatusOr; +using ift::common::SegmentSet; + +namespace ift::encoder { + +void AllPairsMerger::AssignMergedSegment(segment_index_t base, + const SegmentSet& to_merge) { + if (!initialized_) { + return; + } + for (segment_index_t s : to_merge) { + RemoveActiveSegment(s); + segment_epoch_.at(s) = UINT32_MAX; + } + // base's glyph groupings are not finalized until after Apply() (and + // ReprocessChanged() for non-inert merges) completes, so remove it for now; + // the next TryNextAllPairsMerge() call will re-insert it via + // UpdateActiveSegment(). + RemoveActiveSegment(base); + segment_epoch_.at(base) = ++current_epoch_; +} + +void AllPairsMerger::RemoveActiveSegment(segment_index_t s) { + auto& prob = active_segment_prob_.at(s); + if (prob.has_value()) { + active_segments_by_prob_.erase(ProbOrderedSegment{*prob, s}); + prob = std::nullopt; + } +} + +Status AllPairsMerger::UpdateActiveSegment(Merger& merger, segment_index_t s) { + RemoveActiveSegment(s); + if (!merger.CandidateSegments().contains(s) || + s >= merger.OptimizationCutoffSegment()) { + return absl::OkStatus(); + } + uint32_t size = TRY(merger.ExclusivePatchSize(s)); + if (size == 0) { + return absl::OkStatus(); + } + double prob = TRY(merger.AggregateProbability(s)); + active_segments_by_prob_.insert(ProbOrderedSegment{prob, s}); + active_segment_prob_.at(s) = prob; + return absl::OkStatus(); +} + +Status AllPairsMerger::QueueBestCaseCandidate( + Merger& merger, const SegmentSet& merged_segments) { + if (merged_segments.size() <= 1) { + return absl::OkStatus(); + } + segment_index_t base = *merged_segments.min(); + if (CandidateMerge::WouldMixFeaturesAndCodepoints( + merger.Context().SegmentationInfo(), base, merged_segments)) { + return absl::OkStatus(); + } + + auto best_case_delta = CandidateMerge::ComputeCostDelta( + merger, merged_segments, std::nullopt, 0.0); + if (absl::IsCancelled(best_case_delta.status())) { + return absl::OkStatus(); + } + double delta = TRY(best_case_delta); + if (delta < 0.0) { + queue_.push(QueuedCandidate{ + .cost_delta = delta, + .is_exact = false, + .segments = merged_segments, + .created_epoch = current_epoch_, + .evaluated_merge = std::nullopt, + }); + } + return absl::OkStatus(); +} + +Status AllPairsMerger::QueueLowerRankedExclusivePairs( + Merger& merger, btree_set::const_iterator u_it) { + segment_index_t u = u_it->segment_index; + double u_prob = u_it->probability; + uint32_t u_size = TRY(merger.ExclusivePatchSize(u)); + double inert_threshold = + merger.BestCaseInertProbabilityThreshold(u_size, u_prob, 0.0); + + for (auto v_it = std::next(u_it); v_it != active_segments_by_prob_.end(); + ++v_it) { + segment_index_t v = v_it->segment_index; + if (merger.Context().InertSegments().contains(v) && + v_it->probability <= inert_threshold) { + // Iteration is in descending probability order, so all subsequent inert + // segments also fail the threshold. + // TODO(garretrieger): should we keep looking at non inert options beyond this + // point? Once all pairs is done, test the impact of changing this. + break; + } + TRYV(QueueBestCaseCandidate(merger, SegmentSet{u, v})); + } + + return absl::OkStatus(); +} + +Status AllPairsMerger::QueueCompositeCandidates( + Merger& merger, const std::optional& filter_segments) { + if (merger.Context().GetTargetMode() == config::UNICODE_RANGE) { + return absl::OkStatus(); + } + + ActivationCondition last_exclusive = + ActivationCondition::exclusive_segment(UINT32_MAX, 0); + + btree_set unique_merge_sets; + for (auto it = merger.Context().glyph_groupings.OrderedConditions().lower_bound( + last_exclusive); + it != merger.Context().glyph_groupings.OrderedConditions().end(); ++it) { + const auto& next_condition = *it; + if (next_condition.IsFallback() || next_condition.IsExclusive()) { + continue; + } + + SegmentSet triggering_segments = next_condition.TriggeringSegments(); + if (triggering_segments.size() <= 1) { + continue; + } + + if (filter_segments.has_value() && + !triggering_segments.intersects(*filter_segments)) { + continue; + } + + std::optional min = triggering_segments.min(); + if (min.has_value() && *min >= merger.OptimizationCutoffSegment()) { + continue; + } + + if (!triggering_segments.intersects(merger.CandidateSegments()) || + !triggering_segments.is_subset_of(merger.InscopeSegments())) { + continue; + } + + if (triggering_segments.size() == 2) { + segment_index_t u = *triggering_segments.min(); + segment_index_t v = *triggering_segments.max(); + if (IsActiveProbSegment(u) && IsActiveProbSegment(v)) { + // Pair of active exclusive segments is already handled by exclusive + // pair queuing. + continue; + } + } + + unique_merge_sets.insert(std::move(triggering_segments)); + } + + for (const auto& segments : unique_merge_sets) { + TRYV(QueueBestCaseCandidate(merger, segments)); + } + + return absl::OkStatus(); +} + +Status AllPairsMerger::InitPhase1(Merger& merger) { + size_t num_segments = merger.Context().SegmentationInfo().Segments().size(); + current_epoch_ = 0; + segment_epoch_.assign(num_segments, UINT32_MAX); + for (segment_index_t s : merger.CandidateSegments()) { + segment_epoch_.at(s) = 0; + } + active_segments_by_prob_.clear(); + active_segment_prob_.assign(num_segments, std::nullopt); + queue_ = {}; + + for (segment_index_t s : merger.CandidateSegments()) { + if (s >= merger.OptimizationCutoffSegment()) { + break; + } + TRYV(UpdateActiveSegment(merger, s)); + } + + for (auto u_it = active_segments_by_prob_.begin(); + u_it != active_segments_by_prob_.end(); ++u_it) { + TRYV(QueueLowerRankedExclusivePairs(merger, u_it)); + } + + TRYV(QueueCompositeCandidates(merger)); + initialized_ = true; + return absl::OkStatus(); +} + +bool AllPairsMerger::IsCandidateStale(const QueuedCandidate& candidate) const { + for (segment_index_t s : candidate.segments) { + if (segment_epoch_.at(s) > candidate.created_epoch) { + return true; + } + } + return false; +} + +StatusOr> AllPairsMerger::TryNextAllPairsMerge( + Merger& merger) { + if (phase_1_complete_) { + return std::nullopt; + } + + if (!initialized_) { + TRYV(InitPhase1(merger)); + } + + while (!queue_.empty()) { + QueuedCandidate top = + std::move(const_cast(queue_.top())); + queue_.pop(); + + if (IsCandidateStale(top)) { + continue; + } + + if (!top.is_exact) { + segment_index_t base = *top.segments.min(); + auto baseline = CandidateMerge::BaselineCandidate(base, 0.0); + auto candidate_merge = TRY(CandidateMerge::AssessSegmentMerge( + merger, base, top.segments, baseline)); + if (candidate_merge.has_value() && candidate_merge->CostDelta() < 0.0) { + queue_.push(QueuedCandidate{ + .cost_delta = candidate_merge->CostDelta(), + .is_exact = true, + .segments = std::move(top.segments), + .created_epoch = current_epoch_, + .evaluated_merge = std::move(*candidate_merge), + }); + } + continue; + } + + return TRY(top.evaluated_merge->Apply(merger)); + } + + phase_1_complete_ = true; + return std::nullopt; +} + +} // namespace ift::encoder diff --git a/ift/encoder/all_pairs_merger.h b/ift/encoder/all_pairs_merger.h new file mode 100644 index 00000000..0dc75eb8 --- /dev/null +++ b/ift/encoder/all_pairs_merger.h @@ -0,0 +1,113 @@ +#ifndef IFT_ENCODER_ALL_PAIRS_MERGER_H_ +#define IFT_ENCODER_ALL_PAIRS_MERGER_H_ + +#include +#include +#include +#include + +#include "absl/container/btree_set.h" +#include "absl/status/status.h" +#include "absl/status/statusor.h" +#include "ift/common/int_set.h" +#include "ift/encoder/candidate_merge.h" +#include "ift/encoder/invalidation_set.h" +#include "ift/encoder/types.h" + +namespace ift::encoder { + +class Merger; + +// Implements Phase 1 all-pairs cost-reducing segment merging for Merger using a +// lazy two-stage (best-case lower bound -> exact) priority queue. +class AllPairsMerger { + public: + absl::StatusOr> TryNextAllPairsMerge( + Merger& merger); + + void AssignMergedSegment(segment_index_t base, + const ift::common::SegmentSet& to_merge); + + void MarkFinished(segment_index_t s) { + if (!initialized_) { + return; + } + RemoveActiveSegment(s); + } + + private: + struct QueuedCandidate { + double cost_delta; + bool is_exact; + // Canonical set of all segments participating in the merge (size >= 2). + // The surviving base segment is always *segments.min(). + ift::common::SegmentSet segments; + uint32_t created_epoch; + std::optional evaluated_merge; + + bool HigherPriorityThan(const QueuedCandidate& other) const { + if (cost_delta != other.cost_delta) { + return cost_delta < other.cost_delta; + } + if (is_exact != other.is_exact) { + return is_exact && !other.is_exact; + } + return segments < other.segments; + } + + struct LowerPriority { + bool operator()(const QueuedCandidate& a, + const QueuedCandidate& b) const { + return b.HigherPriorityThan(a); + } + }; + }; + + struct ProbOrderedSegment { + double probability; + segment_index_t segment_index; + + bool operator<(const ProbOrderedSegment& other) const { + if (probability != other.probability) { + return probability > other.probability; + } + return segment_index < other.segment_index; + } + + bool operator==(const ProbOrderedSegment& other) const { + return probability == other.probability && + segment_index == other.segment_index; + } + }; + + absl::Status InitPhase1(Merger& merger); + bool IsCandidateStale(const QueuedCandidate& candidate) const; + bool IsActiveProbSegment(segment_index_t s) const { + return active_segment_prob_.at(s).has_value(); + } + absl::Status UpdateActiveSegment(Merger& merger, segment_index_t s); + void RemoveActiveSegment(segment_index_t s); + absl::Status QueueBestCaseCandidate( + Merger& merger, const ift::common::SegmentSet& merged_segments); + absl::Status QueueLowerRankedExclusivePairs( + Merger& merger, + absl::btree_set::const_iterator u_it); + absl::Status QueueCompositeCandidates( + Merger& merger, + const std::optional& filter_segments = + std::nullopt); + + bool initialized_ = false; + bool phase_1_complete_ = false; + uint32_t current_epoch_ = 0; + std::vector segment_epoch_; + absl::btree_set active_segments_by_prob_; + std::vector> active_segment_prob_; + std::priority_queue, + QueuedCandidate::LowerPriority> + queue_; +}; + +} // namespace ift::encoder + +#endif // IFT_ENCODER_ALL_PAIRS_MERGER_H_ diff --git a/ift/encoder/candidate_merge.cc b/ift/encoder/candidate_merge.cc index a29e47f6..28b4828c 100644 --- a/ift/encoder/candidate_merge.cc +++ b/ift/encoder/candidate_merge.cc @@ -198,10 +198,11 @@ Status CandidateMerge::ApplyPatchMerge(Merger& merger) { {}); } -static bool WouldMixFeaturesAndCodepoints( +bool CandidateMerge::WouldMixFeaturesAndCodepoints( const RequestedSegmentationInformation& segment_info, segment_index_t base_segment_index, const SegmentSet& segments) { - const auto& base = segment_info.Segments()[base_segment_index].Definition(); + const auto& base = + segment_info.Segments().at(base_segment_index).Definition(); bool base_codepoints_only = !base.codepoints.empty() && base.feature_tags.empty(); bool base_features_only = @@ -212,7 +213,7 @@ static bool WouldMixFeaturesAndCodepoints( } for (segment_index_t id : segments) { - const auto& s = segment_info.Segments()[id].Definition(); + const auto& s = segment_info.Segments().at(id).Definition(); if (base_codepoints_only && !s.feature_tags.empty()) { return true; diff --git a/ift/encoder/candidate_merge.h b/ift/encoder/candidate_merge.h index 7eaac8b2..d1d623c9 100644 --- a/ift/encoder/candidate_merge.h +++ b/ift/encoder/candidate_merge.h @@ -8,6 +8,7 @@ #include "ift/common/int_set.h" #include "ift/encoder/activation_condition.h" #include "ift/encoder/invalidation_set.h" +#include "ift/encoder/requested_segmentation_information.h" #include "ift/encoder/segment.h" #include "ift/encoder/types.h" #include "ift/freq/probability_bound.h" @@ -191,6 +192,11 @@ struct CandidateMerge { const SubsetDefinition& def, int quality); + static bool WouldMixFeaturesAndCodepoints( + const RequestedSegmentationInformation& segment_info, + segment_index_t base_segment_index, + const ift::common::SegmentSet& segments); + private: static absl::Status ComputeInitFontGlyphDelta( Merger& merger, const ift::common::GlyphSet& moved_glyphs, diff --git a/ift/encoder/merger.cc b/ift/encoder/merger.cc index 918c7f44..e66c20d7 100644 --- a/ift/encoder/merger.cc +++ b/ift/encoder/merger.cc @@ -52,10 +52,8 @@ StatusOr Merger::ComputeAndCacheAggregateProbability( total += bound.Value(); min_total += bound.Min(); } - if (segment_index < cached_agg_prob_.size()) { - cached_agg_prob_[segment_index] = total; - cached_min_agg_prob_[segment_index] = min_total; - } + cached_agg_prob_.at(segment_index) = total; + cached_min_agg_prob_.at(segment_index) = min_total; return total; } @@ -73,9 +71,8 @@ void Merger::EnsurePatchSizeCacheValid() const { StatusOr Merger::ComputeAndCacheExclusivePatchSize( segment_index_t segment_index) const { EnsurePatchSizeCacheValid(); - if (segment_index < cached_exclusive_size_.size() && - cached_exclusive_size_[segment_index] != UINT32_MAX) { - return cached_exclusive_size_[segment_index]; + if (cached_exclusive_size_.at(segment_index) != UINT32_MAX) { + return cached_exclusive_size_.at(segment_index); } const auto& glyphs = context_->glyph_groupings.ExclusiveGlyphs(segment_index); @@ -83,9 +80,7 @@ StatusOr Merger::ComputeAndCacheExclusivePatchSize( if (!glyphs.empty()) { size = TRY(context_->patch_size_cache->GetPatchSize(glyphs)); } - if (segment_index < cached_exclusive_size_.size()) { - cached_exclusive_size_[segment_index] = size; - } + cached_exclusive_size_.at(segment_index) = size; return size; } @@ -145,6 +140,13 @@ StatusOr> Merger::TryNextMerge() { // cost for each script individually and use the sum of the individual costs // as the overall cost. + if (UseAllPairsMerging()) { + auto modified = TRY(all_pairs_merger_.TryNextAllPairsMerge(*this)); + if (modified.has_value()) { + return *modified; + } + } + while (true) { auto it = candidate_segments_.cbegin(); if (it == candidate_segments_.cend()) { @@ -436,17 +438,14 @@ uint32_t Merger::AssignMergedSegment(segment_index_t base, } } for (segment_index_t s : invalidated) { - if (s < cached_agg_prob_.size()) { - cached_agg_prob_[s] = -1.0; - cached_min_agg_prob_[s] = -1.0; - } - if (s < cached_exclusive_size_.size()) { - cached_exclusive_size_[s] = UINT32_MAX; - } + cached_agg_prob_.at(s) = -1.0; + cached_min_agg_prob_.at(s) = -1.0; + cached_exclusive_size_.at(s) = UINT32_MAX; } if (!is_inert) { exclusive_sizes_dirty_ = true; } + all_pairs_merger_.AssignMergedSegment(base, to_merge); candidate_segments_.subtract(to_merge); candidate_segments_.insert(base); return Context().AssignMergedSegment(base, to_merge, merged_segment, diff --git a/ift/encoder/merger.h b/ift/encoder/merger.h index e0f3b9f6..92be77f1 100644 --- a/ift/encoder/merger.h +++ b/ift/encoder/merger.h @@ -8,6 +8,7 @@ #include "absl/flags/declare.h" #include "ift/common/int_set.h" #include "ift/encoder/activation_condition.h" +#include "ift/encoder/all_pairs_merger.h" #include "ift/encoder/candidate_merge.h" #include "ift/encoder/invalidation_set.h" #include "ift/encoder/merge_strategy.h" @@ -92,6 +93,28 @@ class Merger { uint32_t NumInscopeSegments() const { return inscope_segments_.size(); } + const ift::common::SegmentSet& InscopeSegments() const { + return inscope_segments_; + } + + const ift::common::SegmentSet& CandidateSegments() const { + return candidate_segments_; + } + + segment_index_t OptimizationCutoffSegment() const { + return optimization_cutoff_segment_; + } + + // For a merge of an inert base patch with any other possible inert segment, + // this computes the minimum probability the other segment must have for it to + // be possible to produce a delta lower than lowest_cost_delta (regardless of + // it's probability or size). + // + // This assumes that P(base) >= P(other) + double BestCaseInertProbabilityThreshold(uint32_t base_patch_size, + double base_probability, + double lowest_cost_delta) const; + void RecordMergedSizeReduction(double size_reduction) { int32_t reduction_percent = 100.0 * size_reduction; merged_size_reduction_histogram_[reduction_percent]++; @@ -126,9 +149,8 @@ class Merger { // cached segment probabilities when available. absl::StatusOr AggregateProbability( segment_index_t segment_index) const { - if (segment_index < cached_agg_prob_.size() && - cached_agg_prob_[segment_index] >= 0.0) { - return cached_agg_prob_[segment_index]; + if (cached_agg_prob_.at(segment_index) >= 0.0) { + return cached_agg_prob_.at(segment_index); } return ComputeAndCacheAggregateProbability(segment_index); } @@ -138,9 +160,8 @@ class Merger { // probabilities when available. absl::StatusOr MinAggregateProbability( segment_index_t segment_index) const { - if (segment_index < cached_min_agg_prob_.size() && - cached_min_agg_prob_[segment_index] >= 0.0) { - return cached_min_agg_prob_[segment_index]; + if (cached_min_agg_prob_.at(segment_index) >= 0.0) { + return cached_min_agg_prob_.at(segment_index); } TRYV(ComputeAndCacheAggregateProbability(segment_index)); return cached_min_agg_prob_.at(segment_index); @@ -152,9 +173,8 @@ class Merger { segment_index_t segment_index) const { if (!exclusive_sizes_dirty_ && cached_patch_size_cache_ == context_->patch_size_cache.get() && - segment_index < cached_exclusive_size_.size() && - cached_exclusive_size_[segment_index] != UINT32_MAX) { - return cached_exclusive_size_[segment_index]; + cached_exclusive_size_.at(segment_index) != UINT32_MAX) { + return cached_exclusive_size_.at(segment_index); } return ComputeAndCacheExclusivePatchSize(segment_index); } @@ -291,21 +311,14 @@ class Merger { segment_index_t base_segment_index, const ift::common::SegmentSet& to_merge_segments_); - void MarkFinished(segment_index_t s) { candidate_segments_.erase(s); } + void MarkFinished(segment_index_t s) { + candidate_segments_.erase(s); + all_pairs_merger_.MarkFinished(s); + } absl::Status ApplyInitFontMove(const ift::common::GlyphSet& glyphs_to_move, double delta); - // For a merge of an inert base patch with any other possible inert segment, - // this computes the minimum probability the other segment must have for it to - // be possible to produce a delta lower than lowest_cost_delta (regardless of - // it's probability or size). - // - // This assumes that P(base) >= P(other) - double BestCaseInertProbabilityThreshold(uint32_t base_patch_size, - double base_probability, - double lowest_cost_delta) const; - absl::StatusOr InitFontApplyProbabilityThreshold( size_t profile_index = 0) const; ift::common::SegmentSet InitFontSegmentsToCheck( @@ -347,6 +360,8 @@ class Merger { mutable std::optional cached_base_overhead_; mutable const PatchSizeCache* cached_patch_size_cache_ = nullptr; mutable bool exclusive_sizes_dirty_ = true; + + AllPairsMerger all_pairs_merger_; }; } // namespace ift::encoder From 5a697b55025d55d1eb7e7052f75820ecbfe8ea50 Mon Sep 17 00:00:00 2001 From: Garret Rieger Date: Tue, 20 Oct 2026 18:31:47 +0000 Subject: [PATCH 04/13] Implement post merge invalidation in AllPairsMerger. --- docs/all_pairs_merging.md | 2 +- ift/encoder/all_pairs_merger.cc | 162 +++++++++++++++++++++++++++++--- ift/encoder/all_pairs_merger.h | 14 ++- ift/encoder/glyph_groupings.h | 15 +++ ift/encoder/merger.cc | 3 +- 5 files changed, 180 insertions(+), 16 deletions(-) diff --git a/docs/all_pairs_merging.md b/docs/all_pairs_merging.md index aa7591ea..9e520cf9 100644 --- a/docs/all_pairs_merging.md +++ b/docs/all_pairs_merging.md @@ -132,5 +132,5 @@ To handle this cleanly without changing `TryNextMerge()`'s external contract: - [x] **Commit 1: Add `--experimental_all_pairs_merging` flag, per-segment cost metadata cache, and inert 2-segment fast-path.** Declare `ABSL_DECLARE_FLAG(bool, experimental_all_pairs_merging)` in `ift/encoder/merger.h` (default `false`), add cached per-segment aggregate probability and exclusive patch size vectors on `Merger`, and add an $O(1)$ pre-check + fast-path for 2-segment inert cost delta evaluations in `CandidateMerge::ComputeCostDelta` (with unit tests in `ift/encoder/candidate_merge_test.cc`). - [x] **Commit 2: Implement the lazy two-stage all-pairs priority queue and probability-ordered candidate generation in `AllPairsMerger`.** Add `AllPairsMerger` (`ift/encoder/all_pairs_merger.h`/`.cc`) with `QueuedCandidate`, probability-ordered active segment tracking with `BestCaseInertProbabilityThreshold` row pruning, composite condition candidate queuing, and the lazy `BEST_CASE` $\rightarrow$ `EXACT` pop-and-refine loop for Phase 1 cost-reducing merges. -- [ ] **Commit 3: Implement epoch-based invalidation and incremental `dirty_segments_` refresh across `TryNextMerge()` calls.** Track affected segments for both inert and non-inert merges, refresh `dirty_segments_` at the start of `TryNextMerge()` after `SegmentationContext::ReprocessChanged()`, and reset all-pairs state in `Merger::ReassignInitSubset()`. +- [x] **Commit 3: Implement epoch-based invalidation and incremental `dirty_segments_` refresh across `TryNextMerge()` calls.** Track affected segments for both inert and non-inert merges, refresh `dirty_segments_` at the start of `TryNextMerge()` after `SegmentationContext::ReprocessChanged()`, and reset all-pairs state in `Merger::ReassignInitSubset()`. - [ ] **Commit 4: Integrate Phase 2 (`min_group_size` enforcement) and add unit and determinism tests.** Transition to Phase 2 for any undersized segments remaining after Phase 1 completes, and add tests in `ift/encoder/candidate_merge_test.cc` and `ift/encoder/closure_glyph_segmenter_test.cc` verifying that `--experimental_all_pairs_merging=true` avoids single-base greedy traps, handles composite conditions / `min_group_size` / `UNICODE_RANGE` mode, and is deterministic. diff --git a/ift/encoder/all_pairs_merger.cc b/ift/encoder/all_pairs_merger.cc index ebef8b92..a1ee44d6 100644 --- a/ift/encoder/all_pairs_merger.cc +++ b/ift/encoder/all_pairs_merger.cc @@ -19,25 +19,63 @@ using absl::btree_set; using absl::Status; using absl::StatusOr; +using ift::common::GlyphSet; using ift::common::SegmentSet; namespace ift::encoder { -void AllPairsMerger::AssignMergedSegment(segment_index_t base, +void AllPairsMerger::AssignMergedSegment(const Merger& merger, + segment_index_t base, const SegmentSet& to_merge) { - if (!initialized_) { + if (!initialized_ || phase_1_complete_) { return; } + + SegmentSet merged_segments = to_merge; + merged_segments.insert(base); + const auto& groupings = merger.Context().glyph_groupings; + const auto& pre_comb_map = groupings.ConditionsAndGlyphsPreCombination(); + + for (segment_index_t s : merged_segments) { + // precombination representative affected glyphs are recorded this will + // let us locate any conditions involved with those glyphs post-merge. + for (const auto& c : + groupings.TriggeringSegmentToConditionsPreCombination(s)) { + if (c.IsFallback()) { + continue; + } + if (auto rep = pre_comb_map.at(c).min(); rep.has_value()) { + dirty_glyphs_.insert(*rep); + } + } + + // Also record any segments involved in conditions directly impacted by the merge. + for (const auto& c : + groupings.TriggeringSegmentToConditions(s)) { + if (c.IsFallback()) { + continue; + } + if (!c.IsExclusive()) { + dirty_segments_.union_set(c.TriggeringSegments()); + } + } + } + for (segment_index_t s : to_merge) { RemoveActiveSegment(s); segment_epoch_.at(s) = UINT32_MAX; + dirty_segments_.erase(s); + dirty_bases_.erase(s); } // base's glyph groupings are not finalized until after Apply() (and // ReprocessChanged() for non-inert merges) completes, so remove it for now; // the next TryNextAllPairsMerge() call will re-insert it via - // UpdateActiveSegment(). + // RefreshDirtySegments(). RemoveActiveSegment(base); - segment_epoch_.at(base) = ++current_epoch_; + segment_epoch_.at(base) = UINT32_MAX; + + dirty_segments_.insert(base); + dirty_bases_.insert(base); } void AllPairsMerger::RemoveActiveSegment(segment_index_t s) { @@ -56,8 +94,14 @@ Status AllPairsMerger::UpdateActiveSegment(Merger& merger, segment_index_t s) { } uint32_t size = TRY(merger.ExclusivePatchSize(s)); if (size == 0) { + // TODO(garretrieger): active_segments_by_prob is currently restricted to + // just segments with exclusive patches which in turn means we + // only consider exclusive segment pairs for merging, should we + // consider all segment pairs? or keep this as is relying on composite + // condition candidates. return absl::OkStatus(); } + double prob = TRY(merger.AggregateProbability(s)); active_segments_by_prob_.insert(ProbOrderedSegment{prob, s}); active_segment_prob_.at(s) = prob; @@ -160,9 +204,18 @@ Status AllPairsMerger::QueueCompositeCandidates( segment_index_t u = *triggering_segments.min(); segment_index_t v = *triggering_segments.max(); if (IsActiveProbSegment(u) && IsActiveProbSegment(v)) { - // Pair of active exclusive segments is already handled by exclusive - // pair queuing. - continue; + ProbOrderedSegment pos_u{*active_segment_prob_.at(u), u}; + ProbOrderedSegment pos_v{*active_segment_prob_.at(v), v}; + const ProbOrderedSegment& high = (pos_u < pos_v) ? pos_u : pos_v; + const ProbOrderedSegment& low = (pos_u < pos_v) ? pos_v : pos_u; + uint32_t high_size = TRY(merger.ExclusivePatchSize(high.segment_index)); + double inert_threshold = merger.BestCaseInertProbabilityThreshold( + high_size, high.probability, 0.0); + if (low.probability > inert_threshold) { + // Pair of active exclusive segments above the inert threshold is + // already handled by exclusive pair queuing. + continue; + } } } @@ -176,6 +229,51 @@ Status AllPairsMerger::QueueCompositeCandidates( return absl::OkStatus(); } +Status AllPairsMerger::QueueAllCandidates( + Merger& merger, const std::optional& dirty_segments) { + if (!dirty_segments.has_value()) { + for (auto u_it = active_segments_by_prob_.begin(); + u_it != active_segments_by_prob_.end(); ++u_it) { + TRYV(QueueLowerRankedExclusivePairs(merger, u_it)); + } + } else { + for (segment_index_t d : *dirty_segments) { + if (!IsActiveProbSegment(d)) { + continue; + } + double d_prob = *active_segment_prob_.at(d); + auto d_it = active_segments_by_prob_.find(ProbOrderedSegment{d_prob, d}); + if (d_it == active_segments_by_prob_.end()) { + return absl::InternalError( + "Active probability segment missing from ordered set."); + } + + bool d_is_inert = merger.Context().InertSegments().contains(d); + for (auto u_it = active_segments_by_prob_.begin(); u_it != d_it; ++u_it) { + segment_index_t u = u_it->segment_index; + if (dirty_segments->contains(u)) { + // Pair {u, d} is already queued when processing u's lower-ranked + // partners. + continue; + } + if (d_is_inert) { + uint32_t u_size = TRY(merger.ExclusivePatchSize(u)); + double inert_threshold = merger.BestCaseInertProbabilityThreshold( + u_size, u_it->probability, 0.0); + if (d_prob <= inert_threshold) { + continue; + } + } + TRYV(QueueBestCaseCandidate(merger, SegmentSet{u, d})); + } + + TRYV(QueueLowerRankedExclusivePairs(merger, d_it)); + } + } + + return QueueCompositeCandidates(merger, dirty_segments); +} + Status AllPairsMerger::InitPhase1(Merger& merger) { size_t num_segments = merger.Context().SegmentationInfo().Segments().size(); current_epoch_ = 0; @@ -183,6 +281,9 @@ Status AllPairsMerger::InitPhase1(Merger& merger) { for (segment_index_t s : merger.CandidateSegments()) { segment_epoch_.at(s) = 0; } + dirty_segments_.clear(); + dirty_bases_.clear(); + dirty_glyphs_.clear(); active_segments_by_prob_.clear(); active_segment_prob_.assign(num_segments, std::nullopt); queue_ = {}; @@ -194,13 +295,48 @@ Status AllPairsMerger::InitPhase1(Merger& merger) { TRYV(UpdateActiveSegment(merger, s)); } - for (auto u_it = active_segments_by_prob_.begin(); - u_it != active_segments_by_prob_.end(); ++u_it) { - TRYV(QueueLowerRankedExclusivePairs(merger, u_it)); + TRYV(QueueAllCandidates(merger)); + initialized_ = true; + return absl::OkStatus(); +} + +Status AllPairsMerger::RefreshDirtySegments(Merger& merger) { + const auto& groupings = merger.Context().glyph_groupings; + for (glyph_id_t gid : dirty_glyphs_) { + if (auto cond = groupings.GlyphToCondition(gid); + cond.has_value() && !cond->IsFallback()) { + dirty_segments_.union_set(cond->TriggeringSegments()); + } } + dirty_glyphs_.clear(); + + dirty_bases_.intersect(merger.CandidateSegments()); + for (segment_index_t base : dirty_bases_) { + dirty_segments_.insert(base); + for (const auto& c : groupings.TriggeringSegmentToConditions(base)) { + if (!c.IsFallback() && !c.IsExclusive()) { + dirty_segments_.union_set(c.TriggeringSegments()); + } + } + TRYV(UpdateActiveSegment(merger, base)); + } + dirty_bases_.clear(); - TRYV(QueueCompositeCandidates(merger)); - initialized_ = true; + dirty_segments_.intersect(merger.CandidateSegments()); + if (dirty_segments_.empty()) { + return absl::OkStatus(); + } + + ++current_epoch_; + for (segment_index_t s : dirty_segments_) { + segment_epoch_.at(s) = current_epoch_; + if (IsActiveProbSegment(s) && groupings.ExclusiveGlyphs(s).empty()) { + RemoveActiveSegment(s); + } + } + + TRYV(QueueAllCandidates(merger, dirty_segments_)); + dirty_segments_.clear(); return absl::OkStatus(); } @@ -221,6 +357,8 @@ StatusOr> AllPairsMerger::TryNextAllPairsMerge( if (!initialized_) { TRYV(InitPhase1(merger)); + } else { + TRYV(RefreshDirtySegments(merger)); } while (!queue_.empty()) { diff --git a/ift/encoder/all_pairs_merger.h b/ift/encoder/all_pairs_merger.h index 0dc75eb8..76e2ef25 100644 --- a/ift/encoder/all_pairs_merger.h +++ b/ift/encoder/all_pairs_merger.h @@ -25,14 +25,16 @@ class AllPairsMerger { absl::StatusOr> TryNextAllPairsMerge( Merger& merger); - void AssignMergedSegment(segment_index_t base, + void AssignMergedSegment(const Merger& merger, segment_index_t base, const ift::common::SegmentSet& to_merge); void MarkFinished(segment_index_t s) { - if (!initialized_) { + if (!initialized_ || phase_1_complete_) { return; } RemoveActiveSegment(s); + segment_epoch_.at(s) = UINT32_MAX; + dirty_segments_.erase(s); } private: @@ -81,6 +83,7 @@ class AllPairsMerger { }; absl::Status InitPhase1(Merger& merger); + absl::Status RefreshDirtySegments(Merger& merger); bool IsCandidateStale(const QueuedCandidate& candidate) const; bool IsActiveProbSegment(segment_index_t s) const { return active_segment_prob_.at(s).has_value(); @@ -96,11 +99,18 @@ class AllPairsMerger { Merger& merger, const std::optional& filter_segments = std::nullopt); + absl::Status QueueAllCandidates( + Merger& merger, + const std::optional& dirty_segments = + std::nullopt); bool initialized_ = false; bool phase_1_complete_ = false; uint32_t current_epoch_ = 0; std::vector segment_epoch_; + ift::common::SegmentSet dirty_segments_; + ift::common::SegmentSet dirty_bases_; + ift::common::GlyphSet dirty_glyphs_; absl::btree_set active_segments_by_prob_; std::vector> active_segment_prob_; std::priority_queue, diff --git a/ift/encoder/glyph_groupings.h b/ift/encoder/glyph_groupings.h index 4019b497..c83b576b 100644 --- a/ift/encoder/glyph_groupings.h +++ b/ift/encoder/glyph_groupings.h @@ -110,6 +110,21 @@ class GlyphGroupings { return empty; } + // Returns a list of pre-combination conditions which include segment. + const absl::flat_hash_set& + TriggeringSegmentToConditionsPreCombination(segment_index_t segment) const { + static absl::flat_hash_set empty; + auto it = conditions_and_glyphs_pre_combination_ + .TriggeringSegmentToConditions() + .find(segment); + if (it != + conditions_and_glyphs_pre_combination_.TriggeringSegmentToConditions() + .end()) { + return it->second; + } + return empty; + } + // Add a set of glyphs to an existing exclusive group (and_group of one // segment). absl::Status AddGlyphsToExclusiveGroup(segment_index_t exclusive_segment, diff --git a/ift/encoder/merger.cc b/ift/encoder/merger.cc index e66c20d7..0c6799a7 100644 --- a/ift/encoder/merger.cc +++ b/ift/encoder/merger.cc @@ -422,6 +422,7 @@ Status Merger::ReassignInitSubset() { candidate_patch_merge_glyphs_ = std::nullopt; TRYV(ResetSegmentProbabilities()); TRYV(InitOptimizationCutoff()); + all_pairs_merger_ = AllPairsMerger(); return absl::OkStatus(); } @@ -445,7 +446,7 @@ uint32_t Merger::AssignMergedSegment(segment_index_t base, if (!is_inert) { exclusive_sizes_dirty_ = true; } - all_pairs_merger_.AssignMergedSegment(base, to_merge); + all_pairs_merger_.AssignMergedSegment(*this, base, to_merge); candidate_segments_.subtract(to_merge); candidate_segments_.insert(base); return Context().AssignMergedSegment(base, to_merge, merged_segment, From 2a6da654386d0765c9714cc037b265a6b3ebc557 Mon Sep 17 00:00:00 2001 From: Garret Rieger Date: Tue, 20 Oct 2026 23:29:04 +0000 Subject: [PATCH 05/13] After all pairs merging fallback to the per base merging to enforce min group sizes. - Updates per base merging to utilize all pair's segment ordering if present (since all pairs merging can invalidate the input segment ordering). --- docs/all_pairs_merging.md | 2 +- ift/encoder/BUILD | 2 + ift/encoder/all_pairs_merger.cc | 9 +- ift/encoder/all_pairs_merger.h | 42 ++-- ift/encoder/candidate_merge_test.cc | 261 ++++++++++++++++++++ ift/encoder/closure_glyph_segmenter_test.cc | 213 ++++++++++++++++ ift/encoder/merger.cc | 108 ++++---- 7 files changed, 559 insertions(+), 78 deletions(-) diff --git a/docs/all_pairs_merging.md b/docs/all_pairs_merging.md index 9e520cf9..b38a5cba 100644 --- a/docs/all_pairs_merging.md +++ b/docs/all_pairs_merging.md @@ -133,4 +133,4 @@ To handle this cleanly without changing `TryNextMerge()`'s external contract: - [x] **Commit 1: Add `--experimental_all_pairs_merging` flag, per-segment cost metadata cache, and inert 2-segment fast-path.** Declare `ABSL_DECLARE_FLAG(bool, experimental_all_pairs_merging)` in `ift/encoder/merger.h` (default `false`), add cached per-segment aggregate probability and exclusive patch size vectors on `Merger`, and add an $O(1)$ pre-check + fast-path for 2-segment inert cost delta evaluations in `CandidateMerge::ComputeCostDelta` (with unit tests in `ift/encoder/candidate_merge_test.cc`). - [x] **Commit 2: Implement the lazy two-stage all-pairs priority queue and probability-ordered candidate generation in `AllPairsMerger`.** Add `AllPairsMerger` (`ift/encoder/all_pairs_merger.h`/`.cc`) with `QueuedCandidate`, probability-ordered active segment tracking with `BestCaseInertProbabilityThreshold` row pruning, composite condition candidate queuing, and the lazy `BEST_CASE` $\rightarrow$ `EXACT` pop-and-refine loop for Phase 1 cost-reducing merges. - [x] **Commit 3: Implement epoch-based invalidation and incremental `dirty_segments_` refresh across `TryNextMerge()` calls.** Track affected segments for both inert and non-inert merges, refresh `dirty_segments_` at the start of `TryNextMerge()` after `SegmentationContext::ReprocessChanged()`, and reset all-pairs state in `Merger::ReassignInitSubset()`. -- [ ] **Commit 4: Integrate Phase 2 (`min_group_size` enforcement) and add unit and determinism tests.** Transition to Phase 2 for any undersized segments remaining after Phase 1 completes, and add tests in `ift/encoder/candidate_merge_test.cc` and `ift/encoder/closure_glyph_segmenter_test.cc` verifying that `--experimental_all_pairs_merging=true` avoids single-base greedy traps, handles composite conditions / `min_group_size` / `UNICODE_RANGE` mode, and is deterministic. +- [x] **Commit 4: Integrate Phase 2 (`min_group_size` enforcement) and add unit and determinism tests.** Transition to Phase 2 for any undersized segments remaining after Phase 1 completes, and add tests in `ift/encoder/candidate_merge_test.cc` and `ift/encoder/closure_glyph_segmenter_test.cc` verifying that `--experimental_all_pairs_merging=true` avoids single-base greedy traps, handles composite conditions / `min_group_size` / `UNICODE_RANGE` mode, and is deterministic. diff --git a/ift/encoder/BUILD b/ift/encoder/BUILD index 4698b15f..94239a63 100644 --- a/ift/encoder/BUILD +++ b/ift/encoder/BUILD @@ -428,6 +428,7 @@ cc_test( "//ift/config:auto_segmenter_config", "//ift/freq", "//ift/freq:mock_probability_calculator", + "@abseil-cpp//absl/flags:flag", "@googletest//:gtest", "@googletest//:gtest_main", ], @@ -497,6 +498,7 @@ cc_test( "//ift/freq", "//ift/freq:common", "//ift/freq:mock_probability_calculator", + "@abseil-cpp//absl/flags:flag", "@abseil-cpp//absl/log", "@googletest//:gtest", "@googletest//:gtest_main", diff --git a/ift/encoder/all_pairs_merger.cc b/ift/encoder/all_pairs_merger.cc index a1ee44d6..ac561a78 100644 --- a/ift/encoder/all_pairs_merger.cc +++ b/ift/encoder/all_pairs_merger.cc @@ -27,7 +27,7 @@ namespace ift::encoder { void AllPairsMerger::AssignMergedSegment(const Merger& merger, segment_index_t base, const SegmentSet& to_merge) { - if (!initialized_ || phase_1_complete_) { + if (!initialized_) { return; } @@ -195,7 +195,7 @@ Status AllPairsMerger::QueueCompositeCandidates( continue; } - if (!triggering_segments.intersects(merger.CandidateSegments()) || + if (!triggering_segments.is_subset_of(merger.CandidateSegments()) || !triggering_segments.is_subset_of(merger.InscopeSegments())) { continue; } @@ -351,10 +351,6 @@ bool AllPairsMerger::IsCandidateStale(const QueuedCandidate& candidate) const { StatusOr> AllPairsMerger::TryNextAllPairsMerge( Merger& merger) { - if (phase_1_complete_) { - return std::nullopt; - } - if (!initialized_) { TRYV(InitPhase1(merger)); } else { @@ -390,7 +386,6 @@ StatusOr> AllPairsMerger::TryNextAllPairsMerge( return TRY(top.evaluated_merge->Apply(merger)); } - phase_1_complete_ = true; return std::nullopt; } diff --git a/ift/encoder/all_pairs_merger.h b/ift/encoder/all_pairs_merger.h index 76e2ef25..c6bf4984 100644 --- a/ift/encoder/all_pairs_merger.h +++ b/ift/encoder/all_pairs_merger.h @@ -22,6 +22,23 @@ class Merger; // lazy two-stage (best-case lower bound -> exact) priority queue. class AllPairsMerger { public: + struct ProbOrderedSegment { + double probability; + segment_index_t segment_index; + + bool operator<(const ProbOrderedSegment& other) const { + if (probability != other.probability) { + return probability > other.probability; + } + return segment_index < other.segment_index; + } + + bool operator==(const ProbOrderedSegment& other) const { + return probability == other.probability && + segment_index == other.segment_index; + } + }; + absl::StatusOr> TryNextAllPairsMerge( Merger& merger); @@ -29,12 +46,17 @@ class AllPairsMerger { const ift::common::SegmentSet& to_merge); void MarkFinished(segment_index_t s) { - if (!initialized_ || phase_1_complete_) { + if (!initialized_) { return; } RemoveActiveSegment(s); segment_epoch_.at(s) = UINT32_MAX; dirty_segments_.erase(s); + dirty_bases_.erase(s); + } + + const absl::btree_set& ActiveSegmentsByProb() const { + return active_segments_by_prob_; } private: @@ -65,23 +87,6 @@ class AllPairsMerger { }; }; - struct ProbOrderedSegment { - double probability; - segment_index_t segment_index; - - bool operator<(const ProbOrderedSegment& other) const { - if (probability != other.probability) { - return probability > other.probability; - } - return segment_index < other.segment_index; - } - - bool operator==(const ProbOrderedSegment& other) const { - return probability == other.probability && - segment_index == other.segment_index; - } - }; - absl::Status InitPhase1(Merger& merger); absl::Status RefreshDirtySegments(Merger& merger); bool IsCandidateStale(const QueuedCandidate& candidate) const; @@ -105,7 +110,6 @@ class AllPairsMerger { std::nullopt); bool initialized_ = false; - bool phase_1_complete_ = false; uint32_t current_epoch_ = 0; std::vector segment_epoch_; ift::common::SegmentSet dirty_segments_; diff --git a/ift/encoder/candidate_merge_test.cc b/ift/encoder/candidate_merge_test.cc index 66b8a027..bdc528db 100644 --- a/ift/encoder/candidate_merge_test.cc +++ b/ift/encoder/candidate_merge_test.cc @@ -4,6 +4,7 @@ #include #include +#include "absl/flags/flag.h" #include "gtest/gtest.h" #include "ift/common/bazel_data_file_resolver.h" #include "ift/common/font_data.h" @@ -1776,4 +1777,264 @@ TEST_F(CandidateMergeTest, EXPECT_NEAR(*exact_delta, expected_exact, 1e-9); } +namespace { +struct ScopedAllPairsMergingFlag { + explicit ScopedAllPairsMergingFlag(bool enabled) + : prev(absl::GetFlag(FLAGS_experimental_all_pairs_merging)) { + absl::SetFlag(&FLAGS_experimental_all_pairs_merging, enabled); + } + ~ScopedAllPairsMergingFlag() { + absl::SetFlag(&FLAGS_experimental_all_pairs_merging, prev); + } + bool prev; +}; +} // namespace + +TEST_F(CandidateMergeTest, AllPairsMerging_AvoidsSingleBaseGreedyTrap) { + // s0 ('a', gid 69) has a marginally negative merge with s1 ('b', gid 70) + // (delta = -4.0), whereas s1 ('b', gid 70) and s2 ('c', gid 71) have a much + // stronger negative merge (delta = -201.25). + std::vector segments = { + {{'a'}}, + {{'b'}}, + {{'c'}}, + }; + std::vector> probabilities = { + {{{'a'}}, 0.90}, + {{{'b'}}, 0.80}, + {{{'c'}}, 0.79}, + {{{'a', 'b'}}, 0.90}, + {{{'b', 'c'}}, 0.80}, + {{{'a', 'c'}}, 0.99}, + {{{'a', 'b', 'c'}}, 0.99}, + }; + + auto run_merger = [&](bool all_pairs) { + ScopedAllPairsMergingFlag flag_override(all_pairs); + ClosureGlyphSegmenter segmenter(8, 8, PATCH, CLOSURE_ONLY, resolver); + auto context = SegmentationContext::InitializeSegmentationContext( + roboto.get(), {}, segments, segmenter.unmapped_glyph_handling(), + segmenter.condition_analysis_mode(), segmenter.brotli_quality(), + segmenter.init_font_merging_brotli_quality(), resolver); + EXPECT_TRUE(context.ok()) << context.status(); + + auto* size_cache = new MockPatchSizeCache(); + size_cache->SetPatchSize({69}, 200); + size_cache->SetPatchSize({70}, 200); + size_cache->SetPatchSize({71}, 200); + size_cache->SetPatchSize({69, 70}, 440); + size_cache->SetPatchSize({70, 71}, 220); + size_cache->SetPatchSize({69, 71}, 450); + size_cache->SetPatchSize({69, 70, 71}, 650); + context->patch_size_cache.reset(size_cache); + + auto calc = std::make_shared(probabilities); + Merger merger = *Merger::New( + *context, MergeStrategy::CostBased(ProbabilityProfile(calc), 75, 1), + all, all); + + while (true) { + auto modified = merger.TryNextMerge(); + EXPECT_TRUE(modified.ok()) << modified.status(); + if (!modified->has_value()) { + break; + } + EXPECT_TRUE(context->ReprocessChanged(**modified).ok()); + } + + std::vector result; + for (const auto& seg : context->SegmentationInfo().Segments()) { + result.push_back(seg.Definition()); + } + return result; + }; + + // Single-base greedy mode merges {0, 1} first, trapping s2 unmerged. + std::vector single_base_result = run_merger(false); + std::vector expected_single_base = { + {'a', 'b'}, + {}, + {'c'}, + }; + EXPECT_EQ(single_base_result, expected_single_base); + + // All-pairs mode merges {1, 2} first and leaves s0 separate. + std::vector all_pairs_result = run_merger(true); + std::vector expected_all_pairs = { + {'a'}, + {'b', 'c'}, + {}, + }; + EXPECT_EQ(all_pairs_result, expected_all_pairs); +} + +TEST_F(CandidateMergeTest, + AllPairsMerging_Phase2ResumesPhase1AfterForcedMerge) { + // With min_group_size = 2: + // - s0 ('a', gid 69) and s1 ('b', gid 70) each have 1 codepoint; s2 + // ({'c', 'd'}, gids {71, 72}) already has 2 codepoints. + // - All initial 2-segment merges have positive cost deltas, so Phase 1 makes + // no merges and Phase 2 forces s0 to merge with s1 (delta = +6.5). + // - Once s0 becomes {'a', 'b'}, merging s0 with s2 has a negative cost delta + // (delta = -220.25). Even though both s0 and s2 now meet min_group_size = 2, + // AllPairsMerger refreshes dirty segment s0 and resumes Phase 1 to apply + // the cost-reducing merge {0, 2}. + ScopedAllPairsMergingFlag flag_override(true); + std::vector segments = { + {{'a'}}, + {{'b'}}, + {{'c', 'd'}}, + }; + std::vector> probabilities = { + {{{'a'}}, 0.80}, + {{{'b'}}, 0.50}, + {{{'c', 'd'}}, 0.75}, + {{{'a', 'b'}}, 0.80}, + {{{'a', 'c', 'd'}}, 0.95}, + {{{'b', 'c', 'd'}}, 0.95}, + {{{'a', 'b', 'c', 'd'}}, 0.80}, + }; + + ClosureGlyphSegmenter segmenter(8, 8, PATCH, CLOSURE_ONLY, resolver); + auto context = SegmentationContext::InitializeSegmentationContext( + roboto.get(), {}, segments, segmenter.unmapped_glyph_handling(), + segmenter.condition_analysis_mode(), segmenter.brotli_quality(), + segmenter.init_font_merging_brotli_quality(), resolver); + ASSERT_TRUE(context.ok()) << context.status(); + + auto* size_cache = new MockPatchSizeCache(); + size_cache->SetPatchSize({69}, 200); + size_cache->SetPatchSize({70}, 200); + size_cache->SetPatchSize({71, 72}, 240); + size_cache->SetPatchSize({69, 70}, 380); + size_cache->SetPatchSize({69, 71, 72}, 450); + size_cache->SetPatchSize({70, 71, 72}, 450); + size_cache->SetPatchSize({69, 70, 71, 72}, 400); + context->patch_size_cache.reset(size_cache); + + auto calc = std::make_shared(probabilities); + Merger merger = *Merger::New( + *context, MergeStrategy::CostBased(ProbabilityProfile(calc), 75, 2), all, + all); + + // Step 1: Phase 1 finds no negative merges; Phase 2 forces {0, 1}. + auto step1 = merger.TryNextMerge(); + ASSERT_TRUE(step1.ok()) << step1.status(); + ASSERT_TRUE(step1->has_value()); + EXPECT_EQ((*step1)->base_segment, 0u); + EXPECT_EQ((*step1)->segments, (SegmentSet{0, 1})); + ASSERT_TRUE(context->ReprocessChanged(**step1).ok()); + + // Step 2: Phase 1 resumes on dirty segment 0 and applies {0, 2}. + auto step2 = merger.TryNextMerge(); + ASSERT_TRUE(step2.ok()) << step2.status(); + ASSERT_TRUE(step2->has_value()); + EXPECT_EQ((*step2)->base_segment, 0u); + EXPECT_EQ((*step2)->segments, (SegmentSet{0, 2})); + ASSERT_TRUE(context->ReprocessChanged(**step2).ok()); + + auto step3 = merger.TryNextMerge(); + ASSERT_TRUE(step3.ok()) << step3.status(); + EXPECT_FALSE(step3->has_value()); +} + +TEST_F(CandidateMergeTest, + AllPairsMerging_Phase2UsesAllPairsProbabilityOrdering) { + // Initial probabilities are descending: + // s0 ('a', 0.88), s1 ('b', 0.50), s2 ('c', 0.001), s3 ('d', 0.48), + // s4 ('e', 0.48). + // In Phase 1, {3, 4} merges into s3 with P({'d', 'e'}) = 0.66 > P(s1) = 0.50. + // In Phase 2 (min_group_size = 2), CollectExclusiveCandidateMerges iterates + // using AllPairsMerger's probability order (s3 at 0.66, then s1 at 0.50, + // then s2 at 0.001) rather than segment-index order: + // - s3 (0.66) is evaluated before s1 (0.50), so on an equal positive cost + // delta (+1.54), s3 is selected over s1. + // - s2 (0.001) is visited after s3 and s1 and breaks early on + // inert_threshold (0.0065). + ScopedAllPairsMergingFlag flag_override(true); + std::vector segments = { + {{'a'}}, + {{'b'}}, + {{'c'}}, + {{'d'}}, + {{'e'}}, + }; + std::vector> probabilities = { + {{{'a'}}, 0.88}, + {{{'b'}}, 0.50}, + {{{'c'}}, 0.001}, + {{{'d'}}, 0.48}, + {{{'e'}}, 0.48}, + {{{'d', 'e'}}, 0.66}, + {{{'a', 'b'}}, 0.88}, + {{{'a', 'c'}}, 0.88}, + {{{'a', 'd'}}, 0.88}, + {{{'a', 'e'}}, 0.88}, + {{{'a', 'd', 'e'}}, 0.88}, + {{{'b', 'c'}}, 0.501}, + {{{'b', 'd'}}, 0.80}, + {{{'b', 'e'}}, 0.80}, + {{{'b', 'd', 'e'}}, 0.85}, + {{{'c', 'd'}}, 0.481}, + {{{'c', 'e'}}, 0.481}, + {{{'c', 'd', 'e'}}, 0.661}, + }; + + ClosureGlyphSegmenter segmenter(8, 8, PATCH, CLOSURE_ONLY, resolver); + auto context = SegmentationContext::InitializeSegmentationContext( + roboto.get(), {}, segments, segmenter.unmapped_glyph_handling(), + segmenter.condition_analysis_mode(), segmenter.brotli_quality(), + segmenter.init_font_merging_brotli_quality(), resolver); + ASSERT_TRUE(context.ok()) << context.status(); + + // Glyphs: 'a'=69, 'b'=70, 'c'=71, 'd'=72, 'e'=73. + auto* size_cache = new MockPatchSizeCache(); + size_cache->SetPatchSize({69}, 500); + size_cache->SetPatchSize({70}, 200); + size_cache->SetPatchSize({71}, 200); + size_cache->SetPatchSize({72}, 200); + size_cache->SetPatchSize({73}, 200); + // {3, 4} has negative best-case and exact cost delta in Phase 1: + // best-case: 0.66 * (300 + 75) - 2 * 0.48 * (200 + 75) = 247.5 - 264.0 = -16.5. + // exact: 0.66 * (200 + 75) - 264.0 = 181.5 - 264.0 = -82.5. + size_cache->SetPatchSize({72, 73}, 200); + // In Phase 2 for s0 (cost = 0.88 * 575 = 506.0): + // Both {0, 1} and {0, 3} have the exact same positive cost delta (+1.54): + // - With s3={'d', 'e'} (cost = 0.66 * 275 = 181.5): + // size({69, 72, 73}) = 708 -> 0.88 * 783 - 506.0 - 181.5 = 689.04 - 687.5 = +1.54. + // - With s1={'b'} (cost = 0.50 * 275 = 137.5): + // size({69, 70}) = 658 -> 0.88 * 733 - 506.0 - 137.5 = 645.04 - 643.5 = +1.54. + // Because s3 (0.66) precedes s1 (0.50) in AllPairsMerger's probability order, + // s3 is evaluated first and wins the tie. + size_cache->SetPatchSize({69, 72, 73}, 708); + size_cache->SetPatchSize({69, 70}, 658); + size_cache->SetPatchSize({69, 72}, 700); + size_cache->SetPatchSize({69, 73}, 700); + size_cache->SetPatchSize({70, 72}, 400); + size_cache->SetPatchSize({70, 73}, 400); + size_cache->SetPatchSize({70, 72, 73}, 400); + context->patch_size_cache.reset(size_cache); + + auto calc = std::make_shared(probabilities); + Merger merger = *Merger::New( + *context, MergeStrategy::CostBased(ProbabilityProfile(calc), 75, 2), all, + all); + + // Step 1 (Phase 1): {3, 4} merges into s3. + auto step1 = merger.TryNextMerge(); + ASSERT_TRUE(step1.ok()) << step1.status(); + ASSERT_TRUE(step1->has_value()); + EXPECT_EQ((*step1)->base_segment, 3u); + EXPECT_EQ((*step1)->segments, (SegmentSet{3, 4})); + ASSERT_TRUE(context->ReprocessChanged(**step1).ok()); + + // Step 2 (Phase 2): s0 merges with s3 (evaluated before s1 due to higher + // post-merge probability, and breaking before s2 on inert_threshold). + auto step2 = merger.TryNextMerge(); + ASSERT_TRUE(step2.ok()) << step2.status(); + ASSERT_TRUE(step2->has_value()); + EXPECT_EQ((*step2)->base_segment, 0u); + EXPECT_EQ((*step2)->segments, (SegmentSet{0, 3})); +} + } // namespace ift::encoder diff --git a/ift/encoder/closure_glyph_segmenter_test.cc b/ift/encoder/closure_glyph_segmenter_test.cc index 73784aeb..c1bacda8 100644 --- a/ift/encoder/closure_glyph_segmenter_test.cc +++ b/ift/encoder/closure_glyph_segmenter_test.cc @@ -2,6 +2,7 @@ #include +#include "absl/flags/flag.h" #include "gtest/gtest.h" #include "ift/common/bazel_data_file_resolver.h" #include "ift/common/font_data.h" @@ -3390,4 +3391,216 @@ TEST_F(ClosureGlyphSegmenterTest, UnicodeRange_TotalCosts) { EXPECT_NEAR(ur_costs[0].ift_patch_cost, expected_patch_cost, 1e-6); } +namespace { +struct ScopedAllPairsMergingFlag { + explicit ScopedAllPairsMergingFlag(bool enabled) + : prev(absl::GetFlag(FLAGS_experimental_all_pairs_merging)) { + absl::SetFlag(&FLAGS_experimental_all_pairs_merging, enabled); + } + ~ScopedAllPairsMergingFlag() { + absl::SetFlag(&FLAGS_experimental_all_pairs_merging, prev); + } + bool prev; +}; +} // namespace + +TEST_F(ClosureGlyphSegmenterTest, AllPairsMerging_AvoidsGreedyTrap_EndToEnd) { + PatchSizeCacheImpl patch_sizer(roboto.get(), segmenter.brotli_quality()); + double size_a = *patch_sizer.GetPatchSize({69}) + 75.0; + double size_b = *patch_sizer.GetPatchSize({70}) + 75.0; + double size_ab = *patch_sizer.GetPatchSize({69, 70}) + 75.0; + + double p_a = 0.50; + double p_b = 0.45; + double p_c = 0.44; + // Choose P({'a', 'b'}) so that merging {'a'} and {'b'} has a small negative + // cost delta (-5.0 bytes), while {'b'} and {'c'} co-occur (P({'b', 'c'}) = + // 0.45) and have a much larger negative cost delta. + double p_ab = (p_a * size_a + p_b * size_b - 5.0) / size_ab; + + auto calc = std::make_shared( + std::vector>{ + {Segment({'a'}), p_a}, + {Segment({'b'}), p_b}, + {Segment({'c'}), p_c}, + {Segment({'a', 'b'}), p_ab}, + {Segment({'b', 'c'}), 0.45}, + {Segment({'a', 'c'}), 0.94}, + {Segment({'a', 'b', 'c'}), 0.95}, + }); + MergeStrategy strategy = + MergeStrategy::CostBased(ProbabilityProfile(calc), 75, 1); + strategy.SetOptimizationCutoffFraction(0.0); + + std::vector single_base_expected = { + {'a', 'b'}, + {}, + {'c'}, + }; + std::vector all_pairs_expected = { + {'a'}, + {'b', 'c'}, + {}, + }; + + GlyphSegmentation single_base_seg({}, {}, {}); + { + ScopedAllPairsMergingFlag flag_override(false); + auto res = segmenter.CodepointToGlyphSegments( + roboto.get(), {}, {{'a'}, {'b'}, {'c'}}, strategy); + ASSERT_TRUE(res.ok()) << res.status(); + EXPECT_EQ(res->Segments(), single_base_expected); + single_base_seg = *res; + } + + GlyphSegmentation all_pairs_seg({}, {}, {}); + { + ScopedAllPairsMergingFlag flag_override(true); + auto res = segmenter.CodepointToGlyphSegments( + roboto.get(), {}, {{'a'}, {'b'}, {'c'}}, strategy); + ASSERT_TRUE(res.ok()) << res.status(); + EXPECT_EQ(res->Segments(), all_pairs_expected); + all_pairs_seg = *res; + } + + auto single_base_costs = + *segmenter.TotalCosts(roboto.get(), single_base_seg, {calc.get()}); + auto all_pairs_costs = + *segmenter.TotalCosts(roboto.get(), all_pairs_seg, {calc.get()}); + EXPECT_LT(all_pairs_costs[0].ift_patch_cost, + single_base_costs[0].ift_patch_cost); +} + +TEST_F(ClosureGlyphSegmenterTest, + AllPairsMerging_CompositeConditionsAndNonInertMerges) { + ScopedAllPairsMergingFlag flag_override(true); + // Combines conjunctive ligatures ('f', 'i', 'l'), a 3-segment disjunctive + // accent condition (0xC1 Aacute, 0xC9 Eacute, 0xCD Iacute), and inert + // segments ('a', 'b'). + UnicodeFrequencies frequencies{ + {{' ', ' '}, 100}, + {{'a', 'a'}, 95}, + {{'b', 'b'}, 95}, + {{'f', 'f'}, 85}, + {{'i', 'i'}, 85}, + {{'l', 'l'}, 85}, + {{0xC1, 0xC1}, 70}, + {{0xC9, 0xC9}, 70}, + {{0xCD, 0xCD}, 70}, + }; + + MergeStrategy strategy = MergeStrategy::CostBased( + *ProbabilityProfile::Unigram(std::move(frequencies)), 75, 1); + strategy.SetOptimizationCutoffFraction(0.0); + + auto segmentation = CodepointToGlyphSegments( + roboto.get(), {}, + {{'a'}, {'b'}, {'f'}, {'i'}, {'l'}, {0xC1}, {0xC9}, {0xCD}}, strategy); + ASSERT_TRUE(segmentation.ok()) << segmentation.status(); + + std::vector expected_segments = { + {'a', 'b', 'f', 'i', 'l', 0xC1, 0xC9, 0xCD}, + {}, + {}, + {}, + {}, + {}, + {}, + {}, + }; + EXPECT_EQ(segmentation->Segments(), expected_segments); +} + +TEST_F(ClosureGlyphSegmenterTest, + AllPairsMerging_MinGroupSizeAndOptimizationCutoff) { + ScopedAllPairsMergingFlag flag_override(true); + UnicodeFrequencies frequencies{ + {{' ', ' '}, 100}, + {{'a', 'a'}, 95}, + {{'b', 'b'}, 1}, + {{'c', 'c'}, 1}, + {{'d', 'd'}, 1}, + {{'b', 'c'}, 1}, + {{'b', 'd'}, 1}, + {{'c', 'd'}, 1}, + }; + + auto strategy = MergeStrategy::CostBased( + *ProbabilityProfile::Bigram(std::move(frequencies)), 75, 2); + strategy.SetOptimizationCutoffFraction(0.05); + + auto segmentation = CodepointToGlyphSegments( + roboto.get(), {}, {{'a'}, {'b'}, {'c'}, {'d'}}, std::move(strategy)); + ASSERT_TRUE(segmentation.ok()) << segmentation.status(); + + std::vector expected_segments = { + {'a', 'b'}, + {}, + {'c', 'd'}, + {}, + }; + EXPECT_EQ(segmentation->Segments(), expected_segments); +} + +TEST_F(ClosureGlyphSegmenterTest, AllPairsMerging_UnicodeRangeMode) { + ScopedAllPairsMergingFlag flag_override(true); + auto calc = std::make_shared( + std::vector>{ + {Segment({'b'}), 0.6}, + {Segment({'c'}), 0.6}, + {Segment({'f'}), 0.5}, + {Segment({'i'}), 0.5}, + {Segment({'f', 'i'}), 0.75}, + {Segment({'b', 'c'}), 0.84}, + {Segment({'b', 'f', 'i'}), 0.99}, + {Segment({'c', 'f', 'i'}), 0.99}, + {Segment({'b', 'c', 'f', 'i'}), 0.99}, + }); + MergeStrategy strategy = MergeStrategy::CostBased( + ProbabilityProfile(calc), /*network_overhead_cost=*/0, + /*min_group_size=*/1); + strategy.SetUsePatchMerges(false); + + for (uint32_t quality : {0u, 1u}) { + ClosureGlyphSegmenter ur_seg(quality, quality, FIND_CONDITIONS, + CLOSURE_AND_VALIDATE_DEP_GRAPH, resolver, + ift::config::UNICODE_RANGE); + auto ur_segmentation = ur_seg.CodepointToGlyphSegments( + roboto.get(), {'a'}, {{'b'}, {'c'}, {'f'}, {'i'}}, strategy); + ASSERT_TRUE(ur_segmentation.ok()) << ur_segmentation.status(); + for (const auto& cond : ur_segmentation->Conditions()) { + EXPECT_TRUE(cond.IsExclusive()) << cond.ToString(); + } + } +} + +TEST_F(ClosureGlyphSegmenterTest, AllPairsMerging_Determinism) { + ScopedAllPairsMergingFlag flag_override(true); + UnicodeFrequencies frequencies{ + {{' ', ' '}, 100}, {{'a', 'a'}, 90}, {{'b', 'b'}, 90}, + {{'c', 'c'}, 60}, {{'d', 'd'}, 60}, {{'e', 'e'}, 60}, + {{'f', 'f'}, 50}, {{'i', 'i'}, 50}, {{'l', 'l'}, 50}, + {{0xC1, 0xC1}, 30}, {{0x106, 0x106}, 30}, + }; + MergeStrategy strategy = MergeStrategy::CostBased( + *ProbabilityProfile::Unigram(std::move(frequencies)), 75, 2); + strategy.SetOptimizationCutoffFraction(0.0); + + std::vector inputs = { + {'a'}, {'b'}, {'c'}, {'d'}, {'e'}, {'f'}, {'i'}, {'l'}, {0xC1}, {0x106}, + }; + + auto baseline = + CodepointToGlyphSegments(roboto.get(), {}, inputs, strategy); + ASSERT_TRUE(baseline.ok()) << baseline.status(); + + for (int run = 0; run < 5; ++run) { + auto current = + CodepointToGlyphSegments(roboto.get(), {}, inputs, strategy); + ASSERT_TRUE(current.ok()) << current.status(); + EXPECT_EQ(current->Segments(), baseline->Segments()); + EXPECT_EQ(current->ToString(), baseline->ToString()); + } +} + } // namespace ift::encoder \ No newline at end of file diff --git a/ift/encoder/merger.cc b/ift/encoder/merger.cc index 0c6799a7..4a03481b 100644 --- a/ift/encoder/merger.cc +++ b/ift/encoder/merger.cc @@ -148,13 +148,17 @@ StatusOr> Merger::TryNextMerge() { } while (true) { - auto it = candidate_segments_.cbegin(); - if (it == candidate_segments_.cend()) { + segment_index_t base_segment_index; + if (UseAllPairsMerging() && + !all_pairs_merger_.ActiveSegmentsByProb().empty()) { + base_segment_index = + all_pairs_merger_.ActiveSegmentsByProb().begin()->segment_index; + } else if (!candidate_segments_.empty()) { + base_segment_index = *candidate_segments_.cbegin(); + } else { break; } - segment_index_t base_segment_index = *it; - std::optional modified; if (strategy_.UseCosts()) { modified = TRY(MergeSegmentWithCosts(base_segment_index)); @@ -549,34 +553,10 @@ StatusOr Merger::ComputeSegmentCutoff() const { StatusOr> Merger::MergeSegmentWithCosts( uint32_t base_segment_index) { - // TODO(garretrieger): what we are trying to solve here is effectively - // a partitioning problem (finding the partitioning with lowest cost) which is - // NP. - // - // To make this tractable we use a simplistic greedy approach were we - // iteratively select two (or more) segments to merge that lower the overall - // cost. Currently this selects candidates from two sources: - // 1. Start with the highest probability segment, evaluate the cost delta for - // merging it with every other segment. Once no more merges are found, - // move on to the next highest frequency. - // 2. Consider merging the groups of segments that are known to interact as - // these might give slightly better results due to reduction of - // conditional patches. - // - // This approach can likely be improved: - // - Consider all possible pairs instead of just pairs with the highest freq - // item. - // - This could be made tractable by caching the pair wise cost deltas and - // invaldidating specific ones as needed on each merge. - // - After forming an initial greedy based partition try to fine tune by - // randomly moving codepoints between the segments to see if further cost - // reductions can be realized. Can use a computaton budget to set a bound - // on how much time is spent here. - // - // Lastly, currently lacking a good set of frequency data for all unicode - // codepoints this approach has not yet been thoroughly tested. Next steps - // would be to gather some frequency data, test this approach as is, and then - // refine it potentially using some of the proposals noted above. + // TODO(garretrieger): After forming an initial greedy based partition try + // to fine tune by randomly moving codepoints between the segments to + // see if further cost reductions can be realized. Can use a computaton + // budget to set a bound on how much time is spent here. auto base_segment_glyphs = context_->glyph_groupings.ExclusiveGlyphs(base_segment_index); @@ -592,6 +572,11 @@ StatusOr> Merger::MergeSegmentWithCosts( bool min_group_size_met = base_segment.MeetsMinimumGroupSize(strategy_.MinimumGroupSize()); if (min_group_size_met) { + if (UseAllPairsMerging()) { + // Negative cost delta merges have already been exhausted by + // AllPairsMerger. + return std::nullopt; + } // If min group size is met, then we will no longer consider merge's that // have a positive cost delta so start with an existing smallest candidate // set to cost delta 0 which will filter out positive cost delta candidates. @@ -599,11 +584,6 @@ StatusOr> Merger::MergeSegmentWithCosts( CandidateMerge::BaselineCandidate(base_segment_index, 0.0); } - // TODO(garretrieger): On each iteration we should consider all merge pairs - // rather than limiting ourselves just to pairs involving a single - // base_segment_index. This will take some care to keep it performant - // however. We'd likely need a priority queue to cache deltas with a - // way of invalidating any pairs that are changed by each merge operation. TRYV(CollectExclusiveCandidateMerges(base_segment_index, smallest_candidate_merge)); @@ -728,28 +708,28 @@ Status Merger::CollectExclusiveCandidateMerges( base_size, base_probability, smallest_candidate_merge->CostDelta()); } - for (auto it = candidate_segments_.lower_bound(base_segment_index); - it != candidate_segments_.end(); it++) { - if (*it == base_segment_index) { - continue; + auto evaluate_candidate = + [&](segment_index_t segment_index) -> StatusOr { + if (segment_index == base_segment_index) { + return true; } - segment_index_t segment_index = *it; - if (segment_index >= optimization_cutoff_segment_ && smallest_candidate_merge.has_value()) { // We are at the optimization cutoff, so we won't evaluate any further // candidates beyond what is need to select at least one. Since a // candidate already exists, we can stop here. - return absl::OkStatus(); + return false; } - if (context_->InertSegments().contains(segment_index) && - TRY(AggregateProbability(segment_index)) <= inert_threshold) { - // Since we iteration is in probability order from highest to lowest, once - // one segment fails the threshold then we know all further ones will as - // well. - break; + if (context_->InertSegments().contains(segment_index)) { + double segment_probability = TRY(AggregateProbability(segment_index)); + if (segment_probability <= inert_threshold) { + // Since iteration is in probability order from highest to lowest, once + // one segment fails the threshold then we know all further ones will as + // well. + return false; + } } auto segment_glyphs = @@ -757,7 +737,7 @@ Status Merger::CollectExclusiveCandidateMerges( if (segment_glyphs.empty()) { // This segment has no exclusive glyphs, so no need to consider it for a // merge. - continue; + return true; } SegmentSet triggering_segments{segment_index}; @@ -769,6 +749,32 @@ Status Merger::CollectExclusiveCandidateMerges( inert_threshold = BestCaseInertProbabilityThreshold( base_size, base_probability, smallest_candidate_merge->CostDelta()); } + return true; + }; + + if (UseAllPairsMerging()) { + for (const auto& entry : all_pairs_merger_.ActiveSegmentsByProb()) { + if (!TRY(evaluate_candidate(entry.segment_index))) { + break; + } + } + if (!smallest_candidate_merge.has_value()) { + for (auto it = + candidate_segments_.lower_bound(optimization_cutoff_segment_); + it != candidate_segments_.end(); ++it) { + if (!TRY(evaluate_candidate(*it))) { + break; + } + } + } + return absl::OkStatus(); + } + + for (auto it = candidate_segments_.lower_bound(base_segment_index); + it != candidate_segments_.end(); it++) { + if (!TRY(evaluate_candidate(*it))) { + break; + } } return absl::OkStatus(); From 3c15e99adf8439c73349a979f5fefbaaed76f04c Mon Sep 17 00:00:00 2001 From: Garret Rieger Date: Wed, 21 Oct 2026 17:08:46 +0000 Subject: [PATCH 06/13] WIP test out different normalization modes for all pairs. --- ift/encoder/all_pairs_merger.cc | 103 +++++++++++++++++- ift/encoder/all_pairs_merger.h | 19 ++++ ift/encoder/candidate_merge_test.cc | 158 ++++++++++++++++++++++++++++ 3 files changed, 278 insertions(+), 2 deletions(-) diff --git a/ift/encoder/all_pairs_merger.cc b/ift/encoder/all_pairs_merger.cc index ac561a78..f5a00232 100644 --- a/ift/encoder/all_pairs_merger.cc +++ b/ift/encoder/all_pairs_merger.cc @@ -3,11 +3,15 @@ #include #include #include +#include #include #include "absl/container/btree_set.h" +#include "absl/container/flat_hash_set.h" +#include "absl/flags/flag.h" #include "absl/status/status.h" #include "absl/status/statusor.h" +#include "absl/strings/string_view.h" #include "ift/common/int_set.h" #include "ift/common/try.h" #include "ift/encoder/activation_condition.h" @@ -16,6 +20,14 @@ #include "ift/encoder/merger.h" #include "ift/encoder/types.h" +ABSL_FLAG(ift::encoder::AllPairsNormalizationMode, + experimental_all_pairs_normalization, + ift::encoder::AllPairsNormalizationMode::kNone, + "Controls how cost deltas are normalized when ranking candidate " + "merges in all-pairs mode: 'none' (no normalization), 'segment' " + "(divide by merged_segments.size() - 1), or 'byte' (divide by " + "pre-merge patch bytes including network overhead)."); + using absl::btree_set; using absl::Status; using absl::StatusOr; @@ -24,6 +36,36 @@ using ift::common::SegmentSet; namespace ift::encoder { +bool AbslParseFlag(absl::string_view text, AllPairsNormalizationMode* mode, + std::string* error) { + if (text == "none") { + *mode = AllPairsNormalizationMode::kNone; + return true; + } + if (text == "segment" || text == "segments") { + *mode = AllPairsNormalizationMode::kSegment; + return true; + } + if (text == "byte" || text == "bytes") { + *mode = AllPairsNormalizationMode::kByte; + return true; + } + *error = "unknown normalization mode; expected 'none', 'segment', or 'byte'"; + return false; +} + +std::string AbslUnparseFlag(AllPairsNormalizationMode mode) { + switch (mode) { + case AllPairsNormalizationMode::kNone: + return "none"; + case AllPairsNormalizationMode::kSegment: + return "segment"; + case AllPairsNormalizationMode::kByte: + return "byte"; + } + return "none"; +} + void AllPairsMerger::AssignMergedSegment(const Merger& merger, segment_index_t base, const SegmentSet& to_merge) { @@ -108,6 +150,60 @@ Status AllPairsMerger::UpdateActiveSegment(Merger& merger, segment_index_t s) { return absl::OkStatus(); } +StatusOr AllPairsMerger::NormalizationDivisor( + Merger& merger, const SegmentSet& merged_segments) const { + switch (absl::GetFlag(FLAGS_experimental_all_pairs_normalization)) { + case AllPairsNormalizationMode::kNone: + return 1.0; + case AllPairsNormalizationMode::kSegment: + return static_cast(merged_segments.size() - 1); + case AllPairsNormalizationMode::kByte: { + const uint32_t overhead = merger.Strategy().NetworkOverheadCost(); + const auto& context = merger.Context(); + if (merged_segments.size() == 2) { + segment_index_t u = *merged_segments.min(); + segment_index_t v = *merged_segments.max(); + if (context.InertSegments().contains(u) && + context.InertSegments().contains(v)) { + uint32_t size_u = TRY(merger.ExclusivePatchSize(u)); + uint32_t size_v = TRY(merger.ExclusivePatchSize(v)); + if (size_u > 0 && size_v > 0) { + return static_cast(size_u + overhead) + + static_cast(size_v + overhead); + } + } + } + + const bool unicode_range_mode = + (context.GetTargetMode() == config::UNICODE_RANGE); + const auto& groupings = context.glyph_groupings; + const auto& conditions_and_glyphs = groupings.ConditionsAndGlyphs(); + absl::flat_hash_set seen_conditions; + uint64_t total_bytes = 0; + for (segment_index_t s : merged_segments) { + for (const auto& c : groupings.TriggeringSegmentToConditions(s)) { + if (c.IsFallback() || (unicode_range_mode && !c.IsExclusive())) { + continue; + } + auto it = conditions_and_glyphs.find(c); + if (it == conditions_and_glyphs.end()) { + return absl::InternalError( + "Condition which should be present wasn't found."); + } + if (!seen_conditions.insert(&it->first).second) { + continue; + } + uint32_t patch_size = + TRY(context.patch_size_cache->GetPatchSize(it->second)); + total_bytes += static_cast(patch_size + overhead); + } + } + return total_bytes > 0 ? static_cast(total_bytes) : 1.0; + } + } + return 1.0; +} + Status AllPairsMerger::QueueBestCaseCandidate( Merger& merger, const SegmentSet& merged_segments) { if (merged_segments.size() <= 1) { @@ -126,8 +222,10 @@ Status AllPairsMerger::QueueBestCaseCandidate( } double delta = TRY(best_case_delta); if (delta < 0.0) { + double norm_divisor = TRY(NormalizationDivisor(merger, merged_segments)); queue_.push(QueuedCandidate{ - .cost_delta = delta, + .cost_delta = delta / norm_divisor, + .norm_divisor = norm_divisor, .is_exact = false, .segments = merged_segments, .created_epoch = current_epoch_, @@ -373,7 +471,8 @@ StatusOr> AllPairsMerger::TryNextAllPairsMerge( merger, base, top.segments, baseline)); if (candidate_merge.has_value() && candidate_merge->CostDelta() < 0.0) { queue_.push(QueuedCandidate{ - .cost_delta = candidate_merge->CostDelta(), + .cost_delta = candidate_merge->CostDelta() / top.norm_divisor, + .norm_divisor = top.norm_divisor, .is_exact = true, .segments = std::move(top.segments), .created_epoch = current_epoch_, diff --git a/ift/encoder/all_pairs_merger.h b/ift/encoder/all_pairs_merger.h index c6bf4984..44cce868 100644 --- a/ift/encoder/all_pairs_merger.h +++ b/ift/encoder/all_pairs_merger.h @@ -4,11 +4,14 @@ #include #include #include +#include #include #include "absl/container/btree_set.h" +#include "absl/flags/declare.h" #include "absl/status/status.h" #include "absl/status/statusor.h" +#include "absl/strings/string_view.h" #include "ift/common/int_set.h" #include "ift/encoder/candidate_merge.h" #include "ift/encoder/invalidation_set.h" @@ -18,6 +21,16 @@ namespace ift::encoder { class Merger; +enum class AllPairsNormalizationMode { + kNone, + kSegment, + kByte, +}; + +bool AbslParseFlag(absl::string_view text, AllPairsNormalizationMode* mode, + std::string* error); +std::string AbslUnparseFlag(AllPairsNormalizationMode mode); + // Implements Phase 1 all-pairs cost-reducing segment merging for Merger using a // lazy two-stage (best-case lower bound -> exact) priority queue. class AllPairsMerger { @@ -62,6 +75,7 @@ class AllPairsMerger { private: struct QueuedCandidate { double cost_delta; + double norm_divisor = 1.0; bool is_exact; // Canonical set of all segments participating in the merge (size >= 2). // The surviving base segment is always *segments.min(). @@ -95,6 +109,8 @@ class AllPairsMerger { } absl::Status UpdateActiveSegment(Merger& merger, segment_index_t s); void RemoveActiveSegment(segment_index_t s); + absl::StatusOr NormalizationDivisor( + Merger& merger, const ift::common::SegmentSet& merged_segments) const; absl::Status QueueBestCaseCandidate( Merger& merger, const ift::common::SegmentSet& merged_segments); absl::Status QueueLowerRankedExclusivePairs( @@ -124,4 +140,7 @@ class AllPairsMerger { } // namespace ift::encoder +ABSL_DECLARE_FLAG(ift::encoder::AllPairsNormalizationMode, + experimental_all_pairs_normalization); + #endif // IFT_ENCODER_ALL_PAIRS_MERGER_H_ diff --git a/ift/encoder/candidate_merge_test.cc b/ift/encoder/candidate_merge_test.cc index bdc528db..fe4f3cdb 100644 --- a/ift/encoder/candidate_merge_test.cc +++ b/ift/encoder/candidate_merge_test.cc @@ -1788,6 +1788,17 @@ struct ScopedAllPairsMergingFlag { } bool prev; }; + +struct ScopedAllPairsNormalizationFlag { + explicit ScopedAllPairsNormalizationFlag(AllPairsNormalizationMode mode) + : prev(absl::GetFlag(FLAGS_experimental_all_pairs_normalization)) { + absl::SetFlag(&FLAGS_experimental_all_pairs_normalization, mode); + } + ~ScopedAllPairsNormalizationFlag() { + absl::SetFlag(&FLAGS_experimental_all_pairs_normalization, prev); + } + AllPairsNormalizationMode prev; +}; } // namespace TEST_F(CandidateMergeTest, AllPairsMerging_AvoidsSingleBaseGreedyTrap) { @@ -2037,4 +2048,151 @@ TEST_F(CandidateMergeTest, EXPECT_EQ((*step2)->segments, (SegmentSet{0, 3})); } +TEST_F(CandidateMergeTest, AllPairsMerging_NormalizationModes) { + AllPairsNormalizationMode parsed_mode = AllPairsNormalizationMode::kNone; + std::string parse_error; + EXPECT_TRUE(AbslParseFlag("none", &parsed_mode, &parse_error)); + EXPECT_EQ(parsed_mode, AllPairsNormalizationMode::kNone); + EXPECT_EQ(AbslUnparseFlag(parsed_mode), "none"); + EXPECT_TRUE(AbslParseFlag("segment", &parsed_mode, &parse_error)); + EXPECT_EQ(parsed_mode, AllPairsNormalizationMode::kSegment); + EXPECT_EQ(AbslUnparseFlag(parsed_mode), "segment"); + EXPECT_TRUE(AbslParseFlag("byte", &parsed_mode, &parse_error)); + EXPECT_EQ(parsed_mode, AllPairsNormalizationMode::kByte); + EXPECT_EQ(AbslUnparseFlag(parsed_mode), "byte"); + EXPECT_FALSE(AbslParseFlag("invalid", &parsed_mode, &parse_error)); + + // Set up three independent candidate merges with different segment counts + // and pre-merge byte footprints (network overhead = 75, P = 0.50): + // - {0, 1, 2} (aacute, eacute, cacute sharing acute component): + // 3 segments (divisor = 2), 1400 pre-merge bytes (400 + 400 + 400 + 200), + // exact delta = -90.0 -> kNone: -90.0, kSegment: -45.0, kByte: -0.0643 + // - {3, 4} ('d', 'b'): + // 2 segments (divisor = 1), 800 pre-merge bytes (600 + 200), + // exact delta = -68.0 -> kNone: -68.0, kSegment: -68.0, kByte: -0.0850 + // - {5, 6} ('p', 'q'): + // 2 segments (divisor = 1), 400 pre-merge bytes (200 + 200), + // exact delta = -62.0 -> kNone: -62.0, kSegment: -62.0, kByte: -0.1550 + std::vector segments = { + {{0x00E1}}, // s0: aacute + {{0x00E9}}, // s1: eacute + {{0x0107}}, // s2: cacute + {{'d'}}, // s3 + {{'b'}}, // s4 + {{'p'}}, // s5 + {{'q'}}, // s6 + }; + + std::vector> probs = { + {{{0x00E1}}, 0.50}, + {{{0x00E9}}, 0.50}, + {{{0x0107}}, 0.50}, + {{{'d'}}, 0.50}, + {{{'b'}}, 0.50}, + {{{'p'}}, 0.50}, + {{{'q'}}, 0.50}, + {{{0x00E1, 0x00E9, 0x0107}}, 0.50}, + {{{'d', 'b'}}, 0.50}, + {{{'p', 'q'}}, 0.50}, + }; + // Default all other merged combinations to probability 1.0 so their + // best-case cost deltas are strictly positive and never queued. + for (uint32_t mask = 1; mask < (1u << segments.size()); ++mask) { + SubsetDefinition def; + for (size_t i = 0; i < segments.size(); ++i) { + if (mask & (1u << i)) { + def.Union(segments[i].Definition()); + } + } + probs.push_back({Segment(def), 1.0}); + } + + auto run_with_mode = [&](AllPairsNormalizationMode mode) { + ScopedAllPairsMergingFlag all_pairs_flag(true); + ScopedAllPairsNormalizationFlag norm_flag(mode); + + ClosureGlyphSegmenter segmenter(8, 8, PATCH, CLOSURE_ONLY, resolver); + auto context = SegmentationContext::InitializeSegmentationContext( + roboto.get(), {}, segments, segmenter.unmapped_glyph_handling(), + segmenter.condition_analysis_mode(), segmenter.brotli_quality(), + segmenter.init_font_merging_brotli_quality(), resolver); + EXPECT_TRUE(context.ok()) << context.status(); + + const auto& groupings = context->glyph_groupings; + GlyphSet g0 = groupings.ExclusiveGlyphs(0); + GlyphSet g1 = groupings.ExclusiveGlyphs(1); + GlyphSet g2 = groupings.ExclusiveGlyphs(2); + GlyphSet g3 = groupings.ExclusiveGlyphs(3); + GlyphSet g4 = groupings.ExclusiveGlyphs(4); + GlyphSet g5 = groupings.ExclusiveGlyphs(5); + GlyphSet g6 = groupings.ExclusiveGlyphs(6); + GlyphSet g_acute = groupings.ConditionsAndGlyphs().at( + ActivationCondition::or_segments({0, 1, 2}, 0)); + + GlyphSet g012 = g0; + g012.union_set(g1); + g012.union_set(g2); + g012.union_set(g_acute); + GlyphSet g34 = g3; + g34.union_set(g4); + GlyphSet g56 = g5; + g56.union_set(g6); + + auto* size_cache = new MockPatchSizeCache(); + size_cache->SetPatchSize(g0, 325); + size_cache->SetPatchSize(g1, 325); + size_cache->SetPatchSize(g2, 325); + size_cache->SetPatchSize(g_acute, 125); + size_cache->SetPatchSize(g012, 1145); + + size_cache->SetPatchSize(g3, 525); + size_cache->SetPatchSize(g4, 125); + size_cache->SetPatchSize(g34, 589); + + size_cache->SetPatchSize(g5, 125); + size_cache->SetPatchSize(g6, 125); + size_cache->SetPatchSize(g56, 201); + context->patch_size_cache.reset(size_cache); + + MergeStrategy strategy = MergeStrategy::CostBased( + ProbabilityProfile(std::make_shared(probs)), + 75, 1); + + Merger merger = *Merger::New(*context, strategy, all, all); + std::vector merge_order; + while (true) { + auto modified = merger.TryNextMerge(); + EXPECT_TRUE(modified.ok()) << modified.status(); + if (!modified->has_value()) { + break; + } + merge_order.push_back((*modified)->segments); + EXPECT_TRUE(context->ReprocessChanged(**modified).ok()); + } + return merge_order; + }; + + std::vector expected_none = { + SegmentSet{0, 1, 2}, + SegmentSet{3, 4}, + SegmentSet{5, 6}, + }; + EXPECT_EQ(run_with_mode(AllPairsNormalizationMode::kNone), expected_none); + + std::vector expected_segment = { + SegmentSet{3, 4}, + SegmentSet{5, 6}, + SegmentSet{0, 1, 2}, + }; + EXPECT_EQ(run_with_mode(AllPairsNormalizationMode::kSegment), + expected_segment); + + std::vector expected_byte = { + SegmentSet{5, 6}, + SegmentSet{3, 4}, + SegmentSet{0, 1, 2}, + }; + EXPECT_EQ(run_with_mode(AllPairsNormalizationMode::kByte), expected_byte); +} + } // namespace ift::encoder From 452651955189ec2ba8a9e6e2cd055b08ebea5be1 Mon Sep 17 00:00:00 2001 From: Garret Rieger Date: Thu, 22 Oct 2026 23:15:39 +0000 Subject: [PATCH 07/13] In ComputeCostDelta() special case patches which don't change size. Some patches have their condition changed without changing the corresponding patch, thus the new patch has the same size. The delta contribution for these is then (Pnew - Pold) * (patch_size + k) where Pnew >= Pold. As a result we can estimate a best case contribution of 0 without having to calculate Pnew or Pold. Furthermore there are some cases where a condition change, but is effectively the same condition. Which means Pnew = Pold. In this case we can immediately calculate the delta as 0 in non best case analysis without evaluating sizes or probabilities. This updates ComputeCostDelta() to detect these cases and skip probability/size calcs when possible. --- ift/encoder/activation_condition.cc | 16 ++ ift/encoder/activation_condition.h | 6 + ift/encoder/activation_condition_test.cc | 38 ++++ ift/encoder/candidate_merge.cc | 97 +++++++-- ift/encoder/candidate_merge_test.cc | 211 ++++++++++++++++++++ ift/encoder/closure_glyph_segmenter_test.cc | 22 +- 6 files changed, 355 insertions(+), 35 deletions(-) diff --git a/ift/encoder/activation_condition.cc b/ift/encoder/activation_condition.cc index 3151d0f7..0c96c770 100644 --- a/ift/encoder/activation_condition.cc +++ b/ift/encoder/activation_condition.cc @@ -252,6 +252,22 @@ bool ActivationCondition::Intersects(const common::SegmentSet& segments) const { return false; } +bool ActivationCondition::IsUnchangedByMerge( + const common::SegmentSet& merged_segments) const { + size_t merge_size = merged_segments.size(); + if (merge_size <= 1) { + return true; + } + for (const auto& condition_group : conditions_) { + if (condition_group.intersects(merged_segments) && + (condition_group.size() < merge_size || + !merged_segments.is_subset_of(condition_group))) { + return false; + } + } + return true; +} + ActivationCondition ActivationCondition::ReplaceSegments( segment_index_t base_segment, const SegmentSet& segments) const { if (IsPurelyConjunctive()) { diff --git a/ift/encoder/activation_condition.h b/ift/encoder/activation_condition.h index b99ae97c..bd2713a5 100644 --- a/ift/encoder/activation_condition.h +++ b/ift/encoder/activation_condition.h @@ -124,6 +124,12 @@ class ActivationCondition { // Returns true if any of the condition groups intersects with 'segments'. bool Intersects(const common::SegmentSet& segments) const; + // Returns true if merging 'merged_segments' into a single segment does not + // change the effective activation condition (i.e. every disjunctive + // subgroup that intersects 'merged_segments' has 'merged_segments' as a + // subset). + bool IsUnchangedByMerge(const common::SegmentSet& merged_segments) const; + // Generate a new condition which is a copy of this one but with all // occurences of 'segments' replaced with base_segment. New condition will be // simplified to remove any resulting redundancy. diff --git a/ift/encoder/activation_condition_test.cc b/ift/encoder/activation_condition_test.cc index 0cd21e6f..d7580094 100644 --- a/ift/encoder/activation_condition_test.cc +++ b/ift/encoder/activation_condition_test.cc @@ -659,4 +659,42 @@ TEST(ActivationConditionTest, HashCaching) { EXPECT_EQ(cond, cond_copy); } +TEST(ActivationConditionTest, IsUnchangedByMerge) { + // Empty or singleton merge sets never change any condition. + EXPECT_TRUE(ActivationCondition::or_segments({1, 2}, 0).IsUnchangedByMerge({})); + EXPECT_TRUE(ActivationCondition::or_segments({1, 2}, 0).IsUnchangedByMerge({1})); + EXPECT_TRUE(ActivationCondition::True(0).IsUnchangedByMerge({1, 2})); + + // Non-intersecting conditions are unchanged. + EXPECT_TRUE(ActivationCondition::or_segments({3, 4}, 0).IsUnchangedByMerge({1, 2})); + EXPECT_TRUE(ActivationCondition::and_segments({3, 4}, 0).IsUnchangedByMerge({1, 2})); + + // Disjunctive condition that is a superset of the merge group is unchanged. + EXPECT_TRUE(ActivationCondition::or_segments({1, 2, 3}, 0).IsUnchangedByMerge({1, 2})); + EXPECT_TRUE(ActivationCondition::or_segments({1, 2}, 0).IsUnchangedByMerge({1, 2})); + + // Disjunctive condition that partially intersects the merge group is changed. + EXPECT_FALSE(ActivationCondition::or_segments({1, 3}, 0).IsUnchangedByMerge({1, 2})); + + // Conjunctive condition whose clauses are singletons can never have a + // 2+-element merge group as a subset. + EXPECT_FALSE(ActivationCondition::exclusive_segment(1, 0).IsUnchangedByMerge({1, 2})); + EXPECT_FALSE(ActivationCondition::and_segments({1, 2}, 0).IsUnchangedByMerge({1, 2})); + EXPECT_FALSE(ActivationCondition::and_segments({1, 3}, 0).IsUnchangedByMerge({1, 2})); + + // Composite condition: unchanged if all intersecting subgroups contain the + // merge group as a subset. + EXPECT_TRUE(ActivationCondition::composite_condition({{1, 2, 3}, {4, 5}}, 0) + .IsUnchangedByMerge({1, 2})); + EXPECT_TRUE(ActivationCondition::composite_condition({{1, 2}, {1, 2, 4}}, 0) + .IsUnchangedByMerge({1, 2})); + + // Composite condition: changed if any intersecting subgroup does not contain + // the entire merge group. + EXPECT_FALSE(ActivationCondition::composite_condition({{1, 3}, {4, 5}}, 0) + .IsUnchangedByMerge({1, 2})); + EXPECT_FALSE(ActivationCondition::composite_condition({{1, 2, 3}, {2, 4}}, 0) + .IsUnchangedByMerge({1, 2})); +} + } // namespace ift::encoder diff --git a/ift/encoder/candidate_merge.cc b/ift/encoder/candidate_merge.cc index 28b4828c..b3ff016e 100644 --- a/ift/encoder/candidate_merge.cc +++ b/ift/encoder/candidate_merge.cc @@ -783,6 +783,11 @@ StatusOr CandidateMerge::ComputeCostDelta( // Map value is the set of glyphs and patch size metadata associated with the // new condition. struct Info { + const ActivationCondition* first_condition = nullptr; + const GlyphSet* first_glyphs = nullptr; + uint32_t source_count = 0; + bool glyph_set_unchanged = false; + bool skip_new_condition = false; GlyphSet glyphs; uint32_t largest_patch_size = 0; uint32_t largest_incremental_size = 0; @@ -804,6 +809,34 @@ StatusOr CandidateMerge::ComputeCostDelta( double current_delta = 0.0; bool has_unitary = false; const uint32_t per_request_overhead = merger.Strategy().NetworkOverheadCost(); + + auto add_removed_condition = [&](const ActivationCondition* condition, + const GlyphSet* glyphs, + Info& info) -> Status { + uint32_t patch_size = TRY(patch_size_cache->GetPatchSize(*glyphs)); + if (!unicode_range_mode || condition->IsExclusive()) { + double p = TRY(merger.AggregateProbability(*condition)); + double d = p * (patch_size + per_request_overhead); + deltas.push_back(-d); + current_delta -= d; + VLOG(1) << " - (" << p << " * " << (patch_size + per_request_overhead) + << ") -> " << d << " [removed patch " << condition->ToString() + << "]"; + } + + if (!best_case && !info.glyph_set_unchanged) { + info.glyphs.union_set(*glyphs); + } + + uint32_t incremental_size = + (patch_size > base_overhead) ? (patch_size - base_overhead) : 0; + info.largest_patch_size = std::max(info.largest_patch_size, patch_size); + info.largest_incremental_size = + std::max(info.largest_incremental_size, incremental_size); + info.combined_incremental_size += incremental_size; + return absl::OkStatus(); + }; + for (const auto& [condition, glyphs] : modified_conditions) { ActivationCondition updated = condition->ReplaceSegments(base, merged_segments); @@ -818,33 +851,25 @@ StatusOr CandidateMerge::ComputeCostDelta( continue; } - uint32_t patch_size = TRY(patch_size_cache->GetPatchSize(*glyphs)); - if (!unicode_range_mode || condition->IsExclusive()) { - double p = TRY(merger.AggregateProbability(*condition)); - double d = p * (patch_size + per_request_overhead); - deltas.push_back(-d); - current_delta -= d; - VLOG(1) << " - (" << p << " * " << (patch_size + per_request_overhead) - << ") -> " << d << " [removed patch " << condition->ToString() - << "]"; - } - if (!best_case && updated.IsUnitary()) { has_unitary = true; } auto [it, did_insert] = new_conditions.try_emplace(std::move(updated)); auto& info = it->second; - if (!best_case) { - info.glyphs.union_set(*glyphs); + if (did_insert) { + info.first_condition = condition; + info.first_glyphs = glyphs; + info.source_count = 1; + continue; } - uint32_t incremental_size = - (patch_size > base_overhead) ? (patch_size - base_overhead) : 0; - info.largest_patch_size = std::max(info.largest_patch_size, patch_size); - info.largest_incremental_size = - std::max(info.largest_incremental_size, incremental_size); - info.combined_incremental_size += incremental_size; + if (info.source_count == 1) { + TRYV(add_removed_condition(info.first_condition, info.first_glyphs, + info)); + } + info.source_count++; + TRYV(add_removed_condition(condition, glyphs, info)); } if (unicode_range_mode && !best_case && exclusive_gids.has_value() && @@ -852,6 +877,34 @@ StatusOr CandidateMerge::ComputeCostDelta( new_conditions.try_emplace(ActivationCondition::or_segments({base}, 0)); } + for (auto& [c, info] : new_conditions) { + if (info.source_count != 1) { + continue; + } + bool was_active_before_merge = + (!unicode_range_mode || info.first_condition->IsExclusive()); + bool glyph_set_unchanged = true; + if (!best_case && exclusive_gids.has_value()) { + // Check if the provided exclusive gids will cause a change to this condition's glyphs. + // exclusive_gids are added to the unitary condition for this merge, and subtracted + // from everything else. + if (c.IsUnitary()) { + glyph_set_unchanged = exclusive_gids->is_subset_of(*info.first_glyphs); + } else { + glyph_set_unchanged = !info.first_glyphs->intersects(*exclusive_gids); + } + } + info.glyph_set_unchanged = glyph_set_unchanged; + if (glyph_set_unchanged && + (best_case || + (was_active_before_merge && + info.first_condition->IsUnchangedByMerge(merged_segments)))) { + info.skip_new_condition = true; + continue; + } + TRYV(add_removed_condition(info.first_condition, info.first_glyphs, info)); + } + double initial_magnitude_sum = -current_delta; bool fallback_changed = @@ -883,6 +936,10 @@ StatusOr CandidateMerge::ComputeCostDelta( double best_case_reduction_fraction = merger.Strategy().BestCaseSizeReductionFraction(); for (auto& [c, info] : new_conditions) { + if (info.skip_new_condition) { + continue; + } + uint32_t size = 0; if (best_case) { uint32_t extra = @@ -890,6 +947,8 @@ StatusOr CandidateMerge::ComputeCostDelta( extra = std::max((uint32_t)(extra * best_case_reduction_fraction), Merger::BEST_CASE_MERGE_SIZE_DELTA); size = info.largest_patch_size + extra; + } else if (info.glyph_set_unchanged) { + size = info.largest_patch_size; } else { if (exclusive_gids.has_value()) { // When we do not have complete glyph conditions then exclusive_gids diff --git a/ift/encoder/candidate_merge_test.cc b/ift/encoder/candidate_merge_test.cc index fe4f3cdb..4545939b 100644 --- a/ift/encoder/candidate_merge_test.cc +++ b/ift/encoder/candidate_merge_test.cc @@ -2195,4 +2195,215 @@ TEST_F(CandidateMergeTest, AllPairsMerging_NormalizationModes) { EXPECT_EQ(run_with_mode(AllPairsNormalizationMode::kByte), expected_byte); } +TEST_F(CandidateMergeTest, ComputeCostDelta_BestCaseSkipsUnchangedGlyphSetConditions) { + // s0: 'a' -> {69} + // s1: 'f' -> {74} + // s2: 'i' -> {77} + // s1 AND s2: 'f' + 'i' ligature -> {444, 446} + std::vector segments = { + {{'a'}}, + {{'f'}}, + {{'i'}}, + }; + freq::UnicodeFrequencies frequencies{ + {{' ', ' '}, 100}, {{'a', 'a'}, 50}, {{'f', 'f'}, 75}, {{'i', 'i'}, 95}}; + + ClosureGlyphSegmenter segmenter(8, 8, PATCH, CLOSURE_ONLY, resolver); + auto context = SegmentationContext::InitializeSegmentationContext( + roboto.get(), {}, segments, segmenter.unmapped_glyph_handling(), + segmenter.condition_analysis_mode(), segmenter.brotli_quality(), + segmenter.init_font_merging_brotli_quality(), resolver); + ASSERT_TRUE(context.ok()) << context.status(); + + Merger merger = *Merger::New( + *context, + MergeStrategy::CostBased( + *ProbabilityProfile::Unigram(std::move(frequencies)), 75, 4), + all, all); + + MockPatchSizeCache* size_cache = new MockPatchSizeCache(); + size_cache->SetPatchSize({69}, 200); + size_cache->SetPatchSize({74}, 300); + size_cache->SetPatchSize({444, 446}, 150); + context->patch_size_cache.reset(size_cache); + + // When merging {0, 1} ('a' and 'f'): + // - {s0} and {s1} merge into {s0} (glyph sets change, source_count == 2). + // - (s1 AND s2) maps to (s0 AND s2). Its glyph set {444, 446} is unchanged. + // In best-case mode, the delta contribution for (s1 AND s2) is treated as 0, + // and its patch size {444, 446} is never queried. + size_cache->ResetCallCount(); + auto best_case_delta = CandidateMerge::ComputeCostDelta( + merger, {0, 1}, std::nullopt, std::nullopt); + ASSERT_TRUE(best_case_delta.ok()) << best_case_delta.status(); + + // Only the two merging exclusive patches {69} and {74} are queried for size. + EXPECT_EQ(size_cache->GetPatchSizeCallCount(), 2); + + // Expected best-case delta: only reflects {69} and {74} merging into {s0}. + double p_a_or_f = 0.50 + 0.75 - 0.50 * 0.75; + // combined_incremental = (200 - 0) + (300 - 0) = 500, largest = 300. + // extra = max((500 - 300) * 0.5, 6) = 100. + // merged_size = 300 + 100 = 400. + double expected_delta = -0.50 * (200 + 75) - 0.75 * (300 + 75) + + p_a_or_f * (400 + 75); + EXPECT_NEAR(*best_case_delta, expected_delta, 1e-9); +} + +TEST_F(CandidateMergeTest, ComputeCostDelta_SkipsUnchangedEffectiveConditions) { + // s0: aacute (0x00E1) + // s1: eacute (0x00E9) + // s2: cacute (0x0107) + // All three share the acute accent glyphs {117, 169} via (s0 OR s1 OR s2). + // Because they have composite conditions, s0, s1, s2 are NOT inert. + std::vector segments = { + {{0x00E1}}, + {{0x00E9}}, + {{0x0107}}, + }; + freq::UnicodeFrequencies frequencies{ + {{' ', ' '}, 100}, + {{0x00E1, 0x00E1}, 50}, + {{0x00E9, 0x00E9}, 75}, + {{0x0107, 0x0107}, 95}}; + + ClosureGlyphSegmenter segmenter(8, 8, PATCH, CLOSURE_ONLY, resolver); + auto context = SegmentationContext::InitializeSegmentationContext( + roboto.get(), {}, segments, segmenter.unmapped_glyph_handling(), + segmenter.condition_analysis_mode(), segmenter.brotli_quality(), + segmenter.init_font_merging_brotli_quality(), resolver); + ASSERT_TRUE(context.ok()) << context.status(); + ASSERT_FALSE(context->InertSegments().contains(0)); + ASSERT_FALSE(context->InertSegments().contains(1)); + + Merger merger = *Merger::New( + *context, + MergeStrategy::CostBased( + *ProbabilityProfile::Unigram(std::move(frequencies)), 75, 4), + all, all); + + const auto& groupings = context->glyph_groupings; + GlyphSet g0 = groupings.ExclusiveGlyphs(0); + GlyphSet g1 = groupings.ExclusiveGlyphs(1); + GlyphSet g01 = g0; + g01.union_set(g1); + GlyphSet g_acute = groupings.ConditionsAndGlyphs().at( + ActivationCondition::or_segments({0, 1, 2}, 0)); + + MockPatchSizeCache* size_cache = new MockPatchSizeCache(); + size_cache->SetPatchSize(g0, 200); + size_cache->SetPatchSize(g1, 300); + size_cache->SetPatchSize(g01, 450); + size_cache->SetPatchSize(g_acute, 500); // Unchanged condition patch + context->patch_size_cache.reset(size_cache); + + // When merging {0, 1}: + // (s0 OR s1 OR s2) has {0, 1} as a subset of its intersecting subgroup, so + // IsUnchangedByMerge({0, 1}) is true, and its glyph set g_acute does not + // change. Its cost delta is 0 in both best-case and exact mode, and g_acute's + // patch size is never queried. + size_cache->ResetCallCount(); + auto best_delta = CandidateMerge::ComputeCostDelta( + merger, {0, 1}, std::nullopt, std::nullopt); + ASSERT_TRUE(best_delta.ok()) << best_delta.status(); + EXPECT_EQ(size_cache->GetPatchSizeCallCount(), 2); // only g0 and g1 + + double p_0_or_1 = 0.50 + 0.75 - 0.50 * 0.75; + // best-case merged size: 300 + max((500 - 300) * 0.5, 6) = 400. + double expected_best = + -0.50 * (200 + 75) - 0.75 * (300 + 75) + p_0_or_1 * (400 + 75); + EXPECT_NEAR(*best_delta, expected_best, 1e-9); + + size_cache->ResetCallCount(); + auto exact_delta = CandidateMerge::ComputeCostDelta( + merger, {0, 1}, g01, std::nullopt); + ASSERT_TRUE(exact_delta.ok()) << exact_delta.status(); + // In exact mode: g0, g1, and g01 are queried; g_acute is NOT. + EXPECT_EQ(size_cache->GetPatchSizeCallCount(), 3); + + double expected_exact = + -0.50 * (200 + 75) - 0.75 * (300 + 75) + p_0_or_1 * (450 + 75); + EXPECT_NEAR(*exact_delta, expected_exact, 1e-9); +} + +TEST_F(CandidateMergeTest, + ComputeCostDelta_EvaluatesWhenUnchangedEffectiveConditionCollides) { + // s0: aacute (0x00E1) + // s1: eacute (0x00E9) + // s2: cacute (0x0107) + std::vector segments = { + {{0x00E1}}, + {{0x00E9}}, + {{0x0107}}, + }; + freq::UnicodeFrequencies frequencies{ + {{' ', ' '}, 100}, + {{0x00E1, 0x00E1}, 50}, + {{0x00E9, 0x00E9}, 75}, + {{0x0107, 0x0107}, 95}}; + + ClosureGlyphSegmenter segmenter(8, 8, PATCH, CLOSURE_ONLY, resolver); + auto context = SegmentationContext::InitializeSegmentationContext( + roboto.get(), {}, segments, segmenter.unmapped_glyph_handling(), + segmenter.condition_analysis_mode(), segmenter.brotli_quality(), + segmenter.init_font_merging_brotli_quality(), resolver); + ASSERT_TRUE(context.ok()) << context.status(); + + Merger merger = *Merger::New( + *context, + MergeStrategy::CostBased( + *ProbabilityProfile::Unigram(std::move(frequencies)), 75, 4), + all, all); + + const auto& groupings = context->glyph_groupings; + GlyphSet g0 = groupings.ExclusiveGlyphs(0); + GlyphSet g1 = groupings.ExclusiveGlyphs(1); + GlyphSet g01 = g0; + g01.union_set(g1); + GlyphSet g_acute = groupings.ConditionsAndGlyphs().at( + ActivationCondition::or_segments({0, 1, 2}, 0)); + + glyph_id_t gid_d = 72; // in (s0 OR s2) + context->glyph_condition_set.AddOrCondition(gid_d, 0); + context->glyph_condition_set.AddOrCondition(gid_d, 2); + + auto sc = context->glyph_groupings.GroupGlyphs( + context->SegmentationInfo(), context->glyph_condition_set, + *context->glyph_closure_cache, std::nullopt, {gid_d}, {0, 2}, false); + ASSERT_TRUE(sc.ok()) << sc; + + GlyphSet g_acute_and_d = g_acute; + g_acute_and_d.insert(gid_d); + + MockPatchSizeCache* size_cache = new MockPatchSizeCache(); + size_cache->SetPatchSize(g0, 200); + size_cache->SetPatchSize(g1, 300); + size_cache->SetPatchSize(g01, 450); + size_cache->SetPatchSize(g_acute, 100); + size_cache->SetPatchSize({gid_d}, 150); + size_cache->SetPatchSize(g_acute_and_d, 220); + context->patch_size_cache.reset(size_cache); + + // When merging {0, 1}: + // Even though (s0 OR s1 OR s2) individually has an unchanged effective + // condition, it merges into (s0 OR s2) where (s0 OR s2) already exists! + // So their glyph set becomes g_acute U {gid_d}. Both patches must be evaluated. + auto exact_delta = CandidateMerge::ComputeCostDelta( + merger, {0, 1}, g01, std::nullopt); + ASSERT_TRUE(exact_delta.ok()) << exact_delta.status(); + + double p_0_or_1 = 0.50 + 0.75 - 0.50 * 0.75; + double p_0_or_2 = 0.50 + 0.95 - 0.50 * 0.95; + double p_0_or_1_or_2 = 1.0 - (1.0 - 0.50) * (1.0 - 0.75) * (1.0 - 0.95); + // Note that (s0 OR s1 OR s2) and (s0 OR s2) both have their old costs + // subtracted, and their combined patch g_acute U {gid_d} at post-merge + // condition (s0 OR s2) added. Since segment 0 post-merge represents {0, 1}, + // the post-merge probability of (s0 OR s2) is P((s0 OR s1) OR s2) = p_0_or_1_or_2. + double expected_exact = + -0.50 * (200 + 75) - 0.75 * (300 + 75) + p_0_or_1 * (450 + 75) + - p_0_or_1_or_2 * (100 + 75) - p_0_or_2 * (150 + 75) + + p_0_or_1_or_2 * (220 + 75); + EXPECT_NEAR(*exact_delta, expected_exact, 1e-9); +} + } // namespace ift::encoder diff --git a/ift/encoder/closure_glyph_segmenter_test.cc b/ift/encoder/closure_glyph_segmenter_test.cc index c1bacda8..d4c7c72a 100644 --- a/ift/encoder/closure_glyph_segmenter_test.cc +++ b/ift/encoder/closure_glyph_segmenter_test.cc @@ -1165,18 +1165,12 @@ TEST_F(ClosureGlyphSegmenterTest, PatchMerge_WithSimplification) { R"(initial font: { gid0 } p0: { gid3, gid9, gid155 } p1: { gid4, gid10, gid156 } -p2: { gid5, gid6, gid157 } -p3: { gid158 } -p4: { gid14, gid33, gid47, gid60, gid73, gid74, gid75, gid76, gid83, gid91, gid111, gid112, gid145, gid149, gid152, gid190, gid191 } -p5: { gid12, gid13, gid24, gid30, gid38, gid39, gid57, gid59, gid62, gid64, gid68, gid77, gid139, gid140, gid153, gid172, gid174 } -p6: { gid11 } +p2: { gid5, gid6, gid11, gid14, gid33, gid47, gid60, gid73, gid74, gid75, gid76, gid83, gid91, gid111, gid112, gid145, gid149, gid152, gid157, gid158, gid190, gid191 } +p3: { gid12, gid13, gid24, gid30, gid38, gid39, gid57, gid59, gid62, gid64, gid68, gid77, gid139, gid140, gid153, gid172, gid174 } if (s0) then p0 if (s1) then p1 if (s2) then p2 -if (s3) then p3 -if ((s2 OR s3)) then p4 -if ((s0 OR s1 OR s2 OR s3)) then p5 -if ((s0 OR s1) AND s2) then p6 +if ((s0 OR s1 OR s2)) then p3 )"); // With simplification there's no more composite conditions: @@ -1184,16 +1178,12 @@ if ((s0 OR s1) AND s2) then p6 R"(initial font: { gid0 } p0: { gid3, gid9, gid155 } p1: { gid4, gid10, gid156 } -p2: { gid5, gid6, gid11, gid157 } -p3: { gid158 } -p4: { gid14, gid33, gid47, gid60, gid73, gid74, gid75, gid76, gid83, gid91, gid111, gid112, gid145, gid149, gid152, gid190, gid191 } -p5: { gid12, gid13, gid24, gid30, gid38, gid39, gid57, gid59, gid62, gid64, gid68, gid77, gid139, gid140, gid153, gid172, gid174 } +p2: { gid5, gid6, gid11, gid14, gid33, gid47, gid60, gid73, gid74, gid75, gid76, gid83, gid91, gid111, gid112, gid145, gid149, gid152, gid157, gid158, gid190, gid191 } +p3: { gid12, gid13, gid24, gid30, gid38, gid39, gid57, gid59, gid62, gid64, gid68, gid77, gid139, gid140, gid153, gid172, gid174 } if (s0) then p0 if (s1) then p1 if (s2) then p2 -if (s3) then p3 -if ((s2 OR s3)) then p4 -if ((s0 OR s1 OR s2 OR s3)) then p5 +if ((s0 OR s1 OR s2)) then p3 )"); } From 278ffb6f0baa7968dd98f34cf4c5a8e149a2c5d2 Mon Sep 17 00:00:00 2001 From: Garret Rieger Date: Fri, 23 Oct 2026 00:03:30 +0000 Subject: [PATCH 08/13] Don't re-evaluated active segment pairs in all pairs composite collection. --- ift/encoder/all_pairs_merger.cc | 15 +++------------ 1 file changed, 3 insertions(+), 12 deletions(-) diff --git a/ift/encoder/all_pairs_merger.cc b/ift/encoder/all_pairs_merger.cc index f5a00232..48a1eb96 100644 --- a/ift/encoder/all_pairs_merger.cc +++ b/ift/encoder/all_pairs_merger.cc @@ -302,18 +302,9 @@ Status AllPairsMerger::QueueCompositeCandidates( segment_index_t u = *triggering_segments.min(); segment_index_t v = *triggering_segments.max(); if (IsActiveProbSegment(u) && IsActiveProbSegment(v)) { - ProbOrderedSegment pos_u{*active_segment_prob_.at(u), u}; - ProbOrderedSegment pos_v{*active_segment_prob_.at(v), v}; - const ProbOrderedSegment& high = (pos_u < pos_v) ? pos_u : pos_v; - const ProbOrderedSegment& low = (pos_u < pos_v) ? pos_v : pos_u; - uint32_t high_size = TRY(merger.ExclusivePatchSize(high.segment_index)); - double inert_threshold = merger.BestCaseInertProbabilityThreshold( - high_size, high.probability, 0.0); - if (low.probability > inert_threshold) { - // Pair of active exclusive segments above the inert threshold is - // already handled by exclusive pair queuing. - continue; - } + // Pairs of active segments are already considered by QueueLowerRankedExclusivePairs() + // so don't need to check them again here. + continue; } } From 868aabd564df4d0e04b65be2ea65328b8ea5ef4c Mon Sep 17 00:00:00 2001 From: Garret Rieger Date: Fri, 23 Oct 2026 19:24:27 +0000 Subject: [PATCH 09/13] Also log number of gids compressed with brotli. --- ift/encoder/patch_size_cache.h | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/ift/encoder/patch_size_cache.h b/ift/encoder/patch_size_cache.h index 558e4320..8df4828f 100644 --- a/ift/encoder/patch_size_cache.h +++ b/ift/encoder/patch_size_cache.h @@ -33,7 +33,8 @@ class PatchSizeCacheImpl : public PatchSizeCache { {ift::common::FontHelper::kGlyf, ift::common::FontHelper::kGvar, ift::common::FontHelper::kCFF, ift::common::FontHelper::kCFF2}, brotli_quality), - cache_() {} + cache_(), + brotli_quality_(brotli_quality) {} absl::StatusOr GetPatchSize( const ift::common::GlyphSet& gids) override { @@ -43,6 +44,7 @@ class PatchSizeCacheImpl : public PatchSizeCache { } brotli_call_count_++; + brotli_call_gids_ += gids.size(); auto patch_data = TRY(differ_.CreatePatch(gids)); uint32_t size = patch_data.size(); cache_[gids] = size; @@ -50,7 +52,8 @@ class PatchSizeCacheImpl : public PatchSizeCache { } void LogBrotliCallCount() const override { - VLOG(0) << "Total number of calls to brotli = " << brotli_call_count_; + VLOG(0) << "Total brotli ops (q" << brotli_quality_ << ") = " << brotli_call_count_; + VLOG(0) << "Total glyphs compressed with brotli (q" << brotli_quality_ << ") = " << brotli_call_gids_; } private: @@ -58,7 +61,9 @@ class PatchSizeCacheImpl : public PatchSizeCache { ift::common::CompatId id_; GlyphKeyedDiff differ_; absl::flat_hash_map cache_; + uint32_t brotli_quality_; uint64_t brotli_call_count_ = 0; + uint64_t brotli_call_gids_ = 0; }; } // namespace ift::encoder From 31ca4571fa3bae1d5350fa8ebd7a8e837f8488dc Mon Sep 17 00:00:00 2001 From: Garret Rieger Date: Sat, 24 Oct 2026 00:48:43 +0000 Subject: [PATCH 10/13] Add ability to skip all pairs merging during min group size enforcement. --- ift/encoder/all_pairs_merger.cc | 76 ++++++++++++++++++----------- ift/encoder/all_pairs_merger.h | 18 +++++-- ift/encoder/candidate_merge_test.cc | 68 ++++++++++++++++++++++++++ 3 files changed, 130 insertions(+), 32 deletions(-) diff --git a/ift/encoder/all_pairs_merger.cc b/ift/encoder/all_pairs_merger.cc index 48a1eb96..682ddf5f 100644 --- a/ift/encoder/all_pairs_merger.cc +++ b/ift/encoder/all_pairs_merger.cc @@ -28,6 +28,11 @@ ABSL_FLAG(ift::encoder::AllPairsNormalizationMode, "(divide by merged_segments.size() - 1), or 'byte' (divide by " "pre-merge patch bytes including network overhead)."); +ABSL_FLAG(bool, experimental_all_pairs_phase1_in_phase2, true, + "When enabled, Phase 1 all-pairs cost-reducing merging remains active " + "during Phase 2 minimum group size enforcement. When disabled, Phase " + "1 stops once Phase 2 begins."); + using absl::btree_set; using absl::Status; using absl::StatusOr; @@ -69,36 +74,39 @@ std::string AbslUnparseFlag(AllPairsNormalizationMode mode) { void AllPairsMerger::AssignMergedSegment(const Merger& merger, segment_index_t base, const SegmentSet& to_merge) { - if (!initialized_) { + if (phase_ == UNINITIALIZED) { return; } SegmentSet merged_segments = to_merge; merged_segments.insert(base); - const auto& groupings = merger.Context().glyph_groupings; - const auto& pre_comb_map = groupings.ConditionsAndGlyphsPreCombination(); - - for (segment_index_t s : merged_segments) { - // precombination representative affected glyphs are recorded this will - // let us locate any conditions involved with those glyphs post-merge. - for (const auto& c : - groupings.TriggeringSegmentToConditionsPreCombination(s)) { - if (c.IsFallback()) { - continue; - } - if (auto rep = pre_comb_map.at(c).min(); rep.has_value()) { - dirty_glyphs_.insert(*rep); - } - } - // Also record any segments involved in conditions directly impacted by the merge. - for (const auto& c : - groupings.TriggeringSegmentToConditions(s)) { - if (c.IsFallback()) { - continue; + if (phase_ == MERGING) { + // Full dirty segment invalidation is only needed during merging. + const auto& groupings = merger.Context().glyph_groupings; + const auto& pre_comb_map = groupings.ConditionsAndGlyphsPreCombination(); + for (segment_index_t s : merged_segments) { + // precombination representative affected glyphs are recorded this will + // let us locate any conditions involved with those glyphs post-merge. + for (const auto& c : + groupings.TriggeringSegmentToConditionsPreCombination(s)) { + if (c.IsFallback()) { + continue; + } + if (auto rep = pre_comb_map.at(c).min(); rep.has_value()) { + dirty_glyphs_.insert(*rep); + } } - if (!c.IsExclusive()) { - dirty_segments_.union_set(c.TriggeringSegments()); + + // Also record any segments involved in conditions directly impacted by the merge. + for (const auto& c : + groupings.TriggeringSegmentToConditions(s)) { + if (c.IsFallback()) { + continue; + } + if (!c.IsExclusive()) { + dirty_segments_.union_set(c.TriggeringSegments()); + } } } } @@ -363,7 +371,11 @@ Status AllPairsMerger::QueueAllCandidates( return QueueCompositeCandidates(merger, dirty_segments); } -Status AllPairsMerger::InitPhase1(Merger& merger) { +Status AllPairsMerger::Initialize(Merger& merger) { + if (phase_ != UNINITIALIZED) { + return absl::InternalError("AllPairsMerger is already initialized."); + } + size_t num_segments = merger.Context().SegmentationInfo().Segments().size(); current_epoch_ = 0; segment_epoch_.assign(num_segments, UINT32_MAX); @@ -385,7 +397,7 @@ Status AllPairsMerger::InitPhase1(Merger& merger) { } TRYV(QueueAllCandidates(merger)); - initialized_ = true; + phase_ = MERGING; return absl::OkStatus(); } @@ -424,7 +436,9 @@ Status AllPairsMerger::RefreshDirtySegments(Merger& merger) { } } - TRYV(QueueAllCandidates(merger, dirty_segments_)); + if (phase_ == MERGING) { + TRYV(QueueAllCandidates(merger, dirty_segments_)); + } dirty_segments_.clear(); return absl::OkStatus(); } @@ -440,8 +454,9 @@ bool AllPairsMerger::IsCandidateStale(const QueuedCandidate& candidate) const { StatusOr> AllPairsMerger::TryNextAllPairsMerge( Merger& merger) { - if (!initialized_) { - TRYV(InitPhase1(merger)); + + if (phase_ == UNINITIALIZED) { + TRYV(Initialize(merger)); } else { TRYV(RefreshDirtySegments(merger)); } @@ -473,9 +488,12 @@ StatusOr> AllPairsMerger::TryNextAllPairsMerge( continue; } - return TRY(top.evaluated_merge->Apply(merger)); + return top.evaluated_merge->Apply(merger); } + if (!absl::GetFlag(FLAGS_experimental_all_pairs_phase1_in_phase2)) { + phase_ = FINISHED; + } return std::nullopt; } diff --git a/ift/encoder/all_pairs_merger.h b/ift/encoder/all_pairs_merger.h index 44cce868..f093f3fd 100644 --- a/ift/encoder/all_pairs_merger.h +++ b/ift/encoder/all_pairs_merger.h @@ -59,7 +59,7 @@ class AllPairsMerger { const ift::common::SegmentSet& to_merge); void MarkFinished(segment_index_t s) { - if (!initialized_) { + if (phase_ == UNINITIALIZED) { return; } RemoveActiveSegment(s); @@ -101,7 +101,7 @@ class AllPairsMerger { }; }; - absl::Status InitPhase1(Merger& merger); + absl::Status Initialize(Merger& merger); absl::Status RefreshDirtySegments(Merger& merger); bool IsCandidateStale(const QueuedCandidate& candidate) const; bool IsActiveProbSegment(segment_index_t s) const { @@ -125,7 +125,18 @@ class AllPairsMerger { const std::optional& dirty_segments = std::nullopt); - bool initialized_ = false; + enum Phase { + // Merger has not yet been initialized as is not currently tracking + // merges. + UNINITIALIZED, + // Candidate merges remain to check. + MERGING, + // All candidate merges are exhausted, but active segments and prob ordering + // continues to be maintained. + FINISHED + }; + + Phase phase_ = UNINITIALIZED; uint32_t current_epoch_ = 0; std::vector segment_epoch_; ift::common::SegmentSet dirty_segments_; @@ -142,5 +153,6 @@ class AllPairsMerger { ABSL_DECLARE_FLAG(ift::encoder::AllPairsNormalizationMode, experimental_all_pairs_normalization); +ABSL_DECLARE_FLAG(bool, experimental_all_pairs_phase1_in_phase2); #endif // IFT_ENCODER_ALL_PAIRS_MERGER_H_ diff --git a/ift/encoder/candidate_merge_test.cc b/ift/encoder/candidate_merge_test.cc index 4545939b..1015d6b6 100644 --- a/ift/encoder/candidate_merge_test.cc +++ b/ift/encoder/candidate_merge_test.cc @@ -1799,6 +1799,17 @@ struct ScopedAllPairsNormalizationFlag { } AllPairsNormalizationMode prev; }; + +struct ScopedAllPairsPhase1InPhase2Flag { + explicit ScopedAllPairsPhase1InPhase2Flag(bool enabled) + : prev(absl::GetFlag(FLAGS_experimental_all_pairs_phase1_in_phase2)) { + absl::SetFlag(&FLAGS_experimental_all_pairs_phase1_in_phase2, enabled); + } + ~ScopedAllPairsPhase1InPhase2Flag() { + absl::SetFlag(&FLAGS_experimental_all_pairs_phase1_in_phase2, prev); + } + bool prev; +}; } // namespace TEST_F(CandidateMergeTest, AllPairsMerging_AvoidsSingleBaseGreedyTrap) { @@ -1949,6 +1960,63 @@ TEST_F(CandidateMergeTest, EXPECT_FALSE(step3->has_value()); } +TEST_F(CandidateMergeTest, + AllPairsMerging_Phase2DisablesPhase1WhenFlagFalse) { + ScopedAllPairsMergingFlag flag_override(true); + ScopedAllPairsPhase1InPhase2Flag phase1_override(false); + std::vector segments = { + {{'a'}}, + {{'b'}}, + {{'c', 'd'}}, + }; + std::vector> probabilities = { + {{{'a'}}, 0.80}, + {{{'b'}}, 0.50}, + {{{'c', 'd'}}, 0.75}, + {{{'a', 'b'}}, 0.80}, + {{{'a', 'c', 'd'}}, 0.95}, + {{{'b', 'c', 'd'}}, 0.95}, + {{{'a', 'b', 'c', 'd'}}, 0.80}, + }; + + ClosureGlyphSegmenter segmenter(8, 8, PATCH, CLOSURE_ONLY, resolver); + auto context = SegmentationContext::InitializeSegmentationContext( + roboto.get(), {}, segments, segmenter.unmapped_glyph_handling(), + segmenter.condition_analysis_mode(), segmenter.brotli_quality(), + segmenter.init_font_merging_brotli_quality(), resolver); + ASSERT_TRUE(context.ok()) << context.status(); + + auto* size_cache = new MockPatchSizeCache(); + size_cache->SetPatchSize({69}, 200); + size_cache->SetPatchSize({70}, 200); + size_cache->SetPatchSize({71, 72}, 240); + size_cache->SetPatchSize({69, 70}, 380); + size_cache->SetPatchSize({69, 71, 72}, 450); + size_cache->SetPatchSize({70, 71, 72}, 450); + size_cache->SetPatchSize({69, 70, 71, 72}, 400); + context->patch_size_cache.reset(size_cache); + + auto calc = std::make_shared(probabilities); + Merger merger = *Merger::New( + *context, MergeStrategy::CostBased(ProbabilityProfile(calc), 75, 2), all, + all); + + // Step 1: Phase 1 finds no negative merges; Phase 2 forces {0, 1}. + auto step1 = merger.TryNextMerge(); + ASSERT_TRUE(step1.ok()) << step1.status(); + ASSERT_TRUE(step1->has_value()); + EXPECT_EQ((*step1)->base_segment, 0u); + EXPECT_EQ((*step1)->segments, (SegmentSet{0, 1})); + ASSERT_TRUE(context->ReprocessChanged(**step1).ok()); + + // Step 2: With experimental_all_pairs_phase1_in_phase2 = false, Phase 1 does + // not resume on dirty segment 0. Since both s0 and s2 now meet min_group_size = 2, + // no further merges are performed. + auto step2 = merger.TryNextMerge(); + ASSERT_TRUE(step2.ok()) << step2.status(); + EXPECT_FALSE(step2->has_value()); +} + TEST_F(CandidateMergeTest, AllPairsMerging_Phase2UsesAllPairsProbabilityOrdering) { // Initial probabilities are descending: From 56b59e91b54b22a8b80fcef7d522819ab0f8a323 Mon Sep 17 00:00:00 2001 From: Garret Rieger Date: Thu, 12 Nov 2026 20:35:25 +0000 Subject: [PATCH 11/13] Enable all pairs merging by default. - Uses bytes as the normalization divisor. - All pairs merging is not kept separate from min group enforcement merging. This particular configuration demonstrated significant segmenter time and cost reductions in batch testing. --- ift/encoder/BUILD | 2 - ift/encoder/all_pairs_merger.cc | 134 +++----- ift/encoder/all_pairs_merger.h | 17 - ift/encoder/candidate_merge_test.cc | 346 ++++++-------------- ift/encoder/closure_glyph_segmenter_test.cc | 82 ++--- ift/encoder/merger.cc | 74 +---- ift/encoder/merger.h | 5 - 7 files changed, 174 insertions(+), 486 deletions(-) diff --git a/ift/encoder/BUILD b/ift/encoder/BUILD index 94239a63..4698b15f 100644 --- a/ift/encoder/BUILD +++ b/ift/encoder/BUILD @@ -428,7 +428,6 @@ cc_test( "//ift/config:auto_segmenter_config", "//ift/freq", "//ift/freq:mock_probability_calculator", - "@abseil-cpp//absl/flags:flag", "@googletest//:gtest", "@googletest//:gtest_main", ], @@ -498,7 +497,6 @@ cc_test( "//ift/freq", "//ift/freq:common", "//ift/freq:mock_probability_calculator", - "@abseil-cpp//absl/flags:flag", "@abseil-cpp//absl/log", "@googletest//:gtest", "@googletest//:gtest_main", diff --git a/ift/encoder/all_pairs_merger.cc b/ift/encoder/all_pairs_merger.cc index 682ddf5f..f608e975 100644 --- a/ift/encoder/all_pairs_merger.cc +++ b/ift/encoder/all_pairs_merger.cc @@ -3,15 +3,12 @@ #include #include #include -#include #include #include "absl/container/btree_set.h" #include "absl/container/flat_hash_set.h" -#include "absl/flags/flag.h" #include "absl/status/status.h" #include "absl/status/statusor.h" -#include "absl/strings/string_view.h" #include "ift/common/int_set.h" #include "ift/common/try.h" #include "ift/encoder/activation_condition.h" @@ -20,19 +17,6 @@ #include "ift/encoder/merger.h" #include "ift/encoder/types.h" -ABSL_FLAG(ift::encoder::AllPairsNormalizationMode, - experimental_all_pairs_normalization, - ift::encoder::AllPairsNormalizationMode::kNone, - "Controls how cost deltas are normalized when ranking candidate " - "merges in all-pairs mode: 'none' (no normalization), 'segment' " - "(divide by merged_segments.size() - 1), or 'byte' (divide by " - "pre-merge patch bytes including network overhead)."); - -ABSL_FLAG(bool, experimental_all_pairs_phase1_in_phase2, true, - "When enabled, Phase 1 all-pairs cost-reducing merging remains active " - "during Phase 2 minimum group size enforcement. When disabled, Phase " - "1 stops once Phase 2 begins."); - using absl::btree_set; using absl::Status; using absl::StatusOr; @@ -41,36 +25,6 @@ using ift::common::SegmentSet; namespace ift::encoder { -bool AbslParseFlag(absl::string_view text, AllPairsNormalizationMode* mode, - std::string* error) { - if (text == "none") { - *mode = AllPairsNormalizationMode::kNone; - return true; - } - if (text == "segment" || text == "segments") { - *mode = AllPairsNormalizationMode::kSegment; - return true; - } - if (text == "byte" || text == "bytes") { - *mode = AllPairsNormalizationMode::kByte; - return true; - } - *error = "unknown normalization mode; expected 'none', 'segment', or 'byte'"; - return false; -} - -std::string AbslUnparseFlag(AllPairsNormalizationMode mode) { - switch (mode) { - case AllPairsNormalizationMode::kNone: - return "none"; - case AllPairsNormalizationMode::kSegment: - return "segment"; - case AllPairsNormalizationMode::kByte: - return "byte"; - } - return "none"; -} - void AllPairsMerger::AssignMergedSegment(const Merger& merger, segment_index_t base, const SegmentSet& to_merge) { @@ -160,56 +114,47 @@ Status AllPairsMerger::UpdateActiveSegment(Merger& merger, segment_index_t s) { StatusOr AllPairsMerger::NormalizationDivisor( Merger& merger, const SegmentSet& merged_segments) const { - switch (absl::GetFlag(FLAGS_experimental_all_pairs_normalization)) { - case AllPairsNormalizationMode::kNone: - return 1.0; - case AllPairsNormalizationMode::kSegment: - return static_cast(merged_segments.size() - 1); - case AllPairsNormalizationMode::kByte: { - const uint32_t overhead = merger.Strategy().NetworkOverheadCost(); - const auto& context = merger.Context(); - if (merged_segments.size() == 2) { - segment_index_t u = *merged_segments.min(); - segment_index_t v = *merged_segments.max(); - if (context.InertSegments().contains(u) && - context.InertSegments().contains(v)) { - uint32_t size_u = TRY(merger.ExclusivePatchSize(u)); - uint32_t size_v = TRY(merger.ExclusivePatchSize(v)); - if (size_u > 0 && size_v > 0) { - return static_cast(size_u + overhead) + - static_cast(size_v + overhead); - } - } + const uint32_t overhead = merger.Strategy().NetworkOverheadCost(); + const auto& context = merger.Context(); + if (merged_segments.size() == 2) { + segment_index_t u = *merged_segments.min(); + segment_index_t v = *merged_segments.max(); + if (context.InertSegments().contains(u) && + context.InertSegments().contains(v)) { + uint32_t size_u = TRY(merger.ExclusivePatchSize(u)); + uint32_t size_v = TRY(merger.ExclusivePatchSize(v)); + if (size_u > 0 && size_v > 0) { + return static_cast(size_u + overhead) + + static_cast(size_v + overhead); } + } + } - const bool unicode_range_mode = - (context.GetTargetMode() == config::UNICODE_RANGE); - const auto& groupings = context.glyph_groupings; - const auto& conditions_and_glyphs = groupings.ConditionsAndGlyphs(); - absl::flat_hash_set seen_conditions; - uint64_t total_bytes = 0; - for (segment_index_t s : merged_segments) { - for (const auto& c : groupings.TriggeringSegmentToConditions(s)) { - if (c.IsFallback() || (unicode_range_mode && !c.IsExclusive())) { - continue; - } - auto it = conditions_and_glyphs.find(c); - if (it == conditions_and_glyphs.end()) { - return absl::InternalError( - "Condition which should be present wasn't found."); - } - if (!seen_conditions.insert(&it->first).second) { - continue; - } - uint32_t patch_size = - TRY(context.patch_size_cache->GetPatchSize(it->second)); - total_bytes += static_cast(patch_size + overhead); - } + const bool unicode_range_mode = + (context.GetTargetMode() == config::UNICODE_RANGE); + const auto& groupings = context.glyph_groupings; + const auto& conditions_and_glyphs = groupings.ConditionsAndGlyphs(); + absl::flat_hash_set seen_conditions; + uint64_t total_bytes = 0; + for (segment_index_t s : merged_segments) { + for (const auto& c : groupings.TriggeringSegmentToConditions(s)) { + if (c.IsFallback() || (unicode_range_mode && !c.IsExclusive())) { + continue; + } + auto it = conditions_and_glyphs.find(c); + if (it == conditions_and_glyphs.end()) { + return absl::InternalError( + "Condition which should be present wasn't found."); } - return total_bytes > 0 ? static_cast(total_bytes) : 1.0; + if (!seen_conditions.insert(&it->first).second) { + continue; + } + uint32_t patch_size = + TRY(context.patch_size_cache->GetPatchSize(it->second)); + total_bytes += static_cast(patch_size + overhead); } } - return 1.0; + return total_bytes > 0 ? static_cast(total_bytes) : 1.0; } Status AllPairsMerger::QueueBestCaseCandidate( @@ -459,6 +404,9 @@ StatusOr> AllPairsMerger::TryNextAllPairsMerge( TRYV(Initialize(merger)); } else { TRYV(RefreshDirtySegments(merger)); + if (phase_ == FINISHED) { + return std::nullopt; + } } while (!queue_.empty()) { @@ -491,9 +439,7 @@ StatusOr> AllPairsMerger::TryNextAllPairsMerge( return top.evaluated_merge->Apply(merger); } - if (!absl::GetFlag(FLAGS_experimental_all_pairs_phase1_in_phase2)) { - phase_ = FINISHED; - } + phase_ = FINISHED; return std::nullopt; } diff --git a/ift/encoder/all_pairs_merger.h b/ift/encoder/all_pairs_merger.h index f093f3fd..e8fe4230 100644 --- a/ift/encoder/all_pairs_merger.h +++ b/ift/encoder/all_pairs_merger.h @@ -4,14 +4,11 @@ #include #include #include -#include #include #include "absl/container/btree_set.h" -#include "absl/flags/declare.h" #include "absl/status/status.h" #include "absl/status/statusor.h" -#include "absl/strings/string_view.h" #include "ift/common/int_set.h" #include "ift/encoder/candidate_merge.h" #include "ift/encoder/invalidation_set.h" @@ -21,16 +18,6 @@ namespace ift::encoder { class Merger; -enum class AllPairsNormalizationMode { - kNone, - kSegment, - kByte, -}; - -bool AbslParseFlag(absl::string_view text, AllPairsNormalizationMode* mode, - std::string* error); -std::string AbslUnparseFlag(AllPairsNormalizationMode mode); - // Implements Phase 1 all-pairs cost-reducing segment merging for Merger using a // lazy two-stage (best-case lower bound -> exact) priority queue. class AllPairsMerger { @@ -151,8 +138,4 @@ class AllPairsMerger { } // namespace ift::encoder -ABSL_DECLARE_FLAG(ift::encoder::AllPairsNormalizationMode, - experimental_all_pairs_normalization); -ABSL_DECLARE_FLAG(bool, experimental_all_pairs_phase1_in_phase2); - #endif // IFT_ENCODER_ALL_PAIRS_MERGER_H_ diff --git a/ift/encoder/candidate_merge_test.cc b/ift/encoder/candidate_merge_test.cc index 1015d6b6..a925c785 100644 --- a/ift/encoder/candidate_merge_test.cc +++ b/ift/encoder/candidate_merge_test.cc @@ -4,7 +4,6 @@ #include #include -#include "absl/flags/flag.h" #include "gtest/gtest.h" #include "ift/common/bazel_data_file_resolver.h" #include "ift/common/font_data.h" @@ -1777,41 +1776,6 @@ TEST_F(CandidateMergeTest, EXPECT_NEAR(*exact_delta, expected_exact, 1e-9); } -namespace { -struct ScopedAllPairsMergingFlag { - explicit ScopedAllPairsMergingFlag(bool enabled) - : prev(absl::GetFlag(FLAGS_experimental_all_pairs_merging)) { - absl::SetFlag(&FLAGS_experimental_all_pairs_merging, enabled); - } - ~ScopedAllPairsMergingFlag() { - absl::SetFlag(&FLAGS_experimental_all_pairs_merging, prev); - } - bool prev; -}; - -struct ScopedAllPairsNormalizationFlag { - explicit ScopedAllPairsNormalizationFlag(AllPairsNormalizationMode mode) - : prev(absl::GetFlag(FLAGS_experimental_all_pairs_normalization)) { - absl::SetFlag(&FLAGS_experimental_all_pairs_normalization, mode); - } - ~ScopedAllPairsNormalizationFlag() { - absl::SetFlag(&FLAGS_experimental_all_pairs_normalization, prev); - } - AllPairsNormalizationMode prev; -}; - -struct ScopedAllPairsPhase1InPhase2Flag { - explicit ScopedAllPairsPhase1InPhase2Flag(bool enabled) - : prev(absl::GetFlag(FLAGS_experimental_all_pairs_phase1_in_phase2)) { - absl::SetFlag(&FLAGS_experimental_all_pairs_phase1_in_phase2, enabled); - } - ~ScopedAllPairsPhase1InPhase2Flag() { - absl::SetFlag(&FLAGS_experimental_all_pairs_phase1_in_phase2, prev); - } - bool prev; -}; -} // namespace - TEST_F(CandidateMergeTest, AllPairsMerging_AvoidsSingleBaseGreedyTrap) { // s0 ('a', gid 69) has a marginally negative merge with s1 ('b', gid 70) // (delta = -4.0), whereas s1 ('b', gid 70) and s2 ('c', gid 71) have a much @@ -1831,92 +1795,6 @@ TEST_F(CandidateMergeTest, AllPairsMerging_AvoidsSingleBaseGreedyTrap) { {{{'a', 'b', 'c'}}, 0.99}, }; - auto run_merger = [&](bool all_pairs) { - ScopedAllPairsMergingFlag flag_override(all_pairs); - ClosureGlyphSegmenter segmenter(8, 8, PATCH, CLOSURE_ONLY, resolver); - auto context = SegmentationContext::InitializeSegmentationContext( - roboto.get(), {}, segments, segmenter.unmapped_glyph_handling(), - segmenter.condition_analysis_mode(), segmenter.brotli_quality(), - segmenter.init_font_merging_brotli_quality(), resolver); - EXPECT_TRUE(context.ok()) << context.status(); - - auto* size_cache = new MockPatchSizeCache(); - size_cache->SetPatchSize({69}, 200); - size_cache->SetPatchSize({70}, 200); - size_cache->SetPatchSize({71}, 200); - size_cache->SetPatchSize({69, 70}, 440); - size_cache->SetPatchSize({70, 71}, 220); - size_cache->SetPatchSize({69, 71}, 450); - size_cache->SetPatchSize({69, 70, 71}, 650); - context->patch_size_cache.reset(size_cache); - - auto calc = std::make_shared(probabilities); - Merger merger = *Merger::New( - *context, MergeStrategy::CostBased(ProbabilityProfile(calc), 75, 1), - all, all); - - while (true) { - auto modified = merger.TryNextMerge(); - EXPECT_TRUE(modified.ok()) << modified.status(); - if (!modified->has_value()) { - break; - } - EXPECT_TRUE(context->ReprocessChanged(**modified).ok()); - } - - std::vector result; - for (const auto& seg : context->SegmentationInfo().Segments()) { - result.push_back(seg.Definition()); - } - return result; - }; - - // Single-base greedy mode merges {0, 1} first, trapping s2 unmerged. - std::vector single_base_result = run_merger(false); - std::vector expected_single_base = { - {'a', 'b'}, - {}, - {'c'}, - }; - EXPECT_EQ(single_base_result, expected_single_base); - - // All-pairs mode merges {1, 2} first and leaves s0 separate. - std::vector all_pairs_result = run_merger(true); - std::vector expected_all_pairs = { - {'a'}, - {'b', 'c'}, - {}, - }; - EXPECT_EQ(all_pairs_result, expected_all_pairs); -} - -TEST_F(CandidateMergeTest, - AllPairsMerging_Phase2ResumesPhase1AfterForcedMerge) { - // With min_group_size = 2: - // - s0 ('a', gid 69) and s1 ('b', gid 70) each have 1 codepoint; s2 - // ({'c', 'd'}, gids {71, 72}) already has 2 codepoints. - // - All initial 2-segment merges have positive cost deltas, so Phase 1 makes - // no merges and Phase 2 forces s0 to merge with s1 (delta = +6.5). - // - Once s0 becomes {'a', 'b'}, merging s0 with s2 has a negative cost delta - // (delta = -220.25). Even though both s0 and s2 now meet min_group_size = 2, - // AllPairsMerger refreshes dirty segment s0 and resumes Phase 1 to apply - // the cost-reducing merge {0, 2}. - ScopedAllPairsMergingFlag flag_override(true); - std::vector segments = { - {{'a'}}, - {{'b'}}, - {{'c', 'd'}}, - }; - std::vector> probabilities = { - {{{'a'}}, 0.80}, - {{{'b'}}, 0.50}, - {{{'c', 'd'}}, 0.75}, - {{{'a', 'b'}}, 0.80}, - {{{'a', 'c', 'd'}}, 0.95}, - {{{'b', 'c', 'd'}}, 0.95}, - {{{'a', 'b', 'c', 'd'}}, 0.80}, - }; - ClosureGlyphSegmenter segmenter(8, 8, PATCH, CLOSURE_ONLY, resolver); auto context = SegmentationContext::InitializeSegmentationContext( roboto.get(), {}, segments, segmenter.unmapped_glyph_handling(), @@ -1927,43 +1805,43 @@ TEST_F(CandidateMergeTest, auto* size_cache = new MockPatchSizeCache(); size_cache->SetPatchSize({69}, 200); size_cache->SetPatchSize({70}, 200); - size_cache->SetPatchSize({71, 72}, 240); - size_cache->SetPatchSize({69, 70}, 380); - size_cache->SetPatchSize({69, 71, 72}, 450); - size_cache->SetPatchSize({70, 71, 72}, 450); - size_cache->SetPatchSize({69, 70, 71, 72}, 400); + size_cache->SetPatchSize({71}, 200); + size_cache->SetPatchSize({69, 70}, 440); + size_cache->SetPatchSize({70, 71}, 220); + size_cache->SetPatchSize({69, 71}, 450); + size_cache->SetPatchSize({69, 70, 71}, 650); context->patch_size_cache.reset(size_cache); auto calc = std::make_shared(probabilities); Merger merger = *Merger::New( - *context, MergeStrategy::CostBased(ProbabilityProfile(calc), 75, 2), all, + *context, MergeStrategy::CostBased(ProbabilityProfile(calc), 75, 1), all, all); - // Step 1: Phase 1 finds no negative merges; Phase 2 forces {0, 1}. - auto step1 = merger.TryNextMerge(); - ASSERT_TRUE(step1.ok()) << step1.status(); - ASSERT_TRUE(step1->has_value()); - EXPECT_EQ((*step1)->base_segment, 0u); - EXPECT_EQ((*step1)->segments, (SegmentSet{0, 1})); - ASSERT_TRUE(context->ReprocessChanged(**step1).ok()); + while (true) { + auto modified = merger.TryNextMerge(); + ASSERT_TRUE(modified.ok()) << modified.status(); + if (!modified->has_value()) { + break; + } + ASSERT_TRUE(context->ReprocessChanged(**modified).ok()); + } - // Step 2: Phase 1 resumes on dirty segment 0 and applies {0, 2}. - auto step2 = merger.TryNextMerge(); - ASSERT_TRUE(step2.ok()) << step2.status(); - ASSERT_TRUE(step2->has_value()); - EXPECT_EQ((*step2)->base_segment, 0u); - EXPECT_EQ((*step2)->segments, (SegmentSet{0, 2})); - ASSERT_TRUE(context->ReprocessChanged(**step2).ok()); + std::vector result; + for (const auto& seg : context->SegmentationInfo().Segments()) { + result.push_back(seg.Definition()); + } - auto step3 = merger.TryNextMerge(); - ASSERT_TRUE(step3.ok()) << step3.status(); - EXPECT_FALSE(step3->has_value()); + // All-pairs mode merges {1, 2} first and leaves s0 separate. + std::vector expected_all_pairs = { + {'a'}, + {'b', 'c'}, + {}, + }; + EXPECT_EQ(result, expected_all_pairs); } TEST_F(CandidateMergeTest, - AllPairsMerging_Phase2DisablesPhase1WhenFlagFalse) { - ScopedAllPairsMergingFlag flag_override(true); - ScopedAllPairsPhase1InPhase2Flag phase1_override(false); + AllPairsMerging_Phase2DoesNotResumePhase1AfterForcedMerge) { std::vector segments = { {{'a'}}, {{'b'}}, @@ -2009,9 +1887,8 @@ TEST_F(CandidateMergeTest, EXPECT_EQ((*step1)->segments, (SegmentSet{0, 1})); ASSERT_TRUE(context->ReprocessChanged(**step1).ok()); - // Step 2: With experimental_all_pairs_phase1_in_phase2 = false, Phase 1 does - // not resume on dirty segment 0. Since both s0 and s2 now meet min_group_size = 2, - // no further merges are performed. + // Step 2: Phase 1 does not resume during Phase 2 on dirty segment 0. Since + // both s0 and s2 now meet min_group_size = 2, no further merges are performed. auto step2 = merger.TryNextMerge(); ASSERT_TRUE(step2.ok()) << step2.status(); EXPECT_FALSE(step2->has_value()); @@ -2030,7 +1907,6 @@ TEST_F(CandidateMergeTest, // delta (+1.54), s3 is selected over s1. // - s2 (0.001) is visited after s3 and s1 and breaks early on // inert_threshold (0.0065). - ScopedAllPairsMergingFlag flag_override(true); std::vector segments = { {{'a'}}, {{'b'}}, @@ -2116,31 +1992,18 @@ TEST_F(CandidateMergeTest, EXPECT_EQ((*step2)->segments, (SegmentSet{0, 3})); } -TEST_F(CandidateMergeTest, AllPairsMerging_NormalizationModes) { - AllPairsNormalizationMode parsed_mode = AllPairsNormalizationMode::kNone; - std::string parse_error; - EXPECT_TRUE(AbslParseFlag("none", &parsed_mode, &parse_error)); - EXPECT_EQ(parsed_mode, AllPairsNormalizationMode::kNone); - EXPECT_EQ(AbslUnparseFlag(parsed_mode), "none"); - EXPECT_TRUE(AbslParseFlag("segment", &parsed_mode, &parse_error)); - EXPECT_EQ(parsed_mode, AllPairsNormalizationMode::kSegment); - EXPECT_EQ(AbslUnparseFlag(parsed_mode), "segment"); - EXPECT_TRUE(AbslParseFlag("byte", &parsed_mode, &parse_error)); - EXPECT_EQ(parsed_mode, AllPairsNormalizationMode::kByte); - EXPECT_EQ(AbslUnparseFlag(parsed_mode), "byte"); - EXPECT_FALSE(AbslParseFlag("invalid", &parsed_mode, &parse_error)); - - // Set up three independent candidate merges with different segment counts - // and pre-merge byte footprints (network overhead = 75, P = 0.50): +TEST_F(CandidateMergeTest, AllPairsMerging_ByteNormalization) { + // Set up three independent candidate merges with different pre-merge byte + // footprints (network overhead = 75, P = 0.50): // - {0, 1, 2} (aacute, eacute, cacute sharing acute component): - // 3 segments (divisor = 2), 1400 pre-merge bytes (400 + 400 + 400 + 200), - // exact delta = -90.0 -> kNone: -90.0, kSegment: -45.0, kByte: -0.0643 + // 1400 pre-merge bytes (400 + 400 + 400 + 200), + // exact delta = -90.0 -> normalized: -0.0643 // - {3, 4} ('d', 'b'): - // 2 segments (divisor = 1), 800 pre-merge bytes (600 + 200), - // exact delta = -68.0 -> kNone: -68.0, kSegment: -68.0, kByte: -0.0850 + // 800 pre-merge bytes (600 + 200), + // exact delta = -68.0 -> normalized: -0.0850 // - {5, 6} ('p', 'q'): - // 2 segments (divisor = 1), 400 pre-merge bytes (200 + 200), - // exact delta = -62.0 -> kNone: -62.0, kSegment: -62.0, kByte: -0.1550 + // 400 pre-merge bytes (200 + 200), + // exact delta = -62.0 -> normalized: -0.1550 std::vector segments = { {{0x00E1}}, // s0: aacute {{0x00E9}}, // s1: eacute @@ -2175,92 +2038,71 @@ TEST_F(CandidateMergeTest, AllPairsMerging_NormalizationModes) { probs.push_back({Segment(def), 1.0}); } - auto run_with_mode = [&](AllPairsNormalizationMode mode) { - ScopedAllPairsMergingFlag all_pairs_flag(true); - ScopedAllPairsNormalizationFlag norm_flag(mode); - - ClosureGlyphSegmenter segmenter(8, 8, PATCH, CLOSURE_ONLY, resolver); - auto context = SegmentationContext::InitializeSegmentationContext( - roboto.get(), {}, segments, segmenter.unmapped_glyph_handling(), - segmenter.condition_analysis_mode(), segmenter.brotli_quality(), - segmenter.init_font_merging_brotli_quality(), resolver); - EXPECT_TRUE(context.ok()) << context.status(); - - const auto& groupings = context->glyph_groupings; - GlyphSet g0 = groupings.ExclusiveGlyphs(0); - GlyphSet g1 = groupings.ExclusiveGlyphs(1); - GlyphSet g2 = groupings.ExclusiveGlyphs(2); - GlyphSet g3 = groupings.ExclusiveGlyphs(3); - GlyphSet g4 = groupings.ExclusiveGlyphs(4); - GlyphSet g5 = groupings.ExclusiveGlyphs(5); - GlyphSet g6 = groupings.ExclusiveGlyphs(6); - GlyphSet g_acute = groupings.ConditionsAndGlyphs().at( - ActivationCondition::or_segments({0, 1, 2}, 0)); - - GlyphSet g012 = g0; - g012.union_set(g1); - g012.union_set(g2); - g012.union_set(g_acute); - GlyphSet g34 = g3; - g34.union_set(g4); - GlyphSet g56 = g5; - g56.union_set(g6); - - auto* size_cache = new MockPatchSizeCache(); - size_cache->SetPatchSize(g0, 325); - size_cache->SetPatchSize(g1, 325); - size_cache->SetPatchSize(g2, 325); - size_cache->SetPatchSize(g_acute, 125); - size_cache->SetPatchSize(g012, 1145); - - size_cache->SetPatchSize(g3, 525); - size_cache->SetPatchSize(g4, 125); - size_cache->SetPatchSize(g34, 589); - - size_cache->SetPatchSize(g5, 125); - size_cache->SetPatchSize(g6, 125); - size_cache->SetPatchSize(g56, 201); - context->patch_size_cache.reset(size_cache); - - MergeStrategy strategy = MergeStrategy::CostBased( - ProbabilityProfile(std::make_shared(probs)), - 75, 1); - - Merger merger = *Merger::New(*context, strategy, all, all); - std::vector merge_order; - while (true) { - auto modified = merger.TryNextMerge(); - EXPECT_TRUE(modified.ok()) << modified.status(); - if (!modified->has_value()) { - break; - } - merge_order.push_back((*modified)->segments); - EXPECT_TRUE(context->ReprocessChanged(**modified).ok()); - } - return merge_order; - }; + ClosureGlyphSegmenter segmenter(8, 8, PATCH, CLOSURE_ONLY, resolver); + auto context = SegmentationContext::InitializeSegmentationContext( + roboto.get(), {}, segments, segmenter.unmapped_glyph_handling(), + segmenter.condition_analysis_mode(), segmenter.brotli_quality(), + segmenter.init_font_merging_brotli_quality(), resolver); + ASSERT_TRUE(context.ok()) << context.status(); - std::vector expected_none = { - SegmentSet{0, 1, 2}, - SegmentSet{3, 4}, - SegmentSet{5, 6}, - }; - EXPECT_EQ(run_with_mode(AllPairsNormalizationMode::kNone), expected_none); + const auto& groupings = context->glyph_groupings; + GlyphSet g0 = groupings.ExclusiveGlyphs(0); + GlyphSet g1 = groupings.ExclusiveGlyphs(1); + GlyphSet g2 = groupings.ExclusiveGlyphs(2); + GlyphSet g3 = groupings.ExclusiveGlyphs(3); + GlyphSet g4 = groupings.ExclusiveGlyphs(4); + GlyphSet g5 = groupings.ExclusiveGlyphs(5); + GlyphSet g6 = groupings.ExclusiveGlyphs(6); + GlyphSet g_acute = groupings.ConditionsAndGlyphs().at( + ActivationCondition::or_segments({0, 1, 2}, 0)); - std::vector expected_segment = { - SegmentSet{3, 4}, - SegmentSet{5, 6}, - SegmentSet{0, 1, 2}, - }; - EXPECT_EQ(run_with_mode(AllPairsNormalizationMode::kSegment), - expected_segment); + GlyphSet g012 = g0; + g012.union_set(g1); + g012.union_set(g2); + g012.union_set(g_acute); + GlyphSet g34 = g3; + g34.union_set(g4); + GlyphSet g56 = g5; + g56.union_set(g6); + + auto* size_cache = new MockPatchSizeCache(); + size_cache->SetPatchSize(g0, 325); + size_cache->SetPatchSize(g1, 325); + size_cache->SetPatchSize(g2, 325); + size_cache->SetPatchSize(g_acute, 125); + size_cache->SetPatchSize(g012, 1145); + + size_cache->SetPatchSize(g3, 525); + size_cache->SetPatchSize(g4, 125); + size_cache->SetPatchSize(g34, 589); + + size_cache->SetPatchSize(g5, 125); + size_cache->SetPatchSize(g6, 125); + size_cache->SetPatchSize(g56, 201); + context->patch_size_cache.reset(size_cache); + + MergeStrategy strategy = MergeStrategy::CostBased( + ProbabilityProfile(std::make_shared(probs)), + 75, 1); + + Merger merger = *Merger::New(*context, strategy, all, all); + std::vector merge_order; + while (true) { + auto modified = merger.TryNextMerge(); + ASSERT_TRUE(modified.ok()) << modified.status(); + if (!modified->has_value()) { + break; + } + merge_order.push_back((*modified)->segments); + ASSERT_TRUE(context->ReprocessChanged(**modified).ok()); + } std::vector expected_byte = { SegmentSet{5, 6}, SegmentSet{3, 4}, SegmentSet{0, 1, 2}, }; - EXPECT_EQ(run_with_mode(AllPairsNormalizationMode::kByte), expected_byte); + EXPECT_EQ(merge_order, expected_byte); } TEST_F(CandidateMergeTest, ComputeCostDelta_BestCaseSkipsUnchangedGlyphSetConditions) { diff --git a/ift/encoder/closure_glyph_segmenter_test.cc b/ift/encoder/closure_glyph_segmenter_test.cc index d4c7c72a..cb1a4753 100644 --- a/ift/encoder/closure_glyph_segmenter_test.cc +++ b/ift/encoder/closure_glyph_segmenter_test.cc @@ -2,7 +2,6 @@ #include -#include "absl/flags/flag.h" #include "gtest/gtest.h" #include "ift/common/bazel_data_file_resolver.h" #include "ift/common/font_data.h" @@ -1005,8 +1004,8 @@ TEST_F(ClosureGlyphSegmenterTest, SimpleSegmentation_PatchMerge) { std::vector expected_segments = { {'A'}, {'C'}, - {0xC1}, - {0x106}, + {0xC1, 0x106}, + {}, }; ASSERT_EQ(segmentation->Segments(), expected_segments); @@ -1014,8 +1013,8 @@ TEST_F(ClosureGlyphSegmenterTest, SimpleSegmentation_PatchMerge) { R"(initial font: { gid0 } p0: { gid117, gid169, gid640, gid700 } p1: { gid37, gid39 } -if ((s2 OR s3)) then p0 -if ((s0 OR s1 OR s2 OR s3)) then p1 +if (s2) then p0 +if ((s0 OR s1 OR s2)) then p1 )"); } @@ -1046,23 +1045,19 @@ TEST_F(ClosureGlyphSegmenterTest, SimpleSegmentation_NoPatchMerge) { std::vector expected_segments = { {'A'}, {'C'}, - {0xC1}, - {0x106}, + {0xC1, 0x106}, + {}, }; ASSERT_EQ(segmentation->Segments(), expected_segments); ASSERT_EQ(segmentation->ToString(), R"(initial font: { gid0 } -p0: { gid37 } -p1: { gid39 } -p2: { gid117, gid169 } -p3: { gid640 } -p4: { gid700 } -if ((s0 OR s2)) then p0 -if ((s1 OR s3)) then p1 -if ((s2 OR s3)) then p2 -if ((s0 OR s2) AND (s2 OR s3)) then p3 -if ((s1 OR s3) AND (s2 OR s3)) then p4 +p0: { gid117, gid169, gid640, gid700 } +p1: { gid37 } +p2: { gid39 } +if (s2) then p0 +if ((s0 OR s2)) then p1 +if ((s1 OR s2)) then p2 )"); } @@ -1165,12 +1160,14 @@ TEST_F(ClosureGlyphSegmenterTest, PatchMerge_WithSimplification) { R"(initial font: { gid0 } p0: { gid3, gid9, gid155 } p1: { gid4, gid10, gid156 } -p2: { gid5, gid6, gid11, gid14, gid33, gid47, gid60, gid73, gid74, gid75, gid76, gid83, gid91, gid111, gid112, gid145, gid149, gid152, gid157, gid158, gid190, gid191 } +p2: { gid5, gid6, gid14, gid33, gid60, gid91, gid112, gid145, gid152, gid157, gid158, gid190, gid191 } p3: { gid12, gid13, gid24, gid30, gid38, gid39, gid57, gid59, gid62, gid64, gid68, gid77, gid139, gid140, gid153, gid172, gid174 } +p4: { gid11, gid47, gid73, gid74, gid75, gid76, gid83, gid111, gid149 } if (s0) then p0 if (s1) then p1 if (s2) then p2 if ((s0 OR s1 OR s2)) then p3 +if ((s0 OR s1) AND s2) then p4 )"); // With simplification there's no more composite conditions: @@ -3381,19 +3378,6 @@ TEST_F(ClosureGlyphSegmenterTest, UnicodeRange_TotalCosts) { EXPECT_NEAR(ur_costs[0].ift_patch_cost, expected_patch_cost, 1e-6); } -namespace { -struct ScopedAllPairsMergingFlag { - explicit ScopedAllPairsMergingFlag(bool enabled) - : prev(absl::GetFlag(FLAGS_experimental_all_pairs_merging)) { - absl::SetFlag(&FLAGS_experimental_all_pairs_merging, enabled); - } - ~ScopedAllPairsMergingFlag() { - absl::SetFlag(&FLAGS_experimental_all_pairs_merging, prev); - } - bool prev; -}; -} // namespace - TEST_F(ClosureGlyphSegmenterTest, AllPairsMerging_AvoidsGreedyTrap_EndToEnd) { PatchSizeCacheImpl patch_sizer(roboto.get(), segmenter.brotli_quality()); double size_a = *patch_sizer.GetPatchSize({69}) + 75.0; @@ -3422,36 +3406,22 @@ TEST_F(ClosureGlyphSegmenterTest, AllPairsMerging_AvoidsGreedyTrap_EndToEnd) { MergeStrategy::CostBased(ProbabilityProfile(calc), 75, 1); strategy.SetOptimizationCutoffFraction(0.0); - std::vector single_base_expected = { - {'a', 'b'}, - {}, - {'c'}, - }; std::vector all_pairs_expected = { {'a'}, {'b', 'c'}, {}, }; - GlyphSegmentation single_base_seg({}, {}, {}); - { - ScopedAllPairsMergingFlag flag_override(false); - auto res = segmenter.CodepointToGlyphSegments( - roboto.get(), {}, {{'a'}, {'b'}, {'c'}}, strategy); - ASSERT_TRUE(res.ok()) << res.status(); - EXPECT_EQ(res->Segments(), single_base_expected); - single_base_seg = *res; - } + auto single_base_res = segmenter.CodepointToGlyphSegments( + roboto.get(), {}, {{'a', 'b'}, {}, {'c'}}, MergeStrategy::None()); + ASSERT_TRUE(single_base_res.ok()) << single_base_res.status(); + GlyphSegmentation single_base_seg = *single_base_res; - GlyphSegmentation all_pairs_seg({}, {}, {}); - { - ScopedAllPairsMergingFlag flag_override(true); - auto res = segmenter.CodepointToGlyphSegments( - roboto.get(), {}, {{'a'}, {'b'}, {'c'}}, strategy); - ASSERT_TRUE(res.ok()) << res.status(); - EXPECT_EQ(res->Segments(), all_pairs_expected); - all_pairs_seg = *res; - } + auto all_pairs_res = segmenter.CodepointToGlyphSegments( + roboto.get(), {}, {{'a'}, {'b'}, {'c'}}, strategy); + ASSERT_TRUE(all_pairs_res.ok()) << all_pairs_res.status(); + EXPECT_EQ(all_pairs_res->Segments(), all_pairs_expected); + GlyphSegmentation all_pairs_seg = *all_pairs_res; auto single_base_costs = *segmenter.TotalCosts(roboto.get(), single_base_seg, {calc.get()}); @@ -3463,7 +3433,6 @@ TEST_F(ClosureGlyphSegmenterTest, AllPairsMerging_AvoidsGreedyTrap_EndToEnd) { TEST_F(ClosureGlyphSegmenterTest, AllPairsMerging_CompositeConditionsAndNonInertMerges) { - ScopedAllPairsMergingFlag flag_override(true); // Combines conjunctive ligatures ('f', 'i', 'l'), a 3-segment disjunctive // accent condition (0xC1 Aacute, 0xC9 Eacute, 0xCD Iacute), and inert // segments ('a', 'b'). @@ -3503,7 +3472,6 @@ TEST_F(ClosureGlyphSegmenterTest, TEST_F(ClosureGlyphSegmenterTest, AllPairsMerging_MinGroupSizeAndOptimizationCutoff) { - ScopedAllPairsMergingFlag flag_override(true); UnicodeFrequencies frequencies{ {{' ', ' '}, 100}, {{'a', 'a'}, 95}, @@ -3533,7 +3501,6 @@ TEST_F(ClosureGlyphSegmenterTest, } TEST_F(ClosureGlyphSegmenterTest, AllPairsMerging_UnicodeRangeMode) { - ScopedAllPairsMergingFlag flag_override(true); auto calc = std::make_shared( std::vector>{ {Segment({'b'}), 0.6}, @@ -3565,7 +3532,6 @@ TEST_F(ClosureGlyphSegmenterTest, AllPairsMerging_UnicodeRangeMode) { } TEST_F(ClosureGlyphSegmenterTest, AllPairsMerging_Determinism) { - ScopedAllPairsMergingFlag flag_override(true); UnicodeFrequencies frequencies{ {{' ', ' '}, 100}, {{'a', 'a'}, 90}, {{'b', 'b'}, 90}, {{'c', 'c'}, 60}, {{'d', 'd'}, 60}, {{'e', 'e'}, 60}, diff --git a/ift/encoder/merger.cc b/ift/encoder/merger.cc index 4a03481b..ba018f93 100644 --- a/ift/encoder/merger.cc +++ b/ift/encoder/merger.cc @@ -25,22 +25,12 @@ ABSL_FLAG(bool, record_merged_size_reductions, false, "When enabled the merger will record the percent size reductions of " "each assessed merge."); -ABSL_FLAG(bool, experimental_all_pairs_merging, false, - "When enabled the cost-based merger considers all segment pairs " - "globally instead of only pairs with the current highest probability " - "base segment."); - namespace ift::encoder { bool Merger::ShouldRecordMergedSizeReductions() const { return absl::GetFlag(FLAGS_record_merged_size_reductions); } -bool Merger::UseAllPairsMerging() const { - return strategy_.UseCosts() && - absl::GetFlag(FLAGS_experimental_all_pairs_merging); -} - StatusOr Merger::ComputeAndCacheAggregateProbability( segment_index_t segment_index) const { double total = 0.0; @@ -140,7 +130,7 @@ StatusOr> Merger::TryNextMerge() { // cost for each script individually and use the sum of the individual costs // as the overall cost. - if (UseAllPairsMerging()) { + if (strategy_.UseCosts()) { auto modified = TRY(all_pairs_merger_.TryNextAllPairsMerge(*this)); if (modified.has_value()) { return *modified; @@ -149,7 +139,7 @@ StatusOr> Merger::TryNextMerge() { while (true) { segment_index_t base_segment_index; - if (UseAllPairsMerging() && + if (strategy_.UseCosts() && !all_pairs_merger_.ActiveSegmentsByProb().empty()) { base_segment_index = all_pairs_merger_.ActiveSegmentsByProb().begin()->segment_index; @@ -566,24 +556,14 @@ StatusOr> Merger::MergeSegmentWithCosts( return std::nullopt; } - std::optional smallest_candidate_merge; const auto& base_segment = context_->SegmentationInfo().Segments()[base_segment_index]; - bool min_group_size_met = - base_segment.MeetsMinimumGroupSize(strategy_.MinimumGroupSize()); - if (min_group_size_met) { - if (UseAllPairsMerging()) { - // Negative cost delta merges have already been exhausted by - // AllPairsMerger. - return std::nullopt; - } - // If min group size is met, then we will no longer consider merge's that - // have a positive cost delta so start with an existing smallest candidate - // set to cost delta 0 which will filter out positive cost delta candidates. - smallest_candidate_merge = - CandidateMerge::BaselineCandidate(base_segment_index, 0.0); + if (base_segment.MeetsMinimumGroupSize(strategy_.MinimumGroupSize())) { + // Negative cost delta merges have already been exhausted by AllPairsMerger. + return std::nullopt; } + std::optional smallest_candidate_merge; TRYV(CollectExclusiveCandidateMerges(base_segment_index, smallest_candidate_merge)); @@ -598,18 +578,6 @@ StatusOr> Merger::MergeSegmentWithCosts( return std::nullopt; } - if (smallest_candidate_merge->SegmentsToMerge() == - SegmentSet{base_segment_index}) { - // nothing smaller than the baseline was found. - return std::nullopt; - } - - // Enforce a negative cost delta only if this segments has met the minimum - // grouping size. - if (min_group_size_met && smallest_candidate_merge->CostDelta() >= 0.0) { - return std::nullopt; - } - return smallest_candidate_merge->Apply(*this); } @@ -752,28 +720,18 @@ Status Merger::CollectExclusiveCandidateMerges( return true; }; - if (UseAllPairsMerging()) { - for (const auto& entry : all_pairs_merger_.ActiveSegmentsByProb()) { - if (!TRY(evaluate_candidate(entry.segment_index))) { - break; - } - } - if (!smallest_candidate_merge.has_value()) { - for (auto it = - candidate_segments_.lower_bound(optimization_cutoff_segment_); - it != candidate_segments_.end(); ++it) { - if (!TRY(evaluate_candidate(*it))) { - break; - } - } + for (const auto& entry : all_pairs_merger_.ActiveSegmentsByProb()) { + if (!TRY(evaluate_candidate(entry.segment_index))) { + break; } - return absl::OkStatus(); } - - for (auto it = candidate_segments_.lower_bound(base_segment_index); - it != candidate_segments_.end(); it++) { - if (!TRY(evaluate_candidate(*it))) { - break; + if (!smallest_candidate_merge.has_value()) { + for (auto it = + candidate_segments_.lower_bound(optimization_cutoff_segment_); + it != candidate_segments_.end(); ++it) { + if (!TRY(evaluate_candidate(*it))) { + break; + } } } diff --git a/ift/encoder/merger.h b/ift/encoder/merger.h index 92be77f1..fd5526c1 100644 --- a/ift/encoder/merger.h +++ b/ift/encoder/merger.h @@ -5,7 +5,6 @@ #include #include -#include "absl/flags/declare.h" #include "ift/common/int_set.h" #include "ift/encoder/activation_condition.h" #include "ift/encoder/all_pairs_merger.h" @@ -15,8 +14,6 @@ #include "ift/encoder/segmentation_context.h" #include "ift/encoder/types.h" -ABSL_DECLARE_FLAG(bool, experimental_all_pairs_merging); - namespace ift::encoder { // Finds and merge segments/patches from a in progress segmentation. @@ -122,8 +119,6 @@ class Merger { bool ShouldRecordMergedSizeReductions() const; - bool UseAllPairsMerging() const; - void LogMergedSizeHistogram() const; // The total cost of a segmentation is defined as the sum of the costs From 6d38cd15219def021dec24e0ee8b7c5e7c4ff24c Mon Sep 17 00:00:00 2001 From: Garret Rieger Date: Thu, 12 Nov 2026 21:18:17 +0000 Subject: [PATCH 12/13] Update segmenter doc with info on all pairs merging. --- docs/all_pairs_merging.md | 136 -------------------------------------- docs/segmenter.md | 60 ++++++++++++++++- 2 files changed, 58 insertions(+), 138 deletions(-) delete mode 100644 docs/all_pairs_merging.md diff --git a/docs/all_pairs_merging.md b/docs/all_pairs_merging.md deleted file mode 100644 index b38a5cba..00000000 --- a/docs/all_pairs_merging.md +++ /dev/null @@ -1,136 +0,0 @@ -# All-Pairs Cost-Based Segment Merging - -Author: Garret Rieger -Date: Oct 16th, 2026 - -## Introduction - -During cost-based segmentation ([segmenter.md](segmenter.md)), `Merger::TryNextMerge()` iteratively merges segments within each merge group to minimize the expected number of bytes transferred: - -$$\text{cost}(S) = \sum_{m=1}^M \sum_{p_i \in S} P_m(c_i) \times (\text{size}(p_i) + k)$$ - -Currently, candidate selection in `Merger::MergeSegmentWithCosts()` uses a **single-base ("leader-follower")** greedy strategy: - -1. Pick the first active segment in `candidate_segments_` (`base_segment_index`, which starts as the highest-probability segment in the merge group). -2. Evaluate merges between `base_segment_index` and all other active segments (plus composite conditions involving `base_segment_index`). -3. Apply the lowest-cost-delta merge found for `base_segment_index`, reprocess changed glyph conditions, and repeat with the **same** `base_segment_index` until no further merges reduce cost (and `min_group_size` is met), at which point `base_segment_index` is marked finished and removed from `candidate_segments_`. - -While restricting each pass to a single `base_segment_index` avoids maintaining global pairwise state, it has a structural quality limitation: `base_segment_index` ($s_0$) will greedily absorb a lower-probability segment $s_j$ even when $\Delta \text{cost}(s_0, s_j)$ is only marginally negative (e.g. $-5$ bytes) while $s_j$ has a much stronger merge partner $s_k$ (e.g. $\Delta \text{cost}(s_j, s_k) = -150$ bytes). Once $s_0$ absorbs $s_j$, $s_j$'s glyphs are loaded at $s_0$'s higher probability, permanently preventing the $(s_j, s_k)$ merge. - -This document describes a plan to prototype **All-Pairs Agglomerative Merging** in `Merger`, gated behind `--experimental_all_pairs_merging`, while keeping runtime and Brotli compression call counts competitive with (or lower than) the single-base approach. - ---- - -## Performance Challenges of Naive All-Pairs Merging - -Upgrading to all-pairs merging naively introduces two potential performance cliffs: - -1. **Upfront Brotli Compression Cliff**: Evaluating exact cost deltas (`CandidateMerge::ComputeCostDelta`, which compresses candidate glyph sets with Brotli) across all $\binom{N}{2}$ pairs before making the first merge would perform $O(N^2)$ Brotli compressions with a weak pruning threshold (`delta_to_beat = 0.0`). Moreover, every applied merge invalidates up to $2N$ pairs involving the merged segments, discarding most of that upfront Brotli work. -2. **Best-Case Delta (`ComputeCostDelta`) Overhead**: Even without Brotli, calling the general `CandidateMerge::ComputeCostDelta` on all $\binom{N}{2}$ pairs (and $O(N)$ pairs after each merge) is expensive if every evaluation allocates `modified_conditions` and `new_conditions` hash maps, copies `ActivationCondition`s, and computes $O(|s_a| \cdot |s_b|)$ bigram union bounds via `AggregateMergedProbability()`. - ---- - -## Algorithm Design - -### 1. Two-Phase Execution (`Phase 1: Cost Reduction` $\rightarrow$ `Phase 2: Minimum Group Size`) - -In the current single-base merger, a segment that starts below `min_group_size` (e.g. 1 codepoint when `min_group_size = 4`) begins its scan with `delta_to_beat = nullopt`, meaning positive-cost-delta candidates cannot be pruned until a negative-cost-delta candidate is encountered. - -In all-pairs mode, segment merging is split into two phases: - -* **Phase 1 (All-Pairs Cost-Reducing Merges, $\Delta \text{cost} < 0$)**: A global priority queue evaluates and applies merges with strictly negative cost deltas ($\Delta \text{cost} < 0.0$) across all active segments above `optimization_cutoff_segment_`. Because Phase 1 only accepts negative deltas, every candidate evaluation has an immediate upper bound of `delta_to_beat <= 0.0` (tightened further by the current heap minimum), enabling aggressive pruning of positive-delta pairs across the entire merge group. -* **Phase 2 (Minimum Group Size Enforcement)**: Once Phase 1 exhausts all negative-cost-delta merges, any segment that already meets `MinimumGroupSize()` is marked finished in $O(1)$. Remaining undersized segments are merged using `MergeSegmentWithCosts()` in descending probability order to meet `MinimumGroupSize()`. If a forced merge in Phase 2 produces a segment with new negative-cost-delta opportunities, Phase 1 can resume for that segment. -* **Patch Merging**: As before, once all segment merges complete, `TryNextPatchMerge()` runs if `UsePatchMerges()` is enabled. - -### 2. Lazy Two-Stage Priority Queue (`BEST_CASE` $\rightarrow$ `EXACT`) - -Instead of computing exact Brotli-backed cost deltas when inserting candidates into the priority queue, a single min-heap stores both **lower-bound (`is_exact = false`)** and **exact (`is_exact = true`)** entries: - -```cpp -struct QueuedCandidate { - double cost_delta; - bool is_exact; - // Canonical set of all segments participating in the merge (size >= 2). - // The surviving base segment is always *segments.min(). - ift::common::SegmentSet segments; - uint32_t created_epoch; - std::optional evaluated_merge; // Present when is_exact == true. -}; -``` - -#### Deterministic Total Ordering -To guarantee run-to-run determinism, heap entries are ordered by: -1. `cost_delta` ascending (most negative delta first). -2. `is_exact` (`true` before `false`, so an exact candidate tied with a lower bound is popped and applied immediately). -3. `segments` lexicographically ascending (`SegmentSet::operator<`). - -#### Pop-and-Refine Loop -When searching for the next Phase 1 merge: -1. Pop the top `QueuedCandidate` from the min-heap. -2. **Staleness check**: If any $s \in \text{candidate.segments}$ has `segment_epoch_[s] > candidate.created_epoch` (meaning $s$ was removed or had its conditions/glyphs modified after this entry was computed), discard the entry and continue. -3. **If `is_exact == false`**: - * Determine `delta_to_beat`: `0.0` (since Phase 1 requires $\Delta \text{cost} < 0.0$). Note: we cannot use the next heap entry's `cost_delta` as `delta_to_beat` if that entry is still `is_exact == false`, because its exact delta may rise when refined; however, we *can* track the best `is_exact == true` delta currently in the heap as `best_exact_delta_in_heap` and pass `std::min(0.0, best_exact_delta_in_heap)` as `delta_to_beat`. - * Evaluate `CandidateMerge::AssessSegmentMerge(*this, *segments.min(), segments, baseline_with_delta_to_beat)`. - * If a valid `CandidateMerge` is returned with `CostDelta() < 0.0`, push `(merge.CostDelta(), is_exact = true, segments, current_epoch_, merge)` back into the heap. -4. **If `is_exact == true`**: - * Because its exact cost delta is $\le$ the `best_case` lower bound of every other candidate in the heap, it is the globally best merge. Apply `candidate.evaluated_merge->Apply(*this)` and return its `InvalidationSet`. - -**Why this reduces Brotli calls**: In single-base mode, if `s_0` only has a weak best merge (e.g. $\Delta = -3$), `delta_to_beat` is $-3$, so `s_0` runs Brotli on every candidate with `best_case_delta < -3`. In the lazy global heap, the strongest merges across the entire group (e.g. $\Delta = -200$) are popped first; pairs with weaker `best_case_delta` values never run Brotli until the global best delta rises to meet them, by which time many of their constituent segments have already been merged elsewhere. - -### 3. Probability-Ordered Row Pruning & Inert Fast-Path - -In single-base merging, only the lowest-indexed segment ever merges, so iterating `candidate_segments_` in index order automatically iterates in non-increasing probability order and satisfies `P(base) >= P(other)`. Under all-pairs merging, any segment $s$ can merge and increase its probability. - -To preserve `BestCaseInertProbabilityThreshold` pruning and avoid $O(N^2)$ calls to `AggregateMergedProbability()`: - -1. **Per-Segment Cached Metadata**: `Merger` maintains flat vectors indexed by `segment_index_t`: - * `cached_agg_prob_[s]`: `AggregateProbability(s)` - * `cached_exclusive_size_[s]`: `patch_size_cache->GetPatchSize(ExclusiveGlyphs(s))` (or `0` if empty) -2. **Probability-Ordered Active Set**: Active segments below `optimization_cutoff_segment_` with non-empty exclusive glyphs are maintained in an ordered structure sorted by `(cached_agg_prob_[s] descending, s ascending)`. -3. **Row Pruning via `BestCaseInertProbabilityThreshold`**: - * Any exclusive pair $\{u, v\}$ is oriented during candidate generation so that $u$ is higher-ranked in probability order ($P_{\text{agg}}(u) \ge P_{\text{agg}}(v)$). - * When scanning lower-probability partners $v$ for segment $u$ in descending order of $P_{\text{agg}}(v)$, once $u$ is inert and $P_{\text{agg}}(v) \le \text{BestCaseInertProbabilityThreshold}(\text{size}_u, P_{\text{agg}}(u), 0.0)$, all subsequent inert segments $v'$ also have $P_{\text{agg}}(v') \le P_{\text{agg}}(v)$ and cannot achieve a negative cost delta with $u$, allowing the scan to `break` early (or skip remaining inert segments). -4. **$O(1)$ Pre-Check & Inert Fast-Path in `ComputeCostDelta`**: - * When both segments $u$ and $v$ are inert, their only modified conditions are their two unitary exclusive conditions `(u)` and `(v)`, which collapse into a single unitary condition `(min(u, v))`. - * Under `best_case_size_reduction_fraction` ($r_{\text{best}}$), the best-case merged patch size is: - $$S_{\text{best}}(u, v) = \max(S_u, S_v) + \max\left(\lfloor (\min(I_u, I_v)) \cdot r_{\text{best}} \rfloor, \text{BEST\_CASE\_MERGE\_SIZE\_DELTA}\right)$$ - where $I_s = \max(0, S_s - \text{base\_overhead})$. - * Since $P_{\text{agg}}(u \cup v) \ge \max(P_{\text{agg}}(u), P_{\text{agg}}(v)) = P_{\text{agg}}(u)$, we can check in ~5 arithmetic operations—**before** calling `AggregateMergedProbability()` or allocating any hash maps—whether: - $$P_{\text{agg}}(u) \cdot (S_{\text{best}}(u, v) - S_u) - P_{\text{agg}}(v) \cdot (S_v + k_{\text{net}}) \ge \text{delta\_to\_beat}$$ - If this holds, the pair cannot possibly pass `ComputeCostDelta` and is rejected in nanoseconds. - * If it passes the pre-check, the inert fast-path computes `AggregateMergedProbability({u, v})` and evaluates the 3-term cost delta directly without constructing `modified_conditions` or `new_conditions` maps. - -### 4. Composite Condition Candidates - -In single-base mode, `CollectCompositeCandidateMerges(base_segment_index)` scans `OrderedConditions()` for every base segment, potentially evaluating a 3-segment condition $\{s_1, s_2, s_3\}$ multiple times. - -In all-pairs mode (when `target_mode != UNICODE_RANGE`): -* Each composite condition $c$ with triggering segments $S = c.\text{TriggeringSegments()}$ satisfying $|S| > 2$, $*S.\text{min}() < \text{optimization\_cutoff\_segment\_}$, $S \cap \text{candidate\_segments\_} \neq \emptyset$, and $S \subseteq \text{inscope\_segments\_}$ defines a canonical candidate merge set $S$ (with `base = *S.min()`). -* Unique composite sets $S$ are evaluated with `ComputeCostDelta(*this, S, std::nullopt, 0.0)` and, if $< 0.0$, pushed into the same priority queue alongside pairwise candidates. - -### 5. Epoch-Based Invalidation Across `TryNextMerge()` Calls - -In `ClosureGlyphSegmenter::CodepointToGlyphSegments()`, `context.ReprocessChanged(*maybe_modified)` is called *after* `merger.TryNextMerge()` returns. Consequently, updated glyph conditions and groupings for a merge applied in call $k$ are only finalized at the entry of call $k + 1$. - -To handle this cleanly without changing `TryNextMerge()`'s external contract: - -1. **Epoch Tracking**: - * `Merger` maintains a monotonic counter `uint32_t current_epoch_ = 0` and a vector `std::vector segment_epoch_` initialized to `0`. - * Removed segments (`to_merge` or `MarkFinished`) have `segment_epoch_[s]` set to `UINT32_MAX`. -2. **Dirty Segment Recording on Merge Application (Call $k$)**: - * When a merge $M$ into `base` is applied: - * For an **inert merge** (`input_segments_are_inert_ && new_segment_is_inert`), `CandidateMerge::Apply()` updates `glyph_condition_set` and `glyph_groupings` in-place. Only `base` is added to `dirty_segments_`. - * For a **non-inert merge**, before `Apply()` invalidates conditions, `Merger` records in `dirty_segments_`: `base`, every segment in $c.\text{TriggeringSegments()}$ for $c \in \bigcup_{s \in M} \text{TriggeringSegmentToConditions}(s)$, and every segment in the pre-merge condition of `invalidated_glyphs_`. It also saves `pending_non_inert_base_ = base` and `pending_invalidated_glyphs_ = invalidated_glyphs_`. -3. **Refreshing Dirty Segments at Next `TryNextMerge()` Entry (Call $k + 1$)**: - * If `pending_non_inert_base_` is set (meaning `context.ReprocessChanged` just ran), also add to `dirty_segments_` all segments in `TriggeringSegmentToConditions(*pending_non_inert_base_)` and in the post-reprocessing conditions of `pending_invalidated_glyphs_` (intersected with `candidate_segments_`). - * Increment `++current_epoch_`. For each $s \in \text{dirty\_segments\_}$, set `segment_epoch_[s] = current_epoch_` and refresh `cached_agg_prob_[s]`, `cached_exclusive_size_[s]`, and $s$'s position in the probability-ordered active set. - * Generate new pairwise candidates involving `dirty_segments_` (avoiding duplicate `{u, v}` evaluations when both $u, v \in \text{dirty\_segments\_}$) and any composite condition candidates intersecting `dirty_segments_`, then clear `dirty_segments_`. - ---- - -## Implementation Plan / TODO - -- [x] **Commit 1: Add `--experimental_all_pairs_merging` flag, per-segment cost metadata cache, and inert 2-segment fast-path.** Declare `ABSL_DECLARE_FLAG(bool, experimental_all_pairs_merging)` in `ift/encoder/merger.h` (default `false`), add cached per-segment aggregate probability and exclusive patch size vectors on `Merger`, and add an $O(1)$ pre-check + fast-path for 2-segment inert cost delta evaluations in `CandidateMerge::ComputeCostDelta` (with unit tests in `ift/encoder/candidate_merge_test.cc`). -- [x] **Commit 2: Implement the lazy two-stage all-pairs priority queue and probability-ordered candidate generation in `AllPairsMerger`.** Add `AllPairsMerger` (`ift/encoder/all_pairs_merger.h`/`.cc`) with `QueuedCandidate`, probability-ordered active segment tracking with `BestCaseInertProbabilityThreshold` row pruning, composite condition candidate queuing, and the lazy `BEST_CASE` $\rightarrow$ `EXACT` pop-and-refine loop for Phase 1 cost-reducing merges. -- [x] **Commit 3: Implement epoch-based invalidation and incremental `dirty_segments_` refresh across `TryNextMerge()` calls.** Track affected segments for both inert and non-inert merges, refresh `dirty_segments_` at the start of `TryNextMerge()` after `SegmentationContext::ReprocessChanged()`, and reset all-pairs state in `Merger::ReassignInitSubset()`. -- [x] **Commit 4: Integrate Phase 2 (`min_group_size` enforcement) and add unit and determinism tests.** Transition to Phase 2 for any undersized segments remaining after Phase 1 completes, and add tests in `ift/encoder/candidate_merge_test.cc` and `ift/encoder/closure_glyph_segmenter_test.cc` verifying that `--experimental_all_pairs_merging=true` avoids single-base greedy traps, handles composite conditions / `min_group_size` / `UNICODE_RANGE` mode, and is deterministic. diff --git a/docs/segmenter.md b/docs/segmenter.md index ff827dd0..34baefb9 100644 --- a/docs/segmenter.md +++ b/docs/segmenter.md @@ -2,7 +2,7 @@ Author: Garret Rieger Date: Aug 5th, 2026 -Updated: Sep 15th, 2026 +Updated: Nov 12th, 2026 ## Introduction @@ -185,6 +185,58 @@ capitalize on. In the case of initial font merges the cost delta also adds the number of bytes the initial font increases by to the delta value. +### All-Pairs Segment Merging + +During cost-based segment merging ([all_pairs_merger.cc](../ift/encoder/all_pairs_merger.cc)), +candidate merges are selected globally across all active segments in a merge group rather than +greedily merging into one base segment at a time. Evaluating candidates globally avoids greedy traps +where a high-probability segment absorbs a lower-probability segment for a marginal gain and blocks +a much stronger merge between that lower-probability segment and another partner. + +Active segments with exclusive patches above the low-impact optimization cutoff are maintained in +descending probability order so candidate pairs can be pruned early once lower-ranked inert segments +fall below the best-case probability threshold. Multi-segment composite activation conditions are +also queued as candidate merges. Only strictly cost-reducing merges ($\Delta \text{cost} < 0$) are +accepted during all-pairs merging; once all negative cost delta candidates are exhausted, the +all-pairs phase finishes and any remaining undersized segments are handled in a separate pass for +minimum group size enforcement. + +To avoid evaluating $O(n^2)$ Brotli compressions upfront and recomputing unaffected pairs after +every merge, `AllPairsMerger` combines three mechanisms: + +* **Two-Stage Priority Queue**: A single min-heap stores both *best-case lower-bound* entries + (computed from best-case merged patch sizes without running Brotli) and *exact* entries (backed by + Brotli-compressed patch sizes). Heap priorities normalize $\Delta \text{cost}$ by the total + pre-merge byte size (including per-patch network overhead $k$) of the patches affected by the + merged segments, prioritizing merges with the largest expected transfer savings per affected byte + (breaking ties in favor of exact entries, then lexicographically by segment set). On each step the + highest-priority non-stale entry is popped: if it is a lower-bound entry, its exact Brotli cost + delta is evaluated and, if still negative, re-inserted into the heap as an exact entry; if the + popped entry is already exact, its normalized delta is at least as good as every remaining entry + in the heap and the merge is applied immediately. + +* **Segment Epochs**: Rather than scanning or rebuilding the heap to purge invalidated candidates + after a merge, `AllPairsMerger` performs lazy invalidation using a monotonically increasing + `current_epoch` counter and a per-segment epoch array. Each queued candidate records + `current_epoch` when created. Segments that are merged into another segment or marked finished + have their epoch set to `UINT32_MAX`, and segments whose conditions change during a merge have + their epoch updated to the incremented `current_epoch`. When a candidate is popped from the heap, + it is discarded if any of its segments has an epoch greater than the candidate's creation epoch. + +* **Dirty Tracking and Refresh**: Because incremental condition reprocessing for non-inert merges + runs between `TryNextMerge()` calls, post-merge glyph conditions and patch groupings are not + finalized until the start of the next iteration. During merge application (before conditions are + modified), `AssignMergedSegment` records any segments sharing a pre-merge composite condition with + the merged segments, records one representative glyph from each pre-combination condition + triggered by the merged segments, temporarily removes the merged and surviving base segments from + the active set with epoch `UINT32_MAX`, and marks the surviving base segment dirty. At the start + of the next `TryNextAllPairsMerge()` call (after condition reprocessing completes), + `RefreshDirtySegments` looks up the post-reprocessing conditions of those representative glyphs + and the surviving base segment to collect all newly affected triggering segments, updates the + surviving base segment's probability and exclusive patch size in the active set, increments + `current_epoch` across all dirty segments to invalidate stale heap entries, and queues fresh + best-case pairwise and composite candidates involving only the dirty segments. + ### Estimating Activation Condition Probabilities To compute costs we also need to have an estimate for the probability of an activation condition being activated. @@ -486,7 +538,11 @@ help preserve privacy. The current merger implementation has this as a configura minimum group size is configured the merger will never select a merge which has a positive cost delta. When a minimum group size is configured then the merger will accept positive cost delta merges for patches that do not meet the configured minimum group size. When merging to meet minimum -group sizes the merger will still seek out the lowest, least positive, cost delta candidate. +group sizes the merger will still seek out the lowest, least positive, cost delta candidate. For +segment merging, minimum group sizes are enforced in a separate pass after the all-pairs merger has +exhausted all negative cost delta merges: remaining undersized segments are visited in descending +probability order (using the active segment ordering maintained by the all-pairs merger) and merged +per-base without resuming all-pairs cost-reduction searches. #### Caching From 028024445f1fb4b60ce788f784065e919f138892 Mon Sep 17 00:00:00 2001 From: Garret Rieger Date: Thu, 12 Nov 2026 22:37:11 +0000 Subject: [PATCH 13/13] Speed up closure_glyph_segmenter_test by removing full roboto run of non-dep graph modes. These are deprecated and will be removed in the future. --- ift/encoder/closure_glyph_segmenter_test.cc | 7 ------- 1 file changed, 7 deletions(-) diff --git a/ift/encoder/closure_glyph_segmenter_test.cc b/ift/encoder/closure_glyph_segmenter_test.cc index cb1a4753..7002ab0b 100644 --- a/ift/encoder/closure_glyph_segmenter_test.cc +++ b/ift/encoder/closure_glyph_segmenter_test.cc @@ -822,18 +822,11 @@ TEST_F(ClosureGlyphSegmenterTest, FullRoboto_WithFeaturesAndDepGraph) { smcp.feature_tags.insert(HB_TAG('s', 'm', 'c', 'p')); segments.push_back(smcp); - auto segmentation = segmenter.CodepointToGlyphSegments( - roboto.get(), {}, segments, MergeStrategy::Heuristic(4000, 12000)); - auto dep_graph_segmentation = segmenter_dep_graph.CodepointToGlyphSegments( - roboto.get(), {}, segments, MergeStrategy::Heuristic(4000, 12000)); auto dep_graph_only_segmentation = segmenter_dep_graph_only.CodepointToGlyphSegments( roboto.get(), {}, segments, MergeStrategy::Heuristic(4000, 12000)); - ASSERT_TRUE(segmentation.ok()) << segmentation.status(); - ASSERT_TRUE(dep_graph_segmentation.ok()) << dep_graph_segmentation.status(); ASSERT_TRUE(dep_graph_only_segmentation.ok()) << dep_graph_only_segmentation.status(); - ASSERT_EQ(segmentation, dep_graph_segmentation); } TEST_F(ClosureGlyphSegmenterTest, CostRequiresFrequencies) {