Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
60 changes: 58 additions & 2 deletions docs/segmenter.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

Author: Garret Rieger
Date: Aug 5th, 2026
Updated: Sep 15th, 2026
Updated: Nov 12th, 2026

## Introduction

Expand Down Expand Up @@ -185,6 +185,58 @@ capitalize on.
In the case of initial font merges the cost delta also adds the number of bytes the initial font
increases by to the delta value.

### All-Pairs Segment Merging

During cost-based segment merging ([all_pairs_merger.cc](../ift/encoder/all_pairs_merger.cc)),
candidate merges are selected globally across all active segments in a merge group rather than
greedily merging into one base segment at a time. Evaluating candidates globally avoids greedy traps
where a high-probability segment absorbs a lower-probability segment for a marginal gain and blocks
a much stronger merge between that lower-probability segment and another partner.

Active segments with exclusive patches above the low-impact optimization cutoff are maintained in
descending probability order so candidate pairs can be pruned early once lower-ranked inert segments
fall below the best-case probability threshold. Multi-segment composite activation conditions are
also queued as candidate merges. Only strictly cost-reducing merges ($\Delta \text{cost} < 0$) are
accepted during all-pairs merging; once all negative cost delta candidates are exhausted, the
all-pairs phase finishes and any remaining undersized segments are handled in a separate pass for
minimum group size enforcement.

To avoid evaluating $O(n^2)$ Brotli compressions upfront and recomputing unaffected pairs after
every merge, `AllPairsMerger` combines three mechanisms:

* **Two-Stage Priority Queue**: A single min-heap stores both *best-case lower-bound* entries
(computed from best-case merged patch sizes without running Brotli) and *exact* entries (backed by
Brotli-compressed patch sizes). Heap priorities normalize $\Delta \text{cost}$ by the total
pre-merge byte size (including per-patch network overhead $k$) of the patches affected by the
merged segments, prioritizing merges with the largest expected transfer savings per affected byte
(breaking ties in favor of exact entries, then lexicographically by segment set). On each step the
highest-priority non-stale entry is popped: if it is a lower-bound entry, its exact Brotli cost
delta is evaluated and, if still negative, re-inserted into the heap as an exact entry; if the
popped entry is already exact, its normalized delta is at least as good as every remaining entry
in the heap and the merge is applied immediately.

* **Segment Epochs**: Rather than scanning or rebuilding the heap to purge invalidated candidates
after a merge, `AllPairsMerger` performs lazy invalidation using a monotonically increasing
`current_epoch` counter and a per-segment epoch array. Each queued candidate records
`current_epoch` when created. Segments that are merged into another segment or marked finished
have their epoch set to `UINT32_MAX`, and segments whose conditions change during a merge have
their epoch updated to the incremented `current_epoch`. When a candidate is popped from the heap,
it is discarded if any of its segments has an epoch greater than the candidate's creation epoch.

* **Dirty Tracking and Refresh**: Because incremental condition reprocessing for non-inert merges
runs between `TryNextMerge()` calls, post-merge glyph conditions and patch groupings are not
finalized until the start of the next iteration. During merge application (before conditions are
modified), `AssignMergedSegment` records any segments sharing a pre-merge composite condition with
the merged segments, records one representative glyph from each pre-combination condition
triggered by the merged segments, temporarily removes the merged and surviving base segments from
the active set with epoch `UINT32_MAX`, and marks the surviving base segment dirty. At the start
of the next `TryNextAllPairsMerge()` call (after condition reprocessing completes),
`RefreshDirtySegments` looks up the post-reprocessing conditions of those representative glyphs
and the surviving base segment to collect all newly affected triggering segments, updates the
surviving base segment's probability and exclusive patch size in the active set, increments
`current_epoch` across all dirty segments to invalidate stale heap entries, and queues fresh
best-case pairwise and composite candidates involving only the dirty segments.

### Estimating Activation Condition Probabilities

To compute costs we also need to have an estimate for the probability of an activation condition being activated.
Expand Down Expand Up @@ -486,7 +538,11 @@ help preserve privacy. The current merger implementation has this as a configura
minimum group size is configured the merger will never select a merge which has a positive cost
delta. When a minimum group size is configured then the merger will accept positive cost delta
merges for patches that do not meet the configured minimum group size. When merging to meet minimum
group sizes the merger will still seek out the lowest, least positive, cost delta candidate.
group sizes the merger will still seek out the lowest, least positive, cost delta candidate. For
segment merging, minimum group sizes are enforced in a separate pass after the all-pairs merger has
exhausted all negative cost delta merges: remaining undersized segments are visited in descending
probability order (using the active segment ordering maintained by the all-pairs merger) and merged
per-base without resuming all-pairs cost-reduction searches.

#### Caching

Expand Down
3 changes: 3 additions & 0 deletions ift/encoder/BUILD
Original file line number Diff line number Diff line change
Expand Up @@ -78,6 +78,7 @@ cc_library(
cc_library(
name = "segmentation_context",
srcs = [
"all_pairs_merger.cc",
"candidate_merge.cc",
"complex_condition_finder.cc",
"dependency_closure.cc",
Expand All @@ -89,6 +90,7 @@ cc_library(
"woff2_patch_size_cache.cc",
],
hdrs = [
"all_pairs_merger.h",
"candidate_merge.h",
"complex_condition_finder.h",
"condition_to_glyphs_index.h",
Expand Down Expand Up @@ -131,6 +133,7 @@ cc_library(
"@abseil-cpp//absl/status:statusor",
"@abseil-cpp//absl/strings",
"@abseil-cpp//absl/strings:str_format",
"@abseil-cpp//absl/types:span",
"@harfbuzz",
],
)
Expand Down
16 changes: 16 additions & 0 deletions ift/encoder/activation_condition.cc
Original file line number Diff line number Diff line change
Expand Up @@ -252,6 +252,22 @@ bool ActivationCondition::Intersects(const common::SegmentSet& segments) const {
return false;
}

bool ActivationCondition::IsUnchangedByMerge(
const common::SegmentSet& merged_segments) const {
size_t merge_size = merged_segments.size();
if (merge_size <= 1) {
return true;
}
for (const auto& condition_group : conditions_) {
if (condition_group.intersects(merged_segments) &&
(condition_group.size() < merge_size ||
!merged_segments.is_subset_of(condition_group))) {
return false;
}
}
return true;
}

ActivationCondition ActivationCondition::ReplaceSegments(
segment_index_t base_segment, const SegmentSet& segments) const {
if (IsPurelyConjunctive()) {
Expand Down
6 changes: 6 additions & 0 deletions ift/encoder/activation_condition.h
Original file line number Diff line number Diff line change
Expand Up @@ -124,6 +124,12 @@ class ActivationCondition {
// Returns true if any of the condition groups intersects with 'segments'.
bool Intersects(const common::SegmentSet& segments) const;

// Returns true if merging 'merged_segments' into a single segment does not
// change the effective activation condition (i.e. every disjunctive
// subgroup that intersects 'merged_segments' has 'merged_segments' as a
// subset).
bool IsUnchangedByMerge(const common::SegmentSet& merged_segments) const;

// Generate a new condition which is a copy of this one but with all
// occurences of 'segments' replaced with base_segment. New condition will be
// simplified to remove any resulting redundancy.
Expand Down
38 changes: 38 additions & 0 deletions ift/encoder/activation_condition_test.cc
Original file line number Diff line number Diff line change
Expand Up @@ -659,4 +659,42 @@ TEST(ActivationConditionTest, HashCaching) {
EXPECT_EQ(cond, cond_copy);
}

TEST(ActivationConditionTest, IsUnchangedByMerge) {
// Empty or singleton merge sets never change any condition.
EXPECT_TRUE(ActivationCondition::or_segments({1, 2}, 0).IsUnchangedByMerge({}));
EXPECT_TRUE(ActivationCondition::or_segments({1, 2}, 0).IsUnchangedByMerge({1}));
EXPECT_TRUE(ActivationCondition::True(0).IsUnchangedByMerge({1, 2}));

// Non-intersecting conditions are unchanged.
EXPECT_TRUE(ActivationCondition::or_segments({3, 4}, 0).IsUnchangedByMerge({1, 2}));
EXPECT_TRUE(ActivationCondition::and_segments({3, 4}, 0).IsUnchangedByMerge({1, 2}));

// Disjunctive condition that is a superset of the merge group is unchanged.
EXPECT_TRUE(ActivationCondition::or_segments({1, 2, 3}, 0).IsUnchangedByMerge({1, 2}));
EXPECT_TRUE(ActivationCondition::or_segments({1, 2}, 0).IsUnchangedByMerge({1, 2}));

// Disjunctive condition that partially intersects the merge group is changed.
EXPECT_FALSE(ActivationCondition::or_segments({1, 3}, 0).IsUnchangedByMerge({1, 2}));

// Conjunctive condition whose clauses are singletons can never have a
// 2+-element merge group as a subset.
EXPECT_FALSE(ActivationCondition::exclusive_segment(1, 0).IsUnchangedByMerge({1, 2}));
EXPECT_FALSE(ActivationCondition::and_segments({1, 2}, 0).IsUnchangedByMerge({1, 2}));
EXPECT_FALSE(ActivationCondition::and_segments({1, 3}, 0).IsUnchangedByMerge({1, 2}));

// Composite condition: unchanged if all intersecting subgroups contain the
// merge group as a subset.
EXPECT_TRUE(ActivationCondition::composite_condition({{1, 2, 3}, {4, 5}}, 0)
.IsUnchangedByMerge({1, 2}));
EXPECT_TRUE(ActivationCondition::composite_condition({{1, 2}, {1, 2, 4}}, 0)
.IsUnchangedByMerge({1, 2}));

// Composite condition: changed if any intersecting subgroup does not contain
// the entire merge group.
EXPECT_FALSE(ActivationCondition::composite_condition({{1, 3}, {4, 5}}, 0)
.IsUnchangedByMerge({1, 2}));
EXPECT_FALSE(ActivationCondition::composite_condition({{1, 2, 3}, {2, 4}}, 0)
.IsUnchangedByMerge({1, 2}));
}

} // namespace ift::encoder
Loading
Loading