diff --git a/docs/experimental/unicode_range_segmentation.md b/docs/experimental/unicode_range_segmentation.md new file mode 100644 index 00000000..565bc91a --- /dev/null +++ b/docs/experimental/unicode_range_segmentation.md @@ -0,0 +1,98 @@ +# `@font-face unicode-range` Segmentation Mode + +Author: Garret Rieger +Date: Oct 7th, 2026 + +## Introduction + +Note: this is a highly experimental exploration. + +While the IFT encoder's segmentation plan generator ([segmenter.md](segmenter.md)) was designed to partition a font into glyph-keyed and table-keyed patches for Incremental Font Transfer (IFT), the same dependency analysis and frequency-guided merging framework can also be used to produce font subset segmentations for serving fonts via CSS `@font-face` `unicode-range`. + +With `@font-face` `unicode-range`, a font is split into a collection of standalone WOFF2 font subsets that browsers load on demand based on codepoint presence on the page. Supporting `unicode-range` as an optional target mode introduces several key differences compared to standard IFT segmentation: + +| Aspect | `TargetMode::IFT` (Default) | `TargetMode::UNICODE_RANGE` | +| :--- | :--- | :--- | +| **Client Loading Model** | Single font instance incrementally extended by patches | Independent `@font-face` WOFF2 font files matched per character by `unicode-range` | +| **Initial Segment (`config.initial_segment`)** | Loaded once as the base font; initial font merging moves high-probability patches into it | Included in **every** subset (union of `initial_segment` codepoints/glyphs/features + default features and the patch's glyphs; useful for UVS codepoints, etc.). Initial font merging is disabled | +| **Layout Features** | Optional feature patches + table-keyed patches | Codepoints only; each subset statically retains the default layout features plus any explicitly requested in `config.initial_segment.features`. Optional feature segments and table-keyed segments are disallowed | +| **Unmapped Glyphs** | Configurable (defaults to `MOVE_TO_INIT_FONT`) | Must be `FIND_CONDITIONS` or `PATCH` (`AutoSegmenterConfig` sets `FIND_CONDITIONS`; `MOVE_TO_INIT_FONT` is rejected) | +| **Patch Merging** | Optional (`experimental_use_patch_merges`) | Disabled; only segment merges are performed | +| **Interacting Glyphs & Disjoint Conditions** | Can be placed in separate patches (e.g. `fi` in patch `f AND i`, shared component `A` in `A OR À`) | Determined via dependency graph / closure analysis; `MakeActivationConditionsDisjoint` merges triggering segments of non-exclusive conditions whose probability $\ge$ `disjoint_conditions_probability_threshold`. Conditions below the threshold are skipped | +| **Final Subsets & Activation Conditions** | CNF expressions (`(s1 OR s2) AND s3`) | Only **exclusive** patches (`(s_i)`) are emitted in the final plan (any remaining non-exclusive patches are ignored), yielding strictly disjunctive and mutually disjoint `unicode-range` sets | +| **Patch Size / Cost Model** | Brotli-compressed glyph-keyed patch | Standalone WOFF2 font subset for `initial_segment` $\cup$ patch glyphs (`hb_subset` + WOFF2 encoding with `glyf` transform) | + +--- + +## Configuration & Validation + +A `TargetMode` enum (`IFT`, `UNICODE_RANGE`) in [segmenter_config.proto](../ift/config/segmenter_config.proto) selects the segmentation target mode (`target_mode` on `SegmenterConfig`, defaulting to `IFT`), alongside a `disjoint_conditions_probability_threshold` setting. The auto-configuration generator (`AutoSegmenterConfig::GenerateConfig`) also accepts `TargetMode` and automatically configures appropriate defaults when `UNICODE_RANGE` is selected: + +* Disables optional feature segment generation (`generate_feature_segments = false`) and table-keyed segment generation (`generate_table_keyed_segments = false`). +* Sets `unmapped_glyph_handling = FIND_CONDITIONS` so all glyphs in the full closure receive activation conditions. +* Leaves initial font merging thresholds unset and skips IFT-specific base segmentation plan settings (`jump_ahead`, `use_prefetch_lists`). + +To ensure invalid configurations are caught regardless of whether a caller goes through `SegmenterConfigUtil` or invokes `ClosureGlyphSegmenter` directly, `ClosureGlyphSegmenter` validates that when `target_mode == UNICODE_RANGE`: + +1. `unmapped_glyph_handling` is set to either `FIND_CONDITIONS` or `PATCH` (`MOVE_TO_INIT_FONT` is rejected). +2. No input segment definitions contain layout features (optional feature loading is unsupported). +3. Initial font merging is not enabled on any merge group. +4. Patch merging is disabled on all merge groups. +5. `generate_table_keyed_segments` is not enabled. + +### Initial Segment Handling + +In `UNICODE_RANGE` mode, there is no separate initial font loaded prior to patches. Instead, `config.initial_segment` defines a common base subset (codepoints, glyphs, and layout features, combined with the default layout feature list) that is included in **every** generated font subset. + +This is particularly useful for codepoints such as Unicode Variation Selectors (UVS) that may combine with many otherwise unrelated base characters across the font: placing them in `initial_segment` makes them available in every subset and prevents them from creating cross-subset dependencies during condition analysis. + +--- + +## Disjoint Activation Conditions & Interacting Glyphs + +In CSS `@font-face`, each subset is treated as a separate font face during text shaping, and `unicode-range` matching activates subsets purely on per-character codepoint presence. This imposes two requirements on the segmentation: + +1. **Interacting glyphs must reside in the same subset**: If a substitution (such as an `fi` ligature with conjunctive condition `f AND i`) or a shared glyph component (such as base glyph `A` with disjunctive condition `A OR À`) spans multiple segments, splitting those segments into separate `@font-face` subsets would either prevent the substitution from occurring across subset boundaries or require overlapping `unicode-range` definitions. +2. **Activation conditions must be mutually disjoint and exclusive**: Each final font subset must correspond to a single segment (a disjoint set of codepoints forming its `unicode-range`). + +### `MakeActivationConditionsDisjoint` + +After initial condition analysis (and unmapped glyph condition finding when `FIND_CONDITIONS` is used), the segmenter runs a `MakeActivationConditionsDisjoint` pass before starting general cost-based or heuristic merging: + +1. Inspect all non-exclusive patch conditions (conditions involving more than one triggering segment, whether conjunctive or disjunctive). +2. Group segments into connected components by unioning the triggering segments of each non-exclusive condition. +3. Merge the segments within each connected component into a single representative segment and update the segmentation context. + +When all non-exclusive conditions participate in this pass, every multi-segment interaction collapses into a single exclusive segment condition, leaving all surviving segments inert. + +### Breaking Low-Probability Interactions + +In some fonts, rare ligatures or complex contextual rules can chain together large numbers of segments into a single connected component, limiting how finely the font can be split. To control this trade-off, `MakeActivationConditionsDisjoint` supports a configurable `disjoint_conditions_probability_threshold`: + +* For each non-exclusive patch condition, its activation probability is evaluated against the frequency data of the covering cost-based merge group(s). +* If the condition's probability is below `disjoint_conditions_probability_threshold`, that condition is skipped during `MakeActivationConditionsDisjoint` and does not force its triggering segments to be merged. +* During the subsequent merging phase, only segment merges are performed (patch merges are disabled). If the triggering segments of a skipped condition are not later merged by cost-based or heuristic segment merging, the remaining non-exclusive patch is ignored when constructing the final segmentation plan: **only exclusive patches are emitted into the final `unicode-range` segmentation plan**. + +--- + +## WOFF2 Subset Cost Function + +In standard IFT mode, the cost function evaluates candidate merges using the Brotli-compressed size of glyph-keyed patches (which only contain outline tables such as `glyf`/`gvar`/`CFF`/`CFF2` plus a small patch header). + +In `UNICODE_RANGE` mode, each patch corresponds to a complete, standalone WOFF2 font subset. The subset for a patch is defined as the union of `initial_segment` (including default layout features) and the patch's glyphs (along with their associated codepoints). Because every WOFF2 subset must include shared font tables (`cmap`, `head`, `hhea`, `maxp`, `OS/2`, subsetted `GSUB`/`GPOS`, `fvar`, etc.), per-subset table overhead is substantially higher than in glyph-keyed patches. + +To reflect this in the merger's cost delta calculations (`cost(S) = sum P(c_i) * (size(p_i) + k)`): + +* **Exact WOFF2 Sizing (`brotli_quality > 0`)**: A WOFF2-backed `PatchSizeCache` subsets the font to `initial_segment` $\cup$ the candidate patch's glyphs and encodes the resulting font with WOFF2 (including the WOFF2 `glyf` transform) at the configured Brotli quality. When evaluating a merge of two segments $A$ and $B$, the cost delta naturally captures the byte savings of deduplicating the shared non-outline font tables across the two subsets. +* **Estimated WOFF2 Sizing (`brotli_quality == 0`)**: For fast low-quality levels where per-candidate Brotli compression is disabled, subset sizes are estimated using the WOFF2 size of the base `initial_segment` subset (capturing fixed per-subset table overhead) plus the raw outline size of the patch's glyphs scaled by the font's overall WOFF2 outline compression ratio. + +--- + +## Output `SegmentationPlan` Representation + +The resulting `SegmentationPlan` protobuf directly encodes the `@font-face` `unicode-range` subsets for consumption by a downstream subset compiler: + +* `initial_codepoints`, `initial_glyphs`, and `initial_features` specify the common base definition (from `config.initial_segment`) to be unioned into every generated subset alongside default layout features. +* Each entry in `glyph_patch_conditions` is an exclusive condition referencing a single segment `s_i` (`segments[s_i].codepoints` defines the `unicode-range` for that `@font-face` rule) and activating patch `p_i`. +* `glyph_patches[p_i]` lists the disjoint set of glyphs belonging to subset `p_i` (to be unioned with `initial_segment` when producing the subset font). +* `non_glyph_segments` and table-keyed patch fields remain empty. diff --git a/ift/config/BUILD b/ift/config/BUILD index 75c61276..208ca7bc 100644 --- a/ift/config/BUILD +++ b/ift/config/BUILD @@ -106,13 +106,18 @@ cc_library( "//ift/common", "//ift/common:data_file_resolver", "//ift/common:try", + "//ift/encoder:activation_condition", + "//ift/encoder:common", "//ift/encoder:glyph_partition", + "//ift/encoder:segmentation_context", + "//ift/encoder:segmentation_info", "//ift/feature_registry", "//ift/freq", "@abseil-cpp//absl/container:btree", "@abseil-cpp//absl/container:flat_hash_map", "@abseil-cpp//absl/container:flat_hash_set", "@abseil-cpp//absl/log", + "@abseil-cpp//absl/status", "@abseil-cpp//absl/status:statusor", "@abseil-cpp//absl/strings", "@harfbuzz", diff --git a/ift/config/auto_segmenter_config.cc b/ift/config/auto_segmenter_config.cc index 1c91cef0..a090d69b 100644 --- a/ift/config/auto_segmenter_config.cc +++ b/ift/config/auto_segmenter_config.cc @@ -2,6 +2,7 @@ #include #include +#include #include #include #include @@ -11,6 +12,7 @@ #include "absl/container/flat_hash_map.h" #include "absl/container/flat_hash_set.h" #include "absl/log/log.h" +#include "absl/status/status.h" #include "absl/strings/match.h" #include "absl/strings/str_join.h" #include "absl/strings/strip.h" @@ -21,7 +23,14 @@ #include "ift/config/load_codepoints.h" #include "ift/config/segmentation_plan.pb.h" #include "ift/config/segmenter_config.pb.h" +#include "ift/encoder/activation_condition.h" +#include "ift/encoder/dependency_closure.h" +#include "ift/encoder/glyph_closure_cache.h" #include "ift/encoder/glyph_partition.h" +#include "ift/encoder/init_subset_defaults.h" +#include "ift/encoder/requested_segmentation_information.h" +#include "ift/encoder/segment.h" +#include "ift/encoder/subset_definition.h" #include "ift/feature_registry/feature_registry.h" #include "ift/freq/unicode_frequencies.h" @@ -31,9 +40,16 @@ using absl::flat_hash_set; using absl::Status; using absl::StatusOr; using ift::common::CodepointSet; -using ift::common::IntSet; using ift::common::FontHelper; +using ift::common::GlyphSet; +using ift::common::IntSet; +using ift::encoder::AddInitSubsetDefaults; +using ift::encoder::DependencyClosure; +using ift::encoder::GlyphClosureCache; using ift::encoder::GlyphPartition; +using ift::encoder::RequestedSegmentationInformation; +using ift::encoder::Segment; +using ift::encoder::SubsetDefinition; using ift::feature_registry::DefaultFeatureTags; using ift::freq::UnicodeFrequencies; @@ -282,7 +298,19 @@ static btree_set DetectScripts( // characters) are present in many script's frequency data. Filtering // them out from overlap detection helps avoid grouping together a large number // of scripts unnecessarily. -static bool IgnoredForOverlapDetection(hb_codepoint_t cp) { +static bool IgnoredForOverlapDetection(TargetMode target_mode, hb_codepoint_t cp) { + if (target_mode == UNICODE_RANGE) { + // In unicode range mode we more aggressively group interacting scripts since + // we can't effectively handle shared segments across merge groups. + switch (hb_unicode_general_category(hb_unicode_funcs_get_default(), cp)) { + case HB_UNICODE_GENERAL_CATEGORY_CONTROL: + case HB_UNICODE_GENERAL_CATEGORY_SPACE_SEPARATOR: + return true; + default: + return false; + } + } + switch (hb_unicode_general_category(hb_unicode_funcs_get_default(), cp)) { case HB_UNICODE_GENERAL_CATEGORY_CONTROL: case HB_UNICODE_GENERAL_CATEGORY_FORMAT: @@ -299,6 +327,7 @@ static bool IgnoredForOverlapDetection(hb_codepoint_t cp) { // Returns the codepoints of script which are in the font and which are // usable for overlap detection. static CodepointSet OverlapCodepoints( + TargetMode target_mode, const flat_hash_map& freq_list, const std::string& script, const CodepointSet& unicodes) { auto it = freq_list.find(script); @@ -311,7 +340,7 @@ static CodepointSet OverlapCodepoints( CodepointSet result; for (hb_codepoint_t cp : in_script_and_font) { - if (!IgnoredForOverlapDetection(cp)) { + if (!IgnoredForOverlapDetection(target_mode, cp)) { result.insert(cp); } } @@ -319,6 +348,57 @@ static CodepointSet OverlapCodepoints( return result; } +static Status GroupInteractingScripts( + hb_face_t* face, const std::vector& scripts, + const std::vector& script_codepoints, + const ift::common::DataFileResolver& resolver, GlyphPartition& partition) { + if (script_codepoints.size() <= 1) { + return absl::OkStatus(); + } + + std::unique_ptr closure_cache = + TRY(GlyphClosureCache::Create(face, resolver)); + + SubsetDefinition init_segment; + AddInitSubsetDefaults(init_segment); + + std::vector segments; + segments.reserve(script_codepoints.size()); + for (const CodepointSet& codepoints : script_codepoints) { + segments.emplace_back(SubsetDefinition::Codepoints(codepoints)); + } + + std::unique_ptr segmentation_info = + TRY(RequestedSegmentationInformation::Create( + std::move(segments), init_segment, *closure_cache, FIND_CONDITIONS)); + std::unique_ptr dependency_closure = + TRY(DependencyClosure::Create(segmentation_info.get(), face, resolver, + /*allow_context_glyph_analysis=*/true)); + + for (const auto& [gid, condition] : dependency_closure->AllGlyphConditions()) { + if (condition.IsUnitary()) { + continue; + } + + GlyphSet triggering; + triggering.union_set(condition.TriggeringSegments()); + if (triggering.size() > 1) { + uint32_t first = *triggering.min(); + for (uint32_t other : triggering) { + if (TRY(partition.Find(first)) != TRY(partition.Find(other))) { + LOG(INFO) << "Grouping " << scripts[first] << " and " + << scripts[other] + << " together due to glyph condition interaction on g" + << gid << ": " << condition.ToString(); + TRYV(partition.Union(first, other)); + } + } + } + } + + return absl::OkStatus(); +} + // Groups scripts which significantly overlap each other together. Scripts // which don't overlap anything are placed in a group by themselves. // @@ -335,6 +415,8 @@ static CodepointSet OverlapCodepoints( // The returned groups, and the scripts within them, are in a deterministic // order. static StatusOr>> GroupOverlappingScripts( + hb_face_t* face, + TargetMode target_mode, const btree_set& detected_scripts, const flat_hash_map& freq_list, const CodepointSet& unicodes, @@ -344,7 +426,7 @@ static StatusOr>> GroupOverlappingScripts( std::vector codepoints; codepoints.reserve(scripts.size()); for (const auto& script : scripts) { - codepoints.push_back(OverlapCodepoints(freq_list, script, unicodes)); + codepoints.push_back(OverlapCodepoints(target_mode, freq_list, script, unicodes)); } // Only pairs of scripts that share at least one codepoint can overlap. @@ -415,7 +497,7 @@ static StatusOr>> GroupOverlappingScripts( VLOG(1) << "Overlap between " << scripts[i] << " and " << scripts[j] << " is " << fraction << " (" << shared.size() << " shared codepoints)."; - if (fraction >= kScriptOverlapThreshold) { + if (target_mode == UNICODE_RANGE || fraction >= kScriptOverlapThreshold) { LOG(INFO) << "Grouping " << scripts[i] << " and " << scripts[j] << " together, they share " << fraction << " of their probability mass."; @@ -423,6 +505,11 @@ static StatusOr>> GroupOverlappingScripts( } } + if (target_mode == UNICODE_RANGE) { + TRYV(GroupInteractingScripts(face, scripts, codepoints, resolver, + partition)); + } + std::vector> groups; flat_hash_map representative_to_group; for (uint32_t i = 0; i < scripts.size(); i++) { @@ -770,7 +857,11 @@ static void ApplyQualityLevelTo(Quality quality, MergeGroup& merge_group) { static void ApplyQualityLevelTo(Quality quality, SegmenterConfig& config) { config.set_preprocess_merging_group_size_for_ungrouped(kMinimumGroupSize * 3); - config.set_unmapped_glyph_handling(MOVE_TO_INIT_FONT); + if (config.target_mode() == UNICODE_RANGE) { + config.set_unmapped_glyph_handling(FIND_CONDITIONS); + } else { + config.set_unmapped_glyph_handling(MOVE_TO_INIT_FONT); + } if (quality == ONE) { config.set_condition_analysis_mode(DEP_GRAPH_ONLY_WITH_SIMPLIFICATION); @@ -795,21 +886,23 @@ static void ApplyQualityLevelTo(Quality quality, SegmenterConfig& config) { break; } - switch (quality) { - case ONE: - case TWO: - case THREE: - case FOUR: - config.set_brotli_quality_for_initial_font_merging(0); - break; - case FIVE: - case SIX: - config.set_brotli_quality_for_initial_font_merging(9); - break; - case SEVEN: - default: - config.set_brotli_quality_for_initial_font_merging(11); - break; + if (config.target_mode() != UNICODE_RANGE) { + switch (quality) { + case ONE: + case TWO: + case THREE: + case FOUR: + config.set_brotli_quality_for_initial_font_merging(0); + break; + case FIVE: + case SIX: + config.set_brotli_quality_for_initial_font_merging(9); + break; + case SEVEN: + default: + config.set_brotli_quality_for_initial_font_merging(11); + break; + } } ApplyQualityLevelTo(quality, *config.mutable_base_heuristic_config()); @@ -820,7 +913,7 @@ static void ApplyQualityLevelTo(Quality quality, SegmenterConfig& config) { // ift demo. config.mutable_base_cost_config()->set_network_overhead_cost( kDefaultNetworkCost); - config.mutable_base_cost_config()->set_experimental_use_patch_merges(true); + config.mutable_base_cost_config()->set_experimental_use_patch_merges(config.target_mode() != UNICODE_RANGE); for (auto& merge_group : *config.mutable_merge_groups()) { ApplyQualityLevelTo(quality, merge_group); @@ -878,6 +971,90 @@ static CodepointSet CoveredCodepoints( return result; } +static std::optional GroupCodepointSetOverride( + uint32_t group_index, + const std::vector& script_group, + const flat_hash_map& freq_list, + const std::vector& codepoints_added_to_group, + const CodepointSet& unicodes) { + + CodepointSet group_codepoints = + CoveredCodepoints(script_group, freq_list, unicodes); + + CodepointSet added_codepoints = codepoints_added_to_group.at(group_index); + added_codepoints.intersect(group_codepoints); + + CodepointSet removed_codepoints; + for (uint32_t i = 0; i < codepoints_added_to_group.size(); i++) { + if (i != group_index) { + removed_codepoints.union_set(codepoints_added_to_group.at(i)); + } + } + removed_codepoints.intersect(group_codepoints); + + if (added_codepoints.empty() && removed_codepoints.empty()) { + // No changes are needed + return std::nullopt; + } + + group_codepoints.subtract(removed_codepoints); + group_codepoints.union_set(added_codepoints); + return group_codepoints; +} + +// Return is a map from group index to ignore overlap codepoints which should +// be placed exclusively in that group. +static std::vector PlaceIgnoreOverlapCodepoints( + TargetMode target_mode, + const std::string& primary_script, + const std::vector>& script_groups, + const flat_hash_map& freq_list, + const CodepointSet& unicodes) { + + CodepointSet to_be_moved; + flat_hash_map unicode_to_group; + uint32_t group_index = 0; + for (const auto& scripts : script_groups) { + bool has_primary_script = false; + for (const auto& script : scripts) { + if (script == primary_script) { + has_primary_script = true; + } + } + + CodepointSet covered = CoveredCodepoints(scripts, freq_list, unicodes); + for (hb_codepoint_t u : covered) { + if (!IgnoredForOverlapDetection(target_mode, u)) { + continue; + } + + if (!unicode_to_group.contains(u)) { + unicode_to_group[u] = group_index; + continue; + } + + if (has_primary_script) { + // Primary script group takes precedence + unicode_to_group[u] = group_index; + } + + // Anything that's in more than one group will need to be moved + to_be_moved.insert(u); + } + + group_index++; + } + + std::vector out(script_groups.size()); + + for (hb_codepoint_t u : to_be_moved) { + uint32_t group_index = unicode_to_group.at(u); + out.at(group_index).insert(u); + } + + return out; +} + static bool HasFeatureSegments(hb_face_t* face) { for (hb_tag_t tag : FontHelper::GetFeatureTags(face)) { if (!DefaultFeatureTags().contains(tag)) { @@ -966,7 +1143,7 @@ void AutoSegmenterConfig::ConfigureMaxDepth(uint32_t num_segments, // Adds a merge group to config which uses the frequency data of each of // scripts. The script matching primary_script_file (if any) is configured to -// do initial font merging. +// do initial font merging (in IFT mode). // // By default the group covers all segments which the supplied frequency data // sets have data for. If covered_codepoints is provided the group is instead @@ -985,7 +1162,8 @@ static void AddMergeGroup(const std::vector& scripts, auto* freq_data = cost->add_frequency_data(); freq_data->set_built_in_freq_data_name(script); - if (script == primary_script_file) { + if (config.target_mode() != UNICODE_RANGE && + script == primary_script_file) { freq_data->set_initial_font_merge_threshold( -(double)kDefaultNetworkCost * (0.8)); } @@ -1007,14 +1185,18 @@ static void AddMergeGroup(const std::vector& scripts, StatusOr AutoSegmenterConfig::GenerateConfig( hb_face_t* face, const ift::common::DataFileResolver& resolver, std::optional primary_script, - std::optional quality_level) { + std::optional quality_level, TargetMode target_mode) { SegmenterConfig config; - config.set_generate_table_keyed_segments_mode(FROM_FREQ_DATA); - config.set_generate_feature_segments(true); + if (target_mode == UNICODE_RANGE) { + config.set_target_mode(UNICODE_RANGE); + } else { + config.set_generate_table_keyed_segments_mode(FROM_FREQ_DATA); + config.set_generate_feature_segments(true); - auto* base_plan = config.mutable_base_segmentation_plan(); - base_plan->set_jump_ahead(2); - base_plan->set_use_prefetch_lists(true); + auto* base_plan = config.mutable_base_segmentation_plan(); + base_plan->set_jump_ahead(2); + base_plan->set_use_prefetch_lists(true); + } // Collect codepoints auto freq_list = TRY(BuiltInFrequenciesList(resolver)); @@ -1046,21 +1228,35 @@ StatusOr AutoSegmenterConfig::GenerateConfig( std::string primary_script_file = TRY(FindFileName(primary_script.value_or("Script_latin"), freq_list)); - ConfigureMaxDepth( - CountTableKeyedSegments(face, unicodes, freq_list, detected_scripts), - *base_plan); + if (target_mode != UNICODE_RANGE) { + ConfigureMaxDepth( + CountTableKeyedSegments(face, unicodes, freq_list, detected_scripts), + *config.mutable_base_segmentation_plan()); + } // Add a merge group for each group of detected scripts. Scripts which // overlap each other are placed into the same merge group so that their // shared codepoints are cost merged instead of falling back to heuristic // merging. auto script_groups = TRY( - GroupOverlappingScripts(detected_scripts, freq_list, unicodes, resolver)); - for (const auto& script_group : script_groups) { + GroupOverlappingScripts(face, target_mode, detected_scripts, freq_list, unicodes, resolver)); + + std::vector codepoints_added_to_group(script_groups.size()); + if (target_mode == UNICODE_RANGE) { + // Any of ignore overlap codepoints may be in more than one merge group, + // in unicode range mode we need to place thse into a single group. + codepoints_added_to_group = PlaceIgnoreOverlapCodepoints(target_mode, primary_script_file, script_groups, freq_list, unicodes); + } + + for (uint32_t group_index = 0; group_index < script_groups.size(); group_index++) { + const auto& script_group = script_groups.at(group_index); if (!primary_script.has_value() || script_group.size() < 2 || std::find(script_group.begin(), script_group.end(), primary_script_file) == script_group.end()) { - AddMergeGroup(script_group, primary_script_file, std::nullopt, config); + AddMergeGroup(script_group, + primary_script_file, + GroupCodepointSetOverride(group_index, script_group, freq_list, codepoints_added_to_group, unicodes), + config); continue; } @@ -1069,8 +1265,11 @@ StatusOr AutoSegmenterConfig::GenerateConfig( // optimizing it. If it was left grouped with the other scripts it would // be treated as equally important as each of them, so instead give it a // merge group of its own. - AddMergeGroup({primary_script_file}, primary_script_file, std::nullopt, - config); + std::optional primary_script_override = + GroupCodepointSetOverride(group_index, {primary_script_file}, freq_list, codepoints_added_to_group, unicodes); + AddMergeGroup({primary_script_file}, primary_script_file, + primary_script_override, + config); // The rest of the group still needs to be handled, but only for the // codepoints which the primary script's merge group doesn't already @@ -1084,9 +1283,14 @@ StatusOr AutoSegmenterConfig::GenerateConfig( } CodepointSet remaining_codepoints = - CoveredCodepoints(remaining_scripts, freq_list, unicodes); - remaining_codepoints.subtract(CoveredCodepoints( - std::vector{primary_script_file}, freq_list, unicodes)); + GroupCodepointSetOverride(group_index, remaining_scripts, freq_list, codepoints_added_to_group, unicodes) + .value_or(CoveredCodepoints(remaining_scripts, freq_list, unicodes)); + + remaining_codepoints.subtract( + primary_script_override.value_or( + CoveredCodepoints( + std::vector{primary_script_file}, freq_list, unicodes))); + if (remaining_codepoints.empty()) { // The primary script covers everything the rest of the group does, no // additional group is needed. diff --git a/ift/config/auto_segmenter_config.h b/ift/config/auto_segmenter_config.h index b6078cc7..2964dc68 100644 --- a/ift/config/auto_segmenter_config.h +++ b/ift/config/auto_segmenter_config.h @@ -28,10 +28,14 @@ class AutoSegmenterConfig { // quality tradeoff. Lower values have shorter segmenting // times, high values have longer segmenting times but // typically results in better segmentation quality. + // + // target_mode: the target format mode for the generated segmentation config + // (IFT or UNICODE_RANGE). Defaults to IFT. static absl::StatusOr GenerateConfig( hb_face_t* face, const ift::common::DataFileResolver& resolver, std::optional primary_script = std::nullopt, - std::optional quality_level = std::nullopt); + std::optional quality_level = std::nullopt, + TargetMode target_mode = IFT); // Returns the base script for a given language. // For example, "Language_fr" -> "Script_latin". diff --git a/ift/config/auto_segmenter_config_test.cc b/ift/config/auto_segmenter_config_test.cc index 62d9d9ac..790d8b6a 100644 --- a/ift/config/auto_segmenter_config_test.cc +++ b/ift/config/auto_segmenter_config_test.cc @@ -132,6 +132,7 @@ const ScriptGroup kCyrillic = {"Cyrillic", {"Script_cyrillic.riegeli"}}; const ScriptGroup kGreek = {"Greek", {"Script_greek.riegeli"}}; const ScriptGroup kSymbols = {"Symbols", {"Script_symbols.riegeli"}}; const ScriptGroup kLatin = {"Latin", {"Script_latin.riegeli"}}; +const ScriptGroup kLatinCyrillic = {"Cyrillic+Latin", {"Script_cyrillic.riegeli", "Script_latin.riegeli"}}; const ScriptGroup kLanguageFr = {"Language_fr", {"Language_fr.riegeli"}}; const ScriptGroup kFallback = {"Fallback", {"fallback.riegeli"}}; // Most of the emoji code points in the font are ones shared with latin @@ -666,5 +667,199 @@ TEST_F(AutoSegmenterConfigTest, ConfigureMaxDepth) { } } +TEST_F(AutoSegmenterConfigTest, Roboto_UnspecifiedPrimary_UnicodeRange) { + // Since this config will specify explicit segment definitions, use a subsetted + // version of Roboto. + hb_subset_input_t* input = hb_subset_input_create_or_fail(); + ASSERT_TRUE(input); + CodepointSet subset { + 0x20, // Shared w/ All + 0x301, // Shared w/ Latin + Cyrillic + // Latin + 'A', + 'B', + 'C', + // Greek + 0x384, + 0x385, + 0x394, + // Cyrillic + 0x401, + 0x402, + 0x403, + }; + subset.union_into(hb_subset_input_unicode_set(input)); + hb_face_unique_ptr subset_face = make_hb_face(hb_subset_or_fail(face_.get(), input)); + hb_subset_input_destroy(input); + + auto config_or = AutoSegmenterConfig::GenerateConfig( + subset_face.get(), *resolver, std::nullopt, std::nullopt, UNICODE_RANGE); + ASSERT_TRUE(config_or.ok()) << config_or.status(); + + EXPECT_THAT( + GetScripts(*config_or), + UnorderedElementsAre(kLatinCyrillic, kGreek)); + EXPECT_TRUE(GetScriptsWithInitialMergeThreshold(*config_or).empty()); + + std::string config_string; + TextFormat::PrintToString(*config_or, &config_string); + // Because this is unicode range target mode, there's a few unique behaviours: + // - Ignored for overlap codepoints present in multiple groups (ie. 0x20) are moved to exactly one group. + // - This in turn requires explicit segment definitions for affected groups. + // - Combining marks are not ignored for overlap detection so latin + cyrillic are grouped. + ASSERT_EQ(config_string, R"(unmapped_glyph_handling: FIND_CONDITIONS +brotli_quality: 11 +base_heuristic_config { + min_patch_size: 2500 + max_patch_size: 20000 +} +base_cost_config { + network_overhead_cost: 200 + min_group_size: 4 + optimization_cutoff_fraction: 0.005 + experimental_use_patch_merges: false +} +ungrouped_config { + min_patch_size: 2500 + max_patch_size: 20000 +} +preprocess_merging_group_size_for_ungrouped: 12 +merge_groups { + name: "Cyrillic+Latin" + segment_ids { + values: 32 + values: 65 + values: 66 + values: 67 + values: 769 + values: 1025 + values: 1026 + values: 1027 + } + preprocess_merging_group_size: 4 + preprocess_merging_probability_threshold: 0.0005 + cost_config { + frequency_data { + built_in_freq_data_name: "Script_cyrillic.riegeli" + use_bigrams: true + } + frequency_data { + built_in_freq_data_name: "Script_latin.riegeli" + use_bigrams: true + } + } +} +merge_groups { + name: "Greek" + segment_ids { + values: 900 + values: 901 + values: 916 + } + preprocess_merging_group_size: 4 + preprocess_merging_probability_threshold: 0.0005 + cost_config { + frequency_data { + built_in_freq_data_name: "Script_greek.riegeli" + use_bigrams: true + } + } +} +condition_analysis_mode: DEP_GRAPH_ONLY +target_mode: UNICODE_RANGE +)"); +} + +TEST_F(AutoSegmenterConfigTest, UnicodeRange_QualityLevels) { + auto config_q1 = AutoSegmenterConfig::GenerateConfig( + face_.get(), *resolver, std::nullopt, 1, UNICODE_RANGE); + ASSERT_TRUE(config_q1.ok()) << config_q1.status(); + EXPECT_EQ(config_q1->target_mode(), UNICODE_RANGE); + EXPECT_EQ(config_q1->brotli_quality(), 0); + EXPECT_EQ(config_q1->unmapped_glyph_handling(), FIND_CONDITIONS); + EXPECT_FALSE(config_q1->has_brotli_quality_for_initial_font_merging()); + EXPECT_FALSE(config_q1->base_cost_config().experimental_use_patch_merges()); + EXPECT_THAT(GetUseBigrams(*config_q1), Each(false)); + EXPECT_TRUE(GetScriptsWithInitialMergeThreshold(*config_q1).empty()); + + auto config_q7 = AutoSegmenterConfig::GenerateConfig( + face_.get(), *resolver, std::nullopt, 7, UNICODE_RANGE); + ASSERT_TRUE(config_q7.ok()) << config_q7.status(); + EXPECT_EQ(config_q7->target_mode(), UNICODE_RANGE); + EXPECT_EQ(config_q7->brotli_quality(), 11); + EXPECT_EQ(config_q7->unmapped_glyph_handling(), FIND_CONDITIONS); + EXPECT_FALSE(config_q7->has_brotli_quality_for_initial_font_merging()); + EXPECT_FALSE(config_q7->base_cost_config().experimental_use_patch_merges()); + EXPECT_THAT(GetUseBigrams(*config_q7), Each(true)); + EXPECT_TRUE(GetScriptsWithInitialMergeThreshold(*config_q7).empty()); +} + +TEST_F(AutoSegmenterConfigTest, Roboto_InteractingScripts_UnicodeRange) { + hb_subset_input_t* input = hb_subset_input_create_or_fail(); + ASSERT_TRUE(input); + CodepointSet subset{ + // Latin + 'A', + 'B', + 'C', + // Greek (0x386 is a composite glyph using 'A' as a component) + 0x384, + 0x385, + 0x386, + // Cyrillic + 0x401, + 0x402, + 0x403, + }; + subset.union_into(hb_subset_input_unicode_set(input)); + hb_face_unique_ptr subset_face = + make_hb_face(hb_subset_or_fail(face_.get(), input)); + hb_subset_input_destroy(input); + + auto config_or = AutoSegmenterConfig::GenerateConfig( + subset_face.get(), *resolver, std::nullopt, std::nullopt, UNICODE_RANGE); + ASSERT_TRUE(config_or.ok()) << config_or.status(); + + const ScriptGroup kGreekLatin = { + "Greek+Latin", {"Script_greek.riegeli", "Script_latin.riegeli"}}; + EXPECT_THAT(GetScripts(*config_or), + UnorderedElementsAre(kGreekLatin, kCyrillic)); +} + +TEST_F(AutoSegmenterConfigTest, UnicodeRange_PrimaryScriptSplitting) { + ASSERT_TRUE(cjk_face_) << "NotoSansJP-Regular.ttf not found"; + + auto config_or = AutoSegmenterConfig::GenerateConfig( + cjk_face_.get(), *resolver, "Script_japanese", std::nullopt, + UNICODE_RANGE); + ASSERT_TRUE(config_or.ok()) << config_or.status(); + EXPECT_EQ(config_or->target_mode(), UNICODE_RANGE); + EXPECT_FALSE(config_or->has_base_segmentation_plan()); + + const ScriptGroup kRemainingWithoutJapanese = { + "Chinese-simplified+Chinese-traditional+Cyrillic+Emoji+Korean+Latin+" + "Symbols+Fallback", + {"Script_chinese-simplified.riegeli@*", + "Script_chinese-traditional.riegeli@*", "Script_cyrillic.riegeli", + "Script_emoji.riegeli", "Script_korean.riegeli@*", + "Script_latin.riegeli", "Script_symbols.riegeli", "fallback.riegeli"}}; + + EXPECT_THAT( + GetScripts(*config_or), + UnorderedElementsAre(kGreek, kJapanese, kRemainingWithoutJapanese)); + EXPECT_TRUE(GetScriptsWithInitialMergeThreshold(*config_or).empty()); + + auto japanese_covered = GetSegmentIds(*config_or, kJapanese.first); + ASSERT_TRUE(japanese_covered.has_value()); + EXPECT_FALSE(japanese_covered->empty()); + EXPECT_TRUE(japanese_covered->contains(0x20)); + + auto remaining_covered = + GetSegmentIds(*config_or, kRemainingWithoutJapanese.first); + ASSERT_TRUE(remaining_covered.has_value()); + EXPECT_FALSE(remaining_covered->empty()); + EXPECT_FALSE(remaining_covered->intersects(*japanese_covered)); +} + } // namespace } // namespace ift::config diff --git a/ift/config/config_compiler.cc b/ift/config/config_compiler.cc index 0c75d106..c57a9860 100644 --- a/ift/config/config_compiler.cc +++ b/ift/config/config_compiler.cc @@ -17,6 +17,7 @@ #include "ift/encoder/activation_condition.h" #include "ift/encoder/compiler.h" #include "ift/encoder/subset_definition.h" +#include "ift/encoder/unicode_range_compiler.h" using absl::flat_hash_map; using absl::Status; @@ -30,6 +31,7 @@ using ift::encoder::ActivationCondition; using ift::encoder::Compiler; using ift::encoder::design_space_t; using ift::encoder::SubsetDefinition; +using ift::encoder::UnicodeRangeCompiler; namespace ift::config { @@ -57,18 +59,8 @@ static ActivationCondition FromProto( condition.activated_patch()); } -Status ConfigCompiler::Configure(const SegmentationPlan& plan, - Compiler& compiler) { - // First configure the glyph keyed segments, including features deps - for (const auto& [id, gids] : plan.glyph_patches()) { - TRYV(compiler.AddGlyphDataPatch(id, Values(gids))); - } - - std::vector activation_conditions; - for (const auto& c : plan.glyph_patch_conditions()) { - activation_conditions.push_back(FromProto(c)); - } - +static flat_hash_map ParseSegments( + const SegmentationPlan& plan) { flat_hash_map segments; for (const auto& [id, set] : plan.segments()) { auto& segment = segments[id]; @@ -79,15 +71,12 @@ Status ConfigCompiler::Configure(const SegmentationPlan& plan, segment.feature_tags.insert(FontHelper::ToTag(tag)); } } + return segments; +} - auto condition_entries = - TRY(ActivationCondition::ActivationConditionsToPatchMapEntries( - activation_conditions, segments)); - for (const auto& entry : condition_entries) { - TRYV(compiler.AddGlyphDataPatchCondition(entry)); - } - - // Initial subset definition +static StatusOr ParseInitSubset( + const SegmentationPlan& plan, + const flat_hash_map& segments) { auto init_codepoints = Values(plan.initial_codepoints()); auto init_glyphs = Values(plan.initial_glyphs()); auto init_features = TagValues(plan.initial_features()); @@ -112,6 +101,32 @@ Status ConfigCompiler::Configure(const SegmentationPlan& plan, init_subset.feature_tags = init_features; init_subset.design_space = init_design_space; + return init_subset; +} + +Status ConfigCompiler::Configure(const SegmentationPlan& plan, + Compiler& compiler) { + // First configure the glyph keyed segments, including features deps + for (const auto& [id, gids] : plan.glyph_patches()) { + TRYV(compiler.AddGlyphDataPatch(id, Values(gids))); + } + + std::vector activation_conditions; + for (const auto& c : plan.glyph_patch_conditions()) { + activation_conditions.push_back(FromProto(c)); + } + + flat_hash_map segments = ParseSegments(plan); + + auto condition_entries = + TRY(ActivationCondition::ActivationConditionsToPatchMapEntries( + activation_conditions, segments)); + for (const auto& entry : condition_entries) { + TRYV(compiler.AddGlyphDataPatchCondition(entry)); + } + + // Initial subset definition + SubsetDefinition init_subset = TRY(ParseInitSubset(plan, segments)); TRYV(compiler.SetInitSubsetFromDef(init_subset)); // Next configure the table keyed segments @@ -168,4 +183,41 @@ Status ConfigCompiler::Configure(const SegmentationPlan& plan, return absl::OkStatus(); } +Status ConfigCompiler::Configure(const SegmentationPlan& plan, + UnicodeRangeCompiler& compiler) { + if (!plan.non_glyph_codepoint_segmentation().empty() || + !plan.non_glyph_feature_segmentation().empty() || + !plan.non_glyph_design_space_segmentation().empty() || + !plan.non_glyph_segments().empty()) { + return absl::InvalidArgumentError( + "Non-glyph (table-keyed) segmentation is not supported by " + "UnicodeRangeCompiler."); + } + + for (const auto& [id, gids] : plan.glyph_patches()) { + TRYV(compiler.AddGlyphDataPatch(id, Values(gids))); + } + + flat_hash_map segments = ParseSegments(plan); + for (const auto& [id, segment] : segments) { + TRYV(compiler.AddSegment(id, segment)); + } + + for (const auto& c : plan.glyph_patch_conditions()) { + if (c.required_segments_size() != 1 || + c.required_segments(0).values_size() != 1) { + return absl::InvalidArgumentError( + "All glyph_patch_conditions in a unicode-range plan must be " + "exclusive single-segment conditions."); + } + uint32_t segment_id = c.required_segments(0).values(0); + TRYV(compiler.AddGlyphPatchCondition(segment_id, c.activated_patch())); + } + + SubsetDefinition init_subset = TRY(ParseInitSubset(plan, segments)); + TRYV(compiler.SetInitSubsetFromDef(init_subset)); + + return absl::OkStatus(); +} + } // namespace ift::config diff --git a/ift/config/config_compiler.h b/ift/config/config_compiler.h index 54821807..99c7a08d 100644 --- a/ift/config/config_compiler.h +++ b/ift/config/config_compiler.h @@ -4,6 +4,7 @@ #include "absl/status/status.h" #include "ift/config/segmentation_plan.pb.h" #include "ift/encoder/compiler.h" +#include "ift/encoder/unicode_range_compiler.h" namespace ift::config { @@ -12,6 +13,8 @@ class ConfigCompiler { // Configures the compiler based on the provided segmentation plan. static absl::Status Configure(const SegmentationPlan& plan, encoder::Compiler& compiler); + static absl::Status Configure(const SegmentationPlan& plan, + encoder::UnicodeRangeCompiler& compiler); private: ConfigCompiler() = delete; diff --git a/ift/config/segmenter_config.proto b/ift/config/segmenter_config.proto index c6295121..d5df9900 100644 --- a/ift/config/segmenter_config.proto +++ b/ift/config/segmenter_config.proto @@ -59,6 +59,18 @@ enum TableKeyedSegmentMode { FROM_FREQ_DATA = 2; } +enum TargetMode { + // Produces a segmentation plan suitable for Incremental Font Transfer (IFT). + IFT = 0; + + // [EXPERIMENTAL] Produces a segmentation plan suitable for serving font subsets via + // @font-face unicode-range. + // + // Implementation of this is still in progress and performance of produced unicode ranges + // hasn't been verified yet. + UNICODE_RANGE = 1; +} + // This messages provides the configuration details for util/gen_ift_segmentation_plan.cc // // The gen_ift_segmentation_plan tool is used to turn a series of codepoint/feature based @@ -94,10 +106,17 @@ message SegmenterConfig { // Specifies the set of codepoints and layout feature tags that should be placed into the // initial font, and hence be always available. + // + // When target_mode is UNICODE_RANGE, there is no separate initial font; instead, any + // codepoints, glyphs, and layout features specified here (along with default layout features) + // are included in all subsets. SegmentProto initial_segment = 1; // Specifies what to do with unmapped glyphs. Unmapped glyphs are glyphs in which the analysis // is unable to determine activation conditions for. + // + // When target_mode is UNICODE_RANGE, this must be set to either FIND_CONDITIONS or PATCH + // (MOVE_TO_INIT_FONT is invalid). UnmappedGlyphHandling unmapped_glyph_handling = 2 [default = MOVE_TO_INIT_FONT]; // Deprecated: use generate_table_keyed_segments_mode instead. @@ -105,10 +124,14 @@ message SegmenterConfig { // If enabled then the generated segmentation plan will include a set of table keyed segments. // One table keyed segment will be generated per merge group, including the auto generated // ungrouped one. + // + // Invalid when target_mode is UNICODE_RANGE (must be false). bool generate_table_keyed_segments = 3 [default = false, deprecated = true]; // Configures how table keyed segments are generated in the produced // segmentation plan. + // + // When target_mode is UNICODE_RANGE, this must be set to NONE. TableKeyedSegmentMode generate_table_keyed_segments_mode = 16 [default = NONE]; // The set of segments the font is initially broken up into. The key in the map is an ID @@ -116,15 +139,22 @@ message SegmenterConfig { // // If this is not specified then a segmentation where each codepoint in the font is placed // into it's own segment will be used. In this case each segment id is the value of - // the codepoint in that segment + // the codepoint in that segment. + // + // When target_mode is UNICODE_RANGE, segments must not contain layout features. map segments = 4; // When utilizing automated segment generation ('segments' is unspecified), this provides // a list of feature groups that should be added to the automated segment list. Each group // will be one segment. These use their own id space separate from segments. + // + // Invalid when target_mode is UNICODE_RANGE (must be empty). map feature_segments = 5; - // If true and feature_segments is not specified then TODO... + // If true and feature_segments is not specified then a segment will be generated for each + // non-default layout feature in the font. + // + // Invalid when target_mode is UNICODE_RANGE (must be false). bool generate_feature_segments = 14; // When generating compressed patches (to evaluate their size) this is the brotli quality @@ -174,7 +204,20 @@ message SegmenterConfig { // for more informmation. ConditionAnalysisMode condition_analysis_mode = 15 [default = DEP_GRAPH_ONLY]; - // next = 17 + // Configures the target mode for the generated segmentation plan. + TargetMode target_mode = 17 [default = IFT]; + + // When target_mode is UNICODE_RANGE, non-exclusive activation conditions are + // made disjoint by merging their triggering segments together prior to the + // main merging phase. If a non-exclusive activation condition has a + // probability below this threshold, it will be skipped and won't cause its + // triggering segments to be merged. Any remaining non-exclusive patches are + // omitted from the final segmentation plan. + // + // Value is from [0, 1]. + double disjoint_conditions_probability_threshold = 18 [default = 0.0]; + + // next = 19 } // For a given set of segments this configures how merging will be performed. Each merge group @@ -246,6 +289,8 @@ message FrequencyDataConfig { // Init font merging is evaluated per frequency data set: infrequently used languages should not // have their high frequency characters forced into the initial font, so only the data sets which // set this will contribute segments to the initial font. + // + // Invalid when target_mode is UNICODE_RANGE (must not be set). double initial_font_merge_threshold = 4; // Segments below this probability (as computed by this data set) will not be considered for @@ -316,6 +361,8 @@ message CostConfiguration { // If the cost delta (bytes) to move a segment into the initial font is less than this value then, // it will be moved into the initial font. If this is left unset then nothing will be moved to the // init font for this group. + // + // Invalid when target_mode is UNICODE_RANGE (must not be set). double initial_font_merge_threshold = 7 [deprecated = true]; // Deprecated: use FrequencyDataConfig.initial_font_merge_probability_threshold instead. @@ -371,6 +418,8 @@ message CostConfiguration { // // Enabling patch merging will typically result in lower overall costs, but will increase the total // segmentation time due to larger number of patch comparisons made. + // + // Invalid when target_mode is UNICODE_RANGE (must be false). bool experimental_use_patch_merges = 10 [default = false]; } @@ -385,4 +434,4 @@ message HeuristicConfiguration { // Segments will be merged to try and get patch sizes between this min and max. uint32 min_patch_size = 1 [default = 0]; uint32 max_patch_size = 2 [default = 4294967295]; -} \ No newline at end of file +} diff --git a/ift/config/segmenter_config_util.cc b/ift/config/segmenter_config_util.cc index 5f028494..d9980947 100644 --- a/ift/config/segmenter_config_util.cc +++ b/ift/config/segmenter_config_util.cc @@ -354,6 +354,14 @@ SegmenterConfigUtil::ConfigToMergeGroups( const SegmenterConfig& config, const CodepointSet& font_codepoints, const btree_set& font_features, std::vector& segments) { + if (config.target_mode() == UNICODE_RANGE && + (!config.feature_segments().empty() || + config.generate_feature_segments())) { + return absl::InvalidArgumentError( + "Feature segments are not supported when target_mode is " + "UNICODE_RANGE."); + } + SubsetDefinition initial_segment = SegmentProtoToSubsetDefinition(config.initial_segment()); @@ -403,6 +411,11 @@ StatusOr SegmenterConfigUtil::RunSegmenter( hb_face_t* face, const SegmenterConfig& config) { TableKeyedSegmentMode table_keyed_mode = TRY(ResolveTableKeyedSegmentMode(config)); + if (config.target_mode() == UNICODE_RANGE && table_keyed_mode != NONE) { + return absl::InvalidArgumentError( + "Table-keyed segments are not supported when target_mode is " + "UNICODE_RANGE."); + } CodepointSet font_codepoints = FontHelper::ToCodepointsSet(face); btree_set font_features = FontHelper::GetFeatureTags(face); @@ -416,7 +429,8 @@ StatusOr SegmenterConfigUtil::RunSegmenter( ClosureGlyphSegmenter segmenter( config.brotli_quality(), config.brotli_quality_for_initial_font_merging(), config.unmapped_glyph_handling(), config.condition_analysis_mode(), - resolver_); + resolver_, config.target_mode(), + config.disjoint_conditions_probability_threshold()); GlyphSegmentation segmentation = TRY(segmenter.CodepointToGlyphSegments( face, init_segment, segments, merge_groups)); diff --git a/ift/config/segmenter_config_util_test.cc b/ift/config/segmenter_config_util_test.cc index ad352bda..6be79d24 100644 --- a/ift/config/segmenter_config_util_test.cc +++ b/ift/config/segmenter_config_util_test.cc @@ -919,4 +919,137 @@ TEST_F( EXPECT_TRUE(absl::IsInvalidArgument(result.status())) << result.status(); } +TEST_F(SegmenterConfigUtilTest, + RunSegmenter_UnicodeRange_RejectsTableKeyedSegments) { + auto loader = TestFontLoader::Default().value(); + auto face = + loader->LoadFace("ift/common/testdata/Roboto-Regular.ttf").value(); + + SegmenterConfig config; + config.set_target_mode(ift::config::UNICODE_RANGE); + config.set_unmapped_glyph_handling(ift::config::FIND_CONDITIONS); + config.set_generate_table_keyed_segments_mode(FROM_FREQ_DATA); + AddSegment(config, 1, {0x43}); + + SegmenterConfigUtil util("util/testdata/config.txtpb", resolver); + auto result = util.RunSegmenter(face.get(), config); + EXPECT_TRUE(absl::IsInvalidArgument(result.status())) << result.status(); + + config.set_generate_table_keyed_segments_mode(FROM_MERGE_GROUPS); + result = util.RunSegmenter(face.get(), config); + EXPECT_TRUE(absl::IsInvalidArgument(result.status())) << result.status(); +} + +TEST_F(SegmenterConfigUtilTest, + RunSegmenter_UnicodeRange_RejectsFeatureSegments) { + auto loader = TestFontLoader::Default().value(); + auto face = + loader->LoadFace("ift/common/testdata/Roboto-Regular.ttf").value(); + SegmenterConfigUtil util("util/testdata/config.txtpb", resolver); + + { + SegmenterConfig config; + config.set_target_mode(ift::config::UNICODE_RANGE); + config.set_unmapped_glyph_handling(ift::config::FIND_CONDITIONS); + config.set_generate_feature_segments(true); + AddSegment(config, 1, {0x43}); + + std::vector segments_out; + auto groups = + util.ConfigToMergeGroups(config, {0x43}, {}, segments_out); + EXPECT_TRUE(absl::IsInvalidArgument(groups.status())) << groups.status(); + + auto result = util.RunSegmenter(face.get(), config); + EXPECT_TRUE(absl::IsInvalidArgument(result.status())) << result.status(); + } + + { + SegmenterConfig config; + config.set_target_mode(ift::config::UNICODE_RANGE); + config.set_unmapped_glyph_handling(ift::config::FIND_CONDITIONS); + Features features; + features.add_values("smcp"); + (*config.mutable_feature_segments())[1] = features; + AddSegment(config, 1, {0x43}); + + auto result = util.RunSegmenter(face.get(), config); + EXPECT_TRUE(absl::IsInvalidArgument(result.status())) << result.status(); + } + + { + SegmenterConfig config; + config.set_target_mode(ift::config::UNICODE_RANGE); + config.set_unmapped_glyph_handling(ift::config::FIND_CONDITIONS); + AddSegment(config, 1, {0x43}); + (*config.mutable_segments())[1].mutable_features()->add_values("smcp"); + + auto result = util.RunSegmenter(face.get(), config); + EXPECT_TRUE(absl::IsInvalidArgument(result.status())) << result.status(); + } +} + +TEST_F(SegmenterConfigUtilTest, + RunSegmenter_UnicodeRange_ValidatesSegmenterConstraints) { + auto loader = TestFontLoader::Default().value(); + auto face = + loader->LoadFace("ift/common/testdata/Roboto-Regular.ttf").value(); + SegmenterConfigUtil util("util/testdata/config.txtpb", resolver); + + // Default unmapped_glyph_handling (MOVE_TO_INIT_FONT) is rejected. + { + SegmenterConfig config; + config.set_target_mode(ift::config::UNICODE_RANGE); + AddSegment(config, 1, {0x43}); + + auto result = util.RunSegmenter(face.get(), config); + EXPECT_TRUE(absl::IsInvalidArgument(result.status())) << result.status(); + } + + // Initial font merging is rejected. + { + SegmenterConfig config; + config.set_target_mode(ift::config::UNICODE_RANGE); + config.set_unmapped_glyph_handling(ift::config::FIND_CONDITIONS); + AddSegment(config, 1, {0x43}); + auto* group = config.add_merge_groups(); + AddFreqData(*group, "test_freq_data.riegeli"); + group->mutable_cost_config() + ->mutable_frequency_data(0) + ->set_initial_font_merge_threshold(-50.0); + + auto result = util.RunSegmenter(face.get(), config); + EXPECT_TRUE(absl::IsInvalidArgument(result.status())) << result.status(); + } + + // Patch merging is rejected. + { + SegmenterConfig config; + config.set_target_mode(ift::config::UNICODE_RANGE); + config.set_unmapped_glyph_handling(ift::config::FIND_CONDITIONS); + AddSegment(config, 1, {0x43}); + auto* group = config.add_merge_groups(); + AddFreqData(*group, "test_freq_data.riegeli"); + group->mutable_cost_config()->set_experimental_use_patch_merges(true); + + auto result = util.RunSegmenter(face.get(), config); + EXPECT_TRUE(absl::IsInvalidArgument(result.status())) << result.status(); + } + + // Valid UNICODE_RANGE config succeeds. + { + SegmenterConfig config; + config.set_target_mode(ift::config::UNICODE_RANGE); + config.set_unmapped_glyph_handling(ift::config::FIND_CONDITIONS); + config.set_brotli_quality(0); + config.set_brotli_quality_for_initial_font_merging(0); + AddSegment(config, 1, {0x43}); + auto* group = config.add_merge_groups(); + AddFreqData(*group, "test_freq_data.riegeli"); + + auto result = util.RunSegmenter(face.get(), config); + ASSERT_TRUE(result.ok()) << result.status(); + EXPECT_EQ(result->plan.non_glyph_segments_size(), 0); + } +} + // TODO test for feature segment auto generation. \ No newline at end of file diff --git a/ift/dep_graph/unicode_edges.cc b/ift/dep_graph/unicode_edges.cc index b445b3b2..3f070787 100644 --- a/ift/dep_graph/unicode_edges.cc +++ b/ift/dep_graph/unicode_edges.cc @@ -18,6 +18,7 @@ using absl::StatusOr; using ift::common::CodepointSet; using ift::common::DataFileResolver; using ift::common::FontHelper; +using ift::common::GlyphSet; using ift::common::hb_font_unique_ptr; using ift::common::hb_set_unique_ptr; using ift::common::make_hb_font; @@ -223,6 +224,14 @@ void UnicodeEdges::ComputeUVSEdges( } } +UnicodeEdges UnicodeEdges::ComputeCmapAndUVSEdges(hb_face_t* face) { + UnicodeEdges result; + result.unicode_to_gid = UnicodeToGid(face); + result.gid_to_unicode = FontHelper::GidToUnicodeMap(face); + ComputeUVSEdges(face, result.unicode_to_gid, result); + return result; +} + StatusOr UnicodeEdges::ComputeUnicodeDependencyEdges( hb_face_t* face, const DataFileResolver& resolver) { std::string unicode_data_path = TRY(resolver.GetUnicodeDataPath()); @@ -230,18 +239,32 @@ StatusOr UnicodeEdges::ComputeUnicodeDependencyEdges( TRY(resolver.GetDerivedNormalizationPropsPath()); CodepointSet unicodes = FontHelper::ToCodepointsSet(face); - UnicodeEdges result; + UnicodeEdges result = ComputeCmapAndUVSEdges(face); CodepointSet full_composition_exclusions; TRYV(ParseDerivedNormalizationProps(derived_props_path, full_composition_exclusions)); TRYV(ParseUnicodeData(unicode_data_path, unicodes, full_composition_exclusions, result)); - // Compute UVS edges - result.unicode_to_gid = UnicodeToGid(face); - ComputeUVSEdges(face, result.unicode_to_gid, result); - return result; } +CodepointSet UnicodeEdges::CodepointsForGlyphs(const GlyphSet& glyphs) const { + // Codepoints can map to glyphs via either standard cmap mappings, or via + // variation selectors this method captures both. + CodepointSet unicodes; + for (encoder::glyph_id_t gid : glyphs) { + auto unicode = gid_to_unicode.find(gid); + if (unicode != gid_to_unicode.end()) { + unicodes.union_set(unicode->second); + } + + auto vs_unicodes = gid_to_vs.find(gid); + if (vs_unicodes != gid_to_vs.end()) { + unicodes.union_set(vs_unicodes->second); + } + } + return unicodes; +} + } // namespace ift::dep_graph diff --git a/ift/dep_graph/unicode_edges.h b/ift/dep_graph/unicode_edges.h index 04c41137..bc90145c 100644 --- a/ift/dep_graph/unicode_edges.h +++ b/ift/dep_graph/unicode_edges.h @@ -33,11 +33,18 @@ struct UnicodeEdges { absl::flat_hash_map> variation_selector; absl::flat_hash_map unicode_to_gid; + absl::flat_hash_map + gid_to_unicode; absl::flat_hash_map gid_to_vs; static absl::StatusOr ComputeUnicodeDependencyEdges( hb_face_t* face, const ift::common::DataFileResolver& resolver); + static UnicodeEdges ComputeCmapAndUVSEdges(hb_face_t* face); + + ift::common::CodepointSet CodepointsForGlyphs( + const ift::common::GlyphSet& glyphs) const; + private: static void ComputeUVSEdges( hb_face_t* face, diff --git a/ift/encoder/BUILD b/ift/encoder/BUILD index 1b0683b7..0a1c1a03 100644 --- a/ift/encoder/BUILD +++ b/ift/encoder/BUILD @@ -8,10 +8,12 @@ cc_library( srcs = [ "closure_glyph_segmenter.cc", "compiler.cc", + "unicode_range_compiler.cc", ], hdrs = [ "closure_glyph_segmenter.h", "compiler.h", + "unicode_range_compiler.h", ], visibility = [ "//visibility:public", @@ -30,6 +32,7 @@ cc_library( "//ift/config:common_cc_proto", "//ift/config:segmentation_plan_cc_proto", "//ift/config:segmenter_config_cc_proto", + "//ift/dep_graph:unicode_edges", "//ift/feature_registry", "//ift/freq", "//ift/freq:common", @@ -43,6 +46,7 @@ cc_library( "@abseil-cpp//absl/status", "@abseil-cpp//absl/status:statusor", "@abseil-cpp//absl/strings", + "@abseil-cpp//absl/strings:str_format", "@harfbuzz", ], ) @@ -82,6 +86,7 @@ cc_library( "glyph_groupings.cc", "merger.cc", "segmentation_context.cc", + "woff2_patch_size_cache.cc", ], hdrs = [ "candidate_merge.h", @@ -95,6 +100,10 @@ cc_library( "merger.h", "patch_size_cache.h", "segmentation_context.h", + "woff2_patch_size_cache.h", + ], + visibility = [ + "//ift/config:__pkg__", ], deps = [ ":activation_condition", @@ -108,6 +117,7 @@ cc_library( "//ift/common:try", "//ift/config:segmenter_config_cc_proto", "//ift/dep_graph", + "//ift/dep_graph:unicode_edges", "//ift/feature_registry", "//ift/freq", "//ift/freq:common", @@ -136,6 +146,7 @@ cc_library( "requested_segmentation_information.h", ], visibility = [ + "//ift/config:__pkg__", "//ift/dep_graph:__pkg__", ], deps = [ @@ -273,6 +284,30 @@ cc_test( ], ) +cc_test( + name = "unicode_range_compiler_test", + size = "small", + srcs = [ + "unicode_range_compiler_test.cc", + ], + data = [ + "//ift/common:testdata", + ], + deps = [ + ":common", + ":encoder", + "//ift/common", + "//ift/common:data_file_resolver", + "//ift/common:test_font_loader", + "//ift/config:config_compiler", + "//ift/config:segmentation_plan_cc_proto", + "@abseil-cpp//absl/status", + "@googletest//:gtest", + "@googletest//:gtest_main", + "@harfbuzz", + ], +) + cc_test( name = "requested_segmentation_information_test", size = "small", @@ -536,6 +571,25 @@ cc_test( ], ) +cc_test( + name = "woff2_patch_size_cache_test", + size = "small", + srcs = ["woff2_patch_size_cache_test.cc"], + data = [ + "//ift/common:testdata", + ], + deps = [ + ":common", + ":segmentation_context", + ":segmentation_info", + "//ift/common", + "//ift/common:test_font_loader", + "@googletest//:gtest", + "@googletest//:gtest_main", + "@harfbuzz", + ], +) + cc_test( name = "entry_graph_test", size = "small", diff --git a/ift/encoder/candidate_merge.cc b/ift/encoder/candidate_merge.cc index 88aa80b9..24d2b09e 100644 --- a/ift/encoder/candidate_merge.cc +++ b/ift/encoder/candidate_merge.cc @@ -647,14 +647,19 @@ StatusOr CandidateMerge::ComputeCostDelta( struct Info { GlyphSet glyphs; uint32_t largest_patch_size = 0; - uint32_t combined_patch_size = 0; + uint32_t largest_incremental_size = 0; + uint32_t combined_incremental_size = 0; }; flat_hash_map new_conditions; new_conditions.reserve(modified_conditions.size()); segment_index_t base = *merged_segments.min(); const auto& context = merger.Context(); + const bool unicode_range_mode = + (context.GetTargetMode() == config::UNICODE_RANGE); const auto& patch_size_cache = context.patch_size_cache; + uint32_t base_overhead = + unicode_range_mode ? TRY(patch_size_cache->GetPatchSize({})) : 0; // Cost delta contributions are collected here and summed at the end rather // than accumulated as we go. modified_conditions and new_conditions are both @@ -666,21 +671,30 @@ StatusOr CandidateMerge::ComputeCostDelta( bool has_unitary = false; const uint32_t per_request_overhead = merger.Strategy().NetworkOverheadCost(); for (const auto& [condition, glyphs] : modified_conditions) { - uint32_t patch_size = TRY(patch_size_cache->GetPatchSize(*glyphs)); - double p = TRY(merger.AggregateProbability(*condition)); - double d = p * (patch_size + per_request_overhead); - deltas.push_back(-d); - current_delta -= d; - VLOG(1) << " - (" << p << " * " << (patch_size + per_request_overhead) - << ") -> " << d << " [removed patch " << condition->ToString() - << "]"; - ActivationCondition updated = condition->ReplaceSegments(base, merged_segments); if (updated.IsExclusive()) { // Clear is exclusive flag so that de-dup happens properly updated = ActivationCondition::clear_exclusive(std::move(updated)); } + if (unicode_range_mode && !updated.IsUnitary()) { + // In unicode range mode non-exclusive patches are dropped from the final + // segmentation and do not contribute to total cost. Any condition that is + // not unitary after the merge was also non-exclusive before the merge. + continue; + } + + uint32_t patch_size = TRY(patch_size_cache->GetPatchSize(*glyphs)); + if (!unicode_range_mode || condition->IsExclusive()) { + double p = TRY(merger.AggregateProbability(*condition)); + double d = p * (patch_size + per_request_overhead); + deltas.push_back(-d); + current_delta -= d; + VLOG(1) << " - (" << p << " * " << (patch_size + per_request_overhead) + << ") -> " << d << " [removed patch " << condition->ToString() + << "]"; + } + if (!best_case && updated.IsUnitary()) { has_unitary = true; } @@ -691,14 +705,24 @@ StatusOr CandidateMerge::ComputeCostDelta( info.glyphs.union_set(*glyphs); } + uint32_t incremental_size = + (patch_size > base_overhead) ? (patch_size - base_overhead) : 0; info.largest_patch_size = std::max(info.largest_patch_size, patch_size); - info.combined_patch_size += patch_size; + info.largest_incremental_size = + std::max(info.largest_incremental_size, incremental_size); + info.combined_incremental_size += incremental_size; + } + + if (unicode_range_mode && !best_case && exclusive_gids.has_value() && + !exclusive_gids->empty()) { + new_conditions.try_emplace(ActivationCondition::or_segments({base}, 0)); } double initial_magnitude_sum = -current_delta; bool fallback_changed = - !best_case && has_unitary && exclusive_gids.has_value() && + !unicode_range_mode && !best_case && has_unitary && + exclusive_gids.has_value() && context.glyph_groupings.UnmappedGlyphs().intersects(*exclusive_gids); if (fallback_changed) { GlyphSet new_fallback = context.glyph_groupings.UnmappedGlyphs(); @@ -727,7 +751,8 @@ StatusOr CandidateMerge::ComputeCostDelta( for (auto& [c, info] : new_conditions) { uint32_t size = 0; if (best_case) { - uint32_t extra = info.combined_patch_size - info.largest_patch_size; + uint32_t extra = + info.combined_incremental_size - info.largest_incremental_size; extra = std::max((uint32_t)(extra * best_case_reduction_fraction), Merger::BEST_CASE_MERGE_SIZE_DELTA); size = info.largest_patch_size + extra; @@ -749,7 +774,8 @@ StatusOr CandidateMerge::ComputeCostDelta( } size = TRY(patch_size_cache->GetPatchSize(info.glyphs)); if (merger.ShouldRecordMergedSizeReductions()) { - int32_t extra_raw = info.combined_patch_size - info.largest_patch_size; + int32_t extra_raw = + info.combined_incremental_size - info.largest_incremental_size; int32_t extra_actual = ((int32_t)size) - info.largest_patch_size; if (extra_raw != 0.0) { merger.RecordMergedSizeReduction((double)extra_actual / diff --git a/ift/encoder/candidate_merge_test.cc b/ift/encoder/candidate_merge_test.cc index 0641f340..5e5baa10 100644 --- a/ift/encoder/candidate_merge_test.cc +++ b/ift/encoder/candidate_merge_test.cc @@ -1303,4 +1303,67 @@ TEST_F(CandidateMergeTest, EXPECT_EQ(context->glyph_closure_cache->CacheMisses(), misses_before); } +TEST_F(CandidateMergeTest, AssessMerge_CostDeltas_UnicodeRangeIgnoresNonExclusivePatches) { + // s0: 'a' -> {69} + // s1: 'f' -> {74} + // s2: 'i' -> {77} + // Non-exclusive patch (s1 AND s2): 'f' + 'i' -> {444, 446} + std::vector segments = { + {{'a'}}, + {{'f'}}, + {{'i'}}, + }; + freq::UnicodeFrequencies frequencies{ + {{' ', ' '}, 100}, {{'a', 'a'}, 50}, {{'f', 'f'}, 75}, {{'i', 'i'}, 95}}; + + ClosureGlyphSegmenter segmenter(8, 8, PATCH, CLOSURE_ONLY, resolver, + ift::config::UNICODE_RANGE); + auto context = SegmentationContext::InitializeSegmentationContext( + roboto.get(), {}, segments, segmenter.unmapped_glyph_handling(), + segmenter.condition_analysis_mode(), segmenter.brotli_quality(), + segmenter.init_font_merging_brotli_quality(), resolver, + ift::config::UNICODE_RANGE); + ASSERT_TRUE(context.ok()) << context.status(); + + Merger merger = *Merger::New( + *context, + MergeStrategy::CostBased( + *ProbabilityProfile::Unigram(std::move(frequencies)), 75, 4), + all, all); + + MockPatchSizeCache* size_cache = new MockPatchSizeCache(); + size_cache->SetPatchSize({}, 100); + size_cache->SetPatchSize({69}, 200); + size_cache->SetPatchSize({74}, 300); + size_cache->SetPatchSize({77}, 250); + size_cache->SetPatchSize({444, 446}, 150); + size_cache->SetPatchSize({69, 74}, 450); + size_cache->SetPatchSize({74, 77, 444, 446}, 600); + context->patch_size_cache.reset(size_cache); + + // Case 1: Merging s0 ('a') and s1 ('f') modifies (s1 AND s2) -> (s0 AND s2), + // which remains non-exclusive. In UNICODE_RANGE mode non-exclusive patches + // are dropped, so the cost delta only reflects removing {a} and {f} and + // adding {a, f}. + double p_a_or_f = 0.50 + 0.75 - 0.50 * 0.75; + double expected_delta_a_f = -0.50 * (200 + 75) - 0.75 * (300 + 75) + + p_a_or_f * (450 + 75); + auto r_a_f = CandidateMerge::AssessSegmentMerge(merger, 0, {1}, std::nullopt); + ASSERT_TRUE(r_a_f.ok()) << r_a_f.status(); + ASSERT_TRUE(r_a_f->has_value()); + EXPECT_NEAR((*r_a_f)->CostDelta(), expected_delta_a_f, 1e-9); + + // Case 2: Merging s1 ('f') and s2 ('i') turns the previously non-exclusive + // patch {444, 446} into part of the new exclusive patch {74, 77, 444, 446}. + // Since {444, 446} was non-exclusive prior to the merge, its pre-merge cost + // is not subtracted, while the post-merge exclusive patch includes it. + double p_f_or_i = 0.75 + 0.95 - 0.75 * 0.95; + double expected_delta_f_i = -0.75 * (300 + 75) - 0.95 * (250 + 75) + + p_f_or_i * (600 + 75); + auto r_f_i = CandidateMerge::AssessSegmentMerge(merger, 1, {2}, std::nullopt); + ASSERT_TRUE(r_f_i.ok()) << r_f_i.status(); + ASSERT_TRUE(r_f_i->has_value()); + EXPECT_NEAR((*r_f_i)->CostDelta(), expected_delta_f_i, 1e-9); +} + } // namespace ift::encoder diff --git a/ift/encoder/closure_glyph_segmenter.cc b/ift/encoder/closure_glyph_segmenter.cc index d6c04c12..e574143d 100644 --- a/ift/encoder/closure_glyph_segmenter.cc +++ b/ift/encoder/closure_glyph_segmenter.cc @@ -241,24 +241,36 @@ Status ValidateIncrementalGroupings(hb_face_t* face, return absl::OkStatus(); } -static void ClassifySegments( +static size_t ClassifySegments( const std::vector& subset_definitions, const btree_map& merge_groups, + const btree_map& with_shared, bool ignore_empty, SegmentSet& ungrouped_segments, SegmentSet& shared_segments) { std::vector group_count(subset_definitions.size()); for (const auto& [segments, strategy] : merge_groups) { - for (unsigned s : segments) { + const SegmentSet& group_segments = + with_shared.contains(segments) ? with_shared.at(segments) : segments; + for (unsigned s : group_segments) { + if (ignore_empty && subset_definitions[s].Empty()) { + continue; + } group_count[s]++; } } + size_t active_segments = 0; for (unsigned s = 0; s < subset_definitions.size(); s++) { + if (ignore_empty && subset_definitions[s].Empty()) { + continue; + } + active_segments++; if (group_count[s] == 0) { ungrouped_segments.insert(s); } else if (group_count[s] > 1) { shared_segments.insert(s); } } + return active_segments; } // Computes a single representative probability for each segment which is used @@ -275,7 +287,9 @@ static void ClassifySegments( // those the largest of the per strategy averages is used. static StatusOr> ComputeSegmentProbabilities( const std::vector& subset_definitions, - const btree_map& merge_groups) { + const btree_map& merge_groups, + const btree_map& with_shared + ) { std::vector out(subset_definitions.size(), ProbabilityBound::Zero()); @@ -284,7 +298,7 @@ static StatusOr> ComputeSegmentProbabilities( segment_list.emplace_back(def); } - for (const auto& [segments, strategy] : merge_groups) { + for (const auto& [merge_group_segments, strategy] : merge_groups) { if (!strategy.UseCosts()) { continue; } @@ -296,7 +310,12 @@ static StatusOr> ComputeSegmentProbabilities( continue; } - for (segment_index_t s : segments) { + const SegmentSet& in_merge_group = + with_shared.contains(merge_group_segments) + ? with_shared.at(merge_group_segments) + : merge_group_segments; + + for (segment_index_t s : in_merge_group) { double min = 0.0; double max = 0.0; for (const auto& profile : profiles) { @@ -340,7 +359,7 @@ static std::vector PreGroupSegments( const btree_map& merge_groups, const std::vector& ordering, const std::vector& subset_definitions, - std::vector& segment_index_map) { + bool pregroup_segments, std::vector& segment_index_map) { segment_index_map.resize(subset_definitions.size()); std::vector segments; @@ -364,13 +383,17 @@ static std::vector PreGroupSegments( Segment segment = Segment{subset_definitions[o.original_index]}; ordering_it++; + if (!pregroup_segments && segment.Definition().Empty()) { + continue; + } + // Don't pregroup feature segments, these generally have broad interactions // and pre-grouping them blindly can cause poor outcomes. bool is_feature_segment = !segment.Definition().feature_tags.empty(); segment_index_map[o.original_index] = i; - if (strategy != nullptr && !is_feature_segment && + if (pregroup_segments && strategy != nullptr && !is_feature_segment && strategy->PreClosureGroupSize() > 1 && o.probability.Value() <= strategy->PreClosureProbabilityThreshold()) { uint32_t remaining = strategy->PreClosureGroupSize() - 1; @@ -409,7 +432,8 @@ static std::vector PreGroupSegments( static StatusOr> ToOrderedSegments( const std::vector& subset_definitions, btree_map& merge_groups, - btree_map& with_shared) { + btree_map& with_shared, + bool pregroup_segments = true) { // This generates the following ordering: // // merge group 1 segments @@ -425,21 +449,37 @@ static StatusOr> ToOrderedSegments( SegmentSet ungrouped_segments; SegmentSet shared_segments; - ClassifySegments(subset_definitions, merge_groups, ungrouped_segments, - shared_segments); + size_t active_segments = + ClassifySegments(subset_definitions, merge_groups, with_shared, + !pregroup_segments, ungrouped_segments, shared_segments); + + size_t grouped_codepoint_count = 0; + size_t ungrouped_codepoint_count = 0; + size_t shared_codepoint_count = 0; + for (uint32_t s = 0; s < subset_definitions.size(); s++) { + uint32_t count = subset_definitions.at(s).codepoints.size(); + if (ungrouped_segments.contains(s)) { + ungrouped_codepoint_count += count; + } else if (shared_segments.contains(s)) { + shared_codepoint_count += count; + } else { + grouped_codepoint_count += count; + } + } + - VLOG(0) << "Segment classification: " << std::endl - << " " - << subset_definitions.size() - ungrouped_segments.size() - + VLOG(0) << "Segment classification: "; + VLOG(0) << " " + << active_segments - ungrouped_segments.size() - shared_segments.size() - << " segments in exactly one merge groups" << std::endl - << " " << shared_segments.size() - << " segments that in two or more merge groups" << std::endl - << " " << ungrouped_segments.size() - << " segments that are ungrouped"; + << " segments and " << grouped_codepoint_count << " codepoints in exactly one merge groups"; + VLOG(0) << " " << shared_segments.size() + << " segments and " << shared_codepoint_count << " codepoints that in two or more merge groups"; + VLOG(0) << " " << ungrouped_segments.size() << " segments and " << ungrouped_codepoint_count << " codepoints that are ungrouped"; std::vector segment_probabilities = - TRY(ComputeSegmentProbabilities(subset_definitions, merge_groups)); + TRY(ComputeSegmentProbabilities(subset_definitions, merge_groups, with_shared)); + std::vector ordering; uint32_t group_index = 0; for (const auto& [segments, strategy] : merge_groups) { @@ -480,18 +520,27 @@ static StatusOr> ToOrderedSegments( // maps from index in subset_definitions to the new ordering. std::vector segment_index_map; - std::vector segment_defs = PreGroupSegments( - merge_groups, ordering, subset_definitions, segment_index_map); + std::vector segment_defs = + PreGroupSegments(merge_groups, ordering, subset_definitions, + pregroup_segments, segment_index_map); size_t num_segments = segment_defs.size(); - VLOG(0) << segment_defs.size() << " segments after pregrouping."; + if (pregroup_segments) { + VLOG(0) << "Segments pregrouped, " << segment_defs.size() << " remain:"; + } btree_map new_merge_groups; group_index = 0; + btree_map new_with_shared; for (auto& [segments, strategy] : merge_groups) { + const SegmentSet& group_segments = + with_shared.contains(segments) ? with_shared.at(segments) : segments; SegmentSet remapped; SegmentSet remapped_full; CodepointSet unique_codepoints; - for (segment_index_t s : segments) { + for (segment_index_t s : group_segments) { + if (!pregroup_segments && subset_definitions[s].Empty()) { + continue; + } segment_index_t s_prime = segment_index_map[s]; if (!shared_segments.contains(s)) { unique_codepoints.union_set(segment_defs.at(s_prime).Definition().codepoints); @@ -517,10 +566,11 @@ static StatusOr> ToOrderedSegments( return absl::InvalidArgumentError( "Duplicate merge groups are not allowed."); } - with_shared[remapped] = remapped_full; + new_with_shared[remapped] = remapped_full; } merge_groups = std::move(new_merge_groups); + with_shared = std::move(new_with_shared); return segment_defs; } @@ -552,22 +602,241 @@ StatusOr> ToMergers( } static StatusOr ToFinalSegmentation( - SegmentationContext& context, - UnmappedGlyphHandling unmapped_glyph_handling) { + SegmentationContext& context, Span mergers) { context.LogClosureStatistics(); - return context.ToGlyphSegmentation(); + std::vector calculators; + for (const Merger& merger : mergers) { + if (!merger.Strategy().UseCosts()) { + continue; + } + for (const auto& profile : merger.Strategy().ProbabilityProfiles()) { + calculators.push_back(TRY(profile.Calculator())); + } + } + return context.ToGlyphSegmentation(calculators); } -StatusOr ClosureGlyphSegmenter::CodepointToGlyphSegments( - hb_face_t* face, SubsetDefinition initial_segment, +Status ClosureGlyphSegmenter::ValidateInput( const std::vector& subset_definitions, - btree_map merge_groups) const { + const btree_map& merge_groups) const { + if (disjoint_conditions_probability_threshold_ < 0.0 || + disjoint_conditions_probability_threshold_ > 1.0) { + return absl::InvalidArgumentError( + "disjoint_conditions_probability_threshold must be in [0.0, 1.0]."); + } + + if (target_mode_ == ift::config::UNICODE_RANGE) { + if (unmapped_glyph_handling_ != ift::config::FIND_CONDITIONS && + unmapped_glyph_handling_ != ift::config::PATCH) { + return absl::InvalidArgumentError( + "unmapped_glyph_handling must be FIND_CONDITIONS or PATCH when " + "target_mode is UNICODE_RANGE."); + } + + for (const auto& def : subset_definitions) { + if (!def.feature_tags.empty()) { + return absl::InvalidArgumentError( + "Input subset definitions must not contain layout features when " + "target_mode is UNICODE_RANGE."); + } + } + + for (const auto& [segments, strategy] : merge_groups) { + if (strategy.UseCosts() && strategy.HasInitFontMerge()) { + return absl::InvalidArgumentError( + "Initial font merging must not be enabled when target_mode is " + "UNICODE_RANGE."); + } + if (strategy.UsePatchMerges()) { + return absl::InvalidArgumentError( + "Patch merging must be disabled when target_mode is " + "UNICODE_RANGE."); + } + } + } + for (const auto& [segments, strategy] : merge_groups) { if (strategy.UseCosts()) { TRYV(CheckForDisjointCodepoints(subset_definitions, segments)); } } + return absl::OkStatus(); +} + +static void LogMergeForDisjointConditions( + const ActivationCondition& condition, + std::optional condition_probability = std::nullopt) { + if (condition_probability.has_value()) { + VLOG(0) << " Merging segments " + << condition.TriggeringSegments().size() << " segments " + << "(probability = " << *condition_probability << ")"; + } else { + VLOG(0) << " Merging segments " + << condition.TriggeringSegments().size() << " segments"; + } +} + +static StatusOr ShouldMergeForDisjointConditions( + const SegmentationContext& context, const ActivationCondition& condition, + double probability_threshold, + const btree_map& merge_groups) { + if (condition.IsExclusive() || condition.IsFallback()) { + return false; + } + if (merge_groups.empty()) { + LogMergeForDisjointConditions(condition); + return true; + } + + const MergeStrategy* matching_strategy = nullptr; + for (const auto& [merge_group_segments, strategy] : merge_groups) { + if (condition.TriggeringSegments().is_subset_of(merge_group_segments)) { + matching_strategy = &strategy; + break; + } + } + if (matching_strategy == nullptr) { + return false; + } + + if (!matching_strategy->UseCosts() || + matching_strategy->ProbabilityProfiles().empty()) { + LogMergeForDisjointConditions(condition); + return true; + } + + double total_prob = 0.0; + for (const auto& profile : matching_strategy->ProbabilityProfiles()) { + total_prob += + TRY(condition.Probability(context.SegmentationInfo().Segments(), + *TRY(profile.Calculator()))); + } + double avg_prob = + total_prob / (double)matching_strategy->ProbabilityProfiles().size(); + if (probability_threshold <= 0.0 || avg_prob > probability_threshold) { + LogMergeForDisjointConditions(condition, avg_prob); + return true; + } + return false; +} + +static StatusOr PartitionInteractingSegments( + const SegmentationContext& context, double probability_threshold, + const btree_map& merge_groups) { + size_t num_segments = context.SegmentationInfo().Segments().size(); + for (const auto& [_, strategy] : merge_groups) { + if (strategy.UseCosts()) { + TRYV(strategy.ResetSegmentProbabilities(num_segments)); + } + } + + size_t skipped_conditions = 0; + GlyphSet skipped_glyphs; + GlyphPartition segment_partition(num_segments); + for (const auto& condition : context.glyph_groupings.OrderedConditions()) { + SegmentSet triggering = condition.TriggeringSegments(); + if (condition.IsExclusive() || condition.IsFallback() || triggering.size() <= 1) { + continue; + } + + if (TRY(segment_partition.InSameGroup(triggering))) { + // These segments are already planned to be merged. + continue; + } + + if (!TRY(ShouldMergeForDisjointConditions( + context, condition, probability_threshold, merge_groups))) { + skipped_conditions++; + skipped_glyphs.union_set( + context.glyph_groupings.ConditionsAndGlyphs().at(condition)); + continue; + } + + TRYV(segment_partition.Union(triggering)); + } + + VLOG(0) << " Skipped merging for " << skipped_conditions + << " conditions and " << skipped_glyphs.size() << " glyphs."; + return segment_partition; +} + +static Status MergeInteractingSegmentSet( + SegmentationContext& context, const SegmentSet& interacting_segments, + segment_index_t base_segment, const SubsetDefinition& merged_def) { + SegmentSet to_merge = interacting_segments; + to_merge.erase(base_segment); + + GlyphSet gid_conditions_to_update; + for (segment_index_t s : to_merge) { + gid_conditions_to_update.union_set( + context.glyph_condition_set.GlyphsWithSegment(s)); + } + + Segment merged_segment{merged_def}; + context.AssignMergedSegment(base_segment, to_merge, merged_segment, + /*is_inert=*/false); + TRYV(context.InvalidateGlyphInformationForMerge( + gid_conditions_to_update, interacting_segments, base_segment)); + return context.ReprocessChanged( + InvalidationSet(gid_conditions_to_update, to_merge, base_segment)); +} + +static Status MergeInteractingSegmentSets( + SegmentationContext& context, + Span interacting_segment_sets) { + for (const GlyphSet& raw_segment_set : interacting_segment_sets) { + SegmentSet interacting_segments; + interacting_segments.union_set(raw_segment_set); + + segment_index_t base_segment = *interacting_segments.min(); + SubsetDefinition merged_def; + for (segment_index_t s : interacting_segments) { + merged_def.Union( + context.SegmentationInfo().Segments().at(s).Definition()); + } + + TRYV(MergeInteractingSegmentSet(context, interacting_segments, base_segment, + merged_def)); + } + + return absl::OkStatus(); +} + +static Status MakeActivationConditionsDisjoint( + SegmentationContext& context, + double disjoint_conditions_probability_threshold, + btree_map& merge_groups, + btree_map& with_shared) { + + VLOG(0) << "Making activation conditions disjoint for unicode range."; + size_t num_segments = context.SegmentationInfo().Segments().size(); + if (num_segments <= 1) { + return absl::OkStatus(); + } + + GlyphPartition segment_partition = TRY(PartitionInteractingSegments( + context, disjoint_conditions_probability_threshold, merge_groups)); + auto interacting_segment_sets = TRY(segment_partition.NonIdentityGroups()); + if (!interacting_segment_sets.empty()) { + TRYV(MergeInteractingSegmentSets(context, interacting_segment_sets)); + if (!merge_groups.empty()) { + std::vector ordered_segments = TRY(ToOrderedSegments( + context.SegmentationInfo().SegmentSubsetDefinitions(), merge_groups, + with_shared, /*pregroup_segments=*/false)); + TRYV(context.ResetSegments(std::move(ordered_segments))); + } + } + + return absl::OkStatus(); +} + +StatusOr ClosureGlyphSegmenter::CodepointToGlyphSegments( + hb_face_t* face, SubsetDefinition initial_segment, + const std::vector& subset_definitions, + btree_map merge_groups) const { + TRYV(ValidateInput(subset_definitions, merge_groups)); + hb_face_unique_ptr normalized_face = TRY(FontHelper::Normalize(face)); face = normalized_face.get(); @@ -579,7 +848,11 @@ StatusOr ClosureGlyphSegmenter::CodepointToGlyphSegments( TRY(SegmentationContext::InitializeSegmentationContext( face, initial_segment, std::move(segments), unmapped_glyph_handling_, condition_analysis_mode_, brotli_quality_, - init_font_merging_brotli_quality_, resolver_)); + init_font_merging_brotli_quality_, resolver_, target_mode_)); + + if (target_mode_ == ift::config::UNICODE_RANGE) { + TRYV(MakeActivationConditionsDisjoint(context, disjoint_conditions_probability_threshold_, merge_groups, with_shared)); + } std::vector mergers = TRY(ToMergers(context, with_shared, merge_groups)); @@ -628,7 +901,10 @@ StatusOr ClosureGlyphSegmenter::CodepointToGlyphSegments( if (merge_groups.empty()) { // No merging will be needed so we're done. - return ToFinalSegmentation(context, unmapped_glyph_handling_); + if (target_mode_ == ift::config::UNICODE_RANGE) { + TRYV(ValidateIncrementalGroupings(face, context)); + } + return ToFinalSegmentation(context, mergers); } // ### Iteratively merge segments and incrementally reprocess affected data. @@ -678,7 +954,7 @@ StatusOr ClosureGlyphSegmenter::CodepointToGlyphSegments( merger.LogMergedSizeHistogram(); } - return ToFinalSegmentation(context, unmapped_glyph_handling_); + return ToFinalSegmentation(context, mergers); } TRYV(context.ReprocessChanged(std::move(*maybe_modified))); @@ -697,17 +973,31 @@ StatusOr> ClosureGlyphSegmenter::TotalCosts( non_ift.Union(def); } - double init_font_size = TRY(CandidateMerge::Woff2SizeOf( - original_face, segmentation.InitialFontSegment(), 11)); + double init_font_size = 0.0; + if (target_mode_ != ift::config::UNICODE_RANGE) { + init_font_size = TRY(CandidateMerge::Woff2SizeOf( + original_face, segmentation.InitialFontSegment(), 11)); + } double non_ift_font_size = TRY(CandidateMerge::Woff2SizeOf(original_face, non_ift, 11)); double incremental_size = non_ift_font_size / (double)non_ift.codepoints.size(); double init_font_ideal_size = - incremental_size * segmentation.InitialFontSegment().codepoints.size(); + (target_mode_ == ift::config::UNICODE_RANGE) + ? 0.0 + : incremental_size * + segmentation.InitialFontSegment().codepoints.size(); // Use highest quality so we get the true cost. - PatchSizeCacheImpl patch_sizer(original_face, 11); + std::unique_ptr patch_sizer; + if (target_mode_ == ift::config::UNICODE_RANGE) { + SubsetDefinition base_subset = segmentation.InitialFontSegment(); + base_subset.gids.union_set(segmentation.InitialFontGlyphClosure()); + patch_sizer = std::make_unique( + original_face, std::move(base_subset), 11); + } else { + patch_sizer = std::make_unique(original_face, 11); + } CodepointSet covered_codepoints; for (const ProbabilityCalculator* probability_calculator : @@ -729,7 +1019,7 @@ StatusOr> ClosureGlyphSegmenter::TotalCosts( continue; } const GlyphSet& gids = segmentation.GidSegments().at(c.activated()); - double patch_size = (double)TRY(patch_sizer.GetPatchSize(gids)); + double patch_size = (double)TRY(patch_sizer->GetPatchSize(gids)); uncovered_patch_cost += patch_size + kDefaultNetworkCost; } @@ -753,7 +1043,7 @@ StatusOr> ClosureGlyphSegmenter::TotalCosts( for (const auto& c : segmentation.Conditions()) { double Pc = TRY(c.Probability(segments, *probability_calculator)); const GlyphSet& gids = segmentation.GidSegments().at(c.activated()); - double patch_size = (double)TRY(patch_sizer.GetPatchSize(gids)); + double patch_size = (double)TRY(patch_sizer->GetPatchSize(gids)); total_cost += Pc * (patch_size + kDefaultNetworkCost); } diff --git a/ift/encoder/closure_glyph_segmenter.h b/ift/encoder/closure_glyph_segmenter.h index 0fd996e1..40314e20 100644 --- a/ift/encoder/closure_glyph_segmenter.h +++ b/ift/encoder/closure_glyph_segmenter.h @@ -42,12 +42,17 @@ class ClosureGlyphSegmenter { uint32_t brotli_quality, uint32_t init_font_merging_brotli_quality, ift::config::UnmappedGlyphHandling unmapped_glyph_handling, ift::config::ConditionAnalysisMode condition_analysis_mode, - std::shared_ptr resolver) + std::shared_ptr resolver, + ift::config::TargetMode target_mode = ift::config::IFT, + double disjoint_conditions_probability_threshold = 0.0) : brotli_quality_(brotli_quality), init_font_merging_brotli_quality_(init_font_merging_brotli_quality), unmapped_glyph_handling_(unmapped_glyph_handling), condition_analysis_mode_(condition_analysis_mode), - resolver_(std::move(resolver)) {} + resolver_(std::move(resolver)), + target_mode_(target_mode), + disjoint_conditions_probability_threshold_( + disjoint_conditions_probability_threshold) {} /* * Analyzes a set of codepoint segments using a subsetter closure and computes @@ -105,13 +110,24 @@ class ClosureGlyphSegmenter { ift::config::ConditionAnalysisMode condition_analysis_mode() const { return condition_analysis_mode_; } + ift::config::TargetMode target_mode() const { return target_mode_; } + double disjoint_conditions_probability_threshold() const { + return disjoint_conditions_probability_threshold_; + } private: + absl::Status ValidateInput( + const std::vector& subset_definitions, + const absl::btree_map& + merge_groups) const; + uint32_t brotli_quality_; uint32_t init_font_merging_brotli_quality_; ift::config::UnmappedGlyphHandling unmapped_glyph_handling_; ift::config::ConditionAnalysisMode condition_analysis_mode_; std::shared_ptr resolver_; + ift::config::TargetMode target_mode_; + double disjoint_conditions_probability_threshold_; }; } // namespace ift::encoder diff --git a/ift/encoder/closure_glyph_segmenter_test.cc b/ift/encoder/closure_glyph_segmenter_test.cc index 4f103afa..c1a44f08 100644 --- a/ift/encoder/closure_glyph_segmenter_test.cc +++ b/ift/encoder/closure_glyph_segmenter_test.cc @@ -16,6 +16,7 @@ #include "ift/encoder/merger.h" #include "ift/encoder/segmentation_context.h" #include "ift/encoder/subset_definition.h" +#include "ift/encoder/woff2_patch_size_cache.h" #include "ift/freq/bigram_probability_calculator.h" #include "ift/freq/mock_probability_calculator.h" #include "ift/freq/unicode_frequencies.h" @@ -2968,4 +2969,385 @@ TEST_F(ClosureGlyphSegmenterTest, } } +TEST_F(ClosureGlyphSegmenterTest, + UnicodeRangeValidation_RejectsMoveToInitFont) { + ClosureGlyphSegmenter seg(8, 8, MOVE_TO_INIT_FONT, CLOSURE_ONLY, resolver, + ift::config::UNICODE_RANGE); + auto result = seg.CodepointToGlyphSegments(roboto.get(), {'a'}, {{'b'}, {'c'}}); + EXPECT_TRUE(absl::IsInvalidArgument(result.status())) << result.status(); +} + +TEST_F(ClosureGlyphSegmenterTest, + UnicodeRangeValidation_RejectsFeatureSegments) { + ClosureGlyphSegmenter seg(8, 8, FIND_CONDITIONS, CLOSURE_ONLY, resolver, + ift::config::UNICODE_RANGE); + SubsetDefinition smcp; + smcp.feature_tags.insert(HB_TAG('s', 'm', 'c', 'p')); + auto result = + seg.CodepointToGlyphSegments(roboto.get(), {'a'}, {{'b'}, smcp}); + EXPECT_TRUE(absl::IsInvalidArgument(result.status())) << result.status(); +} + +TEST_F(ClosureGlyphSegmenterTest, + UnicodeRangeValidation_RejectsInitFontMerging) { + ClosureGlyphSegmenter seg(8, 8, FIND_CONDITIONS, CLOSURE_ONLY, resolver, + ift::config::UNICODE_RANGE); + MergeStrategy strategy = MergeStrategy::CostBased(ProbabilityProfile( + std::make_shared( + std::vector>{}), + -50.0)); + auto result = seg.CodepointToGlyphSegments(roboto.get(), {'a'}, + {{'b'}, {'c'}}, strategy); + EXPECT_TRUE(absl::IsInvalidArgument(result.status())) << result.status(); +} + +TEST_F(ClosureGlyphSegmenterTest, UnicodeRangeValidation_RejectsPatchMerging) { + ClosureGlyphSegmenter seg(8, 8, FIND_CONDITIONS, CLOSURE_ONLY, resolver, + ift::config::UNICODE_RANGE); + MergeStrategy strategy = MergeStrategy::CostBased( + ProbabilityProfile(std::make_shared( + std::vector>{}))); + strategy.SetUsePatchMerges(true); + auto result = seg.CodepointToGlyphSegments(roboto.get(), {'a'}, + {{'b'}, {'c'}}, strategy); + EXPECT_TRUE(absl::IsInvalidArgument(result.status())) << result.status(); +} + +TEST_F(ClosureGlyphSegmenterTest, + UnicodeRangeValidation_RejectsInvalidThreshold) { + ClosureGlyphSegmenter seg_neg(8, 8, FIND_CONDITIONS, CLOSURE_ONLY, resolver, + ift::config::UNICODE_RANGE, -0.1); + auto result = + seg_neg.CodepointToGlyphSegments(roboto.get(), {'a'}, {{'b'}, {'c'}}); + EXPECT_TRUE(absl::IsInvalidArgument(result.status())) << result.status(); + + ClosureGlyphSegmenter seg_gt1(8, 8, FIND_CONDITIONS, CLOSURE_ONLY, resolver, + ift::config::UNICODE_RANGE, 1.1); + result = + seg_gt1.CodepointToGlyphSegments(roboto.get(), {'a'}, {{'b'}, {'c'}}); + EXPECT_TRUE(absl::IsInvalidArgument(result.status())) << result.status(); +} + +TEST_F(ClosureGlyphSegmenterTest, + UnicodeRange_MakeActivationConditionsDisjoint_AndAndOrConditions) { + std::vector segments = { + {'b'}, {'f'}, {'i'}, {'c', 'd'}, {'d', 'e'}}; + + for (auto analysis_mode : + {CLOSURE_ONLY, CLOSURE_AND_VALIDATE_DEP_GRAPH, DEP_GRAPH_ONLY, + DEP_GRAPH_ONLY_WITH_SIMPLIFICATION}) { + ClosureGlyphSegmenter seg(8, 8, FIND_CONDITIONS, analysis_mode, resolver, + ift::config::UNICODE_RANGE); + auto segmentation = + seg.CodepointToGlyphSegments(roboto.get(), {'a'}, segments); + ASSERT_TRUE(segmentation.ok()) << segmentation.status(); + + // 'f' (s1) and 'i' (s2) merge due to conjunctive ligature condition. + // {'c', 'd'} (s3) and {'d', 'e'} (s4) merge due to disjunctive condition. + std::vector expected_segments = { + {'b'}, {'f', 'i'}, {}, {'c', 'd', 'e'}, {}}; + EXPECT_EQ(segmentation->Segments(), expected_segments); + + for (const auto& cond : segmentation->Conditions()) { + EXPECT_TRUE(cond.IsExclusive()) << cond.ToString(); + } + + EXPECT_EQ(segmentation->ToString(), + R"(initial font: { gid0, gid69 } +p0: { gid70 } +p1: { gid74, gid77, gid444, gid446 } +p2: { gid71, gid72, gid73 } +if (s0) then p0 +if (s1) then p1 +if (s3) then p2 +)"); + } +} + +TEST_F(ClosureGlyphSegmenterTest, + UnicodeRange_MakeActivationConditionsDisjoint_ComplexConditions) { + for (auto analysis_mode : + {CLOSURE_ONLY, CLOSURE_AND_VALIDATE_DEP_GRAPH, DEP_GRAPH_ONLY, + DEP_GRAPH_ONLY_WITH_SIMPLIFICATION}) { + ClosureGlyphSegmenter seg(8, 8, FIND_CONDITIONS, analysis_mode, resolver, + ift::config::UNICODE_RANGE); + auto segmentation = seg.CodepointToGlyphSegments( + noto_nastaliq_urdu.get(), {}, + {{0x20}, {0x62a}, {0x62b}, {0x62c}, {0x62d}}); + ASSERT_TRUE(segmentation.ok()) << segmentation.status(); + + std::vector expected_segments = { + {0x20}, {0x62a, 0x62b, 0x62c, 0x62d}, {}, {}, {}}; + EXPECT_EQ(segmentation->Segments(), expected_segments); + + for (const auto& cond : segmentation->Conditions()) { + EXPECT_TRUE(cond.IsExclusive()) << cond.ToString(); + } + EXPECT_EQ(segmentation->Conditions().size(), 2); + } +} + +TEST_F(ClosureGlyphSegmenterTest, + UnicodeRange_MakeActivationConditionsDisjoint_ProbabilityThreshold) { + // Configure probabilities: + // 'f' (0.8) and 'i' (0.7) -> P(f AND i) = 0.56 >= 0.2 (merged) + // 0xc1 (0.05) and 0x106 (0.01) -> P(0xc1 OR 0x106) = 0.0595 < 0.2 (skipped) + auto calc = std::make_shared( + std::vector>{ + {Segment({'f'}), 0.8}, + {Segment({'i'}), 0.7}, + {Segment({'f', 'i'}), 0.94}, + {Segment({0xc1}), 0.05}, + {Segment({0x106}), 0.01}, + {Segment({0xc1, 0x106}), 0.0595}, + }); + MergeStrategy strategy = + MergeStrategy::CostBased(ProbabilityProfile(calc), + /*network_overhead_cost=*/0, + /*min_group_size=*/1); + strategy.SetOptimizationCutoffFraction(1.0); + strategy.SetUsePatchMerges(false); + + ClosureGlyphSegmenter seg(8, 8, FIND_CONDITIONS, + CLOSURE_AND_VALIDATE_DEP_GRAPH, resolver, + ift::config::UNICODE_RANGE, + /*disjoint_conditions_probability_threshold=*/0.2); + + auto segmentation = seg.CodepointToGlyphSegments( + roboto.get(), {'a'}, {{'f'}, {'i'}, {0xc1}, {0x106}}, strategy); + ASSERT_TRUE(segmentation.ok()) << segmentation.status(); + + // 'f' and 'i' were merged into s0; 0xc1 (s1) and 0x106 (s2) were not merged, + // and their non-exclusive shared patch {gid117, gid169, gid667} was omitted. + std::vector expected_segments = { + {'f', 'i'}, {0xc1}, {0x106}}; + EXPECT_EQ(segmentation->Segments(), expected_segments); + + for (const auto& cond : segmentation->Conditions()) { + EXPECT_TRUE(cond.IsExclusive()) << cond.ToString(); + } + + EXPECT_EQ(segmentation->ToString(), + R"(initial font: { gid0, gid69 } +p0: { gid74, gid77, gid141, gid444, gid446, gid609, gid679 } +p1: { gid37, gid640 } +p2: { gid39, gid700 } +if (s0) then p0 +if (s1) then p1 +if (s2) then p2 +)"); +} + +TEST_F(ClosureGlyphSegmenterTest, + UnicodeRange_InitialSegmentCodepointsAndFeatures) { + ClosureGlyphSegmenter seg(8, 8, FIND_CONDITIONS, + CLOSURE_AND_VALIDATE_DEP_GRAPH, resolver, + ift::config::UNICODE_RANGE); + SubsetDefinition init; + init.codepoints = {'f'}; + init.gids = {70}; // 'b' + init.feature_tags.insert(HB_TAG('s', 'm', 'c', 'p')); + + // Since 'f' is in initial_segment (included in all subsets), 'i' can stay in + // its own segment and its exclusive patch includes the fi ligatures + smcp + // variants without needing 'i' to merge with another segment. + auto segmentation = + seg.CodepointToGlyphSegments(roboto.get(), init, {{'i'}, {'c'}}); + ASSERT_TRUE(segmentation.ok()) << segmentation.status(); + + std::vector expected_segments = {{'i'}, {'c'}}; + EXPECT_EQ(segmentation->Segments(), expected_segments); + EXPECT_TRUE(segmentation->InitialFontSegment().codepoints.contains('f')); + EXPECT_TRUE(segmentation->InitialFontSegment().gids.contains(70)); + EXPECT_TRUE(segmentation->InitialFontSegment().feature_tags.contains( + HB_TAG('s', 'm', 'c', 'p'))); + + auto plan = segmentation->ToSegmentationPlanProto(); + EXPECT_EQ(plan.glyph_patch_conditions_size(), 2); + EXPECT_GT(plan.initial_codepoints().values_size(), 0); + EXPECT_GT(plan.initial_glyphs().values_size(), 0); + EXPECT_EQ(plan.initial_features().values_size(), 1); + EXPECT_EQ(plan.initial_features().values(0), "smcp"); +} + +TEST_F(ClosureGlyphSegmenterTest, + UnicodeRange_MakeActivationConditionsDisjoint_SpansMultipleMergeGroups) { + ClosureGlyphSegmenter seg(8, 8, FIND_CONDITIONS, + CLOSURE_AND_VALIDATE_DEP_GRAPH, resolver, + ift::config::UNICODE_RANGE); + + // s0 ('f') and s1 ('i') are non-shared in Group 0 and interact via 'fi'. + // s2 ('l') and s3 (0xC1) are non-shared in Group 1; 'l' interacts with 'f' + // across Group 0 and Group 1 via 'fl'. + // s4 (0x106) is shared between Group 0 and Group 1 and interacts with 0xC1 + // via the acute accent component. + // MakeActivationConditionsDisjoint only merges within the non-shared segments + // of each group, so 'f' and 'i' merge into {'f', 'i'}, while cross-group + // ('f' + 'l') and shared-segment (0xC1 + 0x106) interactions are not merged. + std::vector segments = { + {'f'}, {'i'}, {'l'}, {0xC1}, {0x106}}; + + auto calc_0 = std::make_shared( + std::vector>{ + {Segment({'f'}), 0.5}, + {Segment({'i'}), 0.4}, + {Segment({'f', 'i'}), 0.7}, + {Segment({0x106}), 0.1}, + }); + auto calc_1 = std::make_shared( + std::vector>{ + {Segment({'l'}), 0.4}, + {Segment({0xC1}), 0.3}, + {Segment({'l', 0xC1}), 1.0}, + {Segment({0x106}), 0.1}, + }); + + MergeStrategy strategy_0 = MergeStrategy::CostBased( + ProbabilityProfile(calc_0), /*network_overhead_cost=*/0, + /*min_group_size=*/1); + strategy_0.SetUsePatchMerges(false); + + MergeStrategy strategy_1 = MergeStrategy::CostBased( + ProbabilityProfile(calc_1), /*network_overhead_cost=*/0, + /*min_group_size=*/1); + strategy_1.SetUsePatchMerges(false); + + btree_map merge_groups = { + {{0, 1, 4}, strategy_0}, + {{2, 3, 4}, strategy_1}, + }; + + auto segmentation = + seg.CodepointToGlyphSegments(roboto.get(), {'a'}, segments, merge_groups); + ASSERT_TRUE(segmentation.ok()) << segmentation.status(); + + std::vector expected_segments = { + {'f', 'i'}, {'l'}, {0xC1}, {0x106}}; + EXPECT_EQ(segmentation->Segments(), expected_segments); +} + +TEST_F(ClosureGlyphSegmenterTest, + UnicodeRange_MakeActivationConditionsDisjoint_ReordersByNewProbability) { + ClosureGlyphSegmenter seg(8, 8, FIND_CONDITIONS, + CLOSURE_AND_VALIDATE_DEP_GRAPH, resolver, + ift::config::UNICODE_RANGE); + + // Initially {'b'} (0.5) > {'f'} (0.4) > {'i'} (0.3) > {'c'} (0.1). + // After MakeActivationConditionsDisjoint merges {'f'} and {'i'} into + // {'f', 'i'} (0.65), {'f', 'i'} has the highest probability and is reordered + // to s0 ahead of {'b'} (s1) and {'c'} (s2). + auto calc = std::make_shared( + std::vector>{ + {Segment({'b'}), 0.5}, + {Segment({'f'}), 0.4}, + {Segment({'i'}), 0.3}, + {Segment({'c'}), 0.1}, + {Segment({'f', 'i'}), 0.65}, + {Segment({'b', 'f', 'i'}), 1.0}, + {Segment({'c', 'f', 'i'}), 1.0}, + {Segment({'b', 'c'}), 1.0}, + }); + + MergeStrategy strategy = MergeStrategy::CostBased( + ProbabilityProfile(calc), /*network_overhead_cost=*/0, + /*min_group_size=*/1); + strategy.SetUsePatchMerges(false); + + auto segmentation = seg.CodepointToGlyphSegments( + roboto.get(), {'a'}, {{'b'}, {'f'}, {'i'}, {'c'}}, strategy); + ASSERT_TRUE(segmentation.ok()) << segmentation.status(); + + std::vector expected_segments = { + {'f', 'i'}, {'b'}, {'c'}}; + EXPECT_EQ(segmentation->Segments(), expected_segments); +} + +TEST_F(ClosureGlyphSegmenterTest, UnicodeRange_Woff2PatchSizeCostFunction) { + // 'b' and 'c' do not interact in Roboto. With P('b') = 0.6, P('c') = 0.6, + // P({'b', 'c'}) = 0.84, and network_overhead_cost = 0: + // - In IFT mode, glyph-keyed patches only have a small patch header, so + // 0.6 * size('b') + 0.6 * size('c') < 0.84 * size({'b', 'c'}) and the + // segments remain separate. + // - In UNICODE_RANGE mode, each patch is a standalone WOFF2 font subset that + // includes shared font table overhead (~1.4KB), so merging deduplicates the + // shared table overhead and reduces total expected bytes for both + // brotli_quality > 0 (Woff2PatchSizeCache) and brotli_quality == 0 + // (EstimatedWoff2PatchSizeCache). + auto calc = std::make_shared( + std::vector>{ + {Segment({'b'}), 0.6}, + {Segment({'c'}), 0.6}, + {Segment({'b', 'c'}), 0.84}, + }); + MergeStrategy strategy = MergeStrategy::CostBased( + ProbabilityProfile(calc), /*network_overhead_cost=*/0, + /*min_group_size=*/1); + strategy.SetUsePatchMerges(false); + + { + ClosureGlyphSegmenter ift_seg(1, 1, FIND_CONDITIONS, + CLOSURE_AND_VALIDATE_DEP_GRAPH, resolver, + ift::config::IFT); + auto ift_segmentation = ift_seg.CodepointToGlyphSegments( + roboto.get(), {'a'}, {{'b'}, {'c'}}, strategy); + ASSERT_TRUE(ift_segmentation.ok()) << ift_segmentation.status(); + std::vector expected_ift = {{'b'}, {'c'}}; + EXPECT_EQ(ift_segmentation->Segments(), expected_ift); + } + + for (uint32_t quality : {0u, 1u}) { + ClosureGlyphSegmenter ur_seg(quality, quality, FIND_CONDITIONS, + CLOSURE_AND_VALIDATE_DEP_GRAPH, resolver, + ift::config::UNICODE_RANGE); + auto ur_segmentation = ur_seg.CodepointToGlyphSegments( + roboto.get(), {'a'}, {{'b'}, {'c'}}, strategy); + ASSERT_TRUE(ur_segmentation.ok()) << ur_segmentation.status(); + std::vector expected_ur = {{'b', 'c'}, {}}; + EXPECT_EQ(ur_segmentation->Segments(), expected_ur) + << "quality=" << quality; + } +} + +TEST_F(ClosureGlyphSegmenterTest, UnicodeRange_TotalCosts) { + UnicodeFrequencies frequencies{ + {{' ', ' '}, 100}, {{'a', 'a'}, 95}, {{'b', 'b'}, 50}, {{'c', 'c'}, 25}, + }; + UnigramProbabilityCalculator calculator(std::move(frequencies)); + + GlyphSegmentation segmentation({'a'}, {}, {}); + auto sc = GlyphSegmentation::ConditionsToSegmentation( + { + {ActivationCondition::exclusive_segment(0, 0), {70}}, + {ActivationCondition::exclusive_segment(1, 0), {71}}, + }, + {}, segmentation); + ASSERT_TRUE(sc.ok()) << sc; + segmentation.CopySegments({{'b'}, {'c'}}); + + ClosureGlyphSegmenter ift_seg(8, 8, FIND_CONDITIONS, CLOSURE_ONLY, resolver, + ift::config::IFT); + std::vector ift_costs = + *ift_seg.TotalCosts(roboto.get(), segmentation, {&calculator}); + + ClosureGlyphSegmenter ur_seg(8, 8, FIND_CONDITIONS, CLOSURE_ONLY, resolver, + ift::config::UNICODE_RANGE); + std::vector ur_costs = + *ur_seg.TotalCosts(roboto.get(), segmentation, {&calculator}); + + EXPECT_EQ(ur_costs[0].ift_init_cost, 0.0); + EXPECT_EQ(ur_costs[0].ideal_init_cost, 0.0); + EXPECT_GT(ur_costs[0].ift_patch_cost, ift_costs[0].ift_patch_cost); + + SubsetDefinition base_subset = segmentation.InitialFontSegment(); + base_subset.gids.union_set(segmentation.InitialFontGlyphClosure()); + Woff2PatchSizeCache woff2_sizer(roboto.get(), base_subset, 11); + double size_0 = (double)*woff2_sizer.GetPatchSize({70}); + double size_1 = (double)*woff2_sizer.GetPatchSize({71}); + double p_b = calculator.ComputeProbability('b').Value(); + double p_c = calculator.ComputeProbability('c').Value(); + double expected_patch_cost = p_b * (size_0 + kDefaultNetworkCost) + + p_c * (size_1 + kDefaultNetworkCost); + EXPECT_NEAR(ur_costs[0].ift_patch_cost, expected_patch_cost, 1e-6); +} + } // namespace ift::encoder \ No newline at end of file diff --git a/ift/encoder/dependency_closure.cc b/ift/encoder/dependency_closure.cc index b0e17962..18fdf65f 100644 --- a/ift/encoder/dependency_closure.cc +++ b/ift/encoder/dependency_closure.cc @@ -87,6 +87,17 @@ Status DependencyClosure::SegmentsMerged(segment_index_t base_segment, return absl::OkStatus(); } +Status DependencyClosure::SegmentsReset() { + VLOG(1) << "DependencyClosure::SegmentsReset()"; + incoming_edges_cache_[0].reset(); + init_font_nodes_.clear(); + node_condition_cache_.clear(); + for (auto& cache : phase_node_condition_cache_) { + cache.clear(); + } + return InitNodeConditionsCache(SegmentSet::all()); +} + Status DependencyClosure::InitNodeConditionsCache( const common::SegmentSet& changed_segments) { TRYV(UpdateAllNodeConditions(changed_segments)); diff --git a/ift/encoder/dependency_closure.h b/ift/encoder/dependency_closure.h index 97a26bd0..1cfe27c7 100644 --- a/ift/encoder/dependency_closure.h +++ b/ift/encoder/dependency_closure.h @@ -119,6 +119,8 @@ class DependencyClosure { absl::Status SegmentsMerged(segment_index_t base_segment, const ift::common::SegmentSet& segments); + absl::Status SegmentsReset(); + // The collection of SegmentsThatInteractWith(...) methods are used to // locate segments that have interactions with unicodes/features/glyphs. // diff --git a/ift/encoder/glyph_closure_cache.cc b/ift/encoder/glyph_closure_cache.cc index 885f68a3..1c33683e 100644 --- a/ift/encoder/glyph_closure_cache.cc +++ b/ift/encoder/glyph_closure_cache.cc @@ -41,7 +41,6 @@ GlyphClosureCache::GlyphClosureCache(hb_face_t* original_face, dep_graph::UnicodeEdges unicode_edges) : original_face_(make_hb_face(hb_face_reference(original_face))), preprocessed_face_(std::move(preprocessed_face)), - gid_to_unicode_(FontHelper::GidToUnicodeMap(preprocessed_face_.get())), unicode_edges_(std::move(unicode_edges)) {} StatusOr GlyphClosureCache::SegmentClosure( @@ -249,21 +248,7 @@ Status GlyphClosureCache::AnalyzeSegment( CodepointSet GlyphClosureCache::CodepointsForGlyphs( const GlyphSet& glyphs) const { - // Codepoints can map to glyphs via either standard cmap mappings, or via - // variation selectors this method captures both. - CodepointSet unicodes; - for (glyph_id_t gid : glyphs) { - auto unicode = gid_to_unicode_.find(gid); - if (unicode != gid_to_unicode_.end()) { - unicodes.union_set(unicode->second); - } - - auto vs_unicodes = unicode_edges_.gid_to_vs.find(gid); - if (vs_unicodes != unicode_edges_.gid_to_vs.end()) { - unicodes.union_set(vs_unicodes->second); - } - } - return unicodes; + return unicode_edges_.CodepointsForGlyphs(glyphs); } StatusOr GlyphClosureCache::ExpandClosure( diff --git a/ift/encoder/glyph_closure_cache.h b/ift/encoder/glyph_closure_cache.h index a7ef1cdb..070aefa8 100644 --- a/ift/encoder/glyph_closure_cache.h +++ b/ift/encoder/glyph_closure_cache.h @@ -76,7 +76,6 @@ class GlyphClosureCache { glyph_closure_cache_; uint64_t glyph_closure_cache_hit_ = 0; uint64_t glyph_closure_cache_miss_ = 0; - absl::flat_hash_map gid_to_unicode_; dep_graph::UnicodeEdges unicode_edges_; }; diff --git a/ift/encoder/glyph_groupings.cc b/ift/encoder/glyph_groupings.cc index 5dd9e21b..9a4e0a96 100644 --- a/ift/encoder/glyph_groupings.cc +++ b/ift/encoder/glyph_groupings.cc @@ -123,36 +123,25 @@ Status GlyphGroupings::AddGlyphsToExclusiveGroup( // Converts this grouping into a finalized GlyphSegmentation. StatusOr GlyphGroupings::ToGlyphSegmentation( - const RequestedSegmentationInformation& segmentation_info) const { + const RequestedSegmentationInformation& segmentation_info, + bool exclusive_only) const { GlyphSegmentation segmentation( segmentation_info.InitFontSegmentWithoutDefaults(), segmentation_info.InitFontGlyphs(), unmapped_glyphs_); segmentation.CopySegments(segmentation_info.SegmentSubsetDefinitions()); - // Recreate the glyph groups based on ConditionsAndGlyphs() which reflects - // the final state (including patch combinations). - btree_map and_glyph_groups; - btree_map or_glyph_groups; - btree_map exclusive_glyph_groups; - for (const auto& [condition, glyphs] : ConditionsAndGlyphs()) { - if (condition.IsExclusive()) { - exclusive_glyph_groups[*condition.TriggeringSegments().begin()] = glyphs; - } else if (condition.conditions().size() == 1) { - SegmentSet segments = condition.TriggeringSegments(); - or_glyph_groups[segments] = glyphs; - } else { - SegmentSet segments = condition.TriggeringSegments(); - and_glyph_groups[segments] = glyphs; - } - } - // The fallback patch isn't stored in ConditionAndGlyphs() so add it in // manually. SegmentSet fallback_segments; btree_map conditions_with_fallback; - conditions_with_fallback.insert(ConditionsAndGlyphs().begin(), - ConditionsAndGlyphs().end()); - if (!unmapped_glyphs_.empty()) { + for (const auto& [condition, glyphs] : ConditionsAndGlyphs()) { + if (exclusive_only && !condition.IsExclusive()) { + continue; + } + conditions_with_fallback.insert({condition, glyphs}); + } + + if (!exclusive_only && !unmapped_glyphs_.empty()) { fallback_segments = segmentation_info.NonEmptySegments(); ActivationCondition non_fallback_cond = diff --git a/ift/encoder/glyph_groupings.h b/ift/encoder/glyph_groupings.h index 9cb7d154..4019b497 100644 --- a/ift/encoder/glyph_groupings.h +++ b/ift/encoder/glyph_groupings.h @@ -150,7 +150,8 @@ class GlyphGroupings { // Converts this grouping into a finalized GlyphSegmentation. absl::StatusOr ToGlyphSegmentation( - const RequestedSegmentationInformation& segmentation_info) const; + const RequestedSegmentationInformation& segmentation_info, + bool exclusive_only = false) const; std::optional GlyphToCondition(glyph_id_t gid) const { auto it = conditions_and_glyphs_.GlyphToCondition().find(gid); diff --git a/ift/encoder/glyph_partition.cc b/ift/encoder/glyph_partition.cc index d92e068e..650abae0 100644 --- a/ift/encoder/glyph_partition.cc +++ b/ift/encoder/glyph_partition.cc @@ -14,6 +14,7 @@ using absl::Span; using absl::Status; using absl::StatusOr; using ift::common::GlyphSet; +using ift::common::IntSet; namespace ift::encoder { @@ -41,7 +42,7 @@ GlyphPartition& GlyphPartition::operator=(const GlyphPartition& other) { return *this; } -Status GlyphPartition::Union(const GlyphSet& glyphs) { +Status GlyphPartition::Union(const IntSet& glyphs) { if (glyphs.empty()) { return absl::OkStatus(); } @@ -122,6 +123,23 @@ StatusOr GlyphPartition::Find(glyph_id_t glyph) const { return parent_[glyph]; } +StatusOr GlyphPartition::InSameGroup(const IntSet& glyphs) const { + if (glyphs.empty()) { + return true; + } + + auto it = glyphs.begin(); + glyph_id_t first_rep = TRY(Find(*it)); + ++it; + + for (; it != glyphs.end(); ++it) { + if (TRY(Find(*it)) != first_rep) { + return false; + } + } + return true; +} + StatusOr GlyphPartition::GlyphsFor(glyph_id_t glyph) const { if (glyph >= parent_.size()) { return absl::InvalidArgumentError( diff --git a/ift/encoder/glyph_partition.h b/ift/encoder/glyph_partition.h index e2870041..663d8b80 100644 --- a/ift/encoder/glyph_partition.h +++ b/ift/encoder/glyph_partition.h @@ -24,7 +24,7 @@ class GlyphPartition { GlyphPartition& operator=(const GlyphPartition& other); // Merge all of the sets that intersect glyphs into a single set. - absl::Status Union(const ift::common::GlyphSet& glyphs); + absl::Status Union(const ift::common::IntSet& glyphs); // Merge the sets containing glyph1 and glyph2 together. absl::Status Union(glyph_id_t glyph1, glyph_id_t glyph2); @@ -35,6 +35,9 @@ class GlyphPartition { // Returns a representative member of the group that glyph belongs too. absl::StatusOr Find(glyph_id_t glyph) const; + // Returns true if all of the given glyphs belong to the same group. + absl::StatusOr InSameGroup(const ift::common::IntSet& glyphs) const; + // Returns all members of the group the glyph belongs too. absl::StatusOr GlyphsFor( glyph_id_t glyph) const; diff --git a/ift/encoder/glyph_partition_test.cc b/ift/encoder/glyph_partition_test.cc index aa4c0938..54c674e2 100644 --- a/ift/encoder/glyph_partition_test.cc +++ b/ift/encoder/glyph_partition_test.cc @@ -237,4 +237,21 @@ TEST_F(GlyphPartitionTest, UnionOtherUnion_Invalid) { ASSERT_TRUE(absl::IsInvalidArgument(gu2.Union(gu1))); } +TEST_F(GlyphPartitionTest, InSameGroup) { + GlyphPartition gu(10); + + ASSERT_TRUE(*gu.InSameGroup({})); + ASSERT_TRUE(*gu.InSameGroup({1})); + ASSERT_FALSE(*gu.InSameGroup({1, 3})); + + ASSERT_TRUE(gu.Union({1, 3, 5}).ok()); + ASSERT_TRUE(*gu.InSameGroup({1, 3})); + ASSERT_TRUE(*gu.InSameGroup({1, 3, 5})); + ASSERT_FALSE(*gu.InSameGroup({1, 3, 4})); + ASSERT_FALSE(*gu.InSameGroup({2, 4})); + + ASSERT_FALSE(gu.InSameGroup({10}).ok()); + ASSERT_FALSE(gu.InSameGroup({1, 3, 10}).ok()); +} + } // namespace ift::encoder \ No newline at end of file diff --git a/ift/encoder/merger.cc b/ift/encoder/merger.cc index bdd3b9e9..19df4c20 100644 --- a/ift/encoder/merger.cc +++ b/ift/encoder/merger.cc @@ -501,8 +501,14 @@ StatusOr> Merger::MergeSegmentWithCosts( // way of invalidating any pairs that are changed by each merge operation. TRYV(CollectExclusiveCandidateMerges(base_segment_index, smallest_candidate_merge)); - TRYV(CollectCompositeCandidateMerges(base_segment_index, - smallest_candidate_merge)); + + if (context_->GetTargetMode() != config::UNICODE_RANGE) { + // Under unicode range composite conditions get dropped, so no benefit + // to trying to merge around them. + TRYV(CollectCompositeCandidateMerges(base_segment_index, + smallest_candidate_merge)); + } + if (!smallest_candidate_merge.has_value()) { return std::nullopt; } @@ -788,10 +794,15 @@ StatusOr> Merger::MergeSegmentWithHeuristic( return std::nullopt; } - auto modified = TRY(TryMergingACompositeCondition(base_segment_index)); - if (modified.has_value()) { - // Return to the parent method so it can reanalyze and reform groups - return *modified; + std::optional modified = std::nullopt; + if (context_->GetTargetMode() != config::UNICODE_RANGE) { + // Under unicode range composite conditions get dropped, so no benefit + // to trying to merge around them. + modified = TRY(TryMergingACompositeCondition(base_segment_index)); + if (modified.has_value()) { + // Return to the parent method so it can reanalyze and reform groups + return *modified; + } } modified = TRY(TryMergingABaseSegment(base_segment_index)); diff --git a/ift/encoder/requested_segmentation_information.cc b/ift/encoder/requested_segmentation_information.cc index b4489483..b33b3d74 100644 --- a/ift/encoder/requested_segmentation_information.cc +++ b/ift/encoder/requested_segmentation_information.cc @@ -64,6 +64,20 @@ RequestedSegmentationInformation::RequestedSegmentationInformation( } } +void RequestedSegmentationInformation::ResetSegments( + std::vector segments) { + segments_ = std::move(segments); + codepoint_to_segments_.clear(); + for (size_t i = 0; i < segments_.size(); ++i) { + auto& s = segments_[i]; + if (s.Definition().codepoints.intersects(init_font_segment_.codepoints)) { + s.Definition().codepoints.subtract(init_font_segment_.codepoints); + } + AddToIndex(i, s.Definition().codepoints); + } + segments_disjoint_ = CheckSegmentsAreDisjoint(init_font_segment_, segments_); +} + SegmentSet RequestedSegmentationInformation::SegmentsForCodepoints( const CodepointSet& codepoints) const { SegmentSet result; diff --git a/ift/encoder/requested_segmentation_information.h b/ift/encoder/requested_segmentation_information.h index 1a7fcfae..7bb5447c 100644 --- a/ift/encoder/requested_segmentation_information.h +++ b/ift/encoder/requested_segmentation_information.h @@ -50,6 +50,8 @@ class RequestedSegmentationInformation { return base_segment.Definition().codepoints.size(); } + void ResetSegments(std::vector segments); + absl::Status ReassignInitSubset(GlyphClosureCache& closure_cache, const SubsetDefinition& new_def) { // TODO XXXXX Can we avoid aggressively expanding the init closure? diff --git a/ift/encoder/segmentation_context.cc b/ift/encoder/segmentation_context.cc index a291e8d8..ef8b7e72 100644 --- a/ift/encoder/segmentation_context.cc +++ b/ift/encoder/segmentation_context.cc @@ -29,10 +29,12 @@ using ift::common::SegmentSet; namespace ift::encoder { Status SegmentationContext::ValidateSegmentation( - const GlyphSegmentation& segmentation) const { + const GlyphSegmentation& segmentation, + absl::Span + probability_calculators) const { GlyphSet visited; const auto& initial_closure = segmentation.InitialFontGlyphClosure(); - for (const auto& [id, gids] : segmentation.GidSegments()) { + auto check_disjoint = [&](const GlyphSet& gids) -> Status { for (glyph_id_t gid : gids) { if (initial_closure.contains(gid)) { return absl::FailedPreconditionError(absl::StrCat( @@ -44,6 +46,53 @@ Status SegmentationContext::ValidateSegmentation( } visited.insert(gid); } + return absl::OkStatus(); + }; + + for (const auto& [id, gids] : segmentation.GidSegments()) { + TRYV(check_disjoint(gids)); + } + + if (target_mode_ == config::UNICODE_RANGE) { + for (const auto& condition : segmentation.Conditions()) { + if (!condition.IsExclusive()) { + return absl::FailedPreconditionError(absl::StrCat( + "Non-exclusive condition found in UNICODE_RANGE mode: ", + condition.ToString())); + } + } + + for (const freq::ProbabilityCalculator* calculator : + probability_calculators) { + calculator->ResetSegmentProbabilities( + segmentation_info_->Segments().size()); + } + + // In unicode range mode not all conditions get record into segmentation, so + // check disjointness across the non-exclusive patches which were skipped. + TRYV(check_disjoint(glyph_groupings.UnmappedGlyphs())); + size_t unreachable_glyph_count = glyph_groupings.UnmappedGlyphs().size(); + for (const auto& cond : glyph_groupings.OrderedConditions()) { + if (cond.IsExclusive()) { + continue; + } + const GlyphSet& gids = glyph_groupings.ConditionsAndGlyphs().at(cond); + TRYV(check_disjoint(gids)); + unreachable_glyph_count += gids.size(); + + double max_prob = 0.0; + for (const freq::ProbabilityCalculator* calculator : + probability_calculators) { + double prob = + TRY(cond.Probability(segmentation_info_->Segments(), *calculator)); + if (prob > max_prob) { + max_prob = prob; + } + } + VLOG(0) << "Unreachable condition " << cond.ToString() << " => " + << gids.ToString() << " (max probability = " << max_prob << ")"; + } + VLOG(0) << "Total unreachable glyphs: " << unreachable_glyph_count; } GlyphSet full_minus_initial = segmentation_info_->FullClosure(); @@ -202,6 +251,14 @@ Status SegmentationContext::ReassignInitSubset(SubsetDefinition new_def) { TRYV(segmentation_info_->ReassignInitSubset(*glyph_closure_cache, new_def)); + if (target_mode_ == config::UNICODE_RANGE) { + estimated_compression_ratio_ = std::nullopt; + estimated_woff2_base_overhead_ = std::nullopt; + patch_size_cache = NewPatchSizeCache(original_face.get(), brotli_quality_); + patch_size_cache_for_init_font = NewPatchSizeCache( + original_face.get(), init_font_brotli_quality_); + } + if (dependency_closure_.has_value()) { // Dep closure doesn't need the expanded affected set, as it will traverse // from the input segments to find the affected subgraph. @@ -245,6 +302,20 @@ Status SegmentationContext::ReassignInitSubset(SubsetDefinition new_def) { return absl::OkStatus(); } +Status SegmentationContext::ResetSegments(std::vector segments) { + segmentation_info_->ResetSegments(std::move(segments)); + if (dependency_closure_.has_value()) { + TRYV((*dependency_closure_)->SegmentsReset()); + } + uint32_t glyph_count = hb_face_get_glyph_count(original_face.get()); + glyph_condition_set = GlyphConditionSet(glyph_count); + glyph_groupings = GlyphGroupings( + glyph_count, + condition_analysis_mode_ == config::DEP_GRAPH_ONLY_WITH_SIMPLIFICATION); + inert_segments_.clear(); + return ReprocessAll(); +} + static void PrintDiff(absl::string_view set_name, const GlyphSet& closure, const GlyphSet& dep) { std::string op = " == "; @@ -326,7 +397,8 @@ SegmentationContext::InitializeSegmentationContext( UnmappedGlyphHandling unmapped_glyph_handling, ConditionAnalysisMode condition_analysis_mode, uint32_t brotli_quality, uint32_t init_font_brotli_quality, - std::shared_ptr resolver) { + std::shared_ptr resolver, + ift::config::TargetMode target_mode) { if (!hb_face_get_glyph_count(face)) { return absl::InvalidArgumentError("Provided font has no glyphs."); } @@ -339,7 +411,7 @@ SegmentationContext::InitializeSegmentationContext( SegmentationContext context = TRY(SegmentationContext::Create( face, initial_segment, segments, unmapped_glyph_handling, condition_analysis_mode, brotli_quality, init_font_brotli_quality, - std::move(resolver))); + std::move(resolver), target_mode)); // ### Generate the initial conditions and groupings by processing all // segments and glyphs. ### diff --git a/ift/encoder/segmentation_context.h b/ift/encoder/segmentation_context.h index 98898afe..af965c5b 100644 --- a/ift/encoder/segmentation_context.h +++ b/ift/encoder/segmentation_context.h @@ -24,6 +24,8 @@ #include "ift/encoder/segment.h" #include "ift/encoder/subset_definition.h" #include "ift/encoder/types.h" +#include "ift/encoder/woff2_patch_size_cache.h" +#include "ift/freq/probability_calculator.h" namespace ift::encoder { @@ -52,7 +54,8 @@ class SegmentationContext { ift::config::UnmappedGlyphHandling unmapped_glyph_handling, ift::config::ConditionAnalysisMode condition_analysis_mode, uint32_t brotli_quality, uint32_t init_font_brotli_quality, - std::shared_ptr resolver) { + std::shared_ptr resolver, + ift::config::TargetMode target_mode = ift::config::IFT) { // TODO(garretrieger): argument list is getting long, switch to a builder // pattern for construction. @@ -66,7 +69,7 @@ class SegmentationContext { SegmentationContext context( face, unmapped_glyph_handling, condition_analysis_mode, brotli_quality, init_font_brotli_quality, std::move(closure_cache), - std::move(segmentation_info), std::move(resolver)); + std::move(segmentation_info), std::move(resolver), target_mode); TRYV(context.InitDependencyClosure()); return context; @@ -85,7 +88,8 @@ class SegmentationContext { original_face.get(), SegmentationInfo().GetUnmappedGlyphHandling(), condition_analysis_mode_, brotli_quality_, init_font_brotli_quality_, std::move(closure_cache), std::move(segmentation_info), resolver_, - estimated_compression_ratio_); + target_mode_, estimated_compression_ratio_, + estimated_woff2_base_overhead_); TRYV(context.InitDependencyClosure()); return std::move(context); @@ -103,7 +107,8 @@ class SegmentationContext { ift::config::UnmappedGlyphHandling unmapped_glyph_handling, ift::config::ConditionAnalysisMode condition_analysis_mode, uint32_t brotli_quality, uint32_t init_font_brotli_quality, - std::shared_ptr resolver); + std::shared_ptr resolver, + ift::config::TargetMode target_mode = ift::config::IFT); private: SegmentationContext( @@ -114,11 +119,11 @@ class SegmentationContext { std::unique_ptr closure_cache, std::unique_ptr segmentation_info, std::shared_ptr resolver, - std::optional estimated_compression_ratio = std::nullopt) + ift::config::TargetMode target_mode = ift::config::IFT, + std::optional estimated_compression_ratio = std::nullopt, + std::optional estimated_woff2_base_overhead = std::nullopt) : estimated_compression_ratio_(estimated_compression_ratio), - patch_size_cache(NewPatchSizeCache(face, brotli_quality)), - patch_size_cache_for_init_font( - NewPatchSizeCache(face, init_font_brotli_quality)), + estimated_woff2_base_overhead_(estimated_woff2_base_overhead), glyph_closure_cache(std::move(closure_cache)), original_face(ift::common::make_hb_face(hb_face_reference(face))), segmentation_info_(std::move(segmentation_info)), @@ -130,7 +135,12 @@ class SegmentationContext { brotli_quality_(brotli_quality), init_font_brotli_quality_(init_font_brotli_quality), condition_analysis_mode_(condition_analysis_mode), - resolver_(std::move(resolver)) {} + target_mode_(target_mode), + resolver_(std::move(resolver)) { + patch_size_cache = NewPatchSizeCache(face, brotli_quality); + patch_size_cache_for_init_font = + NewPatchSizeCache(face, init_font_brotli_quality); + } absl::Status InitDependencyClosure() { if (UsingDepGraph()) { @@ -144,12 +154,16 @@ class SegmentationContext { public: unsigned BrotliQuality() const { return brotli_quality_; } + ift::config::TargetMode GetTargetMode() const { return target_mode_; } + // Convert the information in this context into a finalized GlyphSegmentation // representation. - absl::StatusOr ToGlyphSegmentation() const { - GlyphSegmentation segmentation = - TRY(glyph_groupings.ToGlyphSegmentation(*segmentation_info_)); - TRYV(ValidateSegmentation(segmentation)); + absl::StatusOr ToGlyphSegmentation( + absl::Span + probability_calculators = {}) const { + GlyphSegmentation segmentation = TRY(glyph_groupings.ToGlyphSegmentation( + *segmentation_info_, target_mode_ == config::UNICODE_RANGE)); + TRYV(ValidateSegmentation(segmentation, probability_calculators)); return segmentation; } @@ -265,6 +279,12 @@ class SegmentationContext { */ absl::Status ReassignInitSubset(SubsetDefinition new_def); + /* + * Replaces the segment list, invalidates all condition and grouping + * information, and fully reprocesses all segments. + */ + absl::Status ResetSegments(std::vector segments); + // Regenerates glyphs conditions and groupings for the set of things specified // in invalidation. // @@ -313,7 +333,9 @@ class SegmentationContext { * - Fully covers the full closure. */ absl::Status ValidateSegmentation( - const GlyphSegmentation& segmentation) const; + const GlyphSegmentation& segmentation, + absl::Span + probability_calculators = {}) const; // If the merging strategy specifies a minimum cost threshold, this computes // the segment where we should stop computing optimized merges. This is @@ -324,6 +346,30 @@ class SegmentationContext { // TODO XXXX make this return StatusOr<...> std::unique_ptr NewPatchSizeCache(hb_face_t* face, uint32_t brotli_quality) { + if (target_mode_ == config::UNICODE_RANGE) { + SubsetDefinition base_subset = segmentation_info_->InitFontSegment(); + base_subset.gids.union_set(segmentation_info_->InitFontGlyphs()); + if (brotli_quality == 0) { + if (!estimated_compression_ratio_.has_value() || + !estimated_woff2_base_overhead_.has_value()) { + auto params = EstimatedWoff2PatchSizeCache::EstimateParameters( + face, base_subset); + if (params.ok()) { + estimated_woff2_base_overhead_ = params->first; + estimated_compression_ratio_ = params->second; + } + } + if (estimated_compression_ratio_.has_value() && + estimated_woff2_base_overhead_.has_value()) { + return EstimatedWoff2PatchSizeCache::New( + face, *estimated_woff2_base_overhead_, + *estimated_compression_ratio_); + } + } + return std::make_unique( + face, std::move(base_subset), brotli_quality); + } + if (brotli_quality == 0) { if (!estimated_compression_ratio_.has_value()) { auto r = EstimatedPatchSizeCache::EstimateCompressionRatio(face); @@ -343,6 +389,7 @@ class SegmentationContext { private: std::optional estimated_compression_ratio_; + std::optional estimated_woff2_base_overhead_; public: // Caches and logging @@ -373,6 +420,7 @@ class SegmentationContext { unsigned init_font_brotli_quality_; ift::config::ConditionAnalysisMode condition_analysis_mode_; + ift::config::TargetMode target_mode_; std::shared_ptr resolver_; }; diff --git a/ift/encoder/unicode_range_compiler.cc b/ift/encoder/unicode_range_compiler.cc new file mode 100644 index 00000000..8e7f84ca --- /dev/null +++ b/ift/encoder/unicode_range_compiler.cc @@ -0,0 +1,234 @@ +#include "ift/encoder/unicode_range_compiler.h" + +#include +#include +#include +#include +#include + +#include "absl/status/status.h" +#include "absl/status/statusor.h" +#include "absl/strings/str_cat.h" +#include "absl/strings/str_format.h" +#include "absl/strings/str_join.h" +#include "hb-ot.h" +#include "hb-subset.h" +#include "hb.h" +#include "ift/common/font_data.h" +#include "ift/common/int_set.h" +#include "ift/common/try.h" +#include "ift/common/woff2.h" +#include "ift/encoder/init_subset_defaults.h" +#include "ift/encoder/subset_definition.h" +#include "ift/encoder/types.h" + +using absl::Status; +using absl::StatusOr; +using absl::StrCat; +using absl::StrFormat; +using ift::common::CodepointSet; +using ift::common::FontData; +using ift::common::GlyphSet; +using ift::common::hb_face_unique_ptr; +using ift::common::IntSet; +using ift::common::make_hb_face; +using ift::common::Woff2; + +namespace ift::encoder { + +Status UnicodeRangeCompiler::SetInitSubsetFromDef( + const SubsetDefinition& init_subset) { + if (!init_subset_.Empty()) { + return absl::FailedPreconditionError("Base subset has already been set."); + } + init_subset_ = init_subset; + return absl::OkStatus(); +} + +Status UnicodeRangeCompiler::AddGlyphDataPatch(uint32_t patch_id, + const IntSet& gids) { + if (glyph_data_patches_.contains(patch_id)) { + return absl::FailedPreconditionError( + StrCat("Glyph data patch with id ", patch_id, " already exists.")); + } + if (gids.empty()) { + return absl::InvalidArgumentError( + StrCat("Glyph data patch ", patch_id, " is empty.")); + } + GlyphSet glyph_set; + glyph_set.union_set(gids); + glyph_data_patches_[patch_id] = std::move(glyph_set); + return absl::OkStatus(); +} + +Status UnicodeRangeCompiler::AddSegment(segment_index_t segment_id, + const SubsetDefinition& segment) { + if (segments_.contains(segment_id)) { + return absl::FailedPreconditionError( + StrCat("Segment with id ", segment_id, " already exists.")); + } + if (!segment.feature_tags.empty()) { + return absl::InvalidArgumentError( + StrCat("Segment ", segment_id, + " contains layout features, which are not supported in " + "UNICODE_RANGE mode.")); + } + segments_[segment_id] = segment; + return absl::OkStatus(); +} + +Status UnicodeRangeCompiler::AddGlyphPatchCondition(segment_index_t segment_id, + uint32_t patch_id) { + glyph_patch_conditions_.emplace_back(segment_id, patch_id); + return absl::OkStatus(); +} + +std::string UnicodeRangeCompiler::FormatUnicodeRange( + const CodepointSet& codepoints) { + std::vector parts; + std::optional range_start; + uint32_t range_end = 0; + + auto flush_range = [&]() { + if (!range_start.has_value()) { + return; + } + if (*range_start == range_end) { + parts.push_back(StrFormat("U+%X", *range_start)); + } else { + parts.push_back(StrFormat("U+%X-%X", *range_start, range_end)); + } + range_start = std::nullopt; + }; + + for (uint32_t cp : codepoints) { + if (!range_start.has_value()) { + range_start = cp; + range_end = cp; + } else if (cp == range_end + 1) { + range_end = cp; + } else { + flush_range(); + range_start = cp; + range_end = cp; + } + } + flush_range(); + + return absl::StrJoin(parts, ", "); +} + +static std::optional GetNameEntryUtf8(hb_face_t* face, + hb_ot_name_id_t name_id, + hb_language_t language) { + unsigned int text_size = 0; + unsigned int len = + hb_ot_name_get_utf8(face, name_id, language, &text_size, nullptr); + if (len == 0) { + return std::nullopt; + } + std::string buffer(len + 1, '\0'); + unsigned int buffer_size = buffer.size(); + hb_ot_name_get_utf8(face, name_id, language, &buffer_size, buffer.data()); + buffer.resize(buffer_size); + return buffer; +} + +std::string UnicodeRangeCompiler::ExtractFontFamily(hb_face_t* face) { + if (face == nullptr) { + return "font"; + } + hb_language_t en_lang = hb_language_from_string("en", -1); + for (hb_ot_name_id_t name_id : + {HB_OT_NAME_ID_TYPOGRAPHIC_FAMILY, HB_OT_NAME_ID_FONT_FAMILY}) { + for (hb_language_t lang : {en_lang, HB_LANGUAGE_INVALID}) { + auto name = GetNameEntryUtf8(face, name_id, lang); + if (name.has_value() && !name->empty()) { + return *name; + } + } + } + return "font"; +} + +StatusOr UnicodeRangeCompiler::Compile() const { + if (face_.get() == nullptr) { + return absl::FailedPreconditionError("Input font face is not set."); + } + + hb_face_unique_ptr preprocessed_face = + make_hb_face(hb_subset_preprocess(face_.get())); + + SubsetDefinition base_subset = init_subset_; + AddInitSubsetDefaults(base_subset); + + std::string family = + font_family_.empty() ? ExtractFontFamily(face_.get()) : font_family_; + std::string ext = woff2_encode_ ? ".woff2" : ".ttf"; + std::string format_str = woff2_encode_ ? "woff2" : "truetype"; + + Encoding result; + for (const auto& [segment_id, patch_id] : glyph_patch_conditions_) { + auto seg_it = segments_.find(segment_id); + if (seg_it == segments_.end()) { + return absl::InvalidArgumentError( + StrCat("Segment id ", segment_id, " not found.")); + } + const SubsetDefinition& segment = seg_it->second; + if (segment.codepoints.empty()) { + return absl::InvalidArgumentError( + StrCat("Segment ", segment_id, " has no codepoints.")); + } + + auto patch_it = glyph_data_patches_.find(patch_id); + if (patch_it == glyph_data_patches_.end()) { + return absl::InvalidArgumentError( + StrCat("Glyph patch id ", patch_id, " not found.")); + } + const GlyphSet& patch_gids = patch_it->second; + + SubsetDefinition subset_def = base_subset; + subset_def.Union(segment); + subset_def.gids.union_set(patch_gids); + + hb_subset_input_t* input = hb_subset_input_create_or_fail(); + if (!input) { + return absl::InternalError("Failed to create subset input."); + } + subset_def.ConfigureInput(input, preprocessed_face.get()); + + hb_face_unique_ptr subset_face = + make_hb_face(hb_subset_or_fail(preprocessed_face.get(), input)); + hb_subset_input_destroy(input); + if (!subset_face.get()) { + return absl::InternalError( + StrCat("Failed to create font subset for patch ", patch_id, ".")); + } + + FontData subset_data(subset_face.get()); + if (woff2_encode_) { + subset_data = TRY(Woff2::EncodeWoff2( + subset_data.str(), /*glyf_transform=*/true, brotli_quality_)); + } + + std::string filename = StrCat(font_name_, ".", patch_id, ext); + if (result.subsets.contains(filename)) { + return absl::InvalidArgumentError( + StrCat("Duplicate subset for patch id ", patch_id, ".")); + } + result.subsets.emplace(filename, std::move(subset_data)); + + if (!result.css.empty()) { + absl::StrAppend(&result.css, "\n"); + } + absl::StrAppend(&result.css, "@font-face {\n", " font-family: '", family, + "';\n", " src: url('", filename, "') format('", format_str, + "');\n", + " unicode-range: ", FormatUnicodeRange(segment.codepoints), + ";\n", "}\n"); + } + + return result; +} + +} // namespace ift::encoder diff --git a/ift/encoder/unicode_range_compiler.h b/ift/encoder/unicode_range_compiler.h new file mode 100644 index 00000000..5502af59 --- /dev/null +++ b/ift/encoder/unicode_range_compiler.h @@ -0,0 +1,76 @@ +#ifndef IFT_ENCODER_UNICODE_RANGE_COMPILER_H_ +#define IFT_ENCODER_UNICODE_RANGE_COMPILER_H_ + +#include +#include +#include +#include + +#include "absl/container/btree_map.h" +#include "absl/status/status.h" +#include "absl/status/statusor.h" +#include "absl/strings/string_view.h" +#include "hb.h" +#include "ift/common/font_data.h" +#include "ift/common/int_set.h" +#include "ift/encoder/subset_definition.h" +#include "ift/encoder/types.h" + +namespace ift::encoder { + +// Compiles a font and a UNICODE_RANGE segmentation plan into a set of +// standalone WOFF2 (or OpenType) font subsets and a CSS stylesheet with +// @font-face unicode-range rules. +class UnicodeRangeCompiler { + public: + struct Encoding { + std::string css; + absl::btree_map subsets; + }; + + UnicodeRangeCompiler() : face_(ift::common::make_hb_face(nullptr)) {} + + UnicodeRangeCompiler(const UnicodeRangeCompiler&) = delete; + UnicodeRangeCompiler(UnicodeRangeCompiler&&) = delete; + UnicodeRangeCompiler& operator=(const UnicodeRangeCompiler&) = delete; + UnicodeRangeCompiler& operator=(UnicodeRangeCompiler&&) = delete; + + void SetFace(hb_face_t* face) { face_.reset(hb_face_reference(face)); } + void SetWoff2Encode(bool value) { woff2_encode_ = value; } + void SetBrotliQuality(uint32_t quality) { brotli_quality_ = quality; } + void SetFontName(absl::string_view font_name) { + font_name_ = std::string(font_name); + } + void SetFontFamily(absl::string_view font_family) { + font_family_ = std::string(font_family); + } + + absl::Status SetInitSubsetFromDef(const SubsetDefinition& init_subset); + absl::Status AddGlyphDataPatch(uint32_t patch_id, + const ift::common::IntSet& gids); + absl::Status AddSegment(segment_index_t segment_id, + const SubsetDefinition& segment); + absl::Status AddGlyphPatchCondition(segment_index_t segment_id, + uint32_t patch_id); + + absl::StatusOr Compile() const; + + static std::string FormatUnicodeRange( + const ift::common::CodepointSet& codepoints); + static std::string ExtractFontFamily(hb_face_t* face); + + private: + ift::common::hb_face_unique_ptr face_; + SubsetDefinition init_subset_; + absl::btree_map segments_; + absl::btree_map glyph_data_patches_; + std::vector> glyph_patch_conditions_; + std::string font_name_ = "out"; + std::string font_family_; + bool woff2_encode_ = true; + uint32_t brotli_quality_ = 11; +}; + +} // namespace ift::encoder + +#endif // IFT_ENCODER_UNICODE_RANGE_COMPILER_H_ diff --git a/ift/encoder/unicode_range_compiler_test.cc b/ift/encoder/unicode_range_compiler_test.cc new file mode 100644 index 00000000..ea9677a7 --- /dev/null +++ b/ift/encoder/unicode_range_compiler_test.cc @@ -0,0 +1,186 @@ +#include "ift/encoder/unicode_range_compiler.h" + +#include + +#include "absl/status/status.h" +#include "gtest/gtest.h" +#include "hb.h" +#include "ift/common/bazel_data_file_resolver.h" +#include "ift/common/font_data.h" +#include "ift/common/font_helper.h" +#include "ift/common/int_set.h" +#include "ift/common/test_font_loader.h" +#include "ift/common/woff2.h" +#include "ift/config/config_compiler.h" +#include "ift/config/segmentation_plan.pb.h" +#include "ift/encoder/closure_glyph_segmenter.h" + +using ift::common::BazelDataFileResolver; +using ift::common::CodepointSet; +using ift::common::FontData; +using ift::common::FontHelper; +using ift::common::hb_face_unique_ptr; +using ift::common::Woff2; +using ift::config::CLOSURE_AND_VALIDATE_DEP_GRAPH; +using ift::config::ConfigCompiler; +using ift::config::FIND_CONDITIONS; +using ift::config::SegmentationPlan; + +namespace ift::encoder { + +class UnicodeRangeCompilerTest : public ::testing::Test { + protected: + UnicodeRangeCompilerTest() + : roboto(ift::common::make_hb_face(nullptr)), + noto_sans_jp(ift::common::make_hb_face(nullptr)) { + auto loader = ift::common::TestFontLoader::Default().value(); + roboto = + loader->LoadFace("ift/common/testdata/Roboto-Regular.ttf").value(); + noto_sans_jp = + loader->LoadFace("ift/common/testdata/NotoSansJP-Regular.subset.ttf") + .value(); + resolver = *BazelDataFileResolver::CreateForTest(); + } + + hb_face_unique_ptr roboto; + hb_face_unique_ptr noto_sans_jp; + std::shared_ptr resolver; +}; + +TEST_F(UnicodeRangeCompilerTest, FormatUnicodeRange) { + EXPECT_EQ(UnicodeRangeCompiler::FormatUnicodeRange({}), ""); + EXPECT_EQ(UnicodeRangeCompiler::FormatUnicodeRange({'A'}), "U+41"); + EXPECT_EQ(UnicodeRangeCompiler::FormatUnicodeRange({'A', 'B', 'C', 'D'}), + "U+41-44"); + EXPECT_EQ(UnicodeRangeCompiler::FormatUnicodeRange( + {0x20, 0x41, 0x42, 0x43, 0x61, 0x1F600, 0x1F601}), + "U+20, U+41-43, U+61, U+1F600-1F601"); +} + +TEST_F(UnicodeRangeCompilerTest, ExtractFontFamily) { + EXPECT_EQ(UnicodeRangeCompiler::ExtractFontFamily(roboto.get()), "Roboto"); + EXPECT_EQ(UnicodeRangeCompiler::ExtractFontFamily(noto_sans_jp.get()), + "Noto Sans JP Thin"); +} + +TEST_F(UnicodeRangeCompilerTest, CompileFromSegmentationPlan) { + ClosureGlyphSegmenter segmenter(1, 1, FIND_CONDITIONS, + CLOSURE_AND_VALIDATE_DEP_GRAPH, resolver, + ift::config::UNICODE_RANGE); + auto segmentation = segmenter.CodepointToGlyphSegments( + roboto.get(), {'a'}, {{'f'}, {'i'}, {'b', 'c', 'd'}}); + ASSERT_TRUE(segmentation.ok()) << segmentation.status(); + + SegmentationPlan plan = segmentation->ToSegmentationPlanProto(); + + UnicodeRangeCompiler compiler; + compiler.SetFace(roboto.get()); + compiler.SetFontName("Roboto-Regular"); + compiler.SetBrotliQuality(1); + ASSERT_TRUE(ConfigCompiler::Configure(plan, compiler).ok()); + + auto encoding = compiler.Compile(); + ASSERT_TRUE(encoding.ok()) << encoding.status(); + + EXPECT_EQ(encoding->css, + R"(@font-face { + font-family: 'Roboto'; + src: url('Roboto-Regular.0.woff2') format('woff2'); + unicode-range: U+66, U+69; +} + +@font-face { + font-family: 'Roboto'; + src: url('Roboto-Regular.1.woff2') format('woff2'); + unicode-range: U+62-64; +} +)"); + + ASSERT_EQ(encoding->subsets.size(), 2u); + ASSERT_TRUE(encoding->subsets.contains("Roboto-Regular.0.woff2")); + ASSERT_TRUE(encoding->subsets.contains("Roboto-Regular.1.woff2")); + + // Decode each WOFF2 subset and verify its codepoints include 'a' (from + // initial_segment) plus that subset's segment codepoints, and that subset 0 + // includes the fi and ffi ligature glyphs. + FontData subset_0_ttf = + *Woff2::DecodeWoff2(encoding->subsets.at("Roboto-Regular.0.woff2").str()); + hb_face_unique_ptr face_0 = subset_0_ttf.face(); + CodepointSet cps_0 = FontHelper::ToCodepointsSet(face_0.get()); + // Subset 0 retains 'a' (initial_segment), 'f', 'i', and the 'fi'/'ffi' + // ligature glyphs from patch 0 (which are also mapped in Roboto's cmap to + // U+FB01 and U+FB03). + EXPECT_EQ(cps_0, (CodepointSet{'a', 'f', 'i', 0xFB01, 0xFB03})); + // .notdef, 'a', 'f', 'i', 'fi', 'ffi' + EXPECT_EQ(hb_face_get_glyph_count(face_0.get()), 6u); + + FontData subset_1_ttf = + *Woff2::DecodeWoff2(encoding->subsets.at("Roboto-Regular.1.woff2").str()); + hb_face_unique_ptr face_1 = subset_1_ttf.face(); + CodepointSet cps_1 = FontHelper::ToCodepointsSet(face_1.get()); + EXPECT_EQ(cps_1, (CodepointSet{'a', 'b', 'c', 'd'})); + // .notdef, 'a', 'b', 'c', 'd' + EXPECT_EQ(hb_face_get_glyph_count(face_1.get()), 5u); +} + +TEST_F(UnicodeRangeCompilerTest, FontFamilyOverrideAndUncompressedOutput) { + UnicodeRangeCompiler compiler; + compiler.SetFace(roboto.get()); + compiler.SetFontName("myfont"); + compiler.SetFontFamily("Custom Family"); + compiler.SetWoff2Encode(false); + + SubsetDefinition seg0; + seg0.codepoints = {'a', 'b'}; + ASSERT_TRUE(compiler.AddSegment(0, seg0).ok()); + ASSERT_TRUE(compiler.AddGlyphDataPatch(0, {69, 70}).ok()); + ASSERT_TRUE(compiler.AddGlyphPatchCondition(0, 0).ok()); + + auto encoding = compiler.Compile(); + ASSERT_TRUE(encoding.ok()) << encoding.status(); + + EXPECT_EQ(encoding->css, + R"(@font-face { + font-family: 'Custom Family'; + src: url('myfont.0.ttf') format('truetype'); + unicode-range: U+61-62; +} +)"); + ASSERT_EQ(encoding->subsets.size(), 1u); + ASSERT_TRUE(encoding->subsets.contains("myfont.0.ttf")); + + hb_face_unique_ptr face_0 = encoding->subsets.at("myfont.0.ttf").face(); + EXPECT_EQ(FontHelper::ToCodepointsSet(face_0.get()), + (CodepointSet{'a', 'b'})); +} + +TEST_F(UnicodeRangeCompilerTest, RejectsInvalidPlan) { + // Non-exclusive condition + { + SegmentationPlan plan; + (*plan.mutable_segments())[0].mutable_codepoints()->add_values('a'); + (*plan.mutable_segments())[1].mutable_codepoints()->add_values('b'); + (*plan.mutable_glyph_patches())[0].add_values(69); + auto* cond = plan.add_glyph_patch_conditions(); + auto* req = cond->add_required_segments(); + req->add_values(0); + req->add_values(1); + cond->set_activated_patch(0); + + UnicodeRangeCompiler compiler; + EXPECT_TRUE( + absl::IsInvalidArgument(ConfigCompiler::Configure(plan, compiler))); + } + + // Non-glyph (table-keyed) segmentation + { + SegmentationPlan plan; + plan.add_non_glyph_codepoint_segmentation()->add_values('a'); + + UnicodeRangeCompiler compiler; + EXPECT_TRUE( + absl::IsInvalidArgument(ConfigCompiler::Configure(plan, compiler))); + } +} + +} // namespace ift::encoder diff --git a/ift/encoder/woff2_patch_size_cache.cc b/ift/encoder/woff2_patch_size_cache.cc new file mode 100644 index 00000000..cf4f8776 --- /dev/null +++ b/ift/encoder/woff2_patch_size_cache.cc @@ -0,0 +1,147 @@ +#include "ift/encoder/woff2_patch_size_cache.h" + +#include +#include + +#include "absl/container/flat_hash_map.h" +#include "absl/status/status.h" +#include "absl/status/statusor.h" +#include "hb-subset.h" +#include "hb.h" +#include "ift/common/font_data.h" +#include "ift/common/font_helper.h" +#include "ift/common/hb_set_unique_ptr.h" +#include "ift/common/int_set.h" +#include "ift/common/try.h" +#include "ift/common/woff2.h" +#include "ift/dep_graph/unicode_edges.h" +#include "ift/encoder/init_subset_defaults.h" +#include "ift/encoder/subset_definition.h" + +using absl::StatusOr; +using ift::common::FontData; +using ift::common::FontHelper; +using ift::common::GlyphSet; +using ift::common::hb_face_unique_ptr; +using ift::common::hb_set_unique_ptr; +using ift::common::make_hb_face; +using ift::common::make_hb_set; +using ift::common::Woff2; +using ift::dep_graph::UnicodeEdges; + +namespace ift::encoder { + +Woff2PatchSizeCache::Woff2PatchSizeCache(hb_face_t* original_face, + SubsetDefinition base_subset, + uint32_t brotli_quality) + : preprocessed_face_(make_hb_face(hb_subset_preprocess(original_face))), + base_subset_(std::move(base_subset)), + brotli_quality_(brotli_quality), + unicode_edges_( + UnicodeEdges::ComputeCmapAndUVSEdges(preprocessed_face_.get())), + cache_() { + AddInitSubsetDefaults(base_subset_); +} + +StatusOr Woff2PatchSizeCache::BaseGlyphs() const { + hb_subset_input_t* input = hb_subset_input_create_or_fail(); + if (!input) { + return absl::InternalError("Failed to create subset input."); + } + base_subset_.ConfigureInput(input, preprocessed_face_.get()); + hb_subset_plan_t* plan = + hb_subset_plan_create_or_fail(preprocessed_face_.get(), input); + hb_subset_input_destroy(input); + if (!plan) { + return absl::InternalError("Failed to create base subset plan."); + } + hb_map_t* new_to_old = hb_subset_plan_new_to_old_glyph_mapping(plan); + hb_set_unique_ptr gids = make_hb_set(); + hb_map_values(new_to_old, gids.get()); + hb_subset_plan_destroy(plan); + return GlyphSet(gids); +} + +StatusOr Woff2PatchSizeCache::GetPatchSize(const GlyphSet& gids) { + auto it = cache_.find(gids); + if (it != cache_.end()) { + return it->second; + } + + brotli_call_count_++; + + SubsetDefinition subset_def = base_subset_; + subset_def.gids.union_set(gids); + subset_def.codepoints.union_set(unicode_edges_.CodepointsForGlyphs(gids)); + + hb_subset_input_t* input = hb_subset_input_create_or_fail(); + if (!input) { + return absl::InternalError("Failed to create subset input."); + } + subset_def.ConfigureInput(input, preprocessed_face_.get()); + + hb_face_unique_ptr subset_face = + make_hb_face(hb_subset_or_fail(preprocessed_face_.get(), input)); + hb_subset_input_destroy(input); + if (!subset_face.get()) { + return absl::InternalError("Failed to create font subset."); + } + + FontData subset_data(subset_face.get()); + FontData woff2 = TRY(Woff2::EncodeWoff2( + subset_data.str(), /*glyf_transform=*/true, brotli_quality_)); + uint32_t size = woff2.size(); + cache_[gids] = size; + return size; +} + +StatusOr EstimatedWoff2PatchSizeCache::GetPatchSize( + const GlyphSet& gids) { + auto it = cache_.find(gids); + if (it != cache_.end()) { + return it->second; + } + + uint32_t uncompressed_outline_size = + TRY(FontHelper::TotalGlyphData(face_.get(), gids)); + uint32_t size = + base_woff2_overhead_ + + (uint32_t)((double)uncompressed_outline_size * compression_ratio_); + cache_[gids] = size; + return size; +} + +StatusOr> +EstimatedWoff2PatchSizeCache::EstimateParameters(hb_face_t* original_face, + SubsetDefinition base_subset) { + Woff2PatchSizeCache woff2_sizer(original_face, std::move(base_subset), 11); + uint32_t base_woff2_overhead = TRY(woff2_sizer.GetPatchSize(GlyphSet{})); + + uint32_t glyph_count = hb_face_get_glyph_count(original_face); + if (glyph_count == 0) { + return std::make_pair(base_woff2_overhead, 0.0); + } + + GlyphSet all_gids; + all_gids.insert_range(0, glyph_count - 1); + GlyphSet base_gids = TRY(woff2_sizer.BaseGlyphs()); + + GlyphSet non_base_gids = all_gids; + non_base_gids.subtract(base_gids); + if (non_base_gids.empty()) { + return std::make_pair(base_woff2_overhead, 0.0); + } + + uint32_t uncompressed_size = + TRY(FontHelper::TotalGlyphData(original_face, non_base_gids)); + uint32_t full_woff2_size = TRY(woff2_sizer.GetPatchSize(non_base_gids)); + if (uncompressed_size == 0 || full_woff2_size <= base_woff2_overhead) { + return std::make_pair(base_woff2_overhead, 0.0); + } + + double compression_ratio = (double)(full_woff2_size - base_woff2_overhead) / + (double)uncompressed_size; + return std::make_pair(base_woff2_overhead, compression_ratio); +} + +} // namespace ift::encoder diff --git a/ift/encoder/woff2_patch_size_cache.h b/ift/encoder/woff2_patch_size_cache.h new file mode 100644 index 00000000..ac8cd040 --- /dev/null +++ b/ift/encoder/woff2_patch_size_cache.h @@ -0,0 +1,96 @@ +#ifndef IFT_ENCODER_WOFF2_PATCH_SIZE_CACHE_H_ +#define IFT_ENCODER_WOFF2_PATCH_SIZE_CACHE_H_ + +#include +#include +#include + +#include "absl/container/flat_hash_map.h" +#include "absl/log/log.h" +#include "absl/status/statusor.h" +#include "ift/common/font_data.h" +#include "ift/common/int_set.h" +#include "ift/common/try.h" +#include "ift/dep_graph/unicode_edges.h" +#include "ift/encoder/patch_size_cache.h" +#include "ift/encoder/subset_definition.h" + +namespace ift::encoder { + +// Computes the WOFF2 size of standalone font subsets formed by unioning +// base_subset (the initial segment) with the patch's glyphs and their +// associated codepoints, and caches the result. +class Woff2PatchSizeCache : public PatchSizeCache { + public: + Woff2PatchSizeCache(hb_face_t* original_face, SubsetDefinition base_subset, + uint32_t brotli_quality); + + absl::StatusOr GetPatchSize( + const ift::common::GlyphSet& gids) override; + + void LogBrotliCallCount() const override { + VLOG(0) << "Total number of calls to brotli = " << brotli_call_count_; + } + + uint64_t BrotliCallCount() const { return brotli_call_count_; } + + absl::StatusOr BaseGlyphs() const; + + private: + ift::common::hb_face_unique_ptr preprocessed_face_; + SubsetDefinition base_subset_; + uint32_t brotli_quality_; + dep_graph::UnicodeEdges unicode_edges_; + absl::flat_hash_map cache_; + uint64_t brotli_call_count_ = 0; +}; + +// Estimates the WOFF2 size of a standalone font subset using the WOFF2 size of +// the base subset (initial_segment) plus the raw outline size of the patch's +// glyphs scaled by the font's overall WOFF2 outline compression ratio. +class EstimatedWoff2PatchSizeCache : public PatchSizeCache { + public: + static absl::StatusOr> New( + hb_face_t* face, SubsetDefinition base_subset) { + auto [base_woff2_overhead, compression_ratio] = + TRY(EstimateParameters(face, std::move(base_subset))); + return std::unique_ptr(new EstimatedWoff2PatchSizeCache( + face, base_woff2_overhead, compression_ratio)); + } + + static std::unique_ptr New(hb_face_t* face, + uint32_t base_woff2_overhead, + double compression_ratio) { + return std::unique_ptr(new EstimatedWoff2PatchSizeCache( + face, base_woff2_overhead, compression_ratio)); + } + + absl::StatusOr GetPatchSize( + const ift::common::GlyphSet& gids) override; + + void LogBrotliCallCount() const override {} + + uint32_t BaseWoff2Overhead() const { return base_woff2_overhead_; } + double CompressionRatio() const { return compression_ratio_; } + + static absl::StatusOr> EstimateParameters( + hb_face_t* original_face, SubsetDefinition base_subset); + + private: + EstimatedWoff2PatchSizeCache(hb_face_t* original_face, + uint32_t base_woff2_overhead, + double compression_ratio) + : face_(ift::common::make_hb_face(hb_face_reference(original_face))), + base_woff2_overhead_(base_woff2_overhead), + compression_ratio_(compression_ratio), + cache_() {} + + ift::common::hb_face_unique_ptr face_; + uint32_t base_woff2_overhead_; + double compression_ratio_; + absl::flat_hash_map cache_; +}; + +} // namespace ift::encoder + +#endif // IFT_ENCODER_WOFF2_PATCH_SIZE_CACHE_H_ diff --git a/ift/encoder/woff2_patch_size_cache_test.cc b/ift/encoder/woff2_patch_size_cache_test.cc new file mode 100644 index 00000000..d0aa87de --- /dev/null +++ b/ift/encoder/woff2_patch_size_cache_test.cc @@ -0,0 +1,137 @@ +#include "ift/encoder/woff2_patch_size_cache.h" + +#include +#include + +#include "gtest/gtest.h" +#include "hb-subset.h" +#include "hb.h" +#include "ift/common/font_data.h" +#include "ift/common/font_helper.h" +#include "ift/common/int_set.h" +#include "ift/common/test_font_loader.h" +#include "ift/common/woff2.h" +#include "ift/encoder/init_subset_defaults.h" +#include "ift/encoder/subset_definition.h" + +using ift::common::CodepointSet; +using ift::common::FontData; +using ift::common::FontHelper; +using ift::common::GlyphSet; +using ift::common::hb_face_unique_ptr; +using ift::common::make_hb_face; +using ift::common::Woff2; + +namespace ift::encoder { + +class Woff2PatchSizeCacheTest : public ::testing::Test { + protected: + Woff2PatchSizeCacheTest() : roboto(make_hb_face(nullptr)) { + auto loader = ift::common::TestFontLoader::Default().value(); + auto blob = loader->LoadFontData("ift/common/testdata/Roboto-Regular.ttf") + .value() + .blob(); + roboto = make_hb_face(hb_face_create(blob.get(), 0)); + } + + uint32_t ManualWoff2SubsetSize(SubsetDefinition base_subset, + const GlyphSet& gids, + uint32_t brotli_quality) { + AddInitSubsetDefaults(base_subset); + base_subset.gids.union_set(gids); + auto gid_to_unicode = FontHelper::GidToUnicodeMap(roboto.get()); + for (uint32_t gid : gids) { + auto it = gid_to_unicode.find(gid); + if (it != gid_to_unicode.end()) { + base_subset.codepoints.union_set(it->second); + } + } + + hb_subset_input_t* input = hb_subset_input_create_or_fail(); + base_subset.ConfigureInput(input, roboto.get()); + hb_face_unique_ptr subset_face = + make_hb_face(hb_subset_or_fail(roboto.get(), input)); + hb_subset_input_destroy(input); + + FontData subset_data(subset_face.get()); + FontData woff2 = + Woff2::EncodeWoff2(subset_data.str(), /*glyf_transform=*/true, + brotli_quality) + .value(); + return woff2.size(); + } + + hb_face_unique_ptr roboto; +}; + +TEST_F(Woff2PatchSizeCacheTest, Woff2PatchSizeMatchesManualSubset) { + SubsetDefinition base_subset; + Woff2PatchSizeCache cache(roboto.get(), base_subset, /*brotli_quality=*/1); + + GlyphSet patch_a{44, 47, 49}; + uint32_t size_a = *cache.GetPatchSize(patch_a); + EXPECT_EQ(size_a, ManualWoff2SubsetSize(base_subset, patch_a, 1)); + EXPECT_EQ(cache.BrotliCallCount(), 1u); + + // Repeat call should hit the cache without invoking Brotli again. + EXPECT_EQ(*cache.GetPatchSize(patch_a), size_a); + EXPECT_EQ(cache.BrotliCallCount(), 1u); + + GlyphSet patch_b{45, 48, 50, 51, 52, 53}; + uint32_t size_b = *cache.GetPatchSize(patch_b); + EXPECT_EQ(size_b, ManualWoff2SubsetSize(base_subset, patch_b, 1)); + EXPECT_EQ(cache.BrotliCallCount(), 2u); + EXPECT_GT(size_b, size_a); +} + +TEST_F(Woff2PatchSizeCacheTest, IncludesBaseSubsetInAllPatches) { + SubsetDefinition empty_base; + Woff2PatchSizeCache empty_base_cache(roboto.get(), empty_base, + /*brotli_quality=*/1); + + SubsetDefinition non_empty_base; + non_empty_base.codepoints = CodepointSet{'A', 'B', 'C', 'D', 'E'}; + Woff2PatchSizeCache non_empty_base_cache(roboto.get(), non_empty_base, + /*brotli_quality=*/1); + + uint32_t empty_base_size = *empty_base_cache.GetPatchSize(GlyphSet{}); + uint32_t non_empty_base_size = *non_empty_base_cache.GetPatchSize(GlyphSet{}); + EXPECT_GT(non_empty_base_size, empty_base_size); + EXPECT_EQ(non_empty_base_size, + ManualWoff2SubsetSize(non_empty_base, GlyphSet{}, 1)); + + GlyphSet base_gids = *non_empty_base_cache.BaseGlyphs(); + EXPECT_TRUE(base_gids.contains(0)); + EXPECT_GT(base_gids.size(), 1u); +} + +TEST_F(Woff2PatchSizeCacheTest, EstimatedWoff2PatchSizeCache) { + SubsetDefinition base_subset; + auto [base_overhead, ratio] = + *EstimatedWoff2PatchSizeCache::EstimateParameters(roboto.get(), + base_subset); + EXPECT_EQ(base_overhead, ManualWoff2SubsetSize(base_subset, GlyphSet{}, 11)); + EXPECT_GT(ratio, 0.0); + EXPECT_LT(ratio, 1.0); + + auto estimated = + *EstimatedWoff2PatchSizeCache::New(roboto.get(), base_subset); + EXPECT_EQ(*estimated->GetPatchSize(GlyphSet{}), base_overhead); + + GlyphSet patch_a{44, 47, 49}; + uint32_t raw_a = *FontHelper::TotalGlyphData(roboto.get(), patch_a); + uint32_t est_a = *estimated->GetPatchSize(patch_a); + EXPECT_GT(est_a, base_overhead); + double observed_ratio_a = + (double)(est_a - base_overhead) / (double)raw_a; + EXPECT_NEAR(observed_ratio_a, ratio, 0.02); + + GlyphSet patch_b{45, 48, 50, 51, 52, 53}; + uint32_t raw_b = *FontHelper::TotalGlyphData(roboto.get(), patch_b); + uint32_t est_b = *estimated->GetPatchSize(patch_b); + double observed_ratio_b = + (double)(est_b - base_overhead) / (double)raw_b; + EXPECT_NEAR(observed_ratio_b, ratio, 0.02); +} + +} // namespace ift::encoder diff --git a/util/BUILD b/util/BUILD index e62d3ad0..ff899e29 100644 --- a/util/BUILD +++ b/util/BUILD @@ -40,6 +40,37 @@ cc_binary( ], ) +cc_binary( + name = "font2ur", + srcs = [ + "font2ur.cc", + ], + deps = [ + ":auto_config_flags", + ":path_util", + "//ift/common", + "//ift/common:data_file_resolver", + "//ift/common:try", + "//ift/config:auto_segmenter_config", + "//ift/config:config_compiler", + "//ift/config:load_codepoints", + "//ift/config:segmentation_plan_cc_proto", + "//ift/config:segmenter_config_cc_proto", + "//ift/config:segmenter_config_util", + "//ift/encoder", + "@abseil-cpp//absl/flags:flag", + "@abseil-cpp//absl/flags:parse", + "@abseil-cpp//absl/flags:usage", + "@abseil-cpp//absl/log:globals", + "@abseil-cpp//absl/log:initialize", + "@abseil-cpp//absl/status", + "@abseil-cpp//absl/status:statusor", + "@abseil-cpp//absl/strings", + "@harfbuzz", + "@protobuf", + ], +) + cc_library( name = "path_util", srcs = ["path_util.cc"], diff --git a/util/auto_config_flags.cc b/util/auto_config_flags.cc index 7f17ddc7..3f786a7c 100644 --- a/util/auto_config_flags.cc +++ b/util/auto_config_flags.cc @@ -19,3 +19,8 @@ ABSL_FLAG( "when multiple scripts overlap (eg. like with CJK) merging will " "prioritize optimizing against the primary script/language instead of " "all overlapping scripts equally."); + +ABSL_FLAG( + bool, auto_config_unicode_range, false, + "When auto_config is enabled, generate a segmenter config with target_mode " + "set to UNICODE_RANGE instead of IFT."); diff --git a/util/auto_config_flags.h b/util/auto_config_flags.h index 4f158361..53da5bf3 100644 --- a/util/auto_config_flags.h +++ b/util/auto_config_flags.h @@ -7,5 +7,6 @@ ABSL_DECLARE_FLAG(int, auto_config_quality); ABSL_DECLARE_FLAG(std::string, auto_config_primary_script); +ABSL_DECLARE_FLAG(bool, auto_config_unicode_range); #endif // UTIL_AUTO_CONFIG_FLAGS_H_ diff --git a/util/font2ift.cc b/util/font2ift.cc index 0fbbf01f..187d85f0 100644 --- a/util/font2ift.cc +++ b/util/font2ift.cc @@ -149,6 +149,11 @@ Status write_output(const Compiler::Encoding& encoding) { StatusOr CreateSegmentationPlan( hb_face_t* font, std::shared_ptr resolver) { + if (absl::GetFlag(FLAGS_auto_config_unicode_range)) { + return absl::InvalidArgumentError( + "auto_config_unicode_range is not supported by font2ift."); + } + SegmentationPlan plan; if (absl::GetFlag(FLAGS_plan).empty() || absl::GetFlag(FLAGS_plan) == "auto") { diff --git a/util/font2ur.cc b/util/font2ur.cc new file mode 100644 index 00000000..e1e0eb04 --- /dev/null +++ b/util/font2ur.cc @@ -0,0 +1,239 @@ +#include + +#include +#include +#include +#include +#include +#include +#include + +#include "absl/flags/flag.h" +#include "absl/flags/parse.h" +#include "absl/flags/usage.h" +#include "absl/log/globals.h" +#include "absl/log/initialize.h" +#include "absl/status/status.h" +#include "absl/status/statusor.h" +#include "absl/strings/str_cat.h" +#include "absl/strings/string_view.h" +#include "hb.h" +#include "ift/common/bazel_data_file_resolver.h" +#include "ift/common/data_file_resolver.h" +#include "ift/common/font_data.h" +#include "ift/common/try.h" +#include "ift/config/auto_segmenter_config.h" +#include "ift/config/config_compiler.h" +#include "ift/config/load_codepoints.h" +#include "ift/config/segmentation_plan.pb.h" +#include "ift/config/segmenter_config.pb.h" +#include "ift/config/segmenter_config_util.h" +#include "ift/encoder/unicode_range_compiler.h" +#include "util/auto_config_flags.h" +#include "util/path_util.h" + +using absl::Status; +using absl::StatusOr; +using absl::StrCat; +using ift::common::BazelDataFileResolver; +using ift::common::DataFileResolver; +using ift::common::FontData; +using ift::common::hb_face_unique_ptr; +using ift::config::AutoSegmenterConfig; +using ift::config::ConfigCompiler; +using ift::config::SegmentationPlan; +using ift::encoder::UnicodeRangeCompiler; +using ift::util::JoinAndValidatePath; + +/* + * [EXPERIMENTAL] + * + * Note: this feature is still under development and performance of produced unicode ranges + * + * Utility that converts a standard font file into a set of WOFF2 font subsets + * and a @font-face unicode-range CSS file, optionally following a supplied + * segmentation plan. + * + * Configuration is provided as a textproto file following the + * segmentation_plan.proto schema. + * + * If no configuration is supplied it will be auto generated. + */ + +ABSL_FLAG(std::string, input_font, "in.ttf", + "Name of the font to convert to unicode-range subsets."); + +ABSL_FLAG(std::string, plan, "auto", + "Path to a plan file which is a textproto following the " + "segmentation_plan.proto schema. If set to \"auto\", then " + "segmentation plan will be automatically generated."); + +ABSL_FLAG(std::string, output_path, "./", + "Path to write output files under (CSS and WOFF2 subsets)."); + +ABSL_FLAG(std::string, output_css, "out.css", + "Name of the outputted CSS file."); + +ABSL_FLAG(std::string, font_family, "", + "Optional override for the font-family name in the generated CSS. If " + "empty, extracted from the input font's name table."); + +ABSL_FLAG(bool, woff2_encode, true, + "If enabled the output font subsets will be woff2 encoded."); + +ABSL_FLAG( + int, verbosity, 0, + "Log verbosity level from. 0 is least verbose, higher values are more."); + +static StatusOr load_font(const char* filename) { + return TRY(ift::config::LoadFile(filename)).face(); +} + +static Status write_bytes(const std::string& name, absl::string_view data) { + std::ofstream output(name, + std::ios::out | std::ios::binary | std::ios::trunc); + if (!output.is_open()) { + return absl::NotFoundError(StrCat("File ", name, " was not found.")); + } + output.write(data.data(), data.size()); + if (output.bad()) { + output.close(); + return absl::InternalError(StrCat("Failed to write to ", name, ".")); + } + + output.close(); + return absl::OkStatus(); +} + +static Status write_output(const UnicodeRangeCompiler::Encoding& encoding) { + std::string output_path = absl::GetFlag(FLAGS_output_path); + std::string output_css = absl::GetFlag(FLAGS_output_css); + + std::string css_full_path = TRY(JoinAndValidatePath(output_path, output_css)); + std::cerr << " Writing CSS: " << css_full_path << std::endl; + TRYV(write_bytes(css_full_path, encoding.css)); + + for (const auto& [name, subset] : encoding.subsets) { + std::string subset_full_path = TRY(JoinAndValidatePath(output_path, name)); + std::cerr << " Writing subset: " << subset_full_path << std::endl; + TRYV(write_bytes(subset_full_path, subset.str())); + } + + return absl::OkStatus(); +} + +static StatusOr CreateSegmentationPlan( + hb_face_t* font, std::shared_ptr resolver) { + SegmentationPlan plan; + if (absl::GetFlag(FLAGS_plan).empty() || + absl::GetFlag(FLAGS_plan) == "auto") { + std::cerr << ">> auto generating unicode-range segmentation plan:" + << std::endl; + std::optional quality_level = std::nullopt; + if (absl::GetFlag(FLAGS_auto_config_quality) > 0) { + quality_level = absl::GetFlag(FLAGS_auto_config_quality); + } + + std::optional primary_script = std::nullopt; + if (!absl::GetFlag(FLAGS_auto_config_primary_script).empty()) { + primary_script = absl::GetFlag(FLAGS_auto_config_primary_script); + } + auto config = AutoSegmenterConfig::GenerateConfig( + font, *resolver, primary_script, quality_level, + ift::config::UNICODE_RANGE); + if (!config.ok()) { + return absl::InternalError( + StrCat("Failed to generate config: ", config.status().message())); + } + ift::config::SegmenterConfigUtil config_util("", resolver); + auto result = config_util.RunSegmenter(font, *config); + if (!result.ok()) { + return absl::InternalError( + StrCat("Failed to run segmenter: ", result.status().message())); + } + plan = std::move(result->plan); + } else { + auto config_text = ift::config::LoadFile(absl::GetFlag(FLAGS_plan).c_str()); + if (!config_text.ok()) { + return absl::InternalError(StrCat("Failed to load config file: ", + config_text.status().message())); + } + + if (!google::protobuf::TextFormat::ParseFromString(config_text->str(), + &plan)) { + return absl::InternalError("Failed to parse input config."); + } + } + return plan; +} + +int main(int argc, char** argv) { + absl::SetProgramUsageMessage( + "Converts OpenType and TrueType fonts into @font-face unicode-range " + "WOFF2 subsets and a CSS stylesheet.\n" + "\n" + "Usage: font2ur --input_font=\"myfont.ttf\" --output_path=\"ur/\" " + "--output_css=\"myfont.css\"\n" + "\n" + "Optional: a segmentation plan can be provided with the --plan flag. If " + "one is not given then it will be generated."); + absl::SetStderrThreshold(absl::LogSeverityAtLeast::kInfo); + absl::SetGlobalVLogLevel(absl::GetFlag(FLAGS_verbosity)); + auto args = absl::ParseCommandLine(argc, argv); + absl::InitializeLog(); + + std::string input_font_flag = absl::GetFlag(FLAGS_input_font); + auto font = load_font(input_font_flag.c_str()); + if (!font.ok()) { + std::cerr << "Failed to load input font: " << font.status() << std::endl; + return -1; + } + + auto resolver_status = BazelDataFileResolver::Create(argv[0]); + if (!resolver_status.ok()) { + std::cerr << "Failed to create data file resolver: " + << resolver_status.status() << std::endl; + return -1; + } + auto resolver = std::move(*resolver_status); + + auto plan = CreateSegmentationPlan(font->get(), resolver); + if (!plan.ok()) { + std::cerr << plan.status().message() << std::endl; + return -1; + } + + std::string font_name = std::filesystem::path(input_font_flag).stem().string(); + if (font_name.empty()) { + font_name = "out"; + } + + UnicodeRangeCompiler compiler; + compiler.SetFace(font->get()); + compiler.SetFontName(font_name); + compiler.SetFontFamily(absl::GetFlag(FLAGS_font_family)); + compiler.SetWoff2Encode(absl::GetFlag(FLAGS_woff2_encode)); + + auto sc = ConfigCompiler::Configure(*plan, compiler); + if (!sc.ok()) { + std::cerr << "Failed to apply configuration to the compiler: " << sc + << std::endl; + return -1; + } + + std::cout << ">> compiling subsets:" << std::endl; + auto encoding = compiler.Compile(); + if (!encoding.ok()) { + std::cerr << "Compilation failed: " << encoding.status() << std::endl; + return -1; + } + + std::cout << ">> writing output files:" << std::endl; + auto status = write_output(*encoding); + if (!status.ok()) { + std::cerr << status.message() << std::endl; + return -1; + } + + return 0; +} diff --git a/util/gen_ift_segmentation_plan.cc b/util/gen_ift_segmentation_plan.cc index cbe378ad..2f226424 100644 --- a/util/gen_ift_segmentation_plan.cc +++ b/util/gen_ift_segmentation_plan.cc @@ -121,8 +121,12 @@ static StatusOr LoadConfig(hb_face_t* font, if (!absl::GetFlag(FLAGS_auto_config_primary_script).empty()) { primary_script = absl::GetFlag(FLAGS_auto_config_primary_script); } - return AutoSegmenterConfig::GenerateConfig( - font, resolver, primary_script, quality_level); + ift::config::TargetMode target_mode = + absl::GetFlag(FLAGS_auto_config_unicode_range) + ? ift::config::UNICODE_RANGE + : ift::config::IFT; + return AutoSegmenterConfig::GenerateConfig(font, resolver, primary_script, + quality_level, target_mode); } FontData config_text = @@ -143,6 +147,7 @@ StatusOr LoadFont(const char* filename) { static Status Analysis(hb_face_t* font, btree_map&& merge_groups, const GlyphSegmentation& segmentation, + ift::config::TargetMode target_mode, std::shared_ptr resolver) { // A strategy may have more than one probability profile associated with it, // each one is analyzed separately. @@ -197,7 +202,8 @@ static Status Analysis(hb_face_t* font, } } - ClosureGlyphSegmenter segmenter(11, 11, PATCH, CLOSURE_ONLY, resolver); + ClosureGlyphSegmenter segmenter(11, 11, PATCH, CLOSURE_ONLY, resolver, + target_mode); auto costs = TRY(segmenter.TotalCosts(font, segmentation, strategy_probability_calculators)); @@ -309,7 +315,8 @@ static Status Main(const std::vector args) { config.brotli_quality(), config.brotli_quality_for_initial_font_merging(), config.unmapped_glyph_handling(), config.condition_analysis_mode(), - resolver); + resolver, config.target_mode(), + config.disjoint_conditions_probability_threshold()); TRYV(OutputFallbackGlyphCount(font.get(), segmenter, segmentation)); } @@ -319,7 +326,7 @@ static Status Main(const std::vector args) { std::cerr << ">> Analysis" << std::endl; return Analysis(font.get(), std::move(result.merge_groups), segmentation, - resolver); + config.target_mode(), resolver); } int main(int argc, char** argv) { diff --git a/util/gen_ift_segmenter_config.cc b/util/gen_ift_segmenter_config.cc index 35b79473..879d2cbf 100644 --- a/util/gen_ift_segmenter_config.cc +++ b/util/gen_ift_segmenter_config.cc @@ -35,6 +35,10 @@ ABSL_FLAG(int, quality, 0, "The quality level to use. A value of 0 means auto pick. Valid " "values are 1-8."); +ABSL_FLAG(bool, unicode_range, false, + "If true, generate a segmenter config with target_mode set to " + "UNICODE_RANGE instead of IFT."); + ABSL_FLAG( int, verbosity, 0, "Log verbosity level from. 0 is least verbose, higher values are more."); @@ -60,9 +64,11 @@ static Status Main(const std::vector args) { if (!absl::GetFlag(FLAGS_primary_script).empty()) { primary_script = absl::GetFlag(FLAGS_primary_script); } + ift::config::TargetMode target_mode = absl::GetFlag(FLAGS_unicode_range) + ? ift::config::UNICODE_RANGE + : ift::config::IFT; auto config = TRY(AutoSegmenterConfig::GenerateConfig( - font.get(), *resolver, primary_script, - quality_level)); + font.get(), *resolver, primary_script, quality_level, target_mode)); std::string output; if (!google::protobuf::TextFormat::PrintToString(config, &output)) { @@ -79,7 +85,7 @@ int main(int argc, char** argv) { "Generates a segmenter config to use with a font.\n" "\n" "Usage: gen_ift_segmenter_config --font=\"myfont.ttf\" [--quality=n] " - "[--primary_script=Script_foo]\n"); + "[--primary_script=Script_foo] [--unicode_range]\n"); absl::SetStderrThreshold(absl::LogSeverityAtLeast::kInfo); absl::SetGlobalVLogLevel(absl::GetFlag(FLAGS_verbosity)); auto args = absl::ParseCommandLine(argc, argv);