From a6c67272b18acf88d681313b828773c33f64a66e Mon Sep 17 00:00:00 2001 From: Garret Rieger Date: Fri, 13 Nov 2026 20:19:33 +0000 Subject: [PATCH 1/3] Deprecate and remove closure based condition analysis. Removes all code related to supporting closure baesd condition analysis (and related unmapped glyph handling). Dep graph based condition analysis is much faster and more accurate due to it's ability to find composite conditions, so there isn't really a need to keep these around anymore. --- .../unicode_range_segmentation.md | 13 +- docs/segmenter.md | 12 +- ift/config/auto_segmenter_config.cc | 23 +- ift/config/auto_segmenter_config_test.cc | 30 +- ift/config/common.proto | 16 +- ift/config/segmenter_config.proto | 35 +- ift/config/segmenter_config_util.cc | 23 +- ift/config/segmenter_config_util_test.cc | 100 +++- ift/dep_graph/dependency_graph_test.cc | 8 +- ift/encoder/BUILD | 29 - ift/encoder/activation_condition.cc | 16 +- ift/encoder/activation_condition.h | 14 +- ift/encoder/activation_condition_test.cc | 5 +- ift/encoder/all_pairs_merger.cc | 22 +- ift/encoder/candidate_merge.cc | 266 ++------- ift/encoder/candidate_merge.h | 5 - ift/encoder/candidate_merge_test.cc | 397 ++++++------- ift/encoder/closure_glyph_segmenter.cc | 143 +---- ift/encoder/closure_glyph_segmenter.h | 16 - ift/encoder/closure_glyph_segmenter_test.cc | 541 ++++-------------- ift/encoder/complex_condition_finder.cc | 324 ----------- ift/encoder/complex_condition_finder.h | 34 -- ift/encoder/complex_condition_finder_test.cc | 301 ---------- ift/encoder/dependency_closure.cc | 184 +----- ift/encoder/dependency_closure.h | 80 +-- ift/encoder/dependency_closure_test.cc | 238 ++------ ift/encoder/glyph_closure_cache.cc | 78 --- ift/encoder/glyph_closure_cache.h | 12 - ift/encoder/glyph_closure_cache_test.cc | 61 +- ift/encoder/glyph_groupings.cc | 198 +------ ift/encoder/glyph_groupings.h | 45 +- ift/encoder/glyph_groupings_test.cc | 403 ++----------- ift/encoder/glyph_segmentation.cc | 7 +- ift/encoder/glyph_segmentation.h | 20 +- ift/encoder/glyph_segmentation_test.cc | 14 +- ift/encoder/merger.cc | 16 +- .../requested_segmentation_information.cc | 19 +- .../requested_segmentation_information.h | 14 +- ...requested_segmentation_information_test.cc | 10 +- ift/encoder/segmentation_context.cc | 215 +------ ift/encoder/segmentation_context.h | 125 +--- ift/encoder/unicode_range_compiler_test.cc | 6 +- .../count-fallback-glyphs-for-font.sh | 11 - .../count-fallback-glyphs.sh | 19 - .../fallback_count_config.txtpb | 19 - util/gen_ift_segmentation_plan.cc | 41 +- 46 files changed, 703 insertions(+), 3505 deletions(-) delete mode 100644 ift/encoder/complex_condition_finder.cc delete mode 100644 ift/encoder/complex_condition_finder.h delete mode 100644 ift/encoder/complex_condition_finder_test.cc delete mode 100755 util/fallback_analysis/count-fallback-glyphs-for-font.sh delete mode 100755 util/fallback_analysis/count-fallback-glyphs.sh delete mode 100644 util/fallback_analysis/fallback_count_config.txtpb diff --git a/docs/experimental/unicode_range_segmentation.md b/docs/experimental/unicode_range_segmentation.md index 565bc91a..a26d9d16 100644 --- a/docs/experimental/unicode_range_segmentation.md +++ b/docs/experimental/unicode_range_segmentation.md @@ -16,7 +16,6 @@ With `@font-face` `unicode-range`, a font is split into a collection of standalo | **Client Loading Model** | Single font instance incrementally extended by patches | Independent `@font-face` WOFF2 font files matched per character by `unicode-range` | | **Initial Segment (`config.initial_segment`)** | Loaded once as the base font; initial font merging moves high-probability patches into it | Included in **every** subset (union of `initial_segment` codepoints/glyphs/features + default features and the patch's glyphs; useful for UVS codepoints, etc.). Initial font merging is disabled | | **Layout Features** | Optional feature patches + table-keyed patches | Codepoints only; each subset statically retains the default layout features plus any explicitly requested in `config.initial_segment.features`. Optional feature segments and table-keyed segments are disallowed | -| **Unmapped Glyphs** | Configurable (defaults to `MOVE_TO_INIT_FONT`) | Must be `FIND_CONDITIONS` or `PATCH` (`AutoSegmenterConfig` sets `FIND_CONDITIONS`; `MOVE_TO_INIT_FONT` is rejected) | | **Patch Merging** | Optional (`experimental_use_patch_merges`) | Disabled; only segment merges are performed | | **Interacting Glyphs & Disjoint Conditions** | Can be placed in separate patches (e.g. `fi` in patch `f AND i`, shared component `A` in `A OR À`) | Determined via dependency graph / closure analysis; `MakeActivationConditionsDisjoint` merges triggering segments of non-exclusive conditions whose probability $\ge$ `disjoint_conditions_probability_threshold`. Conditions below the threshold are skipped | | **Final Subsets & Activation Conditions** | CNF expressions (`(s1 OR s2) AND s3`) | Only **exclusive** patches (`(s_i)`) are emitted in the final plan (any remaining non-exclusive patches are ignored), yielding strictly disjunctive and mutually disjoint `unicode-range` sets | @@ -29,16 +28,14 @@ With `@font-face` `unicode-range`, a font is split into a collection of standalo A `TargetMode` enum (`IFT`, `UNICODE_RANGE`) in [segmenter_config.proto](../ift/config/segmenter_config.proto) selects the segmentation target mode (`target_mode` on `SegmenterConfig`, defaulting to `IFT`), alongside a `disjoint_conditions_probability_threshold` setting. The auto-configuration generator (`AutoSegmenterConfig::GenerateConfig`) also accepts `TargetMode` and automatically configures appropriate defaults when `UNICODE_RANGE` is selected: * Disables optional feature segment generation (`generate_feature_segments = false`) and table-keyed segment generation (`generate_table_keyed_segments = false`). -* Sets `unmapped_glyph_handling = FIND_CONDITIONS` so all glyphs in the full closure receive activation conditions. * Leaves initial font merging thresholds unset and skips IFT-specific base segmentation plan settings (`jump_ahead`, `use_prefetch_lists`). To ensure invalid configurations are caught regardless of whether a caller goes through `SegmenterConfigUtil` or invokes `ClosureGlyphSegmenter` directly, `ClosureGlyphSegmenter` validates that when `target_mode == UNICODE_RANGE`: -1. `unmapped_glyph_handling` is set to either `FIND_CONDITIONS` or `PATCH` (`MOVE_TO_INIT_FONT` is rejected). -2. No input segment definitions contain layout features (optional feature loading is unsupported). -3. Initial font merging is not enabled on any merge group. -4. Patch merging is disabled on all merge groups. -5. `generate_table_keyed_segments` is not enabled. +1. No input segment definitions contain layout features (optional feature loading is unsupported). +2. Initial font merging is not enabled on any merge group. +3. Patch merging is disabled on all merge groups. +4. `generate_table_keyed_segments` is not enabled. ### Initial Segment Handling @@ -57,7 +54,7 @@ In CSS `@font-face`, each subset is treated as a separate font face during text ### `MakeActivationConditionsDisjoint` -After initial condition analysis (and unmapped glyph condition finding when `FIND_CONDITIONS` is used), the segmenter runs a `MakeActivationConditionsDisjoint` pass before starting general cost-based or heuristic merging: +After initial condition analysis, the segmenter runs a `MakeActivationConditionsDisjoint` pass before starting general cost-based or heuristic merging: 1. Inspect all non-exclusive patch conditions (conditions involving more than one triggering segment, whether conjunctive or disjunctive). 2. Group segments into connected components by unioning the triggering segments of each non-exclusive condition. diff --git a/docs/segmenter.md b/docs/segmenter.md index 34baefb9..f085ba57 100644 --- a/docs/segmenter.md +++ b/docs/segmenter.md @@ -85,15 +85,9 @@ In the proceeding sections each of these steps will be discussed in more details ## Condition Analysis -Detecting the conditions under which a glyph is needed can be done in a couple of different -ways. The segmentation implementation currently has two main approaches: +Condition analysis infers the activation conditions under which each glyph in the font is needed by analyzing a [glyph dependency graph](dependency_graph.md) and running [dependency graph condition extraction](dependency_graph_condition_extract.md) (previously, an experimental font-subsetting closure approach was also supported; that approach has been removed). -1. Infer glyph conditions via [font subsetting closure](experimental/closure_glyph_segmentation.md). -2. Infer glyph conditions via a [glyph dependency graph](dependency_graph.md) and [dependency graph condition extraction](dependency_graph_condition_extract.md). - -Both approaches ultimately aim to detect conditions that each glyph is needed which will satisfy the glyph closure -requirement. The dependency graph is the default and recommended method for finding conditions. It's generally faster -and produces more detailed conditions. +Using the dependency graph generates detailed activation conditions that satisfy the glyph closure requirement, and the final segmentation is validated against HarfBuzz subsetting closure. ## Merging @@ -373,8 +367,6 @@ In the merger implementation, actual cost delta computation is optimized by firs *aggregate probability* $P_{\text{agg}}(c_i) = \sum_{m=1}^M P_m(c_i)$ of a condition across all frequency data sets in the merge group, and then multiplying that sum once by $(\text{size}(p_i) + k)$ for each affected patch. This avoids redundant patch size evaluations and per-data-set delta loops. -Similarly, because the fallback patch is needed with probability $1.0$ under every frequency data set, -its size delta is multiplied by the number of data sets $M$. ### Prioritizing a Primary Script and Language diff --git a/ift/config/auto_segmenter_config.cc b/ift/config/auto_segmenter_config.cc index f44a597b..3bb13534 100644 --- a/ift/config/auto_segmenter_config.cc +++ b/ift/config/auto_segmenter_config.cc @@ -114,10 +114,6 @@ enum Quality { // To start, the list of parameters we can use to make quality/performance // tradeoffs: // -// - unmapped_glyph_handling (global) -// Lower quality is to not find conditions (so use patch or init font), high -// quality is to find conditions. -// // - generate_feature_segments (global): high quality generate segment per // feature, low quality put all optional features // in one segment. @@ -135,10 +131,10 @@ enum Quality { // Would be reasonable to always have this set to at least the minimum group // size. // -// - condition_analysis_mode: use DEP_GRAPH_ONLY if dependency API is available, -// otherwise CLOSURE_ONLY. DEP_GRAPH_ONLY_WITH_SIMPLIFICATION can significantly -// speed up cases with high condition complexity, at the cost of less specific -// final conditions. +// - condition_analysis_mode: use DEP_GRAPH_ONLY or +// DEP_GRAPH_ONLY_WITH_SIMPLIFICATION. DEP_GRAPH_ONLY_WITH_SIMPLIFICATION can +// significantly speed up cases with high condition complexity, at the cost of +// less specific final conditions. // // Merge group settings: // @@ -403,10 +399,9 @@ static Status GroupInteractingScripts( std::unique_ptr segmentation_info = TRY(RequestedSegmentationInformation::Create( - std::move(segments), init_segment, *closure_cache, FIND_CONDITIONS)); + std::move(segments), init_segment, *closure_cache)); std::unique_ptr dependency_closure = - TRY(DependencyClosure::Create(segmentation_info.get(), face, resolver, - /*allow_context_glyph_analysis=*/true)); + TRY(DependencyClosure::Create(segmentation_info.get(), face, resolver)); GlyphPartition ungrouped_partition(ungrouped_list.size()); flat_hash_map ungrouped_to_scripts; @@ -958,12 +953,6 @@ static void ApplyQualityLevelTo(Quality quality, MergeGroup& merge_group) { static void ApplyQualityLevelTo(Quality quality, SegmenterConfig& config) { config.set_preprocess_merging_group_size_for_ungrouped(kMinimumGroupSize * 3); - if (config.target_mode() == UNICODE_RANGE) { - config.set_unmapped_glyph_handling(FIND_CONDITIONS); - } else { - config.set_unmapped_glyph_handling(MOVE_TO_INIT_FONT); - } - if (quality == ONE) { config.set_condition_analysis_mode(DEP_GRAPH_ONLY_WITH_SIMPLIFICATION); } else { diff --git a/ift/config/auto_segmenter_config_test.cc b/ift/config/auto_segmenter_config_test.cc index 068b890a..6b53be1f 100644 --- a/ift/config/auto_segmenter_config_test.cc +++ b/ift/config/auto_segmenter_config_test.cc @@ -20,9 +20,7 @@ #include "ift/common/test_font_loader.h" #include "ift/config/load_codepoints.h" -using ift::config::CLOSURE_AND_DEP_GRAPH; using ift::config::FIND_CONDITIONS; -using ift::config::MOVE_TO_INIT_FONT; using ift::config::SegmenterConfig; namespace ift::config { @@ -173,8 +171,7 @@ TEST_F(AutoSegmenterConfigTest, Roboto_UnspecifiedPrimary) { std::string config_string; TextFormat::PrintToString(*config_or, &config_string); - ASSERT_EQ(config_string, R"(unmapped_glyph_handling: MOVE_TO_INIT_FONT -brotli_quality: 11 + ASSERT_EQ(config_string, R"(brotli_quality: 11 brotli_quality_for_initial_font_merging: 9 base_heuristic_config { min_patch_size: 2500 @@ -549,12 +546,23 @@ static std::vector GetUseBigrams(const SegmenterConfig& config) { return result; } +#pragma GCC diagnostic push +#pragma GCC diagnostic ignored "-Wdeprecated-declarations" +bool HasUnmappedGlyphHandling(const SegmenterConfig& config) { + return config.has_unmapped_glyph_handling(); +} +UnmappedGlyphHandling GetUnmappedGlyphHandling(const SegmenterConfig& config) { + return config.unmapped_glyph_handling(); +} +#pragma GCC diagnostic pop + TEST_F(AutoSegmenterConfigTest, QualityLevelForcing) { auto config_or = AutoSegmenterConfig::GenerateConfig(face_.get(), *resolver, std::nullopt, 1); ASSERT_TRUE(config_or.ok()) << config_or.status(); EXPECT_EQ(config_or->brotli_quality(), 0); - EXPECT_EQ(config_or->unmapped_glyph_handling(), MOVE_TO_INIT_FONT); + EXPECT_FALSE(HasUnmappedGlyphHandling(*config_or)); + EXPECT_EQ(GetUnmappedGlyphHandling(*config_or), FIND_CONDITIONS); EXPECT_THAT(GetUseBigrams(*config_or), Each(false)); EXPECT_EQ(config_or->brotli_quality_for_initial_font_merging(), 0); EXPECT_EQ(config_or->base_cost_config().optimization_cutoff_fraction(), 0.05); @@ -563,7 +571,8 @@ TEST_F(AutoSegmenterConfigTest, QualityLevelForcing) { std::nullopt, 7); ASSERT_TRUE(config_or_8.ok()) << config_or_8.status(); EXPECT_EQ(config_or_8->brotli_quality(), 11); - EXPECT_EQ(config_or_8->unmapped_glyph_handling(), MOVE_TO_INIT_FONT); + EXPECT_FALSE(HasUnmappedGlyphHandling(*config_or_8)); + EXPECT_EQ(GetUnmappedGlyphHandling(*config_or_8), FIND_CONDITIONS); EXPECT_THAT(GetUseBigrams(*config_or_8), Each(true)); EXPECT_EQ(config_or_8->brotli_quality_for_initial_font_merging(), 11); EXPECT_EQ(config_or_8->base_cost_config().optimization_cutoff_fraction(), @@ -707,8 +716,7 @@ TEST_F(AutoSegmenterConfigTest, Roboto_UnspecifiedPrimary_UnicodeRange) { // - Ignored for overlap codepoints present in multiple groups (ie. 0x20) are moved to exactly one group. // - This in turn requires explicit segment definitions for affected groups. // - Combining marks are not ignored for overlap detection so latin + cyrillic are grouped. - ASSERT_EQ(config_string, R"(unmapped_glyph_handling: FIND_CONDITIONS -brotli_quality: 11 + ASSERT_EQ(config_string, R"(brotli_quality: 11 base_heuristic_config { min_patch_size: 2500 max_patch_size: 20000 @@ -776,7 +784,8 @@ TEST_F(AutoSegmenterConfigTest, UnicodeRange_QualityLevels) { ASSERT_TRUE(config_q1.ok()) << config_q1.status(); EXPECT_EQ(config_q1->target_mode(), UNICODE_RANGE); EXPECT_EQ(config_q1->brotli_quality(), 0); - EXPECT_EQ(config_q1->unmapped_glyph_handling(), FIND_CONDITIONS); + EXPECT_FALSE(HasUnmappedGlyphHandling(*config_q1)); + EXPECT_EQ(GetUnmappedGlyphHandling(*config_q1), FIND_CONDITIONS); EXPECT_FALSE(config_q1->has_brotli_quality_for_initial_font_merging()); EXPECT_FALSE(config_q1->base_cost_config().experimental_use_patch_merges()); EXPECT_THAT(GetUseBigrams(*config_q1), Each(false)); @@ -787,7 +796,8 @@ TEST_F(AutoSegmenterConfigTest, UnicodeRange_QualityLevels) { ASSERT_TRUE(config_q7.ok()) << config_q7.status(); EXPECT_EQ(config_q7->target_mode(), UNICODE_RANGE); EXPECT_EQ(config_q7->brotli_quality(), 11); - EXPECT_EQ(config_q7->unmapped_glyph_handling(), FIND_CONDITIONS); + EXPECT_FALSE(HasUnmappedGlyphHandling(*config_q7)); + EXPECT_EQ(GetUnmappedGlyphHandling(*config_q7), FIND_CONDITIONS); EXPECT_FALSE(config_q7->has_brotli_quality_for_initial_font_merging()); EXPECT_FALSE(config_q7->base_cost_config().experimental_use_patch_merges()); EXPECT_THAT(GetUseBigrams(*config_q7), Each(true)); diff --git a/ift/config/common.proto b/ift/config/common.proto index 0d51abcb..a86b0466 100644 --- a/ift/config/common.proto +++ b/ift/config/common.proto @@ -38,15 +38,17 @@ message AxisRangeProto { float end = 2; } -// This enum is used to specify how unmapped glyphs are handled in the produced segmentation. +// Deprecated: unmapped glyph handling is only used by removed closure analysis +// modes. Specifying anything other than FIND_CONDITIONS will result in an +// error. enum UnmappedGlyphHandling { - // All unmapped glyphs will be placed into a patch which is always loaded. - PATCH = 0; + // Deprecated: specifying this value will result in an error. + PATCH = 0 [deprecated = true]; - // All unmapped glyphs will be moved into the initial font. - MOVE_TO_INIT_FONT = 1; + // Deprecated: specifying this value will result in an error. + MOVE_TO_INIT_FONT = 1 [deprecated = true]; - // Advanced condition analysis will be used (expensive) to discover the conditions - // for unmapped glyphs. + // Default value; triggers no additional behavior in dependency graph + // condition analysis modes. FIND_CONDITIONS = 2; } \ No newline at end of file diff --git a/ift/config/segmenter_config.proto b/ift/config/segmenter_config.proto index d5df9900..f15361a8 100644 --- a/ift/config/segmenter_config.proto +++ b/ift/config/segmenter_config.proto @@ -6,23 +6,14 @@ import "ift/config/segmentation_plan.proto"; import "ift/config/common.proto"; enum ConditionAnalysisMode { - // [DEPRECATED] - // Analyzes glyph conditions using closure analysis only. - // This is a legacy mode, it's much slower and generally produces - // worse results then the dependency graph approach. - CLOSURE_ONLY = 0; - - // Attempts to analyze glyph conditions using a dependency - // graph where possible. When dep graph is expected not to be accurate - // it falls back to closure analysis. This is for many cases much faster - // then pure closure based analysis. - CLOSURE_AND_DEP_GRAPH = 1; - - // Always runs both closure and dependency graph analysis, the - // results are compared and an error is raised if they differ. - // Useful primarily for testing the accuracy of the dependency - // graph approach. - CLOSURE_AND_VALIDATE_DEP_GRAPH = 2; + // Deprecated: specifying this mode will result in an error. + CLOSURE_ONLY = 0 [deprecated = true]; + + // Deprecated: specifying this mode will result in an error. + CLOSURE_AND_DEP_GRAPH = 1 [deprecated = true]; + + // Deprecated: specifying this mode will result in an error. + CLOSURE_AND_VALIDATE_DEP_GRAPH = 2 [deprecated = true]; // [RECOMMENDED] // Analyze glyph conditions using dep graph only. Closure is @@ -112,12 +103,10 @@ message SegmenterConfig { // are included in all subsets. SegmentProto initial_segment = 1; - // Specifies what to do with unmapped glyphs. Unmapped glyphs are glyphs in which the analysis - // is unable to determine activation conditions for. - // - // When target_mode is UNICODE_RANGE, this must be set to either FIND_CONDITIONS or PATCH - // (MOVE_TO_INIT_FONT is invalid). - UnmappedGlyphHandling unmapped_glyph_handling = 2 [default = MOVE_TO_INIT_FONT]; + // Deprecated: only FIND_CONDITIONS is valid; specifying any other value will + // result in an error. + UnmappedGlyphHandling unmapped_glyph_handling = 2 + [default = FIND_CONDITIONS, deprecated = true]; // Deprecated: use generate_table_keyed_segments_mode instead. // diff --git a/ift/config/segmenter_config_util.cc b/ift/config/segmenter_config_util.cc index fec939dc..1caf89aa 100644 --- a/ift/config/segmenter_config_util.cc +++ b/ift/config/segmenter_config_util.cc @@ -176,6 +176,22 @@ static StatusOr ResolveTableKeyedSegmentMode( return config.generate_table_keyed_segments_mode(); } +static absl::Status ValidateConditionAnalysisAndUnmappedGlyphs( + const SegmenterConfig& config) { + if (config.condition_analysis_mode() != DEP_GRAPH_ONLY && + config.condition_analysis_mode() != DEP_GRAPH_ONLY_WITH_SIMPLIFICATION) { + return absl::InvalidArgumentError( + "Closure-based condition_analysis_mode values are deprecated and " + "unsupported; use DEP_GRAPH_ONLY or " + "DEP_GRAPH_ONLY_WITH_SIMPLIFICATION."); + } + if (config.unmapped_glyph_handling() != FIND_CONDITIONS) { + return absl::InvalidArgumentError( + "unmapped_glyph_handling is deprecated and must be FIND_CONDITIONS."); + } + return absl::OkStatus(); +} + #pragma GCC diagnostic pop StatusOr @@ -355,6 +371,8 @@ SegmenterConfigUtil::ConfigToMergeGroups( const SegmenterConfig& config, const CodepointSet& font_codepoints, const btree_set& font_features, std::vector& segments) { + TRYV(ValidateConditionAnalysisAndUnmappedGlyphs(config)); + if (config.target_mode() == UNICODE_RANGE && (!config.feature_segments().empty() || config.generate_feature_segments())) { @@ -410,6 +428,8 @@ SegmenterConfigUtil::ConfigToMergeGroups( StatusOr SegmenterConfigUtil::RunSegmenter( hb_face_t* face, const SegmenterConfig& config) { + TRYV(ValidateConditionAnalysisAndUnmappedGlyphs(config)); + TableKeyedSegmentMode table_keyed_mode = TRY(ResolveTableKeyedSegmentMode(config)); if (config.target_mode() == UNICODE_RANGE && table_keyed_mode != NONE) { @@ -429,8 +449,7 @@ StatusOr SegmenterConfigUtil::RunSegmenter( ClosureGlyphSegmenter segmenter( config.brotli_quality(), config.brotli_quality_for_initial_font_merging(), - config.unmapped_glyph_handling(), config.condition_analysis_mode(), - resolver_, config.target_mode(), + config.condition_analysis_mode(), resolver_, config.target_mode(), config.disjoint_conditions_probability_threshold()); GlyphSegmentation segmentation = TRY(segmenter.CodepointToGlyphSegments( diff --git a/ift/config/segmenter_config_util_test.cc b/ift/config/segmenter_config_util_test.cc index 9814a2de..e6e104ca 100644 --- a/ift/config/segmenter_config_util_test.cc +++ b/ift/config/segmenter_config_util_test.cc @@ -84,6 +84,16 @@ void SetLegacyGenerateTableKeyedSegments(SegmenterConfig& config, bool value) { config.set_generate_table_keyed_segments(value); } +void SetLegacyUnmappedGlyphHandling(SegmenterConfig& config, + ift::config::UnmappedGlyphHandling value) { + config.set_unmapped_glyph_handling(value); +} + +void SetLegacyConditionAnalysisMode(SegmenterConfig& config, + ift::config::ConditionAnalysisMode value) { + config.set_condition_analysis_mode(value); +} + #pragma GCC diagnostic pop MergeStrategy ExpectedCostStrategy( @@ -960,7 +970,6 @@ TEST_F(SegmenterConfigUtilTest, SegmenterConfig config; config.set_target_mode(ift::config::UNICODE_RANGE); - config.set_unmapped_glyph_handling(ift::config::FIND_CONDITIONS); config.set_generate_table_keyed_segments_mode(FROM_FREQ_DATA); AddSegment(config, 1, {0x43}); @@ -983,7 +992,6 @@ TEST_F(SegmenterConfigUtilTest, { SegmenterConfig config; config.set_target_mode(ift::config::UNICODE_RANGE); - config.set_unmapped_glyph_handling(ift::config::FIND_CONDITIONS); config.set_generate_feature_segments(true); AddSegment(config, 1, {0x43}); @@ -999,7 +1007,6 @@ TEST_F(SegmenterConfigUtilTest, { SegmenterConfig config; config.set_target_mode(ift::config::UNICODE_RANGE); - config.set_unmapped_glyph_handling(ift::config::FIND_CONDITIONS); Features features; features.add_values("smcp"); (*config.mutable_feature_segments())[1] = features; @@ -1012,7 +1019,6 @@ TEST_F(SegmenterConfigUtilTest, { SegmenterConfig config; config.set_target_mode(ift::config::UNICODE_RANGE); - config.set_unmapped_glyph_handling(ift::config::FIND_CONDITIONS); AddSegment(config, 1, {0x43}); (*config.mutable_segments())[1].mutable_features()->add_values("smcp"); @@ -1028,21 +1034,26 @@ TEST_F(SegmenterConfigUtilTest, loader->LoadFace("ift/common/testdata/Roboto-Regular.ttf").value(); SegmenterConfigUtil util("util/testdata/config.txtpb", resolver); - // Default unmapped_glyph_handling (MOVE_TO_INIT_FONT) is rejected. + // Explicit MOVE_TO_INIT_FONT or PATCH is rejected. { - SegmenterConfig config; - config.set_target_mode(ift::config::UNICODE_RANGE); - AddSegment(config, 1, {0x43}); - - auto result = util.RunSegmenter(face.get(), config); - EXPECT_TRUE(absl::IsInvalidArgument(result.status())) << result.status(); +#pragma GCC diagnostic push +#pragma GCC diagnostic ignored "-Wdeprecated-declarations" + for (auto handling : {ift::config::MOVE_TO_INIT_FONT, ift::config::PATCH}) { +#pragma GCC diagnostic pop + SegmenterConfig config; + config.set_target_mode(ift::config::UNICODE_RANGE); + SetLegacyUnmappedGlyphHandling(config, handling); + AddSegment(config, 1, {0x43}); + + auto result = util.RunSegmenter(face.get(), config); + EXPECT_TRUE(absl::IsInvalidArgument(result.status())) << result.status(); + } } // Initial font merging is rejected. { SegmenterConfig config; config.set_target_mode(ift::config::UNICODE_RANGE); - config.set_unmapped_glyph_handling(ift::config::FIND_CONDITIONS); AddSegment(config, 1, {0x43}); auto* group = config.add_merge_groups(); AddFreqData(*group, "test_freq_data.riegeli"); @@ -1058,7 +1069,6 @@ TEST_F(SegmenterConfigUtilTest, { SegmenterConfig config; config.set_target_mode(ift::config::UNICODE_RANGE); - config.set_unmapped_glyph_handling(ift::config::FIND_CONDITIONS); AddSegment(config, 1, {0x43}); auto* group = config.add_merge_groups(); AddFreqData(*group, "test_freq_data.riegeli"); @@ -1072,7 +1082,6 @@ TEST_F(SegmenterConfigUtilTest, { SegmenterConfig config; config.set_target_mode(ift::config::UNICODE_RANGE); - config.set_unmapped_glyph_handling(ift::config::FIND_CONDITIONS); config.set_brotli_quality(0); config.set_brotli_quality_for_initial_font_merging(0); AddSegment(config, 1, {0x43}); @@ -1085,4 +1094,67 @@ TEST_F(SegmenterConfigUtilTest, } } +TEST_F(SegmenterConfigUtilTest, + RunSegmenter_RejectsDeprecatedConditionAnalysisModes) { + auto loader = TestFontLoader::Default().value(); + auto face = + loader->LoadFace("ift/common/testdata/Roboto-Regular.ttf").value(); + SegmenterConfigUtil util("util/testdata/config.txtpb", resolver); + +#pragma GCC diagnostic push +#pragma GCC diagnostic ignored "-Wdeprecated-declarations" + for (auto mode : + {ift::config::CLOSURE_ONLY, ift::config::CLOSURE_AND_DEP_GRAPH, + ift::config::CLOSURE_AND_VALIDATE_DEP_GRAPH}) { +#pragma GCC diagnostic pop + SegmenterConfig config; + SetLegacyConditionAnalysisMode(config, mode); + AddSegment(config, 1, {0x43}); + + std::vector segments_out; + auto groups = util.ConfigToMergeGroups(config, {0x43}, {}, segments_out); + EXPECT_TRUE(absl::IsInvalidArgument(groups.status())) << groups.status(); + + auto result = util.RunSegmenter(face.get(), config); + EXPECT_TRUE(absl::IsInvalidArgument(result.status())) << result.status(); + } +} + +TEST_F(SegmenterConfigUtilTest, + RunSegmenter_RejectsInvalidUnmappedGlyphHandling) { + auto loader = TestFontLoader::Default().value(); + auto face = + loader->LoadFace("ift/common/testdata/Roboto-Regular.ttf").value(); + SegmenterConfigUtil util("util/testdata/config.txtpb", resolver); + +#pragma GCC diagnostic push +#pragma GCC diagnostic ignored "-Wdeprecated-declarations" + for (auto handling : {ift::config::PATCH, ift::config::MOVE_TO_INIT_FONT}) { +#pragma GCC diagnostic pop + SegmenterConfig config; + SetLegacyUnmappedGlyphHandling(config, handling); + AddSegment(config, 1, {0x43}); + + std::vector segments_out; + auto groups = util.ConfigToMergeGroups(config, {0x43}, {}, segments_out); + EXPECT_TRUE(absl::IsInvalidArgument(groups.status())) << groups.status(); + + auto result = util.RunSegmenter(face.get(), config); + EXPECT_TRUE(absl::IsInvalidArgument(result.status())) << result.status(); + } + + // Default (unset) and explicit FIND_CONDITIONS succeed. + { + SegmenterConfig config; + config.set_brotli_quality(0); + AddSegment(config, 1, {0x43}); + auto result = util.RunSegmenter(face.get(), config); + ASSERT_TRUE(result.ok()) << result.status(); + + SetLegacyUnmappedGlyphHandling(config, ift::config::FIND_CONDITIONS); + result = util.RunSegmenter(face.get(), config); + ASSERT_TRUE(result.ok()) << result.status(); + } +} + // TODO test for feature segment auto generation. \ No newline at end of file diff --git a/ift/dep_graph/dependency_graph_test.cc b/ift/dep_graph/dependency_graph_test.cc index 01721b64..d7d7d7f0 100644 --- a/ift/dep_graph/dependency_graph_test.cc +++ b/ift/dep_graph/dependency_graph_test.cc @@ -17,8 +17,6 @@ #include "ift/encoder/types.h" #include "ift/freq/probability_bound.h" -using ift::config::PATCH; - using absl::btree_set; using absl::flat_hash_map; using absl::flat_hash_set; @@ -62,7 +60,7 @@ class DependencyGraphTest : public ::testing::Test { loader->LoadFace("ift/common/testdata/NotoSansKR[wght].subset.ttf") .value()), segmentation_info(*RequestedSegmentationInformation::Create( - segments, WithDefaultFeatures({}), *closure_cache, PATCH)), + segments, WithDefaultFeatures({}), *closure_cache)), graph(*DependencyGraph::Create(segmentation_info.get(), face.get(), *resolver)) {} @@ -75,7 +73,7 @@ class DependencyGraphTest : public ::testing::Test { void Reconfigure(SubsetDefinition new_init, std::vector new_segments) { segmentation_info = *RequestedSegmentationInformation::Create( - new_segments, new_init, *closure_cache, PATCH); + new_segments, new_init, *closure_cache); graph = *DependencyGraph::Create(segmentation_info.get(), face.get(), *resolver); } @@ -84,7 +82,7 @@ class DependencyGraphTest : public ::testing::Test { std::vector new_segments) { closure_cache = std::move(*GlyphClosureCache::Create(new_face, *resolver)); segmentation_info = *RequestedSegmentationInformation::Create( - new_segments, new_init, *closure_cache, PATCH); + new_segments, new_init, *closure_cache); graph = *DependencyGraph::Create(segmentation_info.get(), new_face, *resolver); } diff --git a/ift/encoder/BUILD b/ift/encoder/BUILD index 4698b15f..2eb8668b 100644 --- a/ift/encoder/BUILD +++ b/ift/encoder/BUILD @@ -80,7 +80,6 @@ cc_library( srcs = [ "all_pairs_merger.cc", "candidate_merge.cc", - "complex_condition_finder.cc", "dependency_closure.cc", "estimated_patch_size_cache.cc", "glyph_condition_set.cc", @@ -92,7 +91,6 @@ cc_library( hdrs = [ "all_pairs_merger.h", "candidate_merge.h", - "complex_condition_finder.h", "condition_to_glyphs_index.h", "dependency_closure.h", "estimated_patch_size_cache.h", @@ -158,7 +156,6 @@ cc_library( "//ift/common", "//ift/common:data_file_resolver", "//ift/common:try", - "//ift/config:common_cc_proto", "//ift/dep_graph:unicode_edges", "//ift/feature_registry", "@abseil-cpp//absl/container:flat_hash_map", @@ -380,31 +377,6 @@ cc_test( ], ) -cc_test( - name = "complex_condition_finder_test", - size = "medium", - srcs = [ - "complex_condition_finder_test.cc", - ], - data = [ - "//ift:testdata", - "//ift/common:testdata", - ], - deps = [ - ":common", - ":encoder", - ":segmentation_context", - ":segmentation_info", - "//ift/common", - "//ift/common:data_file_resolver", - "//ift/common:test_font_loader", - "//ift/freq:common", - "@abseil-cpp//absl/container:btree", - "@googletest//:gtest", - "@googletest//:gtest_main", - ], -) - cc_test( name = "closure_glyph_segmenter_test", size = "medium", @@ -538,7 +510,6 @@ cc_test( "//ift/common", "//ift/common:data_file_resolver", "//ift/common:test_font_loader", - "//ift/config:common_cc_proto", "//ift/dep_graph", "//ift/freq:common", "@bazel_tools//tools/cpp/runfiles", diff --git a/ift/encoder/activation_condition.cc b/ift/encoder/activation_condition.cc index 0c96c770..f7949b57 100644 --- a/ift/encoder/activation_condition.cc +++ b/ift/encoder/activation_condition.cc @@ -74,19 +74,16 @@ ActivationCondition ActivationCondition::and_segments( } ActivationCondition ActivationCondition::or_segments(const SegmentSet& segments, - patch_id_t activated, - bool is_fallback) { + patch_id_t activated) { ActivationCondition conditions; conditions.activated_ = {activated}; conditions.conditions_.push_back(segments); - conditions.is_fallback_ = is_fallback; return conditions; } ActivationCondition ActivationCondition::composite_condition( - absl::Span groups, patch_id_t activated, - bool is_fallback) { + absl::Span groups, patch_id_t activated) { ActivationCondition conditions; conditions.activated_ = {activated}; for (const auto& group : groups) { @@ -95,7 +92,6 @@ ActivationCondition ActivationCondition::composite_condition( Simplify(conditions.conditions_); - conditions.is_fallback_ = is_fallback; return conditions; } @@ -174,7 +170,6 @@ ActivationCondition ActivationCondition::Or(const ActivationCondition& a, // they must be combined in a commutative and associative way. Otherwise // folding Or() across a group of conditions would produce different results // depending on the order the conditions are visited in. - condition.is_fallback_ = a.is_fallback_ || b.is_fallback_; condition.is_exclusive_ = false; condition.activated_ = std::max(a.activated_, b.activated_); condition.encoding_ = std::max(a.encoding_, b.encoding_); @@ -297,7 +292,6 @@ ActivationCondition ActivationCondition::ReplaceSegments( } new_condition.activated_ = activated_; - new_condition.is_fallback_ = is_fallback_; new_condition.is_exclusive_ = is_exclusive_ || new_condition.conditions_.size() == 1; new_condition.encoding_ = encoding_; @@ -321,7 +315,6 @@ ActivationCondition ActivationCondition::RemoveIntersectingSubgroups( ActivationCondition new_condition = ActivationCondition::True(0); new_condition.activated_ = activated_; new_condition.encoding_ = encoding_; - new_condition.is_fallback_ = is_fallback_; new_condition.is_exclusive_ = is_exclusive_; for (const auto& condition_group : conditions_) { @@ -378,7 +371,6 @@ ActivationCondition ActivationCondition::NonCompositeSuperset() const { } ActivationCondition out; - out.is_fallback_ = is_fallback_; out.is_exclusive_ = is_exclusive_; out.conditions_ = {}; out.activated_ = activated_; @@ -472,10 +464,6 @@ bool ActivationCondition::operator<(const ActivationCondition& other) const { return is_exclusive_; } - if (is_fallback_ != other.is_fallback_) { - return !is_fallback_; - } - return encoding_ < other.encoding_; } diff --git a/ift/encoder/activation_condition.h b/ift/encoder/activation_condition.h index bd2713a5..9830fb6b 100644 --- a/ift/encoder/activation_condition.h +++ b/ift/encoder/activation_condition.h @@ -53,16 +53,14 @@ class ActivationCondition { * (segment_1) OR ... OR inersects(segment_n). */ static ActivationCondition or_segments(const ift::common::SegmentSet& ids, - patch_id_t activated, - bool is_fallback = false); + patch_id_t activated); /* * Constructs a condition that activates when the input intersects: * (s1 OR ..) AND (si OR ...) AND ... */ static ActivationCondition composite_condition( - absl::Span groups, patch_id_t activated, - bool is_fallback = false); + absl::Span groups, patch_id_t activated); /* * Constructs a new condition equal to 'condition' except with the is @@ -105,8 +103,6 @@ class ActivationCondition { bool IsAlwaysTrue() const { return conditions().size() == 0; } - bool IsFallback() const { return is_fallback_; } - /* * Populates out with the set of patch ids that are part of this condition * (excluding the activated patch) @@ -229,8 +225,8 @@ class ActivationCondition { size_t Hash() const { if (!cached_hash_.has_value()) { - cached_hash_ = absl::HashOf(conditions_, activated_, is_fallback_, - is_exclusive_, encoding_); + cached_hash_ = + absl::HashOf(conditions_, activated_, is_exclusive_, encoding_); } return *cached_hash_; } @@ -248,7 +244,6 @@ class ActivationCondition { return false; } return conditions_ == other.conditions_ && activated_ == other.activated_ && - is_fallback_ == other.is_fallback_ && is_exclusive_ == other.is_exclusive_ && encoding_ == other.encoding_; } @@ -270,7 +265,6 @@ class ActivationCondition { ActivationCondition() : conditions_(), activated_({0}), encoding_(proto::GLYPH_KEYED) {} - bool is_fallback_ = false; bool is_exclusive_ = false; // Represents: // (s_1_1 OR s_1_2 OR ...) AND (s_2_1 OR ...) ... diff --git a/ift/encoder/activation_condition_test.cc b/ift/encoder/activation_condition_test.cc index d7580094..42a232e7 100644 --- a/ift/encoder/activation_condition_test.cc +++ b/ift/encoder/activation_condition_test.cc @@ -417,7 +417,6 @@ TEST(ActivationConditionTest, And) { EXPECT_EQ(combined_ab.ToString(), "if ((s1 OR s2) AND s3) then p10"); EXPECT_EQ(combined_ab.activated(), 10); EXPECT_FALSE(combined_ab.IsExclusive()); - EXPECT_FALSE(combined_ab.IsFallback()); auto combined_ba = ActivationCondition::And(b, a); EXPECT_EQ(combined_ab.conditions(), combined_ba.conditions()); @@ -468,7 +467,6 @@ TEST(ActivationConditionTest, Or) { "then p11"); EXPECT_EQ(combined_ab.activated(), 11); EXPECT_FALSE(combined_ab.IsExclusive()); - EXPECT_FALSE(combined_ab.IsFallback()); auto combined_ba = ActivationCondition::Or(b, a); EXPECT_EQ(combined_ab, combined_ba); @@ -477,14 +475,13 @@ TEST(ActivationConditionTest, Or) { TEST(ActivationConditionTest, Or_IsOrderIndependent) { auto a = ActivationCondition::or_segments({1, 2}, 10); a.SetEncoding(ift::proto::DEFAULT_ENCODING); - auto b = ActivationCondition::or_segments({3}, 11, /*is_fallback=*/true); + auto b = ActivationCondition::or_segments({3}, 11); b.SetEncoding(ift::proto::DEFAULT_ENCODING); auto c = ActivationCondition::and_segments({4, 5}, 9); c.SetEncoding(ift::proto::TABLE_KEYED_PARTIAL); auto ab = ActivationCondition::Or(a, b); EXPECT_EQ(ab, ActivationCondition::Or(b, a)); - EXPECT_TRUE(ab.IsFallback()); EXPECT_EQ(ab.activated(), 11); auto ac = ActivationCondition::Or(a, c); diff --git a/ift/encoder/all_pairs_merger.cc b/ift/encoder/all_pairs_merger.cc index f608e975..219f1baa 100644 --- a/ift/encoder/all_pairs_merger.cc +++ b/ift/encoder/all_pairs_merger.cc @@ -44,20 +44,13 @@ void AllPairsMerger::AssignMergedSegment(const Merger& merger, // let us locate any conditions involved with those glyphs post-merge. for (const auto& c : groupings.TriggeringSegmentToConditionsPreCombination(s)) { - if (c.IsFallback()) { - continue; - } if (auto rep = pre_comb_map.at(c).min(); rep.has_value()) { dirty_glyphs_.insert(*rep); } } // Also record any segments involved in conditions directly impacted by the merge. - for (const auto& c : - groupings.TriggeringSegmentToConditions(s)) { - if (c.IsFallback()) { - continue; - } + for (const auto& c : groupings.TriggeringSegmentToConditions(s)) { if (!c.IsExclusive()) { dirty_segments_.union_set(c.TriggeringSegments()); } @@ -138,7 +131,7 @@ StatusOr AllPairsMerger::NormalizationDivisor( uint64_t total_bytes = 0; for (segment_index_t s : merged_segments) { for (const auto& c : groupings.TriggeringSegmentToConditions(s)) { - if (c.IsFallback() || (unicode_range_mode && !c.IsExclusive())) { + if (unicode_range_mode && !c.IsExclusive()) { continue; } auto it = conditions_and_glyphs.find(c); @@ -168,8 +161,8 @@ Status AllPairsMerger::QueueBestCaseCandidate( return absl::OkStatus(); } - auto best_case_delta = CandidateMerge::ComputeCostDelta( - merger, merged_segments, std::nullopt, 0.0); + auto best_case_delta = + CandidateMerge::ComputeCostDelta(merger, merged_segments, 0.0); if (absl::IsCancelled(best_case_delta.status())) { return absl::OkStatus(); } @@ -227,7 +220,7 @@ Status AllPairsMerger::QueueCompositeCandidates( last_exclusive); it != merger.Context().glyph_groupings.OrderedConditions().end(); ++it) { const auto& next_condition = *it; - if (next_condition.IsFallback() || next_condition.IsExclusive()) { + if (next_condition.IsExclusive()) { continue; } @@ -349,8 +342,7 @@ Status AllPairsMerger::Initialize(Merger& merger) { Status AllPairsMerger::RefreshDirtySegments(Merger& merger) { const auto& groupings = merger.Context().glyph_groupings; for (glyph_id_t gid : dirty_glyphs_) { - if (auto cond = groupings.GlyphToCondition(gid); - cond.has_value() && !cond->IsFallback()) { + if (auto cond = groupings.GlyphToCondition(gid); cond.has_value()) { dirty_segments_.union_set(cond->TriggeringSegments()); } } @@ -360,7 +352,7 @@ Status AllPairsMerger::RefreshDirtySegments(Merger& merger) { for (segment_index_t base : dirty_bases_) { dirty_segments_.insert(base); for (const auto& c : groupings.TriggeringSegmentToConditions(base)) { - if (!c.IsFallback() && !c.IsExclusive()) { + if (!c.IsExclusive()) { dirty_segments_.union_set(c.TriggeringSegments()); } } diff --git a/ift/encoder/candidate_merge.cc b/ift/encoder/candidate_merge.cc index b3ff016e..e950fabf 100644 --- a/ift/encoder/candidate_merge.cc +++ b/ift/encoder/candidate_merge.cc @@ -106,39 +106,15 @@ StatusOr CandidateMerge::Apply(Merger& merger) { return InvalidationSet(base_segment_index_); } - // Upon application of this merger if all of the input segments were inert - // then it's likely that the new segment will also be inert (though not - // gauranteed). We can check for this case by running a closure analysis - // and checking there are no non-exclusive gids. If the input segments are - // inert and the new segment is inert then we can directly compute the result - // of the merge without needing to invalidate and reprocess. SegmentSet segments_to_merge_with_base = segments_to_merge_; segments_to_merge_with_base.insert(base_segment_index_); - bool new_segment_is_inert = false; - if (input_segments_are_inert_ && - !merger.Context().IsPureDepGraphAnalysisMode()) { - // Only need to compute the new segment inertness in non pure dep graph - // mode. In pure dep graph mode this will be recomputed directly from the - // graph after the merge is applied. - GlyphSet and_gids, or_gids, exclusive_gids; - TRYV(merger.Context().AnalyzeSegment(segments_to_merge_with_base, and_gids, - or_gids, exclusive_gids)); - new_segment_is_inert = (and_gids.empty() && or_gids.empty()); - if (new_segment_is_inert) { - // When the new segment is inert then invalidated glyphs is - // used to shortcut closure analysis and directly construct - // the new condition and glyph mappings, save all of the - // closure glyphs to it. - invalidated_glyphs_.union_set(exclusive_gids); - } - } const auto& segments = merger.Context().SegmentationInfo().Segments(); uint32_t size_before = segments[base_segment_index_].Definition().codepoints.size(); uint32_t size_after = merger.AssignMergedSegment(base_segment_index_, segments_to_merge_, - *merged_segment_, new_segment_is_inert); + *merged_segment_, input_segments_are_inert_); VLOG(0) << " Merged " << size_before << " codepoints up to " << size_after << " codepoints for segment " << base_segment_index_ << "." @@ -154,24 +130,6 @@ StatusOr CandidateMerge::Apply(Merger& merger) { TRYV(merger.Context().InvalidateGlyphInformationForMerge( invalidated_glyphs_, segments_to_merge_with_base, base_segment_index_)); - if (new_segment_is_inert) { - // The newly formed segment will be inert which means we can construct the - // new condition sets and glyph groupings here instead of using the - // closure analysis to do it. The new segment is simply the union of all - // glyphs associated with each segment that is part of the merge. - // (gid_conditons_to_update) - for (glyph_id_t gid : invalidated_glyphs_) { - merger.Context().glyph_condition_set.AddAndCondition(gid, - base_segment_index_); - } - TRYV(merger.Context().glyph_groupings.AddGlyphsToExclusiveGroup( - base_segment_index_, invalidated_glyphs_)); - - // We've now fully updated information for these glyphs so don't need to - // return them. - invalidated_glyphs_.clear(); - } - return InvalidationSet(invalidated_glyphs_, segments_to_merge_, base_segment_index_); } @@ -251,11 +209,6 @@ static Status FindModifiedConditions( for (auto s : merged_segments) { for (const auto& c : groupings.TriggeringSegmentToConditions(s)) { - if (c.IsFallback()) { - // Ignore fallback for this analysis. - continue; - } - auto glyph_it = conditions_and_glyphs.find(c); if (glyph_it == conditions_and_glyphs.end()) { return absl::InternalError( @@ -296,16 +249,8 @@ static flat_hash_map PatchesWithGlyphsOrSegments( const SegmentSet& segments) { // To more efficiently target our search we can use the glyph_condition_set to // locate conditions that intersect with gids. - GlyphSet fallback_glyphs = context.glyph_groupings.UnmappedGlyphs(); flat_hash_set conditions_of_interest; for (glyph_id_t gid : gids) { - if (fallback_glyphs.contains(gid)) { - // Fallback glyphs are handled separately at the end since the - // conditions in the glyph condition set associated with a fallback - // glyph are not accurate. - continue; - } - auto result = context.glyph_groupings.GlyphToCondition(gid); if (!result.has_value()) { continue; @@ -327,13 +272,6 @@ static flat_hash_map PatchesWithGlyphsOrSegments( context.glyph_groupings.ConditionsAndGlyphs().at(condition))); } - // We also need to check if there's a fallback patch and it intersects gids. - if (!fallback_glyphs.empty() && fallback_glyphs.intersects(gids)) { - ActivationCondition condition = - ActivationCondition::or_segments({}, 0, true); - result.insert(std::make_pair(condition, fallback_glyphs)); - } - return result; } @@ -389,10 +327,8 @@ static StatusOr CostFor(Merger& merger, const freq::ProbabilityCalculator* calculator, const ActivationCondition& condition, const GlyphSet& glyphs) { - double probability = 1.0; - if (!condition.IsFallback()) { - probability = TRY(condition.Probability(merger.Context().SegmentationInfo().Segments(), *calculator)); - } + double probability = TRY(condition.Probability( + merger.Context().SegmentationInfo().Segments(), *calculator)); double patch_size = TRY(merger.Context().patch_size_cache_for_init_font->GetPatchSize( @@ -524,13 +460,10 @@ StatusOr> CandidateMerge::ComputeInitFontCostDelta( } for (const auto& [condition, glyphs] : new_conditions) { - double patch_probability_after = 1.0; - if (!condition.IsFallback()) { - // TODO(garretrieger): XXXX also include the effect of modified segments - // in this calc. Start with finding a test case. - patch_probability_after = - TRY(condition.Probability(merger.Context().SegmentationInfo().Segments(), *calculator)); - } + // TODO(garretrieger): XXXX also include the effect of modified segments + // in this calc. Start with finding a test case. + double patch_probability_after = TRY(condition.Probability( + merger.Context().SegmentationInfo().Segments(), *calculator)); double patch_size_after = TRY(merger.Context().patch_size_cache_for_init_font->GetPatchSize( @@ -576,11 +509,8 @@ StatusOr CandidateMerge::ComputeBestCaseInitFontCostDelta( merger.Context(), glyph_closure_delta, SegmentSet{}); deltas.reserve(affected_conditions.size()); for (const auto& [condition, glyphs] : affected_conditions) { - double patch_probability = 1.0; - if (!condition.IsFallback()) { - patch_probability = - TRY(condition.Probability(merger.Context().SegmentationInfo().Segments(), *calculator)); - } + double patch_probability = TRY(condition.Probability( + merger.Context().SegmentationInfo().Segments(), *calculator)); GlyphSet new_glyphs = glyphs; new_glyphs.subtract(glyph_closure_delta); @@ -641,7 +571,6 @@ template static StatusOr ComputeInertPairCostDelta( Merger& merger, const SegmentSet& merged_segments, segment_index_t u, segment_index_t v, uint32_t size_u, uint32_t size_v, - const std::optional& exclusive_gids, std::optional delta_to_beat) { const auto& context = merger.Context(); const auto& patch_size_cache = context.patch_size_cache; @@ -667,26 +596,13 @@ static StatusOr ComputeInertPairCostDelta( Merger::BEST_CASE_MERGE_SIZE_DELTA); size = largest_patch_size + extra; } else { - const GlyphSet* glyphs_for_size = nullptr; - GlyphSet combined_glyphs; - const auto& glyphs_u = context.glyph_groupings.ExclusiveGlyphs(u); - const auto& glyphs_v = context.glyph_groupings.ExclusiveGlyphs(v); - if (exclusive_gids.has_value() && glyphs_u.is_subset_of(*exclusive_gids) && - glyphs_v.is_subset_of(*exclusive_gids)) { - glyphs_for_size = &*exclusive_gids; - } else { - combined_glyphs = glyphs_u; - combined_glyphs.union_set(glyphs_v); - if (exclusive_gids.has_value()) { - combined_glyphs.union_set(*exclusive_gids); - } - glyphs_for_size = &combined_glyphs; - } - if (glyphs_for_size->empty()) { + GlyphSet combined_glyphs = context.glyph_groupings.ExclusiveGlyphs(u); + combined_glyphs.union_set(context.glyph_groupings.ExclusiveGlyphs(v)); + if (combined_glyphs.empty()) { std::array deltas = {-d_u, -d_v}; return SumDeltas(absl::MakeSpan(deltas)); } - size = TRY(patch_size_cache->GetPatchSize(*glyphs_for_size)); + size = TRY(patch_size_cache->GetPatchSize(combined_glyphs)); if (merger.ShouldRecordMergedSizeReductions() && extra_raw != 0) { int32_t extra_actual = ((int32_t)size) - largest_patch_size; merger.RecordMergedSizeReduction((double)extra_actual / @@ -738,7 +654,7 @@ static StatusOr ComputeInertPairCostDelta( template StatusOr CandidateMerge::ComputeCostDelta( Merger& merger, const SegmentSet& merged_segments, - std::optional exclusive_gids, std::optional delta_to_beat) { + std::optional delta_to_beat) { if (merged_segments.size() <= 1) { return 0; } @@ -756,13 +672,9 @@ StatusOr CandidateMerge::ComputeCostDelta( context.InertSegments().contains(v)) { uint32_t size_u = TRY(merger.ExclusivePatchSize(u)); uint32_t size_v = TRY(merger.ExclusivePatchSize(v)); - if (size_u > 0 && size_v > 0 && - (best_case || unicode_range_mode || !exclusive_gids.has_value() || - !context.glyph_groupings.UnmappedGlyphs().intersects( - *exclusive_gids))) { + if (size_u > 0 && size_v > 0) { return ComputeInertPairCostDelta( - merger, merged_segments, u, v, size_u, size_v, exclusive_gids, - delta_to_beat); + merger, merged_segments, u, v, size_u, size_v, delta_to_beat); } } } @@ -807,7 +719,6 @@ StatusOr CandidateMerge::ComputeCostDelta( deltas.reserve(2 * modified_conditions.size() + 1); double current_delta = 0.0; - bool has_unitary = false; const uint32_t per_request_overhead = merger.Strategy().NetworkOverheadCost(); auto add_removed_condition = [&](const ActivationCondition* condition, @@ -851,9 +762,6 @@ StatusOr CandidateMerge::ComputeCostDelta( continue; } - if (!best_case && updated.IsUnitary()) { - has_unitary = true; - } auto [it, did_insert] = new_conditions.try_emplace(std::move(updated)); auto& info = it->second; @@ -872,33 +780,16 @@ StatusOr CandidateMerge::ComputeCostDelta( TRYV(add_removed_condition(condition, glyphs, info)); } - if (unicode_range_mode && !best_case && exclusive_gids.has_value() && - !exclusive_gids->empty()) { - new_conditions.try_emplace(ActivationCondition::or_segments({base}, 0)); - } - for (auto& [c, info] : new_conditions) { if (info.source_count != 1) { continue; } bool was_active_before_merge = (!unicode_range_mode || info.first_condition->IsExclusive()); - bool glyph_set_unchanged = true; - if (!best_case && exclusive_gids.has_value()) { - // Check if the provided exclusive gids will cause a change to this condition's glyphs. - // exclusive_gids are added to the unitary condition for this merge, and subtracted - // from everything else. - if (c.IsUnitary()) { - glyph_set_unchanged = exclusive_gids->is_subset_of(*info.first_glyphs); - } else { - glyph_set_unchanged = !info.first_glyphs->intersects(*exclusive_gids); - } - } - info.glyph_set_unchanged = glyph_set_unchanged; - if (glyph_set_unchanged && - (best_case || - (was_active_before_merge && - info.first_condition->IsUnchangedByMerge(merged_segments)))) { + info.glyph_set_unchanged = true; + if (best_case || + (was_active_before_merge && + info.first_condition->IsUnchangedByMerge(merged_segments))) { info.skip_new_condition = true; continue; } @@ -906,30 +797,6 @@ StatusOr CandidateMerge::ComputeCostDelta( } double initial_magnitude_sum = -current_delta; - - bool fallback_changed = - !unicode_range_mode && !best_case && has_unitary && - exclusive_gids.has_value() && - context.glyph_groupings.UnmappedGlyphs().intersects(*exclusive_gids); - if (fallback_changed) { - GlyphSet new_fallback = context.glyph_groupings.UnmappedGlyphs(); - new_fallback.subtract(*exclusive_gids); - - // Fallback is always needed with 100% probability so delta is just the size - // difference (new - old). Since the fallback is needed with 100% - // probability under every calculator, it's aggregate probability is the - // number of probability profiles. - double fallback_probability = merger.MaxAggregateProbability(); - double diff = (((double)TRY(patch_size_cache->GetPatchSize(new_fallback))) - - ((double)TRY(patch_size_cache->GetPatchSize( - context.glyph_groupings.UnmappedGlyphs())))) * - fallback_probability; - VLOG(1) << " + " << diff << " [fallback delta]"; - deltas.push_back(diff); - current_delta += diff; - initial_magnitude_sum += std::abs(diff); - } - double initial_delta = current_delta; // Add in cost associated within the new conditions patch pairs. @@ -950,21 +817,6 @@ StatusOr CandidateMerge::ComputeCostDelta( } else if (info.glyph_set_unchanged) { size = info.largest_patch_size; } else { - if (exclusive_gids.has_value()) { - // When we do not have complete glyph conditions then exclusive_gids - // provides a hint at which glyphs will end up in the exclusive patch - // after the merge, incorporate them into the size calculation here. - // Remove them from all other patches and place them into the exclusive - // patch. - info.glyphs.subtract(*exclusive_gids); - if (c.IsUnitary()) { - info.glyphs.union_set(*exclusive_gids); - } - if (info.glyphs.empty()) { - // Patch has no remaining glyphs, so don't add back any cost for it. - continue; - } - } size = TRY(patch_size_cache->GetPatchSize(info.glyphs)); if (merger.ShouldRecordMergedSizeReductions()) { int32_t extra_raw = @@ -1004,11 +856,9 @@ StatusOr CandidateMerge::ComputeCostDelta( template StatusOr CandidateMerge::ComputeCostDelta( Merger& merger, const SegmentSet& merged_segments, - std::optional exclusive_gids, std::optional delta_to_beat); template StatusOr CandidateMerge::ComputeCostDelta( Merger& merger, const SegmentSet& merged_segments, - std::optional exclusive_gids, std::optional delta_to_beat); StatusOr @@ -1179,17 +1029,11 @@ StatusOr CandidateMerge::PatchMergeDetails::ComputePatchMergeCostDelta( return cost_delta; } -static StatusOr ExceedsMaxPatchSize( - const Merger& merger, segment_index_t base_segment_index, - const SegmentSet& merged_segments, - const std::optional& exclusive_gids, uint32_t& new_patch_size) { +static StatusOr ExceedsMaxPatchSize(const Merger& merger, + segment_index_t base_segment_index, + const SegmentSet& merged_segments, + uint32_t& new_patch_size) { uint32_t max_patch_size = merger.Strategy().PatchSizeMaxBytes(); - if (exclusive_gids.has_value() && new_patch_size > max_patch_size) { - VLOG(0) << " Merged exclusive patch for " << base_segment_index - << " is too big (" << new_patch_size << " > " << max_patch_size - << "), skipping."; - return true; - } flat_hash_map modified_conditions; @@ -1215,18 +1059,16 @@ static StatusOr ExceedsMaxPatchSize( } const auto& patch_size_cache = merger.Context().patch_size_cache; - if (!exclusive_gids.has_value()) { - auto unitary_it = new_conditions.find( - ActivationCondition::or_segments({base_segment_index}, 0)); - if (unitary_it != new_conditions.end()) { - new_patch_size = - TRY(patch_size_cache->GetPatchSize(unitary_it->second.glyphs)); - if (new_patch_size > max_patch_size) { - VLOG(0) << " Merged exclusive patch for " << base_segment_index - << " is too big (" << new_patch_size << " > " << max_patch_size - << "), skipping."; - return true; - } + auto unitary_it = new_conditions.find( + ActivationCondition::or_segments({base_segment_index}, 0)); + if (unitary_it != new_conditions.end()) { + new_patch_size = + TRY(patch_size_cache->GetPatchSize(unitary_it->second.glyphs)); + if (new_patch_size > max_patch_size) { + VLOG(0) << " Merged exclusive patch for " << base_segment_index + << " is too big (" << new_patch_size << " > " << max_patch_size + << "), skipping."; + return true; } } @@ -1235,10 +1077,6 @@ static StatusOr ExceedsMaxPatchSize( continue; } - if (exclusive_gids.has_value()) { - entry.glyphs.subtract(*exclusive_gids); - } - if (entry.glyphs.size() <= entry.max_input_glyph_count) { // This non-exclusive condition did not grow in glyph count as a result // of the merge. @@ -1303,7 +1141,7 @@ StatusOr> CandidateMerge::AssessSegmentMerge( // If that doesn't beat the current smallest there's no need to // do more indepth analysis. auto best_case_delta_status = ComputeCostDelta( - merger, segments_to_merge_with_base, std::nullopt, delta_to_beat); + merger, segments_to_merge_with_base, delta_to_beat); if (absl::IsCancelled(best_case_delta_status.status())) { // Delta computation was ended early since it couldn't beat // best_merge_candidate. @@ -1322,44 +1160,16 @@ StatusOr> CandidateMerge::AssessSegmentMerge( // we need to update those (and the down stream and/or glyph groups) to // reflect the removal/change and allow recalculation during the // GroupGlyphs steps - // - // Changes caused by adding new segments into the base segment will be - // handled by the next AnalyzeSegment step. gid_conditions_to_update.union_set( merger.Context().glyph_condition_set.GlyphsWithSegment(segment_index)); } uint32_t new_patch_size = 0; - std::optional exclusive_gids; - if (!merger.Context().IsPureDepGraphAnalysisMode()) { - if (!segments_to_merge_are_inert) { - // When we're not in pure depgraph mode then glyph conditions are - // incomplete. To help improve accuracy run a closure to find the new set - // of exclusive glyphs associated with the merge and incorporate that into - // the cost calculations. - GlyphSet and_gids, or_gids; - exclusive_gids = GlyphSet{}; - TRYV(merger.Context().AnalyzeSegment(segments_to_merge_with_base, - and_gids, or_gids, *exclusive_gids)); - } else { - // When the segments being added to base are all inert then we can assume - // the merged patch is just a combination of the glyphs from the base and - // the input segments. Since the segments being added are intert we know - // that they won't interact with base and will just bring along their own - // glyphs. - exclusive_gids = gid_conditions_to_update; - exclusive_gids->union_set( - merger.Context().glyph_groupings.ExclusiveGlyphs(base_segment_index)); - } - new_patch_size = - TRY(merger.Context().patch_size_cache->GetPatchSize(*exclusive_gids)); - } - double cost_delta = 0.0; if (merger.Strategy().UseCosts()) { // Cost delta values are only needed when using cost based merge strategy. auto cost_delta_status = ComputeCostDelta( - merger, segments_to_merge_with_base, exclusive_gids, delta_to_beat); + merger, segments_to_merge_with_base, delta_to_beat); if (absl::IsCancelled(cost_delta_status.status())) { // Delta computation was ended early since it couldn't beat // best_merge_candidate. @@ -1367,8 +1177,8 @@ StatusOr> CandidateMerge::AssessSegmentMerge( } cost_delta = TRY(cost_delta_status); } else if (TRY(ExceedsMaxPatchSize(merger, base_segment_index, - segments_to_merge_with_base, exclusive_gids, - new_patch_size))) { + segments_to_merge_with_base, + new_patch_size))) { return std::nullopt; } diff --git a/ift/encoder/candidate_merge.h b/ift/encoder/candidate_merge.h index d1d623c9..617dca1a 100644 --- a/ift/encoder/candidate_merge.h +++ b/ift/encoder/candidate_merge.h @@ -138,10 +138,6 @@ struct CandidateMerge { // Computes the predicted change to the total cost if merged_segments // are joined together into a new segment. // - // exclusive_gids is an optional hint that specifies which glyphs - // will end up in the exclusive patch after merged_segments are merged - // together. - // // If best_case is true then this will compute an estimated best possible cost // delta from the merge (computationally cheap) instead of the real delta. // @@ -150,7 +146,6 @@ struct CandidateMerge { template static absl::StatusOr ComputeCostDelta( Merger& merger, const ift::common::SegmentSet& merged_segments, - std::optional exclusive_gids, std::optional delta_to_beat); // Computes the predicted change to the toal cost if moved_glyphs are diff --git a/ift/encoder/candidate_merge_test.cc b/ift/encoder/candidate_merge_test.cc index a925c785..d9919de5 100644 --- a/ift/encoder/candidate_merge_test.cc +++ b/ift/encoder/candidate_merge_test.cc @@ -22,10 +22,8 @@ using absl::flat_hash_map; -using ift::config::CLOSURE_ONLY; using ift::config::DEP_GRAPH_ONLY; using ift::config::DEP_GRAPH_ONLY_WITH_SIMPLIFICATION; -using ift::config::PATCH; using ift::common::BazelDataFileResolver; using ift::common::CodepointSet; @@ -134,11 +132,11 @@ TEST_F(CandidateMergeTest, AssessMerge_CostDeltas) { auto probability_calculator = std::make_unique(segment_probabilities); - ClosureGlyphSegmenter segmenter(8, 8, PATCH, CLOSURE_ONLY, resolver); + ClosureGlyphSegmenter segmenter(8, 8, DEP_GRAPH_ONLY, resolver); auto context = SegmentationContext::InitializeSegmentationContext( - roboto.get(), {}, segments, segmenter.unmapped_glyph_handling(), - segmenter.condition_analysis_mode(), segmenter.brotli_quality(), - segmenter.init_font_merging_brotli_quality(), resolver); + roboto.get(), {}, segments, segmenter.condition_analysis_mode(), + segmenter.brotli_quality(), segmenter.init_font_merging_brotli_quality(), + resolver); ASSERT_TRUE(context.ok()) << context.status(); Merger merger = *Merger::New( @@ -203,11 +201,11 @@ TEST_F(CandidateMergeTest, AssessMerge_WithBestCandidate) { auto probability_calculator = std::make_unique(segment_probabilities); - ClosureGlyphSegmenter segmenter(8, 8, PATCH, CLOSURE_ONLY, resolver); + ClosureGlyphSegmenter segmenter(8, 8, DEP_GRAPH_ONLY, resolver); auto context = SegmentationContext::InitializeSegmentationContext( - roboto.get(), {}, segments, segmenter.unmapped_glyph_handling(), - segmenter.condition_analysis_mode(), segmenter.brotli_quality(), - segmenter.init_font_merging_brotli_quality(), resolver); + roboto.get(), {}, segments, segmenter.condition_analysis_mode(), + segmenter.brotli_quality(), segmenter.init_font_merging_brotli_quality(), + resolver); ASSERT_TRUE(context.ok()) << context.status(); Merger merger = *Merger::New( @@ -263,11 +261,11 @@ TEST_F(CandidateMergeTest, AssessMerge_CostDeltas_Complex) { {{'i'}}, }; - ClosureGlyphSegmenter segmenter(8, 8, PATCH, CLOSURE_ONLY, resolver); + ClosureGlyphSegmenter segmenter(8, 8, DEP_GRAPH_ONLY, resolver); auto context = SegmentationContext::InitializeSegmentationContext( - roboto.get(), {}, segments, segmenter.unmapped_glyph_handling(), - segmenter.condition_analysis_mode(), segmenter.brotli_quality(), - segmenter.init_font_merging_brotli_quality(), resolver); + roboto.get(), {}, segments, segmenter.condition_analysis_mode(), + segmenter.brotli_quality(), segmenter.init_font_merging_brotli_quality(), + resolver); ASSERT_TRUE(context.ok()) << context.status(); Merger merger = *Merger::New( @@ -319,11 +317,11 @@ TEST_F(CandidateMergeTest, AssessMerge_CostDeltas_Complex_ModifiedConditions) { freq::UnicodeFrequencies frequencies{ {{' ', ' '}, 100}, {{'a', 'a'}, 50}, {{'f', 'f'}, 75}, {{'i', 'i'}, 95}}; - ClosureGlyphSegmenter segmenter(8, 8, PATCH, CLOSURE_ONLY, resolver); + ClosureGlyphSegmenter segmenter(8, 8, DEP_GRAPH_ONLY, resolver); auto context = SegmentationContext::InitializeSegmentationContext( - roboto.get(), {}, segments, segmenter.unmapped_glyph_handling(), - segmenter.condition_analysis_mode(), segmenter.brotli_quality(), - segmenter.init_font_merging_brotli_quality(), resolver); + roboto.get(), {}, segments, segmenter.condition_analysis_mode(), + segmenter.brotli_quality(), segmenter.init_font_merging_brotli_quality(), + resolver); ASSERT_TRUE(context.ok()) << context.status(); Merger merger = *Merger::New( @@ -394,11 +392,11 @@ TEST_F(CandidateMergeTest, AssessMerge_CostDeltas_MultipleCalculators) { strategy.AddProbabilityProfile(ProbabilityProfile( std::make_shared(probabilities_2))); - ClosureGlyphSegmenter segmenter(8, 8, PATCH, CLOSURE_ONLY, resolver); + ClosureGlyphSegmenter segmenter(8, 8, DEP_GRAPH_ONLY, resolver); auto context = SegmentationContext::InitializeSegmentationContext( - roboto.get(), {}, segments, segmenter.unmapped_glyph_handling(), - segmenter.condition_analysis_mode(), segmenter.brotli_quality(), - segmenter.init_font_merging_brotli_quality(), resolver); + roboto.get(), {}, segments, segmenter.condition_analysis_mode(), + segmenter.brotli_quality(), segmenter.init_font_merging_brotli_quality(), + resolver); ASSERT_TRUE(context.ok()) << context.status(); Merger merger = *Merger::New(*context, strategy, all, all); @@ -461,11 +459,11 @@ TEST_F(CandidateMergeTest, AssessMerge_CostDeltas_UnrelatedCalculator) { strategy.AddProbabilityProfile(ProbabilityProfile( std::make_shared(probabilities_2))); - ClosureGlyphSegmenter segmenter(8, 8, PATCH, CLOSURE_ONLY, resolver); + ClosureGlyphSegmenter segmenter(8, 8, DEP_GRAPH_ONLY, resolver); auto context = SegmentationContext::InitializeSegmentationContext( - roboto.get(), {}, segments, segmenter.unmapped_glyph_handling(), - segmenter.condition_analysis_mode(), segmenter.brotli_quality(), - segmenter.init_font_merging_brotli_quality(), resolver); + roboto.get(), {}, segments, segmenter.condition_analysis_mode(), + segmenter.brotli_quality(), segmenter.init_font_merging_brotli_quality(), + resolver); ASSERT_TRUE(context.ok()) << context.status(); Merger merger = *Merger::New(*context, strategy, all, all); @@ -532,11 +530,11 @@ TEST_F(CandidateMergeTest, AssessPatchMerge) { auto probability_calculator = std::make_unique(segment_probabilities); - ClosureGlyphSegmenter segmenter(8, 8, PATCH, CLOSURE_ONLY, resolver); + ClosureGlyphSegmenter segmenter(8, 8, DEP_GRAPH_ONLY, resolver); auto context = SegmentationContext::InitializeSegmentationContext( - roboto.get(), {}, segments, segmenter.unmapped_glyph_handling(), - segmenter.condition_analysis_mode(), segmenter.brotli_quality(), - segmenter.init_font_merging_brotli_quality(), resolver); + roboto.get(), {}, segments, segmenter.condition_analysis_mode(), + segmenter.brotli_quality(), segmenter.init_font_merging_brotli_quality(), + resolver); ASSERT_TRUE(context.ok()) << context.status(); Merger merger = *Merger::New( @@ -590,11 +588,11 @@ TEST_F(CandidateMergeTest, AssessPatchMerge_RequiresPatches) { auto probability_calculator = std::make_unique(segment_probabilities); - ClosureGlyphSegmenter segmenter(8, 8, PATCH, CLOSURE_ONLY, resolver); + ClosureGlyphSegmenter segmenter(8, 8, DEP_GRAPH_ONLY, resolver); auto context = SegmentationContext::InitializeSegmentationContext( - roboto.get(), {}, segments, segmenter.unmapped_glyph_handling(), - segmenter.condition_analysis_mode(), segmenter.brotli_quality(), - segmenter.init_font_merging_brotli_quality(), resolver); + roboto.get(), {}, segments, segmenter.condition_analysis_mode(), + segmenter.brotli_quality(), segmenter.init_font_merging_brotli_quality(), + resolver); ASSERT_TRUE(context.ok()) << context.status(); Merger merger = *Merger::New( @@ -635,11 +633,11 @@ TEST_F(CandidateMergeTest, AssessPatchMerge_NonDisjunctive) { MockPatchSizeCache* size_cache = new MockPatchSizeCache(); - ClosureGlyphSegmenter segmenter(8, 8, PATCH, CLOSURE_ONLY, resolver); + ClosureGlyphSegmenter segmenter(8, 8, DEP_GRAPH_ONLY, resolver); auto context = SegmentationContext::InitializeSegmentationContext( - roboto.get(), {}, segments, segmenter.unmapped_glyph_handling(), - segmenter.condition_analysis_mode(), segmenter.brotli_quality(), - segmenter.init_font_merging_brotli_quality(), resolver); + roboto.get(), {}, segments, segmenter.condition_analysis_mode(), + segmenter.brotli_quality(), segmenter.init_font_merging_brotli_quality(), + resolver); ASSERT_TRUE(context.ok()) << context.status(); Merger merger = *Merger::New( @@ -664,10 +662,9 @@ TEST_F(CandidateMergeTest, AssessPatchMerge_NonDisjunctive) { context->glyph_condition_set.AddAndCondition(gid_A, 0); // Update glyph groupings. - auto sc = context->glyph_groupings.GroupGlyphs( - context->SegmentationInfo(), context->glyph_condition_set, - *context->glyph_closure_cache, std::nullopt, {gid_A, gid_B}, {0, 1, 2}, - false); + auto sc = context->glyph_groupings.GroupGlyphs(context->SegmentationInfo(), + context->glyph_condition_set, + {gid_A, gid_B}, {0, 1, 2}); ASSERT_TRUE(sc.ok()) << sc; context->patch_size_cache.reset(size_cache); @@ -716,11 +713,11 @@ TEST_F(CandidateMergeTest, AssessPatchMerge_MultipleCalculators) { MockPatchSizeCache* size_cache = new MockPatchSizeCache(); - ClosureGlyphSegmenter segmenter(8, 8, PATCH, CLOSURE_ONLY, resolver); + ClosureGlyphSegmenter segmenter(8, 8, DEP_GRAPH_ONLY, resolver); auto context = SegmentationContext::InitializeSegmentationContext( - roboto.get(), {}, segments, segmenter.unmapped_glyph_handling(), - segmenter.condition_analysis_mode(), segmenter.brotli_quality(), - segmenter.init_font_merging_brotli_quality(), resolver); + roboto.get(), {}, segments, segmenter.condition_analysis_mode(), + segmenter.brotli_quality(), segmenter.init_font_merging_brotli_quality(), + resolver); ASSERT_TRUE(context.ok()) << context.status(); Merger merger = *Merger::New(*context, strategy, all, all); @@ -741,10 +738,9 @@ TEST_F(CandidateMergeTest, AssessPatchMerge_MultipleCalculators) { context->glyph_condition_set.AddAndCondition(gid_A, 0); // Update glyph groupings. - auto sc = context->glyph_groupings.GroupGlyphs( - context->SegmentationInfo(), context->glyph_condition_set, - *context->glyph_closure_cache, std::nullopt, {gid_A, gid_B}, {0, 1, 2}, - false); + auto sc = context->glyph_groupings.GroupGlyphs(context->SegmentationInfo(), + context->glyph_condition_set, + {gid_A, gid_B}, {0, 1, 2}); ASSERT_TRUE(sc.ok()) << sc; context->patch_size_cache.reset(size_cache); @@ -786,12 +782,12 @@ TEST_F(CandidateMergeTest, AssessPatchMerge_NonDisjunctive_WithSimplification) { MockPatchSizeCache* size_cache = new MockPatchSizeCache(); - ClosureGlyphSegmenter segmenter(8, 8, PATCH, - DEP_GRAPH_ONLY_WITH_SIMPLIFICATION, resolver); + ClosureGlyphSegmenter segmenter(8, 8, DEP_GRAPH_ONLY_WITH_SIMPLIFICATION, + resolver); auto context = SegmentationContext::InitializeSegmentationContext( - roboto.get(), {}, segments, segmenter.unmapped_glyph_handling(), - segmenter.condition_analysis_mode(), segmenter.brotli_quality(), - segmenter.init_font_merging_brotli_quality(), resolver); + roboto.get(), {}, segments, segmenter.condition_analysis_mode(), + segmenter.brotli_quality(), segmenter.init_font_merging_brotli_quality(), + resolver); ASSERT_TRUE(context.ok()) << context.status(); Merger merger = *Merger::New( @@ -816,10 +812,9 @@ TEST_F(CandidateMergeTest, AssessPatchMerge_NonDisjunctive_WithSimplification) { context->glyph_condition_set.AddAndCondition(gid_A, 0); // Update glyph groupings. - auto sc = context->glyph_groupings.GroupGlyphs( - context->SegmentationInfo(), context->glyph_condition_set, - *context->glyph_closure_cache, std::nullopt, {gid_A, gid_B}, {0, 1, 2}, - false); + auto sc = context->glyph_groupings.GroupGlyphs(context->SegmentationInfo(), + context->glyph_condition_set, + {gid_A, gid_B}, {0, 1, 2}); ASSERT_TRUE(sc.ok()) << sc; context->patch_size_cache.reset(size_cache); @@ -866,11 +861,11 @@ TEST_F(CandidateMergeTest, ComputeInitFontCostDelta) { auto probability_calculator = std::make_unique(segment_probabilities); - ClosureGlyphSegmenter segmenter(8, 8, PATCH, CLOSURE_ONLY, resolver); + ClosureGlyphSegmenter segmenter(8, 8, DEP_GRAPH_ONLY, resolver); auto context = SegmentationContext::InitializeSegmentationContext( - roboto.get(), {'a'}, segments, segmenter.unmapped_glyph_handling(), - segmenter.condition_analysis_mode(), segmenter.brotli_quality(), - segmenter.init_font_merging_brotli_quality(), resolver); + roboto.get(), {'a'}, segments, segmenter.condition_analysis_mode(), + segmenter.brotli_quality(), segmenter.init_font_merging_brotli_quality(), + resolver); ASSERT_TRUE(context.ok()) << context.status(); Merger merger = *Merger::New( @@ -1010,11 +1005,11 @@ TEST_F(CandidateMergeTest, ComputeInitFontCostDelta_TracksSmallestDelta) { auto probability_calculator = std::make_unique(segment_probabilities); - ClosureGlyphSegmenter segmenter(8, 8, PATCH, CLOSURE_ONLY, resolver); + ClosureGlyphSegmenter segmenter(8, 8, DEP_GRAPH_ONLY, resolver); auto context = SegmentationContext::InitializeSegmentationContext( - roboto.get(), {'a'}, segments, segmenter.unmapped_glyph_handling(), - segmenter.condition_analysis_mode(), segmenter.brotli_quality(), - segmenter.init_font_merging_brotli_quality(), resolver); + roboto.get(), {'a'}, segments, segmenter.condition_analysis_mode(), + segmenter.brotli_quality(), segmenter.init_font_merging_brotli_quality(), + resolver); ASSERT_TRUE(context.ok()) << context.status(); Merger merger = *Merger::New( @@ -1076,11 +1071,11 @@ TEST_F(CandidateMergeTest, ComputeCostDelta_EarlyExitAndBoundary) { freq::UnicodeFrequencies frequencies{ {{' ', ' '}, 100}, {{'a', 'a'}, 50}, {{'f', 'f'}, 75}, {{'i', 'i'}, 95}}; - ClosureGlyphSegmenter segmenter(8, 8, PATCH, CLOSURE_ONLY, resolver); + ClosureGlyphSegmenter segmenter(8, 8, DEP_GRAPH_ONLY, resolver); auto context = SegmentationContext::InitializeSegmentationContext( - roboto.get(), {}, segments, segmenter.unmapped_glyph_handling(), - segmenter.condition_analysis_mode(), segmenter.brotli_quality(), - segmenter.init_font_merging_brotli_quality(), resolver); + roboto.get(), {}, segments, segmenter.condition_analysis_mode(), + segmenter.brotli_quality(), segmenter.init_font_merging_brotli_quality(), + resolver); ASSERT_TRUE(context.ok()) << context.status(); Merger merger = *Merger::New( @@ -1096,32 +1091,30 @@ TEST_F(CandidateMergeTest, ComputeCostDelta_EarlyExitAndBoundary) { size_cache->SetPatchSize({69, 74}, 450); context->patch_size_cache.reset(size_cache); - GlyphSet exclusive_gids = {69, 74}; - auto exact_false = CandidateMerge::ComputeCostDelta( - merger, {0, 1}, exclusive_gids, std::nullopt); + auto exact_false = + CandidateMerge::ComputeCostDelta(merger, {0, 1}, std::nullopt); ASSERT_TRUE(exact_false.ok()) << exact_false.status(); - auto exact_true = CandidateMerge::ComputeCostDelta( - merger, {0, 1}, std::nullopt, std::nullopt); + auto exact_true = + CandidateMerge::ComputeCostDelta(merger, {0, 1}, std::nullopt); ASSERT_TRUE(exact_true.ok()) << exact_true.status(); // Well below the actual delta: should cancel early. auto cancelled_false = CandidateMerge::ComputeCostDelta( - merger, {0, 1}, exclusive_gids, *exact_false - 10.0); + merger, {0, 1}, *exact_false - 10.0); EXPECT_TRUE(absl::IsCancelled(cancelled_false.status())); auto cancelled_true = CandidateMerge::ComputeCostDelta( - merger, {0, 1}, std::nullopt, *exact_true - 10.0); + merger, {0, 1}, *exact_true - 10.0); EXPECT_TRUE(absl::IsCancelled(cancelled_true.status())); // Within the FP rounding margin of the actual delta (e.g. equal to or 1 ULP // below): should NOT cancel early, and should return the exact sorted sum. auto boundary_false = CandidateMerge::ComputeCostDelta( - merger, {0, 1}, exclusive_gids, - std::nextafter(*exact_false, -INFINITY)); + merger, {0, 1}, std::nextafter(*exact_false, -INFINITY)); ASSERT_TRUE(boundary_false.ok()) << boundary_false.status(); EXPECT_EQ(*boundary_false, *exact_false); auto boundary_true = CandidateMerge::ComputeCostDelta( - merger, {0, 1}, std::nullopt, std::nextafter(*exact_true, -INFINITY)); + merger, {0, 1}, std::nextafter(*exact_true, -INFINITY)); ASSERT_TRUE(boundary_true.ok()) << boundary_true.status(); EXPECT_EQ(*boundary_true, *exact_true); @@ -1140,61 +1133,6 @@ TEST_F(CandidateMergeTest, ComputeCostDelta_EarlyExitAndBoundary) { EXPECT_EQ((*better_by_1_ulp)->CostDelta(), *exact_false); } -TEST_F(CandidateMergeTest, ComputeCostDelta_FallbackChangedBeforeEarlyExit) { - std::vector segments = { - {{0x54}}, // s0 -> gid 56 - {{0x6C}}, // s1 -> gid 80 - {{0x13C}}, // s2 - {{0x21A}}, // s3 - {{0xF6C3}}, // s4 - }; - freq::UnicodeFrequencies frequencies{ - {{' ', ' '}, 100}, {{0x54, 0x54}, 50}, {{0x6C, 0x6C}, 75}, - {{0x13C, 0x13C}, 10}, {{0x21A, 0x21A}, 10}, {{0xF6C3, 0xF6C3}, 10}}; - - ClosureGlyphSegmenter segmenter(8, 8, PATCH, CLOSURE_ONLY, resolver); - auto context = SegmentationContext::InitializeSegmentationContext( - roboto.get(), {}, segments, segmenter.unmapped_glyph_handling(), - segmenter.condition_analysis_mode(), segmenter.brotli_quality(), - segmenter.init_font_merging_brotli_quality(), resolver); - ASSERT_TRUE(context.ok()) << context.status(); - - Merger merger = *Merger::New( - *context, - MergeStrategy::CostBased( - *ProbabilityProfile::Unigram(std::move(frequencies)), 75, 4), - all, all); - - // Include an unmapped glyph in exclusive_gids so fallback_changed is - // triggered and reduces the fallback patch size by 1000 bytes. - GlyphSet old_fallback = context->glyph_groupings.UnmappedGlyphs(); - ASSERT_FALSE(old_fallback.empty()); - glyph_id_t unmapped_gid = *old_fallback.min(); - GlyphSet exclusive_gids = {56, 80, unmapped_gid}; - GlyphSet new_fallback = old_fallback; - new_fallback.subtract(exclusive_gids); - - MockPatchSizeCache* size_cache = new MockPatchSizeCache(); - size_cache->SetPatchSize({56}, 200); - size_cache->SetPatchSize({80}, 300); - size_cache->SetPatchSize(exclusive_gids, 450); - size_cache->SetPatchSize(old_fallback, 2000); - size_cache->SetPatchSize(new_fallback, 1000); - context->patch_size_cache.reset(size_cache); - - auto exact_with_fallback = CandidateMerge::ComputeCostDelta( - merger, {0, 1}, exclusive_gids, std::nullopt); - ASSERT_TRUE(exact_with_fallback.ok()) << exact_with_fallback.status(); - - // Without the -1000 fallback delta applied prior to the second loop, the - // running delta during the second loop would be +1000 higher and would - // falsely exceed delta_to_beat = *exact_with_fallback + 100. - auto with_threshold = CandidateMerge::ComputeCostDelta( - merger, {0, 1}, exclusive_gids, *exact_with_fallback + 100.0); - ASSERT_TRUE(with_threshold.ok()) << with_threshold.status(); - EXPECT_EQ(*with_threshold, *exact_with_fallback); -} - TEST_F(CandidateMergeTest, AssessMerge_Heuristic_NonExclusiveMaxPatchSize) { // s0: 'a' -> {69} // s1: 'i' -> {77}, s1 AND s3 -> {444, 446} @@ -1207,11 +1145,11 @@ TEST_F(CandidateMergeTest, AssessMerge_Heuristic_NonExclusiveMaxPatchSize) { {{'f'}}, }; - ClosureGlyphSegmenter segmenter(8, 8, PATCH, CLOSURE_ONLY, resolver); + ClosureGlyphSegmenter segmenter(8, 8, DEP_GRAPH_ONLY, resolver); auto context = SegmentationContext::InitializeSegmentationContext( - roboto.get(), {}, segments, segmenter.unmapped_glyph_handling(), - segmenter.condition_analysis_mode(), segmenter.brotli_quality(), - segmenter.init_font_merging_brotli_quality(), resolver); + roboto.get(), {}, segments, segmenter.condition_analysis_mode(), + segmenter.brotli_quality(), segmenter.init_font_merging_brotli_quality(), + resolver); ASSERT_TRUE(context.ok()) << context.status(); Merger merger = @@ -1260,11 +1198,11 @@ TEST_F(CandidateMergeTest, {{'f'}}, }; - ClosureGlyphSegmenter segmenter(8, 8, PATCH, DEP_GRAPH_ONLY, resolver); + ClosureGlyphSegmenter segmenter(8, 8, DEP_GRAPH_ONLY, resolver); auto context = SegmentationContext::InitializeSegmentationContext( - roboto.get(), {}, segments, segmenter.unmapped_glyph_handling(), - segmenter.condition_analysis_mode(), segmenter.brotli_quality(), - segmenter.init_font_merging_brotli_quality(), resolver); + roboto.get(), {}, segments, segmenter.condition_analysis_mode(), + segmenter.brotli_quality(), segmenter.init_font_merging_brotli_quality(), + resolver); ASSERT_TRUE(context.ok()) << context.status(); Merger merger = @@ -1316,13 +1254,12 @@ TEST_F(CandidateMergeTest, AssessMerge_CostDeltas_UnicodeRangeIgnoresNonExclusiv freq::UnicodeFrequencies frequencies{ {{' ', ' '}, 100}, {{'a', 'a'}, 50}, {{'f', 'f'}, 75}, {{'i', 'i'}, 95}}; - ClosureGlyphSegmenter segmenter(8, 8, PATCH, CLOSURE_ONLY, resolver, + ClosureGlyphSegmenter segmenter(8, 8, DEP_GRAPH_ONLY, resolver, ift::config::UNICODE_RANGE); auto context = SegmentationContext::InitializeSegmentationContext( - roboto.get(), {}, segments, segmenter.unmapped_glyph_handling(), - segmenter.condition_analysis_mode(), segmenter.brotli_quality(), - segmenter.init_font_merging_brotli_quality(), resolver, - ift::config::UNICODE_RANGE); + roboto.get(), {}, segments, segmenter.condition_analysis_mode(), + segmenter.brotli_quality(), segmenter.init_font_merging_brotli_quality(), + resolver, ift::config::UNICODE_RANGE); ASSERT_TRUE(context.ok()) << context.status(); Merger merger = *Merger::New( @@ -1418,18 +1355,17 @@ TEST_F(CandidateMergeTest, {{'l', 'l'}, 40}, }; - ClosureGlyphSegmenter segmenter(8, 8, PATCH, CLOSURE_ONLY, resolver); + ClosureGlyphSegmenter segmenter(8, 8, DEP_GRAPH_ONLY, resolver); auto context = SegmentationContext::InitializeSegmentationContext( - roboto.get(), {}, segments, segmenter.unmapped_glyph_handling(), - segmenter.condition_analysis_mode(), segmenter.brotli_quality(), - segmenter.init_font_merging_brotli_quality(), resolver); + roboto.get(), {}, segments, segmenter.condition_analysis_mode(), + segmenter.brotli_quality(), segmenter.init_font_merging_brotli_quality(), + resolver); ASSERT_TRUE(context.ok()) << context.status(); GlyphSet g0 = context->glyph_groupings.ExclusiveGlyphs(0); GlyphSet g1 = context->glyph_groupings.ExclusiveGlyphs(1); ASSERT_FALSE(g0.empty()); ASSERT_FALSE(g1.empty()); - ASSERT_TRUE(context->glyph_groupings.UnmappedGlyphs().empty()); ASSERT_TRUE(context->glyph_groupings.ConditionsAndGlyphs().contains( ActivationCondition::or_segments({0, 1}, 0))); @@ -1499,27 +1435,27 @@ TEST_F(CandidateMergeTest, // g_acute has now collapsed into the exclusive patch for segment 0. EXPECT_EQ(context->glyph_groupings.ExclusiveGlyphs(0), g01); - // Step 2 & 3: Patch merging combines composite ligature patches. + // Step 2: Patch merging combines composite ligature patches (s2 AND s3) and + // (s2 AND s4) into s2 AND (s3 OR s4). auto step2 = merger.TryNextMerge(); ASSERT_TRUE(step2.ok()) << step2.status(); ASSERT_TRUE(step2->has_value()); ASSERT_TRUE(context->ReprocessChanged(**step2).ok()); + // Step 3: No further merges remain. The exclusive patch for segment 0 must + // NOT be merged into any composite patch. auto step3 = merger.TryNextMerge(); ASSERT_TRUE(step3.ok()) << step3.status(); - ASSERT_TRUE(step3->has_value()); - ASSERT_TRUE(context->ReprocessChanged(**step3).ok()); - - // Step 4: No further merges remain. The exclusive patch for segment 0 must - // NOT be merged into any composite patch. - auto step4 = merger.TryNextMerge(); - ASSERT_TRUE(step4.ok()) << step4.status(); - EXPECT_FALSE(step4->has_value()); + EXPECT_FALSE(step3->has_value()); EXPECT_EQ(context->glyph_groupings.ExclusiveGlyphs(0), g01); EXPECT_EQ(context->glyph_groupings.ConditionsAndGlyphs().at( ActivationCondition::exclusive_segment(0, 0)), g01); + EXPECT_EQ(context->glyph_groupings.ConditionsAndGlyphs().at( + ActivationCondition::composite_condition( + {SegmentSet{2}, SegmentSet{3, 4}}, 0)), + g_fi_fl); } TEST_F(CandidateMergeTest, @@ -1541,12 +1477,12 @@ TEST_F(CandidateMergeTest, {{0x00F9, 0x00F9}, 40}, }; - ClosureGlyphSegmenter segmenter( - 8, 8, PATCH, DEP_GRAPH_ONLY_WITH_SIMPLIFICATION, resolver); + ClosureGlyphSegmenter segmenter(8, 8, DEP_GRAPH_ONLY_WITH_SIMPLIFICATION, + resolver); auto context = SegmentationContext::InitializeSegmentationContext( - roboto.get(), {}, segments, segmenter.unmapped_glyph_handling(), - segmenter.condition_analysis_mode(), segmenter.brotli_quality(), - segmenter.init_font_merging_brotli_quality(), resolver); + roboto.get(), {}, segments, segmenter.condition_analysis_mode(), + segmenter.brotli_quality(), segmenter.init_font_merging_brotli_quality(), + resolver); ASSERT_TRUE(context.ok()) << context.status(); GlyphSet g0 = context->glyph_groupings.ExclusiveGlyphs(0); @@ -1638,11 +1574,11 @@ TEST_F(CandidateMergeTest, ComputeCostDelta_InertPairFastPathAndPreCheck) { freq::UnicodeFrequencies frequencies{ {{' ', ' '}, 100}, {{'a', 'a'}, 80}, {{'b', 'b'}, 5}, {{'c', 'c'}, 75}}; - ClosureGlyphSegmenter segmenter(8, 8, PATCH, CLOSURE_ONLY, resolver); + ClosureGlyphSegmenter segmenter(8, 8, DEP_GRAPH_ONLY, resolver); auto context = SegmentationContext::InitializeSegmentationContext( - roboto.get(), {}, segments, segmenter.unmapped_glyph_handling(), - segmenter.condition_analysis_mode(), segmenter.brotli_quality(), - segmenter.init_font_merging_brotli_quality(), resolver); + roboto.get(), {}, segments, segmenter.condition_analysis_mode(), + segmenter.brotli_quality(), segmenter.init_font_merging_brotli_quality(), + resolver); ASSERT_TRUE(context.ok()) << context.status(); ASSERT_TRUE(context->InertSegments().contains(0)); ASSERT_TRUE(context->InertSegments().contains(1)); @@ -1664,8 +1600,8 @@ TEST_F(CandidateMergeTest, ComputeCostDelta_InertPairFastPathAndPreCheck) { // First evaluation warms the per-segment exclusive patch size cache for s0 // and s2. size_cache->ResetCallCount(); - auto best_case_0_2 = CandidateMerge::ComputeCostDelta( - merger, {0, 2}, std::nullopt, std::nullopt); + auto best_case_0_2 = + CandidateMerge::ComputeCostDelta(merger, {0, 2}, std::nullopt); ASSERT_TRUE(best_case_0_2.ok()) << best_case_0_2.status(); EXPECT_EQ(size_cache->GetPatchSizeCallCount(), 2); @@ -1679,8 +1615,8 @@ TEST_F(CandidateMergeTest, ComputeCostDelta_InertPairFastPathAndPreCheck) { // Subsequent best-case evaluation of {0, 2} hits the per-segment size cache // with zero calls to PatchSizeCache::GetPatchSize. size_cache->ResetCallCount(); - auto best_case_0_2_cached = CandidateMerge::ComputeCostDelta( - merger, {0, 2}, std::nullopt, std::nullopt); + auto best_case_0_2_cached = + CandidateMerge::ComputeCostDelta(merger, {0, 2}, std::nullopt); ASSERT_TRUE(best_case_0_2_cached.ok()) << best_case_0_2_cached.status(); EXPECT_EQ(*best_case_0_2_cached, *best_case_0_2); EXPECT_EQ(size_cache->GetPatchSizeCallCount(), 0); @@ -1691,13 +1627,12 @@ TEST_F(CandidateMergeTest, ComputeCostDelta_InertPairFastPathAndPreCheck) { // 0.80*(400+75) - 0.80*(200+75) - 0.05*(300+75) = 160 - 18.75 = +141.25 > 0, // so the O(1) pre-check cancels immediately. auto cancelled_precheck = - CandidateMerge::ComputeCostDelta(merger, {0, 1}, std::nullopt, 0.0); + CandidateMerge::ComputeCostDelta(merger, {0, 1}, 0.0); EXPECT_TRUE(absl::IsCancelled(cancelled_precheck.status())); // Exact evaluation of {0, 2} via the inert fast-path. - GlyphSet exclusive_0_2 = {69, 71}; - auto exact_0_2 = CandidateMerge::ComputeCostDelta( - merger, {0, 2}, exclusive_0_2, std::nullopt); + auto exact_0_2 = + CandidateMerge::ComputeCostDelta(merger, {0, 2}, std::nullopt); ASSERT_TRUE(exact_0_2.ok()) << exact_0_2.status(); double expected_exact_0_2 = -0.80 * (200 + 75) - 0.75 * (240 + 75) + p_0_or_2 * (320 + 75); @@ -1705,7 +1640,7 @@ TEST_F(CandidateMergeTest, ComputeCostDelta_InertPairFastPathAndPreCheck) { // Boundary check: 1 ULP below exact_0_2 does not cancel early. auto boundary_0_2 = CandidateMerge::ComputeCostDelta( - merger, {0, 2}, exclusive_0_2, std::nextafter(*exact_0_2, -INFINITY)); + merger, {0, 2}, std::nextafter(*exact_0_2, -INFINITY)); ASSERT_TRUE(boundary_0_2.ok()) << boundary_0_2.status(); EXPECT_EQ(*boundary_0_2, *exact_0_2); @@ -1715,7 +1650,9 @@ TEST_F(CandidateMergeTest, ComputeCostDelta_InertPairFastPathAndPreCheck) { CandidateMerge::AssessSegmentMerge(merger, 0, {2}, std::nullopt); ASSERT_TRUE(candidate.ok()) << candidate.status(); ASSERT_TRUE(candidate->has_value()); - ASSERT_TRUE((*candidate)->Apply(merger).ok()); + auto applied = (*candidate)->Apply(merger); + ASSERT_TRUE(applied.ok()) << applied.status(); + ASSERT_TRUE(context->ReprocessChanged(*applied).ok()); size_cache->ResetCallCount(); EXPECT_EQ(*merger.ExclusivePatchSize(1), 300u); @@ -1734,13 +1671,12 @@ TEST_F(CandidateMergeTest, freq::UnicodeFrequencies frequencies{ {{' ', ' '}, 100}, {{'a', 'a'}, 80}, {{'c', 'c'}, 75}}; - ClosureGlyphSegmenter segmenter(8, 8, PATCH, CLOSURE_ONLY, resolver, + ClosureGlyphSegmenter segmenter(8, 8, DEP_GRAPH_ONLY, resolver, ift::config::UNICODE_RANGE); auto context = SegmentationContext::InitializeSegmentationContext( - roboto.get(), {}, segments, segmenter.unmapped_glyph_handling(), - segmenter.condition_analysis_mode(), segmenter.brotli_quality(), - segmenter.init_font_merging_brotli_quality(), resolver, - ift::config::UNICODE_RANGE); + roboto.get(), {}, segments, segmenter.condition_analysis_mode(), + segmenter.brotli_quality(), segmenter.init_font_merging_brotli_quality(), + resolver, ift::config::UNICODE_RANGE); ASSERT_TRUE(context.ok()) << context.status(); Merger merger = *Merger::New( @@ -1762,16 +1698,15 @@ TEST_F(CandidateMergeTest, double p_0_or_1 = 0.80 + 0.75 - 0.80 * 0.75; double expected_best = -0.80 * (200 + 75) - 0.75 * (240 + 75) + p_0_or_1 * (290 + 75); - auto best_delta = CandidateMerge::ComputeCostDelta( - merger, {0, 1}, std::nullopt, std::nullopt); + auto best_delta = + CandidateMerge::ComputeCostDelta(merger, {0, 1}, std::nullopt); ASSERT_TRUE(best_delta.ok()) << best_delta.status(); EXPECT_NEAR(*best_delta, expected_best, 1e-9); - GlyphSet exclusive_gids = {69, 71}; double expected_exact = -0.80 * (200 + 75) - 0.75 * (240 + 75) + p_0_or_1 * (320 + 75); - auto exact_delta = CandidateMerge::ComputeCostDelta( - merger, {0, 1}, exclusive_gids, std::nullopt); + auto exact_delta = + CandidateMerge::ComputeCostDelta(merger, {0, 1}, std::nullopt); ASSERT_TRUE(exact_delta.ok()) << exact_delta.status(); EXPECT_NEAR(*exact_delta, expected_exact, 1e-9); } @@ -1795,11 +1730,11 @@ TEST_F(CandidateMergeTest, AllPairsMerging_AvoidsSingleBaseGreedyTrap) { {{{'a', 'b', 'c'}}, 0.99}, }; - ClosureGlyphSegmenter segmenter(8, 8, PATCH, CLOSURE_ONLY, resolver); + ClosureGlyphSegmenter segmenter(8, 8, DEP_GRAPH_ONLY, resolver); auto context = SegmentationContext::InitializeSegmentationContext( - roboto.get(), {}, segments, segmenter.unmapped_glyph_handling(), - segmenter.condition_analysis_mode(), segmenter.brotli_quality(), - segmenter.init_font_merging_brotli_quality(), resolver); + roboto.get(), {}, segments, segmenter.condition_analysis_mode(), + segmenter.brotli_quality(), segmenter.init_font_merging_brotli_quality(), + resolver); ASSERT_TRUE(context.ok()) << context.status(); auto* size_cache = new MockPatchSizeCache(); @@ -1857,11 +1792,11 @@ TEST_F(CandidateMergeTest, {{{'a', 'b', 'c', 'd'}}, 0.80}, }; - ClosureGlyphSegmenter segmenter(8, 8, PATCH, CLOSURE_ONLY, resolver); + ClosureGlyphSegmenter segmenter(8, 8, DEP_GRAPH_ONLY, resolver); auto context = SegmentationContext::InitializeSegmentationContext( - roboto.get(), {}, segments, segmenter.unmapped_glyph_handling(), - segmenter.condition_analysis_mode(), segmenter.brotli_quality(), - segmenter.init_font_merging_brotli_quality(), resolver); + roboto.get(), {}, segments, segmenter.condition_analysis_mode(), + segmenter.brotli_quality(), segmenter.init_font_merging_brotli_quality(), + resolver); ASSERT_TRUE(context.ok()) << context.status(); auto* size_cache = new MockPatchSizeCache(); @@ -1935,11 +1870,11 @@ TEST_F(CandidateMergeTest, {{{'c', 'd', 'e'}}, 0.661}, }; - ClosureGlyphSegmenter segmenter(8, 8, PATCH, CLOSURE_ONLY, resolver); + ClosureGlyphSegmenter segmenter(8, 8, DEP_GRAPH_ONLY, resolver); auto context = SegmentationContext::InitializeSegmentationContext( - roboto.get(), {}, segments, segmenter.unmapped_glyph_handling(), - segmenter.condition_analysis_mode(), segmenter.brotli_quality(), - segmenter.init_font_merging_brotli_quality(), resolver); + roboto.get(), {}, segments, segmenter.condition_analysis_mode(), + segmenter.brotli_quality(), segmenter.init_font_merging_brotli_quality(), + resolver); ASSERT_TRUE(context.ok()) << context.status(); // Glyphs: 'a'=69, 'b'=70, 'c'=71, 'd'=72, 'e'=73. @@ -2038,11 +1973,11 @@ TEST_F(CandidateMergeTest, AllPairsMerging_ByteNormalization) { probs.push_back({Segment(def), 1.0}); } - ClosureGlyphSegmenter segmenter(8, 8, PATCH, CLOSURE_ONLY, resolver); + ClosureGlyphSegmenter segmenter(8, 8, DEP_GRAPH_ONLY, resolver); auto context = SegmentationContext::InitializeSegmentationContext( - roboto.get(), {}, segments, segmenter.unmapped_glyph_handling(), - segmenter.condition_analysis_mode(), segmenter.brotli_quality(), - segmenter.init_font_merging_brotli_quality(), resolver); + roboto.get(), {}, segments, segmenter.condition_analysis_mode(), + segmenter.brotli_quality(), segmenter.init_font_merging_brotli_quality(), + resolver); ASSERT_TRUE(context.ok()) << context.status(); const auto& groupings = context->glyph_groupings; @@ -2118,11 +2053,11 @@ TEST_F(CandidateMergeTest, ComputeCostDelta_BestCaseSkipsUnchangedGlyphSetCondit freq::UnicodeFrequencies frequencies{ {{' ', ' '}, 100}, {{'a', 'a'}, 50}, {{'f', 'f'}, 75}, {{'i', 'i'}, 95}}; - ClosureGlyphSegmenter segmenter(8, 8, PATCH, CLOSURE_ONLY, resolver); + ClosureGlyphSegmenter segmenter(8, 8, DEP_GRAPH_ONLY, resolver); auto context = SegmentationContext::InitializeSegmentationContext( - roboto.get(), {}, segments, segmenter.unmapped_glyph_handling(), - segmenter.condition_analysis_mode(), segmenter.brotli_quality(), - segmenter.init_font_merging_brotli_quality(), resolver); + roboto.get(), {}, segments, segmenter.condition_analysis_mode(), + segmenter.brotli_quality(), segmenter.init_font_merging_brotli_quality(), + resolver); ASSERT_TRUE(context.ok()) << context.status(); Merger merger = *Merger::New( @@ -2143,8 +2078,8 @@ TEST_F(CandidateMergeTest, ComputeCostDelta_BestCaseSkipsUnchangedGlyphSetCondit // In best-case mode, the delta contribution for (s1 AND s2) is treated as 0, // and its patch size {444, 446} is never queried. size_cache->ResetCallCount(); - auto best_case_delta = CandidateMerge::ComputeCostDelta( - merger, {0, 1}, std::nullopt, std::nullopt); + auto best_case_delta = + CandidateMerge::ComputeCostDelta(merger, {0, 1}, std::nullopt); ASSERT_TRUE(best_case_delta.ok()) << best_case_delta.status(); // Only the two merging exclusive patches {69} and {74} are queried for size. @@ -2177,11 +2112,11 @@ TEST_F(CandidateMergeTest, ComputeCostDelta_SkipsUnchangedEffectiveConditions) { {{0x00E9, 0x00E9}, 75}, {{0x0107, 0x0107}, 95}}; - ClosureGlyphSegmenter segmenter(8, 8, PATCH, CLOSURE_ONLY, resolver); + ClosureGlyphSegmenter segmenter(8, 8, DEP_GRAPH_ONLY, resolver); auto context = SegmentationContext::InitializeSegmentationContext( - roboto.get(), {}, segments, segmenter.unmapped_glyph_handling(), - segmenter.condition_analysis_mode(), segmenter.brotli_quality(), - segmenter.init_font_merging_brotli_quality(), resolver); + roboto.get(), {}, segments, segmenter.condition_analysis_mode(), + segmenter.brotli_quality(), segmenter.init_font_merging_brotli_quality(), + resolver); ASSERT_TRUE(context.ok()) << context.status(); ASSERT_FALSE(context->InertSegments().contains(0)); ASSERT_FALSE(context->InertSegments().contains(1)); @@ -2213,8 +2148,8 @@ TEST_F(CandidateMergeTest, ComputeCostDelta_SkipsUnchangedEffectiveConditions) { // change. Its cost delta is 0 in both best-case and exact mode, and g_acute's // patch size is never queried. size_cache->ResetCallCount(); - auto best_delta = CandidateMerge::ComputeCostDelta( - merger, {0, 1}, std::nullopt, std::nullopt); + auto best_delta = + CandidateMerge::ComputeCostDelta(merger, {0, 1}, std::nullopt); ASSERT_TRUE(best_delta.ok()) << best_delta.status(); EXPECT_EQ(size_cache->GetPatchSizeCallCount(), 2); // only g0 and g1 @@ -2225,8 +2160,8 @@ TEST_F(CandidateMergeTest, ComputeCostDelta_SkipsUnchangedEffectiveConditions) { EXPECT_NEAR(*best_delta, expected_best, 1e-9); size_cache->ResetCallCount(); - auto exact_delta = CandidateMerge::ComputeCostDelta( - merger, {0, 1}, g01, std::nullopt); + auto exact_delta = + CandidateMerge::ComputeCostDelta(merger, {0, 1}, std::nullopt); ASSERT_TRUE(exact_delta.ok()) << exact_delta.status(); // In exact mode: g0, g1, and g01 are queried; g_acute is NOT. EXPECT_EQ(size_cache->GetPatchSizeCallCount(), 3); @@ -2252,11 +2187,11 @@ TEST_F(CandidateMergeTest, {{0x00E9, 0x00E9}, 75}, {{0x0107, 0x0107}, 95}}; - ClosureGlyphSegmenter segmenter(8, 8, PATCH, CLOSURE_ONLY, resolver); + ClosureGlyphSegmenter segmenter(8, 8, DEP_GRAPH_ONLY, resolver); auto context = SegmentationContext::InitializeSegmentationContext( - roboto.get(), {}, segments, segmenter.unmapped_glyph_handling(), - segmenter.condition_analysis_mode(), segmenter.brotli_quality(), - segmenter.init_font_merging_brotli_quality(), resolver); + roboto.get(), {}, segments, segmenter.condition_analysis_mode(), + segmenter.brotli_quality(), segmenter.init_font_merging_brotli_quality(), + resolver); ASSERT_TRUE(context.ok()) << context.status(); Merger merger = *Merger::New( @@ -2277,9 +2212,9 @@ TEST_F(CandidateMergeTest, context->glyph_condition_set.AddOrCondition(gid_d, 0); context->glyph_condition_set.AddOrCondition(gid_d, 2); - auto sc = context->glyph_groupings.GroupGlyphs( - context->SegmentationInfo(), context->glyph_condition_set, - *context->glyph_closure_cache, std::nullopt, {gid_d}, {0, 2}, false); + auto sc = context->glyph_groupings.GroupGlyphs(context->SegmentationInfo(), + context->glyph_condition_set, + {gid_d}, {0, 2}); ASSERT_TRUE(sc.ok()) << sc; GlyphSet g_acute_and_d = g_acute; @@ -2298,8 +2233,8 @@ TEST_F(CandidateMergeTest, // Even though (s0 OR s1 OR s2) individually has an unchanged effective // condition, it merges into (s0 OR s2) where (s0 OR s2) already exists! // So their glyph set becomes g_acute U {gid_d}. Both patches must be evaluated. - auto exact_delta = CandidateMerge::ComputeCostDelta( - merger, {0, 1}, g01, std::nullopt); + auto exact_delta = + CandidateMerge::ComputeCostDelta(merger, {0, 1}, std::nullopt); ASSERT_TRUE(exact_delta.ok()) << exact_delta.status(); double p_0_or_1 = 0.50 + 0.75 - 0.50 * 0.75; diff --git a/ift/encoder/closure_glyph_segmenter.cc b/ift/encoder/closure_glyph_segmenter.cc index e4c6ccc4..8bca135f 100644 --- a/ift/encoder/closure_glyph_segmenter.cc +++ b/ift/encoder/closure_glyph_segmenter.cc @@ -23,6 +23,7 @@ #include "ift/common/try.h" #include "ift/common/woff2.h" #include "ift/config/auto_segmenter_config.h" +#include "ift/config/common.pb.h" #include "ift/encoder/activation_condition.h" #include "ift/encoder/glyph_groupings.h" #include "ift/encoder/glyph_segmentation.h" @@ -40,11 +41,9 @@ using ift::config::FROM_FREQ_DATA; using ift::config::FROM_MERGE_GROUPS; using ift::config::kDefaultNetworkCost; -using ift::config::MOVE_TO_INIT_FONT; using ift::config::SegmentationPlan; using ift::config::SegmentsProto; using ift::config::TableKeyedSegmentMode; -using ift::config::UnmappedGlyphHandling; using absl::btree_map; using absl::btree_set; @@ -128,69 +127,6 @@ static void PrintDiff(const GlyphGroupings& a, const GlyphGroupings& b) { } } -// Returns true if the glyph groupings a and b are equivalent (not necessarily -// equal). -// -// Complex condition finding can return multiple possible valid conditions for a -// glyph so for these we allowe differences as long as the closure condition is -// satisfied (that is there are no additional conditions.) -static StatusOr GlyphGroupingsAreEquivalent( - const SegmentationContext& context, const GlyphGroupings& a, - const GlyphGroupings& b) { - if (a.ConditionsAndGlyphs() == b.ConditionsAndGlyphs()) { - return true; - } - - for (glyph_id_t gid = 0; - gid < hb_face_get_glyph_count(context.original_face.get()); gid++) { - auto maybe_cond_a = a.GlyphToCondition(gid); - auto maybe_cond_b = b.GlyphToCondition(gid); - - if (maybe_cond_a.has_value() != maybe_cond_b.has_value()) { - return false; - } - - if (!maybe_cond_a.has_value()) { - continue; - } - - ActivationCondition cond_a = *maybe_cond_a; - ActivationCondition cond_b = *maybe_cond_b; - - if (cond_a == cond_b) { - continue; - } - - if (!a.FoundConditionGlyphs().contains(gid) || - !b.FoundConditionGlyphs().contains(gid)) { - // diffs only allowed if gid is a found conditions gid in both a and b. - return false; - } - - if (cond_a.conditions().size() > 1 || cond_b.conditions().size() > 1) { - // diffs only allowed for purely disjunctive conditions - return false; - } - - if (TRY(context.glyph_closure_cache->HasAdditionalConditions( - &context.SegmentationInfo(), cond_a.TriggeringSegments(), - GlyphSet{gid}))) { - return false; - } - - if (TRY(context.glyph_closure_cache->HasAdditionalConditions( - &context.SegmentationInfo(), cond_b.TriggeringSegments(), - GlyphSet{gid}))) { - return false; - } - - // If no additional conditions are present on either side then we allow the - // diff. - } - - return true; -} - /* * Checks that the incrementally generated glyph conditions and groupings in * context match what would have been produced by a non incremental process. @@ -211,20 +147,13 @@ Status ValidateIncrementalGroupings(hb_face_t* face, } TRYV(non_incremental_context.ReprocessAll()); - bool glyph_groupings_diffs_allowed = false; if (non_incremental_context.glyph_groupings.ConditionsAndGlyphs() != context.glyph_groupings.ConditionsAndGlyphs()) { - if (!TRY(GlyphGroupingsAreEquivalent( - context, context.glyph_groupings, - non_incremental_context.glyph_groupings))) { - VLOG(0) << "-- incremental grouping"; - VLOG(0) << "++ non-incremental grouping"; - PrintDiff(context.glyph_groupings, - non_incremental_context.glyph_groupings); - return absl::FailedPreconditionError( - "conditions_and_glyphs aren't correct."); - } - glyph_groupings_diffs_allowed = true; + VLOG(0) << "-- incremental grouping"; + VLOG(0) << "++ non-incremental grouping"; + PrintDiff(context.glyph_groupings, non_incremental_context.glyph_groupings); + return absl::FailedPreconditionError( + "conditions_and_glyphs aren't correct."); } if (non_incremental_context.glyph_condition_set != @@ -232,8 +161,7 @@ Status ValidateIncrementalGroupings(hb_face_t* face, return absl::FailedPreconditionError("glyph_condition_set isn't correct."); } - if (!glyph_groupings_diffs_allowed && - non_incremental_context.glyph_groupings != context.glyph_groupings) { + if (non_incremental_context.glyph_groupings != context.glyph_groupings) { return absl::FailedPreconditionError("glyph groups aren't correct."); } @@ -609,6 +537,15 @@ static StatusOr ToFinalSegmentation( Status ClosureGlyphSegmenter::ValidateInput( const std::vector& subset_definitions, const btree_map& merge_groups) const { + if (condition_analysis_mode_ != ift::config::DEP_GRAPH_ONLY && + condition_analysis_mode_ != + ift::config::DEP_GRAPH_ONLY_WITH_SIMPLIFICATION) { + return absl::InvalidArgumentError( + "Closure-based condition_analysis_mode values are deprecated and " + "unsupported; use DEP_GRAPH_ONLY or " + "DEP_GRAPH_ONLY_WITH_SIMPLIFICATION."); + } + if (disjoint_conditions_probability_threshold_ < 0.0 || disjoint_conditions_probability_threshold_ > 1.0) { return absl::InvalidArgumentError( @@ -616,13 +553,6 @@ Status ClosureGlyphSegmenter::ValidateInput( } if (target_mode_ == ift::config::UNICODE_RANGE) { - if (unmapped_glyph_handling_ != ift::config::FIND_CONDITIONS && - unmapped_glyph_handling_ != ift::config::PATCH) { - return absl::InvalidArgumentError( - "unmapped_glyph_handling must be FIND_CONDITIONS or PATCH when " - "target_mode is UNICODE_RANGE."); - } - for (const auto& def : subset_definitions) { if (!def.feature_tags.empty()) { return absl::InvalidArgumentError( @@ -671,7 +601,7 @@ static StatusOr ShouldMergeForDisjointConditions( const SegmentationContext& context, const ActivationCondition& condition, double probability_threshold, const btree_map& merge_groups) { - if (condition.IsExclusive() || condition.IsFallback()) { + if (condition.IsExclusive()) { return false; } if (merge_groups.empty()) { @@ -726,7 +656,7 @@ static StatusOr PartitionInteractingSegments( GlyphPartition segment_partition(num_segments); for (const auto& condition : context.glyph_groupings.OrderedConditions()) { SegmentSet triggering = condition.TriggeringSegments(); - if (condition.IsExclusive() || condition.IsFallback() || triggering.size() <= 1) { + if (condition.IsExclusive() || triggering.size() <= 1) { continue; } @@ -836,9 +766,9 @@ StatusOr ClosureGlyphSegmenter::CodepointToGlyphSegments( TRY(ToOrderedSegments(subset_definitions, merge_groups, with_shared)); SegmentationContext context = TRY(SegmentationContext::InitializeSegmentationContext( - face, initial_segment, std::move(segments), unmapped_glyph_handling_, - condition_analysis_mode_, brotli_quality_, - init_font_merging_brotli_quality_, resolver_, target_mode_)); + face, initial_segment, std::move(segments), condition_analysis_mode_, + brotli_quality_, init_font_merging_brotli_quality_, resolver_, + target_mode_)); if (target_mode_ == ift::config::UNICODE_RANGE) { std::vector calculators; @@ -880,20 +810,6 @@ StatusOr ClosureGlyphSegmenter::CodepointToGlyphSegments( } } - // Once we've gotten standard segments placed into the initial font as needed, - // if requested any remaining fallback glyphs are also moved into the init - // font. - GlyphSet fallback_glyphs = context.glyph_groupings.UnmappedGlyphs(); - if (unmapped_glyph_handling_ == MOVE_TO_INIT_FONT && - !fallback_glyphs.empty()) { - VLOG(0) << "Moving " << fallback_glyphs.size() - << " fallback glyphs into the initial font." << std::endl; - SubsetDefinition new_def = context.SegmentationInfo().InitFontSegment(); - new_def.gids.union_set(fallback_glyphs); - TRYV(context.ReassignInitSubset(new_def)); - init_font_changed = true; - } - if (init_font_changed) { for (Merger& merger : mergers) { // Any init font moves above can cause segment removals that affect other @@ -1072,23 +988,6 @@ StatusOr> ClosureGlyphSegmenter::TotalCosts( return out; } -Status ClosureGlyphSegmenter::FallbackCost( - hb_face_t* original_face, const GlyphSegmentation& segmentation, - uint32_t& fallback_glyphs_size, uint32_t& all_glyphs_size) const { - GlyphSet all_glyphs = segmentation.InitialFontGlyphClosure(); - for (const auto& [_, gids] : segmentation.GidSegments()) { - all_glyphs.union_set(gids); - } - - GlyphSet fallback_glyphs = segmentation.UnmappedGlyphs(); - - PatchSizeCacheImpl patch_sizer(original_face, 11); - all_glyphs_size = TRY(patch_sizer.GetPatchSize(all_glyphs)); - fallback_glyphs_size = TRY(patch_sizer.GetPatchSize(fallback_glyphs)); - - return absl::OkStatus(); -} - static std::vector TableKeyedSegmentsFromMergeGroups( const btree_map& merge_groups, const std::vector& segments, diff --git a/ift/encoder/closure_glyph_segmenter.h b/ift/encoder/closure_glyph_segmenter.h index 40314e20..a96984f7 100644 --- a/ift/encoder/closure_glyph_segmenter.h +++ b/ift/encoder/closure_glyph_segmenter.h @@ -8,7 +8,6 @@ #include "absl/container/btree_map.h" #include "absl/status/statusor.h" #include "ift/common/data_file_resolver.h" -#include "ift/config/common.pb.h" #include "ift/config/segmentation_plan.pb.h" #include "ift/config/segmenter_config.pb.h" #include "ift/encoder/glyph_segmentation.h" @@ -40,14 +39,12 @@ class ClosureGlyphSegmenter { public: ClosureGlyphSegmenter( uint32_t brotli_quality, uint32_t init_font_merging_brotli_quality, - ift::config::UnmappedGlyphHandling unmapped_glyph_handling, ift::config::ConditionAnalysisMode condition_analysis_mode, std::shared_ptr resolver, ift::config::TargetMode target_mode = ift::config::IFT, double disjoint_conditions_probability_threshold = 0.0) : brotli_quality_(brotli_quality), init_font_merging_brotli_quality_(init_font_merging_brotli_quality), - unmapped_glyph_handling_(unmapped_glyph_handling), condition_analysis_mode_(condition_analysis_mode), resolver_(std::move(resolver)), target_mode_(target_mode), @@ -82,15 +79,6 @@ class ClosureGlyphSegmenter { absl::Span probability_calculators) const; - /* - * Computes the total cost of the fallback patch (expected number of bytes - * transferred) - */ - absl::Status FallbackCost(hb_face_t* original_face, - const GlyphSegmentation& segmentation, - uint32_t& fallback_glyphs_size, - uint32_t& all_glyphs_size) const; - static absl::Status AddTableKeyedSegments( ift::config::SegmentationPlan& plan, const absl::btree_map& @@ -104,9 +92,6 @@ class ClosureGlyphSegmenter { uint32_t init_font_merging_brotli_quality() const { return init_font_merging_brotli_quality_; } - ift::config::UnmappedGlyphHandling unmapped_glyph_handling() const { - return unmapped_glyph_handling_; - } ift::config::ConditionAnalysisMode condition_analysis_mode() const { return condition_analysis_mode_; } @@ -123,7 +108,6 @@ class ClosureGlyphSegmenter { uint32_t brotli_quality_; uint32_t init_font_merging_brotli_quality_; - ift::config::UnmappedGlyphHandling unmapped_glyph_handling_; ift::config::ConditionAnalysisMode condition_analysis_mode_; std::shared_ptr resolver_; ift::config::TargetMode target_mode_; diff --git a/ift/encoder/closure_glyph_segmenter_test.cc b/ift/encoder/closure_glyph_segmenter_test.cc index 7002ab0b..c47d2af5 100644 --- a/ift/encoder/closure_glyph_segmenter_test.cc +++ b/ift/encoder/closure_glyph_segmenter_test.cc @@ -22,14 +22,9 @@ #include "ift/freq/unicode_frequencies.h" #include "ift/freq/unigram_probability_calculator.h" -using ift::config::CLOSURE_AND_VALIDATE_DEP_GRAPH; -using ift::config::CLOSURE_ONLY; using ift::config::DEP_GRAPH_ONLY; using ift::config::DEP_GRAPH_ONLY_WITH_SIMPLIFICATION; -using ift::config::FIND_CONDITIONS; using ift::config::kDefaultNetworkCost; -using ift::config::MOVE_TO_INIT_FONT; -using ift::config::PATCH; using absl::btree_map; using absl::StatusOr; @@ -58,20 +53,9 @@ class ClosureGlyphSegmenterTest : public ::testing::Test { noto_sans_jp(make_hb_face(nullptr)), resolver(*BazelDataFileResolver::CreateForTest()), - segmenter(8, 8, PATCH, CLOSURE_ONLY, resolver), - segmenter_find_conditions(8, 8, FIND_CONDITIONS, CLOSURE_ONLY, - resolver), - segmenter_move_to_init_font(8, 8, MOVE_TO_INIT_FONT, CLOSURE_ONLY, - resolver), - segmenter_dep_graph(8, 8, PATCH, CLOSURE_AND_VALIDATE_DEP_GRAPH, - resolver), - segmenter_dep_graph_only(8, 8, PATCH, DEP_GRAPH_ONLY, resolver), + segmenter(8, 8, DEP_GRAPH_ONLY, resolver), segmenter_dep_graph_only_with_simplification( - 8, 8, PATCH, DEP_GRAPH_ONLY_WITH_SIMPLIFICATION, resolver), - segmenter_find_conditions_dep_graph( - 8, 8, FIND_CONDITIONS, CLOSURE_AND_VALIDATE_DEP_GRAPH, resolver), - segmenter_move_to_init_font_dep_graph( - 8, 8, MOVE_TO_INIT_FONT, CLOSURE_AND_VALIDATE_DEP_GRAPH, resolver) { + 8, 8, DEP_GRAPH_ONLY_WITH_SIMPLIFICATION, resolver) { roboto = from_file("ift/common/testdata/Roboto-Regular.ttf"); noto_nastaliq_urdu = from_file("ift/common/testdata/NotoNastaliqUrdu.subset.ttf"); @@ -87,98 +71,16 @@ class ClosureGlyphSegmenterTest : public ::testing::Test { hb_face_t* face, SubsetDefinition initial_segment, const std::vector& subset_definitions, std::optional strategy = std::nullopt) const { - auto segmentation = TRY(segmenter.CodepointToGlyphSegments( - face, initial_segment, subset_definitions, strategy)); - auto dep_graph_segmentation = - TRY(segmenter_dep_graph.CodepointToGlyphSegments( - face, initial_segment, subset_definitions, strategy)); - auto dep_graph_only_segmentation = - TRY(segmenter_dep_graph_only.CodepointToGlyphSegments( - face, initial_segment, subset_definitions, strategy)); - - if (segmentation != dep_graph_segmentation) { - return absl::InternalError( - "Basic segmenter: segmentation from dep graph is not equivalent to " - "the non-dep graph version."); - } - if (segmentation != dep_graph_only_segmentation) { - std::cerr << "Closure segmentation: " << segmentation.ToString() - << std::endl; - std::cerr << "Dep graph only segmentation: " - << dep_graph_only_segmentation.ToString() << std::endl; - return absl::InternalError( - "Basic segmenter: segmentation from dep graph only is not equivalent " - "to " - "the non-dep graph version."); - } - - return segmentation; + return segmenter.CodepointToGlyphSegments(face, initial_segment, + subset_definitions, strategy); } StatusOr CodepointToGlyphSegments( hb_face_t* face, SubsetDefinition initial_segment, const std::vector& subset_definitions, btree_map merge_groups) const { - auto segmentation = TRY(segmenter.CodepointToGlyphSegments( - face, initial_segment, subset_definitions, merge_groups)); - auto dep_graph_segmentation = - TRY(segmenter_dep_graph.CodepointToGlyphSegments( - face, initial_segment, subset_definitions, merge_groups)); - auto dep_graph_only_segmentation = - TRY(segmenter_dep_graph_only.CodepointToGlyphSegments( - face, initial_segment, subset_definitions, merge_groups)); - - if (segmentation != dep_graph_segmentation) { - return absl::InternalError( - "Basic segmenter: segmentation from dep graph is not equivalent to " - "the non-dep graph version."); - } - if (segmentation != dep_graph_only_segmentation) { - std::cerr << "Closure segmentation: " << segmentation.ToString() - << std::endl; - std::cerr << "Dep graph only segmentation: " - << dep_graph_only_segmentation.ToString() << std::endl; - return absl::InternalError( - "Basic segmenter: segmentation from dep graph only is not equivalent " - "to " - "the non-dep graph version."); - } - - return segmentation; - } - - StatusOr FindConditionsCodepointToGlyphSegments( - hb_face_t* face, SubsetDefinition initial_segment, - const std::vector& subset_definitions, - std::optional strategy = std::nullopt) const { - auto segmentation = segmenter_find_conditions.CodepointToGlyphSegments( - face, initial_segment, subset_definitions, strategy); - auto dep_graph_segmentation = - segmenter_find_conditions_dep_graph.CodepointToGlyphSegments( - face, initial_segment, subset_definitions, strategy); - if (segmentation != dep_graph_segmentation) { - return absl::InternalError( - "Find conditions segmenter: segmentation from dep graph is not " - "equivalent to the non-dep graph version."); - } - return segmentation; - } - - StatusOr MoveToInitFontCodepointToGlyphSegments( - hb_face_t* face, SubsetDefinition initial_segment, - const std::vector& subset_definitions, - absl::btree_map merge_groups) const { - auto segmentation = segmenter_move_to_init_font.CodepointToGlyphSegments( - face, initial_segment, subset_definitions, merge_groups); - auto dep_graph_segmentation = - segmenter_move_to_init_font_dep_graph.CodepointToGlyphSegments( - face, initial_segment, subset_definitions, merge_groups); - if (segmentation != dep_graph_segmentation) { - return absl::InternalError( - "Move to init font segmenter: segmentation from dep graph is not " - "equivalent to the non-dep graph version."); - } - return segmentation; + return segmenter.CodepointToGlyphSegments(face, initial_segment, + subset_definitions, merge_groups); } hb_face_unique_ptr roboto; @@ -187,13 +89,7 @@ class ClosureGlyphSegmenterTest : public ::testing::Test { std::shared_ptr resolver; ClosureGlyphSegmenter segmenter; - ClosureGlyphSegmenter segmenter_find_conditions; - ClosureGlyphSegmenter segmenter_move_to_init_font; - ClosureGlyphSegmenter segmenter_dep_graph; - ClosureGlyphSegmenter segmenter_dep_graph_only; ClosureGlyphSegmenter segmenter_dep_graph_only_with_simplification; - ClosureGlyphSegmenter segmenter_find_conditions_dep_graph; - ClosureGlyphSegmenter segmenter_move_to_init_font_dep_graph; }; TEST_F(ClosureGlyphSegmenterTest, SimpleSegmentation) { @@ -335,16 +231,9 @@ if ((s0 OR s1)) then p1 } TEST_F(ClosureGlyphSegmenterTest, SegmentationWithAdditionalConditionOverlap) { - auto segmentation = segmenter.CodepointToGlyphSegments( - roboto.get(), {'a'}, {{'f'}, {'i'}, {'f', 'i'}}); - auto dep_graph_segmentation = segmenter_dep_graph.CodepointToGlyphSegments( - roboto.get(), {'a'}, {{'f'}, {'i'}, {'f', 'i'}}); - auto dep_graph_only_segmentation = - segmenter_dep_graph_only.CodepointToGlyphSegments( - roboto.get(), {'a'}, {{'f'}, {'i'}, {'f', 'i'}}); + auto segmentation = + CodepointToGlyphSegments(roboto.get(), {'a'}, {{'f'}, {'i'}, {'f', 'i'}}); ASSERT_TRUE(segmentation.ok()) << segmentation.status(); - ASSERT_TRUE(dep_graph_segmentation.ok()) << segmentation.status(); - ASSERT_TRUE(dep_graph_only_segmentation.ok()) << segmentation.status(); std::vector expected_segments = {{'f'}, {'i'}, {'f', 'i'}}; ASSERT_EQ(segmentation->Segments(), expected_segments); @@ -356,17 +245,6 @@ p1: { gid77 } p2: { gid444, gid446 } if ((s0 OR s2)) then p0 if ((s1 OR s2)) then p1 -if ((s0 OR s1 OR s2)) then p2 -)"); - ASSERT_EQ(segmentation->ToString(), dep_graph_segmentation->ToString()); - - ASSERT_EQ(dep_graph_only_segmentation->ToString(), - R"(initial font: { gid0, gid69 } -p0: { gid74 } -p1: { gid77 } -p2: { gid444, gid446 } -if ((s0 OR s2)) then p0 -if ((s1 OR s2)) then p1 if ((s0 OR s2) AND (s1 OR s2)) then p2 )"); } @@ -589,101 +467,12 @@ if (s0 AND s1) then p3 )"); } -TEST_F(ClosureGlyphSegmenterTest, UnmappedGlyphs_FallbackSegment) { - auto segmentation = segmenter.CodepointToGlyphSegments( - noto_nastaliq_urdu.get(), {}, {{0x62a}, {0x62b}, {0x62c}, {0x62d}}); - auto dep_graph_segmentation = segmenter_dep_graph.CodepointToGlyphSegments( - noto_nastaliq_urdu.get(), {}, {{0x62a}, {0x62b}, {0x62c}, {0x62d}}); - auto dep_graph_only_segmentation = - segmenter_dep_graph_only.CodepointToGlyphSegments( - noto_nastaliq_urdu.get(), {}, {{0x62a}, {0x62b}, {0x62c}, {0x62d}}); - ASSERT_TRUE(segmentation.ok()) << segmentation.status(); - ASSERT_TRUE(dep_graph_segmentation.ok()) << segmentation.status(); - ASSERT_TRUE(dep_graph_only_segmentation.ok()) << segmentation.status(); - - ASSERT_EQ(segmentation->UnmappedGlyphs().size(), 12); - - ASSERT_EQ(segmentation->ToString(), - R"(initial font: { gid0 } -p0: { gid3, gid9, gid155 } -p1: { gid4, gid10, gid156 } -p2: { gid5, gid6, gid11, gid157 } -p3: { gid158 } -p4: { gid12, gid13, gid24, gid30, gid38, gid39, gid57, gid59, gid62, gid68, gid139, gid140, gid153, gid172 } -p5: { gid14, gid33, gid60, gid91, gid112, gid145, gid152 } -p6: { gid47, gid64, gid73, gid74, gid75, gid76, gid77, gid83, gid111, gid149, gid174, gid190, gid191 } -if (s0) then p0 -if (s1) then p1 -if (s2) then p2 -if (s3) then p3 -if ((s0 OR s1)) then p4 -if ((s2 OR s3)) then p5 -if ((s0 OR s1 OR s2 OR s3)) then p6 -)"); - ASSERT_EQ(segmentation->ToString(), dep_graph_segmentation->ToString()); - - ASSERT_EQ(dep_graph_only_segmentation->ToString(), - R"(initial font: { gid0 } -p0: { gid3, gid9, gid155 } -p1: { gid4, gid10, gid156 } -p2: { gid5, gid6, gid157 } -p3: { gid158 } -p4: { gid12, gid13, gid24, gid30, gid38, gid39, gid57, gid59, gid62, gid64, gid68, gid77, gid139, gid140, gid153, gid172 } -p5: { gid14, gid33, gid60, gid91, gid112, gid145, gid152, gid190, gid191 } -p6: { gid174 } -p7: { gid11 } -p8: { gid47, gid73, gid74, gid75, gid76, gid83, gid111, gid149 } -if (s0) then p0 -if (s1) then p1 -if (s2) then p2 -if (s3) then p3 -if ((s0 OR s1)) then p4 -if ((s2 OR s3)) then p5 -if ((s0 OR s1 OR s2 OR s3)) then p6 -if ((s0 OR s1) AND s2) then p7 -if ((s0 OR s1) AND (s2 OR s3)) then p8 -)"); -} - -TEST_F(ClosureGlyphSegmenterTest, UnmappedGlyphs_FindConditions) { - auto segmentation = FindConditionsCodepointToGlyphSegments( - noto_nastaliq_urdu.get(), {}, - {{0x20}, {0x62a}, {0x62b}, {0x62c}, {0x62d}}, std::nullopt); - ASSERT_TRUE(segmentation.ok()) << segmentation.status(); - - ASSERT_TRUE(segmentation->UnmappedGlyphs().empty()) - << segmentation->UnmappedGlyphs().ToString(); - - ASSERT_EQ(segmentation->ToString(), - R"(initial font: { gid0 } -p0: { gid1 } -p1: { gid3, gid9, gid155 } -p2: { gid4, gid10, gid156 } -p3: { gid5, gid6, gid11, gid157 } -p4: { gid158 } -p5: { gid12, gid13, gid24, gid30, gid38, gid39, gid57, gid59, gid62, gid68, gid139, gid140, gid153, gid172 } -p6: { gid14, gid33, gid60, gid91, gid112, gid145, gid152 } -p7: { gid47, gid64, gid73, gid74, gid75, gid76, gid77, gid83, gid111, gid149, gid174, gid190, gid191 } -if (s0) then p0 -if (s1) then p1 -if (s2) then p2 -if (s3) then p3 -if (s4) then p4 -if ((s1 OR s2)) then p5 -if ((s3 OR s4)) then p6 -if ((s1 OR s2 OR s3 OR s4)) then p7 -)"); -} - TEST_F(ClosureGlyphSegmenterTest, DepGraphOnly_FindConditions) { - auto segmentation = segmenter_dep_graph_only.CodepointToGlyphSegments( + auto segmentation = segmenter.CodepointToGlyphSegments( noto_nastaliq_urdu.get(), {}, {{0x20}, {0x62a}, {0x62b}, {0x62c}, {0x62d}}); ASSERT_TRUE(segmentation.ok()) << segmentation.status(); - ASSERT_TRUE(segmentation->UnmappedGlyphs().empty()) - << segmentation->UnmappedGlyphs().ToString(); - ASSERT_EQ(segmentation->ToString(), R"(initial font: { gid0 } p0: { gid1 } @@ -716,9 +505,6 @@ TEST_F(ClosureGlyphSegmenterTest, DepGraphOnly_WithSimplification) { {{0x20}, {0x62a}, {0x62b}, {0x62c}, {0x62d}}); ASSERT_TRUE(segmentation.ok()) << segmentation.status(); - ASSERT_TRUE(segmentation->UnmappedGlyphs().empty()) - << segmentation->UnmappedGlyphs().ToString(); - ASSERT_EQ(segmentation->ToString(), R"(initial font: { gid0 } p0: { gid1 } @@ -740,61 +526,6 @@ if ((s1 OR s2 OR s3 OR s4)) then p7 )"); } -TEST_F(ClosureGlyphSegmenterTest, UnmappedGlyphs_FindConditions_IsFallback) { - // Here the found conditions are equal to the fallback segment, this ensures - // everything works properly in this case. - auto segmentation = FindConditionsCodepointToGlyphSegments( - noto_nastaliq_urdu.get(), {}, {{0x62a}, {0x62b}, {0x62c}, {0x62d}}, - std::nullopt); - ASSERT_TRUE(segmentation.ok()) << segmentation.status(); - ASSERT_TRUE(segmentation->UnmappedGlyphs().empty()) - << segmentation->UnmappedGlyphs().ToString(); - ASSERT_EQ(segmentation->ToString(), - R"(initial font: { gid0 } -p0: { gid3, gid9, gid155 } -p1: { gid4, gid10, gid156 } -p2: { gid5, gid6, gid11, gid157 } -p3: { gid158 } -p4: { gid12, gid13, gid24, gid30, gid38, gid39, gid57, gid59, gid62, gid68, gid139, gid140, gid153, gid172 } -p5: { gid14, gid33, gid60, gid91, gid112, gid145, gid152 } -p6: { gid47, gid64, gid73, gid74, gid75, gid76, gid77, gid83, gid111, gid149, gid174, gid190, gid191 } -if (s0) then p0 -if (s1) then p1 -if (s2) then p2 -if (s3) then p3 -if ((s0 OR s1)) then p4 -if ((s2 OR s3)) then p5 -if ((s0 OR s1 OR s2 OR s3)) then p6 -)"); -} - -TEST_F(ClosureGlyphSegmenterTest, - UnmappedGlyphs_FallbackSegmentMovedToInitFont) { - auto segmentation = MoveToInitFontCodepointToGlyphSegments( - noto_nastaliq_urdu.get(), {}, {{0x62a}, {0x62b}, {0x62c}, {0x62d}}, - btree_map{}); - ASSERT_TRUE(segmentation.ok()) << segmentation.status(); - - ASSERT_EQ(segmentation->UnmappedGlyphs().size(), 0); - - ASSERT_EQ( - segmentation->ToString(), - R"(initial font: { gid0, gid47, gid64, gid73, gid74, gid75, gid76, gid77, gid83, gid111, gid149, gid174, gid190, gid191 } -p0: { gid3, gid9, gid155 } -p1: { gid4, gid10, gid156 } -p2: { gid5, gid6, gid11, gid157 } -p3: { gid158 } -p4: { gid12, gid13, gid24, gid30, gid38, gid39, gid57, gid59, gid62, gid68, gid139, gid140, gid153, gid172 } -p5: { gid14, gid33, gid60, gid91, gid112, gid145, gid152 } -if (s0) then p0 -if (s1) then p1 -if (s2) then p2 -if (s3) then p3 -if ((s0 OR s1)) then p4 -if ((s2 OR s3)) then p5 -)"); -} - TEST_F(ClosureGlyphSegmenterTest, FullRoboto_WithFeaturesAndDepGraph) { auto codepoints = FontHelper::ToCodepointsSet(roboto.get()); @@ -822,9 +553,8 @@ TEST_F(ClosureGlyphSegmenterTest, FullRoboto_WithFeaturesAndDepGraph) { smcp.feature_tags.insert(HB_TAG('s', 'm', 'c', 'p')); segments.push_back(smcp); - auto dep_graph_only_segmentation = - segmenter_dep_graph_only.CodepointToGlyphSegments( - roboto.get(), {}, segments, MergeStrategy::Heuristic(4000, 12000)); + auto dep_graph_only_segmentation = segmenter.CodepointToGlyphSegments( + roboto.get(), {}, segments, MergeStrategy::Heuristic(4000, 12000)); ASSERT_TRUE(dep_graph_only_segmentation.ok()) << dep_graph_only_segmentation.status(); } @@ -983,15 +713,15 @@ TEST_F(ClosureGlyphSegmenterTest, SimpleSegmentation_PatchMerge) { strategy.SetOptimizationCutoffFraction(0.0); strategy.SetUsePatchMerges(true); - auto segmentation = segmenter_dep_graph_only.CodepointToGlyphSegments( - roboto.get(), {}, - { - {'A'}, - {'C'}, - {0xC1}, /* Aacute */ - {0x106}, /* Cacute */ - }, - strategy); + auto segmentation = + segmenter.CodepointToGlyphSegments(roboto.get(), {}, + { + {'A'}, + {'C'}, + {0xC1}, /* Aacute */ + {0x106}, /* Cacute */ + }, + strategy); ASSERT_TRUE(segmentation.ok()) << segmentation.status(); std::vector expected_segments = { @@ -1024,15 +754,15 @@ TEST_F(ClosureGlyphSegmenterTest, SimpleSegmentation_NoPatchMerge) { strategy.SetOptimizationCutoffFraction(0.0); strategy.SetUsePatchMerges(false); - auto segmentation = segmenter_dep_graph_only.CodepointToGlyphSegments( - roboto.get(), {}, - { - {'A'}, - {'C'}, - {0xC1}, /* Aacute */ - {0x106}, /* Cacute */ - }, - strategy); + auto segmentation = + segmenter.CodepointToGlyphSegments(roboto.get(), {}, + { + {'A'}, + {'C'}, + {0xC1}, /* Aacute */ + {0x106}, /* Cacute */ + }, + strategy); ASSERT_TRUE(segmentation.ok()) << segmentation.status(); std::vector expected_segments = { @@ -1066,15 +796,15 @@ TEST_F(ClosureGlyphSegmenterTest, SimpleSegmentation_PatchMerge_MinGroupSize) { strategy.SetOptimizationCutoffFraction(0.0); strategy.SetUsePatchMerges(true); - auto segmentation = segmenter_dep_graph_only.CodepointToGlyphSegments( - roboto.get(), {}, - { - {'A'}, - {'C'}, - {0xC1}, /* Aacute */ - {0x106}, /* Cacute */ - }, - strategy); + auto segmentation = + segmenter.CodepointToGlyphSegments(roboto.get(), {}, + { + {'A'}, + {'C'}, + {0xC1}, /* Aacute */ + {0x106}, /* Cacute */ + }, + strategy); ASSERT_TRUE(segmentation.ok()) << segmentation.status(); std::vector expected_segments = { @@ -1103,15 +833,14 @@ if ((s1 OR s3) AND (s2 OR s3)) then p4 // Now with min group size larger merges will be done to reach min group size strategy.SetMinimumGroupSize(3); - segmentation = segmenter_dep_graph_only.CodepointToGlyphSegments( - roboto.get(), {}, - { - {'A'}, - {'C'}, - {0xC1}, /* Aacute */ - {0x106}, /* Cacute */ - }, - strategy); + segmentation = segmenter.CodepointToGlyphSegments(roboto.get(), {}, + { + {'A'}, + {'C'}, + {0xC1}, /* Aacute */ + {0x106}, /* Cacute */ + }, + strategy); ASSERT_TRUE(segmentation.ok()) << segmentation.status(); ASSERT_EQ(segmentation->Segments(), expected_segments); // Group size minimum is met for everything other than the last condition @@ -1139,7 +868,7 @@ TEST_F(ClosureGlyphSegmenterTest, PatchMerge_WithSimplification) { strategy.SetOptimizationCutoffFraction(0.0); strategy.SetUsePatchMerges(true); - auto segmentation = segmenter_dep_graph_only.CodepointToGlyphSegments( + auto segmentation = segmenter.CodepointToGlyphSegments( noto_nastaliq_urdu.get(), {}, {{0x62a}, {0x62b}, {0x62c}, {0x62d}}, strategy); auto segmentation_with_simplification = @@ -1311,11 +1040,11 @@ TEST_F(ClosureGlyphSegmenterTest, TotalCost) { UnigramProbabilityCalculator calculator(std::move(frequencies)); // Basic no segment case. - GlyphSegmentation segmentation1({'a', 'b', 'c'}, {}, {}); - auto sc = GlyphSegmentation::ConditionsToSegmentation({}, {}, segmentation1); + GlyphSegmentation segmentation1({'a', 'b', 'c'}, {}); + auto sc = GlyphSegmentation::ConditionsToSegmentation({}, segmentation1); ASSERT_TRUE(sc.ok()) << sc; - ClosureGlyphSegmenter segmenter(8, 8, PATCH, CLOSURE_ONLY, resolver); + ClosureGlyphSegmenter segmenter(8, 8, DEP_GRAPH_ONLY, resolver); std::vector base_cost = *segmenter.TotalCosts(roboto.get(), segmentation1, {&calculator}); ASSERT_GT(base_cost[0].ift_init_cost, 1000); @@ -1324,13 +1053,13 @@ TEST_F(ClosureGlyphSegmenterTest, TotalCost) { ASSERT_EQ(base_cost[0].uncovered_ift_patch_cost, 0); // Add some patches - GlyphSegmentation segmentation2({'a', 'b', 'c'}, {}, {}); + GlyphSegmentation segmentation2({'a', 'b', 'c'}, {}); sc = GlyphSegmentation::ConditionsToSegmentation( { {ActivationCondition::exclusive_segment(0, 0), {100, 101, 102}}, {ActivationCondition::exclusive_segment(1, 0), {103, 104, 105}}, }, - {}, segmentation2); + segmentation2); ASSERT_TRUE(sc.ok()) << sc; std::vector segments{ @@ -1347,7 +1076,7 @@ TEST_F(ClosureGlyphSegmenterTest, TotalCost) { ASSERT_EQ(with_patches_cost[0].uncovered_ift_patch_cost, 0); // Add uncovered segments ('f' and 'g' are not in frequencies). - GlyphSegmentation segmentation3({'a', 'b', 'c'}, {}, {}); + GlyphSegmentation segmentation3({'a', 'b', 'c'}, {}); sc = GlyphSegmentation::ConditionsToSegmentation( { {ActivationCondition::exclusive_segment(0, 0), {100, 101, 102}}, @@ -1355,7 +1084,7 @@ TEST_F(ClosureGlyphSegmenterTest, TotalCost) { {ActivationCondition::and_segments({0, 2}, 0), {105}}, {ActivationCondition::and_segments({2, 3}, 0), {106, 107}}, }, - {}, segmentation3); + segmentation3); ASSERT_TRUE(sc.ok()) << sc; std::vector segments3{ @@ -2153,62 +1882,20 @@ TEST_F(ClosureGlyphSegmenterTest, ConjunctiveAdditionalConditions) { smcp.feature_tags.insert(HB_TAG('s', 'm', 'c', 'p')); smcp.feature_tags.insert(HB_TAG('c', '2', 's', 'c')); - auto segmentation = segmenter.CodepointToGlyphSegments(roboto.get(), {0xE4}, - { - {0xD6}, // Ö - {0xF6}, // ö - smcp, - }, - MergeStrategy::None()); - auto dep_graph_segmentation = - segmenter_dep_graph.CodepointToGlyphSegments(roboto.get(), {0xE4}, - { - {0xD6}, // Ö - {0xF6}, // ö - smcp, - }, - MergeStrategy::None()); - auto dep_graph_only_segmentation = - segmenter_dep_graph_only.CodepointToGlyphSegments(roboto.get(), {0xE4}, - { - {0xD6}, // Ö - {0xF6}, // ö - smcp, - }, - MergeStrategy::None()); + auto segmentation = CodepointToGlyphSegments(roboto.get(), {0xE4}, + { + {0xD6}, // Ö + {0xF6}, // ö + smcp, + }, + MergeStrategy::None()); ASSERT_TRUE(segmentation.ok()) << segmentation.status(); - ASSERT_TRUE(dep_graph_segmentation.ok()) << dep_graph_segmentation.status(); - ASSERT_TRUE(dep_graph_only_segmentation.ok()) - << dep_graph_only_segmentation.status(); - // Note: gid839 is the small caps O dieresis, it's currently mapped with - // condition 'if {smcp}', the true condition is 'if {smcp} AND {Ö or ö}' - // but the closure segmenter does not currently discover composite conditions. ASSERT_EQ(segmentation->ToString(), R"(initial font: { gid0, gid69, gid106, gid670 } p0: { gid51, gid660 } p1: { gid83, gid687 } -p2: { gid477, gid563, gid822, gid839 } -if (s0) then p0 -if (s1) then p1 -if (s2) then p2 -)"); - - ASSERT_EQ(dep_graph_segmentation->ToString(), - R"(initial font: { gid0, gid69, gid106, gid670 } -p0: { gid51, gid660 } -p1: { gid83, gid687 } -p2: { gid477, gid563, gid822, gid839 } -if (s0) then p0 -if (s1) then p1 -if (s2) then p2 -)"); - - ASSERT_EQ(dep_graph_only_segmentation->ToString(), - R"(initial font: { gid0, gid69, gid106, gid670 } -p0: { gid51, gid660 } -p1: { gid83, gid687 } p2: { gid563, gid822 } p3: { gid477, gid839 } if (s0) then p0 @@ -2328,7 +2015,7 @@ TEST_F(ClosureGlyphSegmenterTest, InitFontMergingAndFindConditions) { profile.init_font_merge_threshold = -75; MergeStrategy strategy = MergeStrategy::CostBased(std::move(profile)); - auto segmentation = FindConditionsCodepointToGlyphSegments( + auto segmentation = CodepointToGlyphSegments( noto_nastaliq_urdu.get(), {}, {{0x20}, {0x62a}, {0x62b}, {0x62c}, {0x62d}}, strategy); ASSERT_TRUE(segmentation.ok()) << segmentation.status(); @@ -2339,12 +2026,10 @@ TEST_F(ClosureGlyphSegmenterTest, InitFontMergingAndFindConditions) { }; ASSERT_EQ(segmentation->Segments(), expected_segments); - // Complex condition if ((s1 OR s2 OR s3 OR s4)) then ... is moved to the init - // font. ASSERT_EQ( segmentation->ToString(), - R"(initial font: { gid0, gid1, gid3, gid4, gid9, gid10, gid12, gid13, gid24, gid30, gid38, gid39, gid57, gid59, gid62, gid68, gid139, gid140, gid153, gid155, gid156, gid172, gid174 } -p0: { gid5, gid6, gid11, gid14, gid33, gid47, gid60, gid64, gid73, gid74, gid75, gid76, gid77, gid83, gid91, gid111, gid112, gid145, gid149, gid152, gid157, gid158, gid190, gid191 } + R"(initial font: { gid0, gid1, gid3, gid4, gid9, gid10, gid12, gid13, gid24, gid30, gid38, gid39, gid57, gid59, gid62, gid64, gid68, gid77, gid139, gid140, gid153, gid155, gid156, gid172, gid174 } +p0: { gid5, gid6, gid11, gid14, gid33, gid47, gid60, gid73, gid74, gid75, gid76, gid83, gid91, gid111, gid112, gid145, gid149, gid152, gid157, gid158, gid190, gid191 } if (s3) then p0 )"); } @@ -2704,7 +2389,7 @@ TEST_F(ClosureGlyphSegmenterTest, // When max_patch_size allows the merged exclusive patch {77, 80} but is // smaller than the combined non-exclusive patch {444, 445, 446, 447}, s0 and // s1 must not be merged. - for (ClosureGlyphSegmenter* seg : {&segmenter, &segmenter_dep_graph_only}) { + for (ClosureGlyphSegmenter* seg : {&segmenter}) { auto not_merged = seg->CodepointToGlyphSegments( roboto.get(), {}, segments, MergeStrategy::Heuristic(merged_exclusive_size, @@ -2745,7 +2430,7 @@ TEST_F(ClosureGlyphSegmenterTest, PatchSizeCacheImpl patch_sizer(roboto.get(), segmenter.brotli_quality()); uint32_t i_j_size = *patch_sizer.GetPatchSize({77, 78}); - for (ClosureGlyphSegmenter* seg : {&segmenter, &segmenter_dep_graph_only}) { + for (ClosureGlyphSegmenter* seg : {&segmenter}) { auto result = seg->CodepointToGlyphSegments( roboto.get(), {}, segments, MergeStrategy::Heuristic(i_j_size)); ASSERT_TRUE(result.ok()) << result.status(); @@ -2780,7 +2465,7 @@ TEST_F(ClosureGlyphSegmenterTest, // gid77 ('i'), gid444 ('fi'), gid446 ('ffi'), gid640 (Aacute). uint32_t s0_s3_size = *patch_sizer.GetPatchSize({37, 74, 77, 444, 446, 640}); - for (ClosureGlyphSegmenter* seg : {&segmenter, &segmenter_dep_graph_only}) { + for (ClosureGlyphSegmenter* seg : {&segmenter}) { auto result = seg->CodepointToGlyphSegments( roboto.get(), {}, segments, MergeStrategy::Heuristic(s0_s3_size)); ASSERT_TRUE(result.ok()) << result.status(); @@ -2814,7 +2499,7 @@ TEST_F(ClosureGlyphSegmenterTest, PatchSizeCacheImpl patch_sizer(roboto.get(), segmenter.brotli_quality()); uint32_t s_t_size = *patch_sizer.GetPatchSize({87, 88}); - for (ClosureGlyphSegmenter* seg : {&segmenter, &segmenter_dep_graph_only}) { + for (ClosureGlyphSegmenter* seg : {&segmenter}) { auto result = seg->CodepointToGlyphSegments( roboto.get(), {'f', 'F'}, segments, MergeStrategy::Heuristic(s_t_size, s_t_size)); @@ -2853,7 +2538,7 @@ TEST_F(ClosureGlyphSegmenterTest, {{0, 1, 2}, MergeStrategy::Heuristic(a_b_size)}, }; - for (ClosureGlyphSegmenter* seg : {&segmenter, &segmenter_dep_graph_only}) { + for (ClosureGlyphSegmenter* seg : {&segmenter}) { auto result = seg->CodepointToGlyphSegments(roboto.get(), {}, segments, merge_groups); ASSERT_TRUE(result.ok()) << result.status(); @@ -2880,10 +2565,9 @@ TEST_F(ClosureGlyphSegmenterTest, HeuristicStrategy_BatchesInertBaseSegments) { *patch_sizer.GetPatchSize({69, 70, 71, 72, 73, 75}); auto context = SegmentationContext::InitializeSegmentationContext( - roboto.get(), {}, segments, segmenter_dep_graph_only.unmapped_glyph_handling(), - segmenter_dep_graph_only.condition_analysis_mode(), - segmenter_dep_graph_only.brotli_quality(), - segmenter_dep_graph_only.init_font_merging_brotli_quality(), resolver); + roboto.get(), {}, segments, segmenter.condition_analysis_mode(), + segmenter.brotli_quality(), segmenter.init_font_merging_brotli_quality(), + resolver); ASSERT_TRUE(context.ok()) << context.status(); SegmentSet all_segments{0, 1, 2, 3, 4, 5, 6, 7}; @@ -2933,7 +2617,7 @@ TEST_F(ClosureGlyphSegmenterTest, // Glyphs for 'a', 'b', 'c', 'd' are 69, 70, 71, 72. uint32_t a_b_c_d_size = *patch_sizer.GetPatchSize({69, 70, 71, 72}); - for (ClosureGlyphSegmenter* seg : {&segmenter, &segmenter_dep_graph_only}) { + for (ClosureGlyphSegmenter* seg : {&segmenter}) { auto result = seg->CodepointToGlyphSegments( roboto.get(), {}, segments, MergeStrategy::Heuristic(a_b_c_d_size, a_b_c_d_size)); @@ -2950,17 +2634,28 @@ TEST_F(ClosureGlyphSegmenterTest, } } +#pragma GCC diagnostic push +#pragma GCC diagnostic ignored "-Wdeprecated-declarations" + TEST_F(ClosureGlyphSegmenterTest, - UnicodeRangeValidation_RejectsMoveToInitFont) { - ClosureGlyphSegmenter seg(8, 8, MOVE_TO_INIT_FONT, CLOSURE_ONLY, resolver, - ift::config::UNICODE_RANGE); - auto result = seg.CodepointToGlyphSegments(roboto.get(), {'a'}, {{'b'}, {'c'}}); - EXPECT_TRUE(absl::IsInvalidArgument(result.status())) << result.status(); + Validation_RejectsDeprecatedConditionAnalysisModes) { + for (auto target_mode : {ift::config::IFT, ift::config::UNICODE_RANGE}) { + for (auto mode : + {ift::config::CLOSURE_ONLY, ift::config::CLOSURE_AND_DEP_GRAPH, + ift::config::CLOSURE_AND_VALIDATE_DEP_GRAPH}) { + ClosureGlyphSegmenter seg(8, 8, mode, resolver, target_mode); + auto result = + seg.CodepointToGlyphSegments(roboto.get(), {'a'}, {{'b'}, {'c'}}); + EXPECT_TRUE(absl::IsInvalidArgument(result.status())) << result.status(); + } + } } +#pragma GCC diagnostic pop + TEST_F(ClosureGlyphSegmenterTest, UnicodeRangeValidation_RejectsFeatureSegments) { - ClosureGlyphSegmenter seg(8, 8, FIND_CONDITIONS, CLOSURE_ONLY, resolver, + ClosureGlyphSegmenter seg(8, 8, DEP_GRAPH_ONLY, resolver, ift::config::UNICODE_RANGE); SubsetDefinition smcp; smcp.feature_tags.insert(HB_TAG('s', 'm', 'c', 'p')); @@ -2971,7 +2666,7 @@ TEST_F(ClosureGlyphSegmenterTest, TEST_F(ClosureGlyphSegmenterTest, UnicodeRangeValidation_RejectsInitFontMerging) { - ClosureGlyphSegmenter seg(8, 8, FIND_CONDITIONS, CLOSURE_ONLY, resolver, + ClosureGlyphSegmenter seg(8, 8, DEP_GRAPH_ONLY, resolver, ift::config::UNICODE_RANGE); MergeStrategy strategy = MergeStrategy::CostBased(ProbabilityProfile( std::make_shared( @@ -2983,7 +2678,7 @@ TEST_F(ClosureGlyphSegmenterTest, } TEST_F(ClosureGlyphSegmenterTest, UnicodeRangeValidation_RejectsPatchMerging) { - ClosureGlyphSegmenter seg(8, 8, FIND_CONDITIONS, CLOSURE_ONLY, resolver, + ClosureGlyphSegmenter seg(8, 8, DEP_GRAPH_ONLY, resolver, ift::config::UNICODE_RANGE); MergeStrategy strategy = MergeStrategy::CostBased( ProbabilityProfile(std::make_shared( @@ -2996,13 +2691,13 @@ TEST_F(ClosureGlyphSegmenterTest, UnicodeRangeValidation_RejectsPatchMerging) { TEST_F(ClosureGlyphSegmenterTest, UnicodeRangeValidation_RejectsInvalidThreshold) { - ClosureGlyphSegmenter seg_neg(8, 8, FIND_CONDITIONS, CLOSURE_ONLY, resolver, + ClosureGlyphSegmenter seg_neg(8, 8, DEP_GRAPH_ONLY, resolver, ift::config::UNICODE_RANGE, -0.1); auto result = seg_neg.CodepointToGlyphSegments(roboto.get(), {'a'}, {{'b'}, {'c'}}); EXPECT_TRUE(absl::IsInvalidArgument(result.status())) << result.status(); - ClosureGlyphSegmenter seg_gt1(8, 8, FIND_CONDITIONS, CLOSURE_ONLY, resolver, + ClosureGlyphSegmenter seg_gt1(8, 8, DEP_GRAPH_ONLY, resolver, ift::config::UNICODE_RANGE, 1.1); result = seg_gt1.CodepointToGlyphSegments(roboto.get(), {'a'}, {{'b'}, {'c'}}); @@ -3015,9 +2710,8 @@ TEST_F(ClosureGlyphSegmenterTest, {'b'}, {'f'}, {'i'}, {'c', 'd'}, {'d', 'e'}}; for (auto analysis_mode : - {CLOSURE_ONLY, CLOSURE_AND_VALIDATE_DEP_GRAPH, DEP_GRAPH_ONLY, - DEP_GRAPH_ONLY_WITH_SIMPLIFICATION}) { - ClosureGlyphSegmenter seg(8, 8, FIND_CONDITIONS, analysis_mode, resolver, + {DEP_GRAPH_ONLY, DEP_GRAPH_ONLY_WITH_SIMPLIFICATION}) { + ClosureGlyphSegmenter seg(8, 8, analysis_mode, resolver, ift::config::UNICODE_RANGE); auto segmentation = seg.CodepointToGlyphSegments(roboto.get(), {'a'}, segments); @@ -3048,9 +2742,8 @@ if (s3) then p2 TEST_F(ClosureGlyphSegmenterTest, UnicodeRange_MakeActivationConditionsDisjoint_ComplexConditions) { for (auto analysis_mode : - {CLOSURE_ONLY, CLOSURE_AND_VALIDATE_DEP_GRAPH, DEP_GRAPH_ONLY, - DEP_GRAPH_ONLY_WITH_SIMPLIFICATION}) { - ClosureGlyphSegmenter seg(8, 8, FIND_CONDITIONS, analysis_mode, resolver, + {DEP_GRAPH_ONLY, DEP_GRAPH_ONLY_WITH_SIMPLIFICATION}) { + ClosureGlyphSegmenter seg(8, 8, analysis_mode, resolver, ift::config::UNICODE_RANGE); auto segmentation = seg.CodepointToGlyphSegments( noto_nastaliq_urdu.get(), {}, @@ -3089,8 +2782,7 @@ TEST_F(ClosureGlyphSegmenterTest, strategy.SetOptimizationCutoffFraction(1.0); strategy.SetUsePatchMerges(false); - ClosureGlyphSegmenter seg(8, 8, FIND_CONDITIONS, - CLOSURE_AND_VALIDATE_DEP_GRAPH, resolver, + ClosureGlyphSegmenter seg(8, 8, DEP_GRAPH_ONLY, resolver, ift::config::UNICODE_RANGE, /*disjoint_conditions_probability_threshold=*/0.2); @@ -3099,7 +2791,8 @@ TEST_F(ClosureGlyphSegmenterTest, ASSERT_TRUE(segmentation.ok()) << segmentation.status(); // 'f' and 'i' were merged into s0; 0xc1 (s1) and 0x106 (s2) were not merged, - // and their non-exclusive shared patch {gid117, gid169, gid667} was omitted. + // and non-exclusive patches ({gid117, gid169, gid667} and {gid141, gid609, + // gid679}) below the probability threshold were omitted. std::vector expected_segments = { {'f', 'i'}, {0xc1}, {0x106}}; EXPECT_EQ(segmentation->Segments(), expected_segments); @@ -3110,7 +2803,7 @@ TEST_F(ClosureGlyphSegmenterTest, EXPECT_EQ(segmentation->ToString(), R"(initial font: { gid0, gid69 } -p0: { gid74, gid77, gid141, gid444, gid446, gid609, gid679 } +p0: { gid74, gid77, gid444, gid446 } p1: { gid37, gid640 } p2: { gid39, gid700 } if (s0) then p0 @@ -3135,8 +2828,8 @@ TEST_F(ClosureGlyphSegmenterTest, Segment({'f'}), Segment({'i'}), Segment({0xc1}), Segment({0x106})}; auto context = SegmentationContext::InitializeSegmentationContext( - roboto.get(), {'a'}, segments, FIND_CONDITIONS, CLOSURE_ONLY, 8, 8, - resolver, ift::config::UNICODE_RANGE); + roboto.get(), {'a'}, segments, DEP_GRAPH_ONLY, 8, 8, resolver, + ift::config::UNICODE_RANGE); ASSERT_TRUE(context.ok()) << context.status(); ASSERT_TRUE(context->CacheInitialGlyphProbabilities({calc.get()}).ok()); @@ -3161,8 +2854,7 @@ TEST_F(ClosureGlyphSegmenterTest, TEST_F(ClosureGlyphSegmenterTest, UnicodeRange_InitialSegmentCodepointsAndFeatures) { - ClosureGlyphSegmenter seg(8, 8, FIND_CONDITIONS, - CLOSURE_AND_VALIDATE_DEP_GRAPH, resolver, + ClosureGlyphSegmenter seg(8, 8, DEP_GRAPH_ONLY, resolver, ift::config::UNICODE_RANGE); SubsetDefinition init; init.codepoints = {'f'}; @@ -3193,8 +2885,7 @@ TEST_F(ClosureGlyphSegmenterTest, TEST_F(ClosureGlyphSegmenterTest, UnicodeRange_MakeActivationConditionsDisjoint_SpansMultipleMergeGroups) { - ClosureGlyphSegmenter seg(8, 8, FIND_CONDITIONS, - CLOSURE_AND_VALIDATE_DEP_GRAPH, resolver, + ClosureGlyphSegmenter seg(8, 8, DEP_GRAPH_ONLY, resolver, ift::config::UNICODE_RANGE); // s0 ('f') and s1 ('i') are non-shared in Group 0 and interact via 'fi'. @@ -3249,8 +2940,7 @@ TEST_F(ClosureGlyphSegmenterTest, TEST_F(ClosureGlyphSegmenterTest, UnicodeRange_MakeActivationConditionsDisjoint_ReordersByNewProbability) { - ClosureGlyphSegmenter seg(8, 8, FIND_CONDITIONS, - CLOSURE_AND_VALIDATE_DEP_GRAPH, resolver, + ClosureGlyphSegmenter seg(8, 8, DEP_GRAPH_ONLY, resolver, ift::config::UNICODE_RANGE); // Initially {'b'} (0.5) > {'f'} (0.4) > {'i'} (0.3) > {'c'} (0.1). @@ -3306,8 +2996,7 @@ TEST_F(ClosureGlyphSegmenterTest, UnicodeRange_Woff2PatchSizeCostFunction) { strategy.SetUsePatchMerges(false); { - ClosureGlyphSegmenter ift_seg(1, 1, FIND_CONDITIONS, - CLOSURE_AND_VALIDATE_DEP_GRAPH, resolver, + ClosureGlyphSegmenter ift_seg(1, 1, DEP_GRAPH_ONLY, resolver, ift::config::IFT); auto ift_segmentation = ift_seg.CodepointToGlyphSegments( roboto.get(), {'a'}, {{'b'}, {'c'}}, strategy); @@ -3317,8 +3006,7 @@ TEST_F(ClosureGlyphSegmenterTest, UnicodeRange_Woff2PatchSizeCostFunction) { } for (uint32_t quality : {0u, 1u}) { - ClosureGlyphSegmenter ur_seg(quality, quality, FIND_CONDITIONS, - CLOSURE_AND_VALIDATE_DEP_GRAPH, resolver, + ClosureGlyphSegmenter ur_seg(quality, quality, DEP_GRAPH_ONLY, resolver, ift::config::UNICODE_RANGE); auto ur_segmentation = ur_seg.CodepointToGlyphSegments( roboto.get(), {'a'}, {{'b'}, {'c'}}, strategy); @@ -3335,22 +3023,22 @@ TEST_F(ClosureGlyphSegmenterTest, UnicodeRange_TotalCosts) { }; UnigramProbabilityCalculator calculator(std::move(frequencies)); - GlyphSegmentation segmentation({'a'}, {}, {}); + GlyphSegmentation segmentation({'a'}, {}); auto sc = GlyphSegmentation::ConditionsToSegmentation( { {ActivationCondition::exclusive_segment(0, 0), {70}}, {ActivationCondition::exclusive_segment(1, 0), {71}}, }, - {}, segmentation); + segmentation); ASSERT_TRUE(sc.ok()) << sc; segmentation.CopySegments({{'b'}, {'c'}}); - ClosureGlyphSegmenter ift_seg(8, 8, FIND_CONDITIONS, CLOSURE_ONLY, resolver, + ClosureGlyphSegmenter ift_seg(8, 8, DEP_GRAPH_ONLY, resolver, ift::config::IFT); std::vector ift_costs = *ift_seg.TotalCosts(roboto.get(), segmentation, {&calculator}); - ClosureGlyphSegmenter ur_seg(8, 8, FIND_CONDITIONS, CLOSURE_ONLY, resolver, + ClosureGlyphSegmenter ur_seg(8, 8, DEP_GRAPH_ONLY, resolver, ift::config::UNICODE_RANGE); std::vector ur_costs = *ur_seg.TotalCosts(roboto.get(), segmentation, {&calculator}); @@ -3512,8 +3200,7 @@ TEST_F(ClosureGlyphSegmenterTest, AllPairsMerging_UnicodeRangeMode) { strategy.SetUsePatchMerges(false); for (uint32_t quality : {0u, 1u}) { - ClosureGlyphSegmenter ur_seg(quality, quality, FIND_CONDITIONS, - CLOSURE_AND_VALIDATE_DEP_GRAPH, resolver, + ClosureGlyphSegmenter ur_seg(quality, quality, DEP_GRAPH_ONLY, resolver, ift::config::UNICODE_RANGE); auto ur_segmentation = ur_seg.CodepointToGlyphSegments( roboto.get(), {'a'}, {{'b'}, {'c'}, {'f'}, {'i'}}, strategy); diff --git a/ift/encoder/complex_condition_finder.cc b/ift/encoder/complex_condition_finder.cc deleted file mode 100644 index ad3b6c12..00000000 --- a/ift/encoder/complex_condition_finder.cc +++ /dev/null @@ -1,324 +0,0 @@ -#include "ift/encoder/complex_condition_finder.h" - -#include - -#include "absl/log/log.h" -#include "absl/status/status.h" -#include "ift/common/int_set.h" -#include "ift/encoder/glyph_closure_cache.h" -#include "ift/encoder/requested_segmentation_information.h" -#include "ift/encoder/subset_definition.h" -#include "ift/encoder/types.h" - -using absl::btree_map; -using absl::Status; -using absl::StatusOr; -using ift::common::GlyphSet; -using ift::common::SegmentSet; - -// For more information on this process see the explanation in: -// ../../docs/experimental/closure_glyph_segmentation_complex_conditions.md - -namespace ift::encoder { - -// One unit of work for the analysis. One from to_be_tested will be checked. -struct Task { - // These are the segments of the full condition found so far. Should be - // excluded from the analysis now, does not include elements of sub_condition. - SegmentSet full_condition; - - // These segments have been determined to be part of a sub condition. - SegmentSet sub_condition; - - // These segments have not yet been tested. - SegmentSet to_be_tested; - - // The subset definition of the union of segments in sub_condition, - // to_be_tested, and the initial font subset definition. - SubsetDefinition subset_definition; - - // The set of glyphs in scope for analysis. - GlyphSet glyphs; -}; - -struct Context { - const SegmentSet all_segments; - const SegmentSet inscope_segments; - const RequestedSegmentationInformation* segmentation_info; - GlyphClosureCache* glyph_closure_cache; - std::vector queue; - - public: - Status ScheduleInitialTasks( - GlyphSet glyphs, btree_map existing_conditions) { - if (glyphs.intersects(segmentation_info->InitFontGlyphs())) { - return absl::InvalidArgumentError( - "Can't analyze glyphs that are in the init font."); - } - - // Each existing condition will map to one initial task that excludes the - // existing condition from the analysis. - for (const auto& [segments, glyph_sub_group] : existing_conditions) { - if (!TRY(InClosure(segments, glyph_sub_group))) { - return absl::InvalidArgumentError( - "The glyphs of existing conditions must be in the closure of " - "condition segments."); - } - TRYV(ScheduleExistingConditionTask(segments, glyph_sub_group, glyphs)); - } - - if (glyphs.empty()) { - return absl::OkStatus(); - } - - if (!TRY(InClosure(all_segments, glyphs))) { - return absl::InvalidArgumentError( - "glyphs to analyze must be in the closure of all segments."); - } - - SegmentSet to_be_tested = all_segments; - to_be_tested.intersect(inscope_segments); - - // If any glyphs remain that do not have existing conditions these are - // covered by a task with no excluded segments. - queue.push_back(CreateTask({}, {}, to_be_tested, glyphs)); - - return absl::OkStatus(); - } - - Status ProcessQueue(btree_map& glyph_to_conditions) { - // TODO(garretrieger): to reduce runtime of this analysis the processing of - // the queue could be parallelized by using a threadpool to run tasks. The - // tasks are fully independent so this should be straightforward. - while (!queue.empty()) { - Task next = std::move(queue.back()); - queue.pop_back(); - TRYV(RunAnalysisTask(next, glyph_to_conditions)); - } - return absl::OkStatus(); - } - - private: - Task CreateTask(SegmentSet full_condition, SegmentSet sub_condition, - SegmentSet to_be_tested, GlyphSet glyphs) { - SegmentSet all = sub_condition; - all.union_set(to_be_tested); - SubsetDefinition task_definition = - segmentation_info->CombinedDefinition(all); - return Task{ - .full_condition = full_condition, - .sub_condition = sub_condition, - .to_be_tested = to_be_tested, - .subset_definition = task_definition, - .glyphs = glyphs, - }; - } - - Task CreateSubTask(const Task& task, GlyphSet new_glyphs, - segment_index_t tested, bool keep) { - SegmentSet new_to_be_tested = task.to_be_tested; - new_to_be_tested.erase(tested); - - SegmentSet new_sub_condition = task.sub_condition; - SubsetDefinition new_sub_definition = task.subset_definition; - if (keep) { - new_sub_condition.insert(tested); - } else { - new_sub_definition.Subtract( - segmentation_info->Segments().at(tested).Definition()); - } - - return Task{ - .full_condition = task.full_condition, - .sub_condition = new_sub_condition, - .to_be_tested = new_to_be_tested, - .subset_definition = new_sub_definition, - .glyphs = new_glyphs, - }; - } - - // Returns true if all glyphs are in the closure of segments. - StatusOr InClosure(const SegmentSet& segments, const GlyphSet& glyphs) { - GlyphSet closure = - TRY(glyph_closure_cache->SegmentClosure(segmentation_info, segments)); - return glyphs.is_subset_of(closure); - } - - StatusOr> HasAdditionalConditions( - const SegmentSet& segments, const GlyphSet& glyphs) { - SegmentSet except = all_segments; - except.subtract(segments); - GlyphSet closure_glyphs = - TRY(glyph_closure_cache->SegmentClosure(segmentation_info, except)); - closure_glyphs.intersect(glyphs); - - except.intersect(inscope_segments); - return std::make_pair(closure_glyphs, std::move(except)); - } - - Status ScheduleExistingConditionTask(const SegmentSet& condition, - const GlyphSet& condition_glyphs, - GlyphSet& all_glyphs) { - // We need to check if there are any additional conditions, - // if there aren't there is no need to schedule the analysis. - auto [glyphs_with_additional_conditions, except] = - TRY(HasAdditionalConditions(condition, condition_glyphs)); - - if (glyphs_with_additional_conditions.empty()) { - return absl::OkStatus(); - } - - queue.push_back( - CreateTask(condition, {}, except, glyphs_with_additional_conditions)); - all_glyphs.subtract(condition_glyphs); - - return absl::OkStatus(); - } - - // Each analysis step checks one segment to see for which glyphs that segment - // is relevant. The supplied task data structure gives the specific state - // around which the segment is tested. - // - // To test a segment a closure is run without the segment being tested: - // - For inscope glyphs which appear in the closure the test segment is not - // relevant for these glyphs - // - For inscope glyphs which do not appear in the closure the test segment is - // relevant for these glyphs. - // - // Based on the anlysis results up to two more analysis steps are spawned (one - // for glyphs where segment is relevant, the other where it is not relevant) - // to test the next segment. - // - // Once all segments are tested the resulting sub condition segments - // is recorded in out. Lastly, the non-relevant segments are checked to see - // if additional conditions are present, if they are another analysis task is - // queued to discover the additional conditions. - Status RunAnalysisTask( - Task task, btree_map& glyph_to_conditions) { - if (task.glyphs.empty()) { - // Nothing left to check. - return absl::OkStatus(); - } - - if (task.to_be_tested.empty()) { - return RecordSubCondition(task, glyph_to_conditions); - } - - segment_index_t test_segment = *task.to_be_tested.min(); - - SubsetDefinition subset_definition = task.subset_definition; - subset_definition.Subtract( - segmentation_info->Segments().at(test_segment).Definition()); - GlyphSet closure_glyphs = - TRY(glyph_closure_cache->GlyphClosure(subset_definition)); - - GlyphSet needs_test_segment = task.glyphs; - needs_test_segment.subtract(closure_glyphs); - GlyphSet doesnt_need_test_segment = task.glyphs; - doesnt_need_test_segment.intersect(closure_glyphs); - - queue.push_back( - CreateSubTask(task, doesnt_need_test_segment, test_segment, false)); - queue.push_back( - CreateSubTask(task, needs_test_segment, test_segment, true)); - - return absl::OkStatus(); - } - - // A sub condition has been found, record it and kick off any - // further analysis needed for additional conditions. - Status RecordSubCondition( - Task task, btree_map& glyph_to_conditions) { - for (glyph_id_t gid : task.glyphs) { - glyph_to_conditions[gid].union_set(task.sub_condition); - } - - // We have identified a sub condition for glyphs, however as usual - // there may be remaining additional conditions which we need to - // check for - task.full_condition.union_set(task.sub_condition); - auto [additional_condition_glyphs, remaining] = - TRY(HasAdditionalConditions(task.full_condition, task.glyphs)); - - if (!additional_condition_glyphs.empty() && remaining.empty()) { - return absl::InternalError( - "Additional conditions check has failed, but there are no more " - "non-excluded segments to utilize."); - } - - // Anything left in glyphs has additional conditions, recurse again to - // analyze them further - queue.push_back(CreateTask(task.full_condition, {}, remaining, - additional_condition_glyphs)); - return absl::OkStatus(); - } -}; - -static btree_map ExistingConditions( - const GlyphConditionSet& glyph_condition_set, const GlyphSet& glyphs, - btree_map& glyph_to_conditions) { - btree_map existing_conditions; - for (glyph_id_t gid : glyphs) { - const auto& condition = glyph_condition_set.ConditionsFor(gid).activation(); - if (condition.IsExclusive() || !condition.IsPurelyDisjunctive()) { - continue; - } - - SegmentSet or_segments = condition.TriggeringSegments(); - existing_conditions[or_segments].insert(gid); - glyph_to_conditions[gid].union_set(or_segments); - } - return existing_conditions; -} - -StatusOr> FindSupersetDisjunctiveConditionsFor( - const RequestedSegmentationInformation& segmentation_info, - const GlyphConditionSet& glyph_condition_set, - GlyphClosureCache& closure_cache, GlyphSet glyphs, - SegmentSet inscope_segments) { - if (!segmentation_info.SegmentsAreDisjoint()) { - return absl::InvalidArgumentError( - "Complex condition finding requires disjoint segments."); - } - - VLOG(0) << "Analyzing " << glyphs.size() - << " unmapped glyphs with the complex condition detector."; - - // TODO(garretrieger): we should see which unicodes (and thus which segments) - // may interact with the GSUB table. Any segments which don't interact with - // GSUB will already have relavent conditions discovered via the standard - // closure analysis. Only segments which interact with GSUB may be part of - // complex conditions (since complex conditions require at least one 'AND' - // which only GSUB can introduce). As a result we can exclude any segments - // with no GSUB interaction from this analysis which should significantly - // speed things up. - Context context{ - .all_segments = segmentation_info.NonEmptySegments(), - .inscope_segments = inscope_segments, - .segmentation_info = &segmentation_info, - .glyph_closure_cache = &closure_cache, - .queue = {}, - }; - - // We may already have some partial conditions generated for the fallback - // glyphs, preload these into the output and schedule the initial tasks - // excluding those segments. - btree_map glyph_to_conditions; - btree_map existing_conditions = - ExistingConditions(glyph_condition_set, glyphs, glyph_to_conditions); - TRYV(context.ScheduleInitialTasks(std::move(glyphs), existing_conditions)); - - TRYV(context.ProcessQueue(glyph_to_conditions)); - - btree_map grouped_out; - for (const auto& [gid, segments] : glyph_to_conditions) { - grouped_out[segments].insert(gid); - } - - VLOG(0) << "Found " << grouped_out.size() - << " new conditions for the unmapped glyphs."; - - return grouped_out; -} - -} // namespace ift::encoder \ No newline at end of file diff --git a/ift/encoder/complex_condition_finder.h b/ift/encoder/complex_condition_finder.h deleted file mode 100644 index 057ab76d..00000000 --- a/ift/encoder/complex_condition_finder.h +++ /dev/null @@ -1,34 +0,0 @@ -#ifndef IFT_ENCODER_COMPLEX_CONDITION_FINDER_H_ -#define IFT_ENCODER_COMPLEX_CONDITION_FINDER_H_ - -#include "absl/status/statusor.h" -#include "ift/common/int_set.h" -#include "ift/encoder/glyph_closure_cache.h" -#include "ift/encoder/glyph_condition_set.h" -#include "ift/encoder/requested_segmentation_information.h" - -namespace ift::encoder { - -// Finds superset purely disjunctive conditions that activate each -// provided glyph. Returns a map from each condition to the activated -// glyphs. -// -// Takes a glyph condition set which will be used as a starting point. -// -// A superset purely disjunctive condition will activate at least -// whenever the true condition would. It will only ever include segments -// that appear in the true condition. There are typically multiple -// possible superset conditions. This will find one of them. -// -// For example if a glyph has the true condition (a and b) or (b and c) -// this could find the condition (a or c). -absl::StatusOr> -FindSupersetDisjunctiveConditionsFor( - const RequestedSegmentationInformation& segmentation_info, - const GlyphConditionSet& glyph_condition_set, - GlyphClosureCache& closure_cache, ift::common::GlyphSet glyphs, - ift::common::SegmentSet inscope_segments); - -} // namespace ift::encoder - -#endif // IFT_ENCODER_COMPLEX_CONDITION_FINDER_H_ \ No newline at end of file diff --git a/ift/encoder/complex_condition_finder_test.cc b/ift/encoder/complex_condition_finder_test.cc deleted file mode 100644 index f6a22922..00000000 --- a/ift/encoder/complex_condition_finder_test.cc +++ /dev/null @@ -1,301 +0,0 @@ -#include "ift/encoder/complex_condition_finder.h" - -#include "absl/container/btree_map.h" -#include "gtest/gtest.h" -#include "ift/common/bazel_data_file_resolver.h" -#include "ift/common/font_data.h" -#include "ift/common/int_set.h" -#include "ift/common/test_font_loader.h" -#include "ift/encoder/closure_glyph_segmenter.h" -#include "ift/encoder/requested_segmentation_information.h" -#include "ift/encoder/segmentation_context.h" -#include "ift/encoder/types.h" -#include "ift/freq/probability_bound.h" - -using ift::config::CLOSURE_ONLY; -using ift::config::PATCH; - -using absl::btree_map; -using absl::StatusOr; -using ift::common::BazelDataFileResolver; -using ift::common::DataFileResolver; -using ift::common::FontData; -using ift::common::GlyphSet; -using ift::common::hb_face_unique_ptr; -using ift::common::make_hb_face; -using ift::common::SegmentSet; -using ift::freq::ProbabilityBound; - -namespace ift::encoder { - -class ComplexConditionFinderTest : public ::testing::Test { - protected: - ComplexConditionFinderTest() - : roboto(make_hb_face(nullptr)), - resolver(*BazelDataFileResolver::CreateForTest()), - segmenter(1, 1, PATCH, CLOSURE_ONLY, resolver) { - roboto = from_file("ift/common/testdata/Roboto-Regular.ttf"); - } - - hb_face_unique_ptr from_file(const char* filename) { - auto loader = ift::common::TestFontLoader::Default().value(); - return loader->LoadFace(filename).value(); - } - - SubsetDefinition CombinedDefinition(const SegmentationContext& context, - const SegmentSet& segments) { - SubsetDefinition def; - for (segment_index_t s : segments) { - def.Union(context.SegmentationInfo().Segments().at(s).Definition()); - } - return def; - } - - StatusOr SegmentClosure(SegmentationContext& context, - const SegmentSet& segments) { - SubsetDefinition closure_def = CombinedDefinition(context, segments); - return context.glyph_closure_cache->GlyphClosure(closure_def); - } - - SegmentationContext TestContext(bool basic_closure_analysis) { - auto context = *SegmentationContext::InitializeSegmentationContext( - roboto.get(), {'f'}, - { - /* 0 */ {{0x54}}, - /* 1 */ {{0x6C}}, - /* 2 */ {{0x6E}}, - /* 3 */ {{0x13C}}, - /* 4 */ {{0x146}}, - /* 5 */ {{0x21A}}, - /* 6 */ {{0xF6C3}}, - /* 7 */ {{0x69}}, - }, - segmenter.unmapped_glyph_handling(), - segmenter.condition_analysis_mode(), segmenter.brotli_quality(), - segmenter.init_font_merging_brotli_quality(), resolver); - - if (!basic_closure_analysis) { - // initialaztion populates the basic conditions, clear those - // out so we can control them. - context.glyph_condition_set.InvalidateGlyphInformation( - {748, 756, 782}, {0, 1, 2, 3, 4, 5, 6}); - } - - return context; - } - - hb_face_unique_ptr roboto; - std::shared_ptr resolver; - ClosureGlyphSegmenter segmenter; - - // Expected complex conditions: - // - // 0xF6C3, 0x54, 0x21A => g782 - // 0xF6C3, 0x6C, 0x13C => g748 - // 0xF6C3, 0x6E, 0x146 => g756 - btree_map expected = { - {{6, 1, 3}, {748}}, - {{6, 2, 4}, {756}}, - {{6, 0, 5}, {782}}, - }; -}; - -TEST_F(ComplexConditionFinderTest, FindConditions) { - SegmentationContext context = TestContext(false); - - auto r = FindSupersetDisjunctiveConditionsFor(context.SegmentationInfo(), - context.glyph_condition_set, - *context.glyph_closure_cache, - { - 748, - 756, - 782, - }, - SegmentSet::all()); - ASSERT_TRUE(r.ok()) << r.status(); - ASSERT_EQ(expected, *r); - - // Verify that the closure requirement is met. If all segments from - // the minimal condition are excluded then the mapped gid should not - // appear in the closure. - const SegmentSet all = {0, 1, 2, 3, 4, 5, 6}; - for (const auto& [segments, gids] : *r) { - SegmentSet except = all; - except.subtract(segments); - - GlyphSet closure = *SegmentClosure(context, except); - ASSERT_FALSE(closure.intersects(gids)); - - closure = *SegmentClosure(context, segments); - ASSERT_TRUE(gids.is_subset_of(closure)); - } -} - -TEST_F(ComplexConditionFinderTest, FindConditions_Partial) { - SegmentationContext context = TestContext(false); - - auto r = FindSupersetDisjunctiveConditionsFor(context.SegmentationInfo(), - context.glyph_condition_set, - *context.glyph_closure_cache, - { - 748, - }, - SegmentSet::all()); - ASSERT_TRUE(r.ok()) << r.status(); - expected.erase(SegmentSet{6, 0, 5}); - expected.erase(SegmentSet{6, 2, 4}); - ASSERT_EQ(expected, *r); -} - -TEST_F(ComplexConditionFinderTest, FindConditions_IncompleteExistingCondition) { - SegmentationContext context = TestContext(false); - - context.glyph_condition_set.AddOrCondition(748, 6); - auto r = FindSupersetDisjunctiveConditionsFor(context.SegmentationInfo(), - context.glyph_condition_set, - *context.glyph_closure_cache, - { - 748, - }, - SegmentSet::all()); - ASSERT_TRUE(absl::IsInvalidArgument(r.status())) << r.status(); -} - -TEST_F(ComplexConditionFinderTest, FindConditions_GlyphsNotInClosure) { - SegmentationContext context = TestContext(false); - - auto r = FindSupersetDisjunctiveConditionsFor( - context.SegmentationInfo(), context.glyph_condition_set, - *context.glyph_closure_cache, - { - 748, - 40 // this is not in the full closure. - }, - SegmentSet::all()); - ASSERT_TRUE(absl::IsInvalidArgument(r.status())) << r.status(); -} - -TEST_F(ComplexConditionFinderTest, - FindConditions_WithExistingConditions_FromClosureAnalysis) { - SegmentationContext context = TestContext(true); - - auto r = FindSupersetDisjunctiveConditionsFor(context.SegmentationInfo(), - context.glyph_condition_set, - *context.glyph_closure_cache, - { - 748, - 756, - 782, - }, - SegmentSet::all()); - ASSERT_TRUE(r.ok()) << r.status(); - ASSERT_EQ(expected, *r); -} - -TEST_F(ComplexConditionFinderTest, FindConditions_WithExistingConditions) { - SegmentationContext context = TestContext(false); - - context.glyph_condition_set.AddOrCondition(748, 1); - context.glyph_condition_set.AddOrCondition(748, 6); - - auto r = FindSupersetDisjunctiveConditionsFor(context.SegmentationInfo(), - context.glyph_condition_set, - *context.glyph_closure_cache, - { - 748, - 756, - 782, - }, - SegmentSet::all()); - ASSERT_TRUE(r.ok()) << r.status(); - ASSERT_EQ(expected, *r); -} - -TEST_F(ComplexConditionFinderTest, - FindConditions_WithExistingConditions_NoAdditionalConditions) { - SegmentationContext context = TestContext(false); - - context.glyph_condition_set.AddOrCondition(748, 1); - context.glyph_condition_set.AddOrCondition(748, 3); - context.glyph_condition_set.AddOrCondition(748, 6); - - auto r = FindSupersetDisjunctiveConditionsFor(context.SegmentationInfo(), - context.glyph_condition_set, - *context.glyph_closure_cache, - { - 748, - 756, - 782, - }, - SegmentSet::all()); - ASSERT_TRUE(r.ok()) << r.status(); - ASSERT_EQ(expected, *r); -} - -TEST_F(ComplexConditionFinderTest, FindConditions_RejectsInitFontGlyphs) { - SegmentationContext context = TestContext(false); - - auto r = FindSupersetDisjunctiveConditionsFor( - context.SegmentationInfo(), context.glyph_condition_set, - *context.glyph_closure_cache, - { - 748, - 74, // f - in the init closure - }, - SegmentSet::all()); - ASSERT_TRUE(absl::IsInvalidArgument(r.status())) << r.status(); -} - -TEST_F(ComplexConditionFinderTest, FindConditions_ClosureRespectsInitFont) { - SegmentationContext context = TestContext(false); - - auto r = FindSupersetDisjunctiveConditionsFor( - context.SegmentationInfo(), context.glyph_condition_set, - *context.glyph_closure_cache, - { - 446, // fi ligature - combines i with f from the init font - }, - SegmentSet::all()); - ASSERT_TRUE(r.ok()) << r.status(); - expected = { - {{7}, {446}}, - }; - ASSERT_EQ(expected, *r); -} - -TEST_F(ComplexConditionFinderTest, FindConditions_WithInscope) { - SegmentationContext context = TestContext(false); - - auto r = FindSupersetDisjunctiveConditionsFor(context.SegmentationInfo(), - context.glyph_condition_set, - *context.glyph_closure_cache, - { - 748, - 756, - }, - {1, 2, 3, 4, 6}); - ASSERT_TRUE(r.ok()) << r.status(); - - expected.erase({6, 0, 5}); - ASSERT_EQ(expected, *r); -} - -TEST_F(ComplexConditionFinderTest, FindConditions_WithBadInscope) { - SegmentationContext context = TestContext(false); - - auto r = FindSupersetDisjunctiveConditionsFor(context.SegmentationInfo(), - context.glyph_condition_set, - *context.glyph_closure_cache, - { - 748, - 756, - }, - { - 2, - 3, - 4, // missing 1, 6 - }); - ASSERT_TRUE(absl::IsInternal(r.status())); -} - -} // namespace ift::encoder \ No newline at end of file diff --git a/ift/encoder/dependency_closure.cc b/ift/encoder/dependency_closure.cc index 18fdf65f..caa72471 100644 --- a/ift/encoder/dependency_closure.cc +++ b/ift/encoder/dependency_closure.cc @@ -37,39 +37,7 @@ namespace ift::encoder { Status DependencyClosure::InitFontChanged(const SegmentSet& segments) { VLOG(1) << "DependencyClosure::InitFontChanged()"; - TRYV(InitNodeConditionsCache(segments)); - - if (!context_glyphs_.has_value() || !init_font_context_glyphs_.has_value()) { - // Context glyphs aren't need for this instance, so skip generating them. - return absl::OkStatus(); - } - - SegmentSet start_segments; - for (segment_index_t s = 0; s < segmentation_info_->Segments().size(); s++) { - if (segmentation_info_->Segments().at(s).Definition().Empty()) { - continue; - } - start_segments.insert(s); - } - - Traversal traversal = TRY(graph_.ClosureTraversal(start_segments, DependencyGraph::COLLECT_CONTEXT)); - context_glyphs_ = traversal.ContextGlyphs(); - - // The init font may have reachable glyphs which are not in the init font - // closure, we need to record the context glyphs from these as they are - // potential interaction points. - init_font_context_glyphs_ = GlyphSet{}; - Traversal init_font_traversal = TRY(graph_.ClosureTraversal( - {Node::InitFont()}, &segmentation_info_->FullClosure(), - &segmentation_info_->FullDefinition().codepoints, DependencyGraph::COLLECT_CONTEXT)); - for (const auto& [g, context] : init_font_traversal.ContextPerGlyph()) { - if (segmentation_info_->NonInitFontGlyphs().contains(g)) { - context_glyphs_->union_set(context); - init_font_context_glyphs_->union_set(context); - } - } - - return absl::OkStatus(); + return InitNodeConditionsCache(segments); } Status DependencyClosure::SegmentsMerged(segment_index_t base_segment, @@ -188,156 +156,6 @@ flat_hash_set DependencyClosure::SegmentsToAffectedNodeConditions( return nodes; } -StatusOr DependencyClosure::AnalyzeSegment( - const SegmentSet& segments, GlyphSet& and_gids, GlyphSet& or_gids, - GlyphSet& exclusive_gids, bool with_simplification) { - AnalysisResult result = TRY(AnalyzeSegmentInternal(segments, with_simplification)); - if (result.accuracy == INACCURATE) { - inaccurate_results_++; - } else { - accurate_results_++; - } - - and_gids.union_set(result.and_gids); - or_gids.union_set(result.or_gids); - exclusive_gids.union_set(result.exclusive_gids); - return result.accuracy; -} - -// Filters out invalid and empty segment ids. -StatusOr DependencyClosure::FilterSegments( - const SegmentSet& segments) const { - SegmentSet out; - for (segment_index_t segment_id : segments) { - if (segment_id >= segmentation_info_->Segments().size()) { - return absl::InvalidArgumentError( - absl::StrCat("Segment index ", segment_id, " is out of bounds.")); - } - - if (segmentation_info_->Segments().at(segment_id).Definition().Empty()) { - // Empty segments are ignored. - continue; - } - - out.insert(segment_id); - } - return out; -} - -StatusOr -DependencyClosure::AnalyzeSegmentInternal( - const SegmentSet& segments_input, bool with_simplification) const { - // This uses a dependency graph (from harfbuzz) to infer how 'segment_id' - // appears in the activation conditions of any glyphs reachable from it. - // This aims to have identical output to GlyphClosureCache::AnalyzeSegment() - // which uses harfbuzz glyph closure to infer conditions. - // - // Condition analysis using the dep graph is not always able to accurately - // reproduce the closure based conditions, so this method returns an - // indication of the accuracy of the analysis. AnalysisAccuracy::INACCURATE is - // returned when something in the dep graph is encountered which may cause - // results to diverge from the closure approach. - - AnalysisResult result; - result.accuracy = ACCURATE; - - SegmentSet segments = TRY(FilterSegments(segments_input)); - if (segments.empty()) { - return result; - } - - auto inscope_nodes = SegmentsToAffectedNodeConditions(segments); - GlyphSet inscope_glyphs; - for (Node node : inscope_nodes) { - if (node.IsGlyph() && - segmentation_info_->NonInitFontGlyphs().contains(node.Id())) { - inscope_glyphs.insert(node.Id()); - } - } - result.reached_glyphs = inscope_glyphs; - - if ((context_glyphs_.has_value() && - inscope_glyphs.intersects(*context_glyphs_)) || - (init_font_context_glyphs_.has_value() && - inscope_glyphs.intersects(*init_font_context_glyphs_))) { - // context glyphs may result in an inaccurate analysis, we can - // still do the analysis but mark it as inaccurate. - result.accuracy = INACCURATE; - } - - // Classify each glyph based on it's conditions - for (glyph_id_t gid : inscope_glyphs) { - auto it = glyph_condition_cache_.find(gid); - if (it == glyph_condition_cache_.end()) { - continue; - } - ActivationCondition condition = it->second; - segment_index_t segment = *segments.min(); - if (segments.size() > 1) { - // Pretend the input segments are merged into one for this analysis. - condition = condition.ReplaceSegments(segment, segments); - } - - if (!condition.TriggeringSegments().contains(segment)) { - // Shouldn't happen, something is wrong with the glyph condition cache. - return absl::InternalError(absl::StrCat("condition in cache for g", gid, - " does not include s", segment)); - } - - if (with_simplification) { - condition = condition.NonCompositeSuperset(); - } - - // Possible cases: - // 1. condition is unitary (only one segment), then glyph is exclusive to - // the input. - if (condition.IsUnitary()) { - result.exclusive_gids.insert(gid); - continue; - } - - // 2. condition is purely disjunctive, glyph is disjunctive to the input. - if (condition.IsPurelyDisjunctive()) { - result.or_gids.insert(gid); - } - - if (condition.conditions().size() > 1) { - bool in_all = true; - bool mixed = false; - for (const auto& disjunctive_group : condition.conditions()) { - if (!disjunctive_group.contains(segment)) { - in_all = false; - continue; - } - - if (disjunctive_group.size() > 1) { - // input segment is involved in a mix of conjunction and disjunction. - mixed = true; - } - } - - // 2. condition has conjunction (two or more sub-groups), and input - // segment appears in every disjunctive sub group, glyph is disjunctive to - // the input. - if (in_all) { - result.or_gids.insert(gid); - } - - // 3. condition has conjunction (two or more sub-groups), and the - // sub-group that contains the input segment(s) equals the input - // segment(s), then glyph is conjunctive to the input. - if (!mixed) { - result.and_gids.insert(gid); - } - - // 4. otherwise gid does not appear in the analysis results due to more - // complicated conditions. - } - } - - return result; -} - StatusOr DependencyClosure::SegmentsThatInteractWith( const SubsetDefinition& def) const { flat_hash_set nodes; diff --git a/ift/encoder/dependency_closure.h b/ift/encoder/dependency_closure.h index 1cfe27c7..62cbfd74 100644 --- a/ift/encoder/dependency_closure.h +++ b/ift/encoder/dependency_closure.h @@ -23,73 +23,26 @@ namespace ift::encoder { class RequestedSegmentationInformation; /* - * Performs closure analysis (like GlyphClosureCache) using a dependency graph - * instead of closure. The dependency graph is not always accurate - * (overestimating the true closure in some cases) so this returns a signal on - * the accuracy of the analysis. + * Extracts glyph activation conditions and interaction sets using a dependency + * graph instead of repeated subsetting closure. In some cases may overestimate + * activation conditions versus real subsetting closure due to reliance on the + * dependency graph. */ class DependencyClosure { public: - // If allow_context_glyph_analysis is true then AnalyzeSegment() will not - // report INNACCURATE for cases that involve context glyphs. static absl::StatusOr> Create( const RequestedSegmentationInformation* segmentation_info, - hb_face_t* face, const ift::common::DataFileResolver& resolver, - bool allow_context_glyph_analysis = false) { + hb_face_t* face, const ift::common::DataFileResolver& resolver) { dep_graph::DependencyGraph graph = TRY( dep_graph::DependencyGraph::Create(segmentation_info, face, resolver)); auto result = std::unique_ptr( new DependencyClosure(std::move(graph), segmentation_info, face)); - if (!allow_context_glyph_analysis) { - result->context_glyphs_ = common::GlyphSet{}; - result->init_font_context_glyphs_ = common::GlyphSet{}; - } TRYV(result->InitFontChanged(ift::common::SegmentSet::all())); return result; } - enum AnalysisAccuracy { - // The analysis is accurate and should match true glyph closure. - ACCURATE, - - // The analysis may not be accurate and as a result may overestimate - // the true glyph closure. - INACCURATE, - }; - - // Attempts to analyze the given segment using a glyph dependency graph - // from harfbuzz. The return value signals if an analysis was able to be - // performed or not. - // - // If ACCURATE is returned then the dep graph is able to produce an analysis - // which should match GlyphClosureCache::AnalyzeSegment(). In this case - // the three output sets (and_gids, or_gids, and exclusive_gids) will be - // populated with the anaysis results. - // - // Otherwise if INACCURATE is returned then the dep graph analysis was found - // to possibly not match GlyphClosureCache::AnalyzeSegment(). The three - // output sets will not be modified in this case. - // - // The three output sets have the following interpretation: - // and_gids: these gids have the union of input segments as a conjunctive - // condition. - // ie. (s_1 U ... U s_n) AND ... -> and_gids - // - // or_gids: these gids have the union of input segments as a disjunctive - // condition. - // ie. (s_1 U ... U s_n) OR ... -> or_gids - // - // exclusive_gids: these gids are exclusively needed by the union of input - // segments. - // ie. (s_1 U ... U s_n) -> exclusive_gids - absl::StatusOr AnalyzeSegment( - const ift::common::SegmentSet& segments, ift::common::GlyphSet& and_gids, - ift::common::GlyphSet& or_gids, ift::common::GlyphSet& exclusive_gids, - bool with_simplification - ); - // Extracts the full activations conditions (as specified by the dependency // graph) for all glyphs. In some cases may overestimate activation conditions // versus real subsetting closure due to reliance on the dependency graph. @@ -137,9 +90,6 @@ class DependencyClosure { absl::StatusOr SegmentsThatInteractWith( const absl::flat_hash_set nodes) const; - uint64_t AccurateResults() const { return accurate_results_; } - uint64_t InaccurateResults() const { return inaccurate_results_; } - private: // Extracts the full activations conditions (as specified by the dependency @@ -194,20 +144,6 @@ class DependencyClosure { original_face_(ift::common::make_hb_face(hb_face_reference(face))), graph_(std::move(graph)) {} - struct AnalysisResult { - AnalysisAccuracy accuracy; - ift::common::GlyphSet reached_glyphs; - ift::common::GlyphSet and_gids; - ift::common::GlyphSet or_gids; - ift::common::GlyphSet exclusive_gids; - }; - - absl::StatusOr AnalyzeSegmentInternal( - const ift::common::SegmentSet& segments, bool with_simplification) const; - - absl::StatusOr FilterSegments( - const common::SegmentSet& segments) const; - absl::Status InitNodeConditionsCache( const common::SegmentSet& changed_segments); void UpdateNodeConditionsCache(segment_index_t base_segment, @@ -217,9 +153,6 @@ class DependencyClosure { ift::common::hb_face_unique_ptr original_face_; dep_graph::DependencyGraph graph_; - std::optional context_glyphs_ = std::nullopt; - std::optional init_font_context_glyphs_ = std::nullopt; - // Inert segments are segments that: // - Appear only in a single unitary condition, that is // they do not interact with any other segments. @@ -246,9 +179,6 @@ class DependencyClosure { // Cache of previously processed InitFont nodes, used to determine newly added // init items. absl::flat_hash_set init_font_nodes_; - - uint64_t accurate_results_ = 0; - uint64_t inaccurate_results_ = 0; }; } // namespace ift::encoder diff --git a/ift/encoder/dependency_closure_test.cc b/ift/encoder/dependency_closure_test.cc index 30476d15..de767156 100644 --- a/ift/encoder/dependency_closure_test.cc +++ b/ift/encoder/dependency_closure_test.cc @@ -9,7 +9,6 @@ #include "ift/common/font_helper.h" #include "ift/common/int_set.h" #include "ift/common/test_font_loader.h" -#include "ift/config/common.pb.h" #include "ift/dep_graph/node.h" #include "ift/encoder/glyph_closure_cache.h" #include "ift/encoder/init_subset_defaults.h" @@ -20,8 +19,6 @@ #include "ift/freq/probability_bound.h" #include "tools/cpp/runfiles/runfiles.h" -using ift::config::PATCH; - using absl::btree_set; using absl::flat_hash_map; using absl::flat_hash_set; @@ -54,7 +51,7 @@ class DependencyClosureTest : public ::testing::Test { resolver(*BazelDataFileResolver::CreateForTest()), closure_cache(*GlyphClosureCache::Create(face.get(), *resolver)), segmentation_info(*RequestedSegmentationInformation::Create( - segments, WithDefaultFeatures(), *closure_cache, PATCH)), + segments, WithDefaultFeatures(), *closure_cache)), dependency_closure(*DependencyClosure::Create(segmentation_info.get(), face.get(), *resolver)) {} @@ -85,67 +82,64 @@ class DependencyClosureTest : public ::testing::Test { /* 5 */ {{0xC1 /* Aacute */}}, }; - void Reconfigure(SubsetDefinition new_init, std::vector new_segments, - bool context_analysis_allowed = false) { + void Reconfigure(SubsetDefinition new_init, + std::vector new_segments) { segmentation_info = *RequestedSegmentationInformation::Create( - new_segments, new_init, *closure_cache, PATCH); - dependency_closure = - *DependencyClosure::Create(segmentation_info.get(), face.get(), - *resolver, context_analysis_allowed); + new_segments, new_init, *closure_cache); + dependency_closure = *DependencyClosure::Create(segmentation_info.get(), + face.get(), *resolver); } void Reconfigure(hb_face_t* new_face, SubsetDefinition new_init, std::vector new_segments) { closure_cache = *GlyphClosureCache::Create(new_face, *resolver); segmentation_info = *RequestedSegmentationInformation::Create( - new_segments, new_init, *closure_cache, PATCH); + new_segments, new_init, *closure_cache); dependency_closure = *DependencyClosure::Create(segmentation_info.get(), new_face, *resolver); } - Status RejectedAnalysis(segment_index_t segment) { - GlyphSet and_gids; - GlyphSet or_gids; - GlyphSet exclusive_gids; - - if (TRY(dependency_closure->AnalyzeSegment({segment}, and_gids, or_gids, - exclusive_gids, false)) == - DependencyClosure::ACCURATE) { - return absl::InternalError( - "Dependency closure analysis should have been rejected."); + GlyphSet ActivatedGlyphs(const SegmentSet& segments) const { + GlyphSet out; + for (const auto& [gid, condition] : + dependency_closure->AllGlyphConditions()) { + bool matches = true; + for (const auto& group : condition.conditions()) { + if (!group.intersects(segments)) { + matches = false; + break; + } + } + if (matches) { + out.insert(gid); + } } - return absl::OkStatus(); + return out; + } + + absl::StatusOr ExpectedClosure(const SegmentSet& segments) const { + GlyphSet gids = + TRY(closure_cache->SegmentClosure(segmentation_info.get(), segments)); + gids.subtract(segmentation_info->InitFontGlyphs()); + return gids; } Status CompareAnalysis(SegmentSet segments) { - GlyphSet and_gids; - GlyphSet or_gids; - GlyphSet exclusive_gids; - - if (TRY(dependency_closure->AnalyzeSegment(segments, and_gids, or_gids, - exclusive_gids, false)) != - DependencyClosure::ACCURATE) { - return absl::InternalError( - "Dependency closure analysis rejected unexpectedly."); + SegmentSet other; + for (segment_index_t i = 0; i < segmentation_info->Segments().size(); ++i) { + if (!segments.contains(i)) { + other.insert(i); + } } - GlyphSet expected_and_gids; - GlyphSet expected_or_gids; - GlyphSet expected_exclusive_gids; - TRYV(closure_cache->AnalyzeSegment(*segmentation_info, segments, - expected_and_gids, expected_or_gids, - expected_exclusive_gids)); - std::string message; bool success = true; - if (!CompareGids("AND", and_gids, expected_and_gids, message)) { - success = false; - } - if (!CompareGids("OR", or_gids, expected_or_gids, message)) { + if (!CompareGids("SEGMENTS", ActivatedGlyphs(segments), + TRY(ExpectedClosure(segments)), message)) { success = false; } - if (!CompareGids("EXCLUSIVE", exclusive_gids, expected_exclusive_gids, - message)) { + if (!CompareGids("OTHER", ActivatedGlyphs(other), + TRY(ExpectedClosure(other)), message)) { success = false; } @@ -180,98 +174,6 @@ class DependencyClosureTest : public ::testing::Test { std::unique_ptr dependency_closure; }; -TEST_F(DependencyClosureTest, AnalyzeSegment_InaccurateAnalysisPopulatesSets) { - Reconfigure(WithDefaultFeatures({'i'}), - { - /* 0 */ {{0x300 /* gravecomb */}}, - /* 1 */ {{0x485}}, - }); - - GlyphSet and_gids; - GlyphSet or_gids; - GlyphSet exclusive_gids; - - // Segment 0 contains gravecomb (0x300), which intersects with - // init_font_context_glyphs_ ('i' in init font + 0x300 form a context lookup). - // Therefore, AnalyzeSegment will return INACCURATE. - auto accuracy = dependency_closure->AnalyzeSegment( - {0}, and_gids, or_gids, exclusive_gids, /*with_simplification=*/false); - ASSERT_TRUE(accuracy.ok()) << accuracy.status(); - EXPECT_EQ(*accuracy, DependencyClosure::INACCURATE); - - // Verify that the analysis was still performed and populated the glyph sets - // despite being INACCURATE. - auto font = make_hb_font(hb_font_create(face.get())); - hb_codepoint_t gravecomb_gid = 0; - ASSERT_TRUE(hb_font_get_nominal_glyph(font.get(), 0x300, &gravecomb_gid)); - EXPECT_TRUE(exclusive_gids.contains(gravecomb_gid)); -} - -TEST_F(DependencyClosureTest, AnalyzeSegment_WithSimplification) { - SubsetDefinition aalt; - aalt.feature_tags.insert(HB_TAG('a', 'a', 'l', 't')); - SubsetDefinition jp78; - jp78.feature_tags.insert(HB_TAG('j', 'p', '7', '8')); - - Reconfigure(noto_sans_jp_vf.get(), {}, - { - /* 0 */ {{0x6717}}, - /* 1 */ {{0x7891}}, - /* 2 */ {{0x798f}}, - /* 3 */ {{0x6406}}, - /* 4 */ {{0xe0100}}, - /* 5 */ {{0xfe00}}, - /* 6 */ {aalt}, - /* 7 */ {jp78}, - }); - - // g8 (glyph 8) has condition: s1 AND (s4 OR s5). - // Without simplification, s1 is part of a mixed condition for g8, so g8 is - // not classified as exclusive. - { - GlyphSet and_gids; - GlyphSet or_gids; - GlyphSet exclusive_gids; - - auto r = dependency_closure->AnalyzeSegment( - {1}, and_gids, or_gids, exclusive_gids, /*with_simplification=*/false); - ASSERT_TRUE(r.ok()) << r.status(); - EXPECT_EQ(*r, DependencyClosure::ACCURATE); - - EXPECT_FALSE(exclusive_gids.contains(8)); - } - - // With simplification, the condition for g8 simplifies to s1, so g8 is - // classified as exclusive. - { - GlyphSet and_gids; - GlyphSet or_gids; - GlyphSet exclusive_gids; - - auto r = dependency_closure->AnalyzeSegment( - {1}, and_gids, or_gids, exclusive_gids, /*with_simplification=*/true); - ASSERT_TRUE(r.ok()) << r.status(); - EXPECT_EQ(*r, DependencyClosure::ACCURATE); - - EXPECT_TRUE(exclusive_gids.contains(8)); - } -} - -TEST_F(DependencyClosureTest, AddsToSets) { - GlyphSet and_gids{101}; - GlyphSet or_gids{102}; - GlyphSet exclusive_gids{103}; - - auto r = dependency_closure->AnalyzeSegment({0}, and_gids, or_gids, - exclusive_gids, false); - ASSERT_TRUE(r.ok()) << r.status(); - ASSERT_EQ(*r, DependencyClosure::ACCURATE); - - ASSERT_EQ(and_gids, (GlyphSet{101, 667})); - ASSERT_EQ(or_gids, (GlyphSet{102})); - ASSERT_EQ(exclusive_gids, (GlyphSet{69, 103})); -} - TEST_F(DependencyClosureTest, ExtractAllGlyphConditions) { Reconfigure(WithDefaultFeatures(), { @@ -682,57 +584,6 @@ TEST_F(DependencyClosureTest, AlreadyInInitFont) { ASSERT_TRUE(s.ok()) << s; } -TEST_F(DependencyClosureTest, SegmentOutOfBounds) { - Status s = CompareAnalysis({10}); - ASSERT_TRUE(absl::IsInvalidArgument(s)) << s; -} - -TEST_F(DependencyClosureTest, Rejected_LookAheadGlyphs) { - Reconfigure(WithDefaultFeatures({}), { - {{'i'}}, - {{0x485}}, - }); - - Status s = CompareAnalysis({0}); - ASSERT_TRUE(s.ok()) << s; - - // glyph for 0x485 is part of a lookahead on a chain context sub - // so analysis should be rejected. - s = RejectedAnalysis(1); - ASSERT_TRUE(s.ok()) << s; -} - -TEST_F(DependencyClosureTest, Allowed_LookAheadGlyphs) { - Reconfigure(WithDefaultFeatures({}), - { - {{'i'}}, - {{0x485}}, - }, - true); - - Status s = CompareAnalysis({0}); - ASSERT_TRUE(s.ok()) << s; - // glyph for 0x485 is part of a lookahead on a chain context sub - // since context analysis has been turned on then analysis will - // not be rejected. - s = CompareAnalysis({1}); - ASSERT_TRUE(s.ok()) << s; -} - -TEST_F(DependencyClosureTest, Rejected_InitFontContext) { - Reconfigure(WithDefaultFeatures({'i'}), - { - {{0x300 /* gravecomb */}}, - {{0x485}}, - }); - - // Gravecomb doesn't pass through any contextual dependencies, but - // it shows up as a context glyph for the init font subset. - // Thus it should be rejected. - Status s = RejectedAnalysis(0); - ASSERT_TRUE(s.ok()) << s; -} - TEST_F(DependencyClosureTest, Noop_Features) { SubsetDefinition features; features.feature_tags.insert(HB_TAG('a', 'b', 'c', 'd')); @@ -822,19 +673,6 @@ TEST_F(DependencyClosureTest, UvsAndFeatures_ConflictingConjunctiveConditions) { ASSERT_TRUE(s.ok()) << s; } -TEST_F(DependencyClosureTest, Rejected_FullySatisfiedContext) { - Reconfigure(WithDefaultFeatures({}), - { - {{'i', 0x300 /* gravecomb */}}, - }); - - // The segment contains everything needed to activated the i + gravecomb - // contextual lookup, but should still be rejected on account of passing - // through a contextual lookup. - Status s = RejectedAnalysis(0); - ASSERT_TRUE(s.ok()) << s; -} - TEST_F(DependencyClosureTest, BidiMirroring) { // Test that the dep graph analysis accounts for bidi mirroring in the // harfbuzz closure diff --git a/ift/encoder/glyph_closure_cache.cc b/ift/encoder/glyph_closure_cache.cc index 1c33683e..2e0c93ac 100644 --- a/ift/encoder/glyph_closure_cache.cc +++ b/ift/encoder/glyph_closure_cache.cc @@ -10,8 +10,6 @@ #include "ift/encoder/subset_definition.h" #include "ift/encoder/types.h" -using ift::config::Glyphs; - using absl::Status; using absl::StatusOr; using ift::common::CodepointSet; @@ -170,82 +168,6 @@ StatusOr GlyphClosureCache::GlyphClosure( return GlyphSet(gids); } -StatusOr GlyphClosureCache::CodepointsToOrGids( - const RequestedSegmentationInformation& segmentation_info, - const SegmentSet& segment_ids) { - GlyphSet and_gids; - GlyphSet or_gids; - GlyphSet exclusive_gids; - TRYV(AnalyzeSegment(segmentation_info, segment_ids, and_gids, or_gids, - exclusive_gids)); - - return or_gids; -} - -Status GlyphClosureCache::AnalyzeSegment( - const RequestedSegmentationInformation& segmentation_info, - const SegmentSet& segment_ids, GlyphSet& and_gids, GlyphSet& or_gids, - GlyphSet& exclusive_gids) { - if (segment_ids.empty()) { - return absl::OkStatus(); - } - - // This function tests various closures using the segment codepoints to - // determine what conditions are present for the inclusion of closure glyphs. - // - // At a high level we do the following (where s_i is the segment being - // tested): - // - // * Set A: glyph closure on original font of the union of all segments. - // * Set B: glyph closure on original font of the union of all segments except - // for s_i - // * Set I: (glyph closure on original font of s_0 union s_i) - (glyph closure - // on original font of s_0) - // * Set D: A - B, the set of glyphs that are dropped when s_i is removed. - // - // Then we know the following: - // * Glyphs in I should be included whenever s_i is activated. - // * s_i is necessary for glyphs in D to be required, but other segments may - // be needed too. - // - // Furthermore we can intersect I and D to produce three sets: - // * D - I: the activation condition for these glyphs is s_i AND … - // Where … is one or more additional segments. - // * I - D: the activation conditions for these glyphs is s_i OR … - // Where … is one or more additional segments. - // * D intersection I: the activation conditions for these glyphs is only s_i - - SubsetDefinition - combined; // This is the subset definition of the unions of segment_ids. - for (segment_index_t s_id : segment_ids) { - combined.Union(segmentation_info.Segments()[s_id].Definition()); - } - - SubsetDefinition except_segment = - ComputeExceptSegment(segmentation_info, segment_ids, combined); - auto B_except_segment_closure = TRY(GlyphClosure(except_segment)); - - SubsetDefinition only_segment = combined; - only_segment.Union(segmentation_info.InitFontSegment()); - - auto I_only_segment_closure = TRY(GlyphClosure(only_segment)); - I_only_segment_closure.subtract(segmentation_info.InitFontGlyphs()); - - GlyphSet D_dropped = segmentation_info.FullClosure(); - D_dropped.subtract(B_except_segment_closure); - - and_gids.union_set(D_dropped); - and_gids.subtract(I_only_segment_closure); - - or_gids.union_set(I_only_segment_closure); - or_gids.subtract(D_dropped); - - exclusive_gids.union_set(I_only_segment_closure); - exclusive_gids.intersect(D_dropped); - - return absl::OkStatus(); -} - CodepointSet GlyphClosureCache::CodepointsForGlyphs( const GlyphSet& glyphs) const { return unicode_edges_.CodepointsForGlyphs(glyphs); diff --git a/ift/encoder/glyph_closure_cache.h b/ift/encoder/glyph_closure_cache.h index 070aefa8..0f097613 100644 --- a/ift/encoder/glyph_closure_cache.h +++ b/ift/encoder/glyph_closure_cache.h @@ -46,18 +46,6 @@ class GlyphClosureCache { const ift::common::SegmentSet& segments, const ift::common::GlyphSet& glyphs); - // Analyzes the provided segments with AnalyzeSegment() and returns just the - // or_gids - absl::StatusOr CodepointsToOrGids( - const RequestedSegmentationInformation& segmentation_info, - const ift::common::SegmentSet& segment_ids); - - absl::Status AnalyzeSegment( - const RequestedSegmentationInformation& segmentation_info, - const ift::common::SegmentSet& segment_ids, - ift::common::GlyphSet& and_gids, ift::common::GlyphSet& or_gids, - ift::common::GlyphSet& exclusive_gids); - absl::StatusOr ExpandClosure( const SubsetDefinition& definition); diff --git a/ift/encoder/glyph_closure_cache_test.cc b/ift/encoder/glyph_closure_cache_test.cc index 93b83a26..a9958958 100644 --- a/ift/encoder/glyph_closure_cache_test.cc +++ b/ift/encoder/glyph_closure_cache_test.cc @@ -12,8 +12,6 @@ #include "ift/encoder/subset_definition.h" #include "ift/freq/probability_bound.h" -using ift::config::PATCH; - using absl::StatusOr; using ift::common::BazelDataFileResolver; using ift::common::CodepointSet; @@ -63,8 +61,7 @@ TEST_F(GlyphClosureCacheTest, SegmentClosure) { SubsetDefinition init; init.feature_tags = {HB_TAG('l', 'i', 'g', 'a')}; - auto info = - RequestedSegmentationInformation::Create(segments, init, **cache, PATCH); + auto info = RequestedSegmentationInformation::Create(segments, init, **cache); ASSERT_TRUE(info.ok()); auto closure = (*cache)->SegmentClosure(info->get(), {0}); @@ -90,8 +87,7 @@ TEST_F(GlyphClosureCacheTest, HasAdditionalConditions) { // s0 or s1 -> g37 (A) - auto info = - RequestedSegmentationInformation::Create(segments, {}, **cache, PATCH); + auto info = RequestedSegmentationInformation::Create(segments, {}, **cache); ASSERT_TRUE(info.ok()); ASSERT_TRUE( @@ -118,45 +114,15 @@ TEST_F(GlyphClosureCacheTest, HasAdditionalConditions_IncludesInitFont) { SubsetDefinition init; init.feature_tags = {HB_TAG('c', '2', 's', 'c')}; - auto info = - RequestedSegmentationInformation::Create(segments, init, **cache, PATCH); + auto info = RequestedSegmentationInformation::Create(segments, init, **cache); ASSERT_TRUE(info.ok()); // s0 or s1 -> g563 (smcap A) - EXPECT_EQ(*(*cache)->CodepointsToOrGids(*info->get(), {0}), - (GlyphSet{37, 563})); - EXPECT_EQ(*(*cache)->CodepointsToOrGids(*info->get(), {1}), - (GlyphSet{37, 563})); - EXPECT_TRUE(*(*cache)->HasAdditionalConditions(info->get(), {0}, {563})); EXPECT_TRUE(*(*cache)->HasAdditionalConditions(info->get(), {1}, {563})); EXPECT_FALSE(*(*cache)->HasAdditionalConditions(info->get(), {0, 1}, {563})); } -TEST_F(GlyphClosureCacheTest, AnalyzeSegment) { - auto cache = GlyphClosureCache::Create(roboto.get(), *resolver); - ASSERT_TRUE(cache.ok()) << cache.status(); - std::vector segments{ - {{'f'}}, - {{'i'}}, - }; - - SubsetDefinition init; - init.feature_tags = {HB_TAG('l', 'i', 'g', 'a')}; - auto info = - RequestedSegmentationInformation::Create(segments, init, **cache, PATCH); - ASSERT_TRUE(info.ok()); - - GlyphSet and_gids, or_gids, exclusive_gids; - auto status = (*cache)->AnalyzeSegment(*info->get(), {0}, and_gids, or_gids, - exclusive_gids); - ASSERT_TRUE(status.ok()); - - EXPECT_EQ(exclusive_gids, (GlyphSet{74 /* f */})); - EXPECT_EQ(or_gids, (GlyphSet{})); - EXPECT_EQ(and_gids, (GlyphSet{444 /* fi */, 446 /* ffi */})); -} - TEST_F(GlyphClosureCacheTest, ExpandClosure) { auto cache = GlyphClosureCache::Create(roboto.get(), *resolver); ASSERT_TRUE(cache.ok()) << cache.status(); @@ -174,27 +140,6 @@ TEST_F(GlyphClosureCacheTest, ExpandClosure) { ASSERT_EQ(*expanded, expected); } -TEST_F(GlyphClosureCacheTest, CodepointsToOrGids) { - auto cache = GlyphClosureCache::Create(roboto.get(), *resolver); - ASSERT_TRUE(cache.ok()) << cache.status(); - std::vector segments{ - {{'A'}}, - {{0xC1 /* Aacute */}}, - }; - - SubsetDefinition init_font; - auto info = RequestedSegmentationInformation::Create(segments, init_font, - **cache, PATCH); - ASSERT_TRUE(info.ok()); - - SegmentSet segs; - segs.insert(0); - - auto or_gids = (*cache)->CodepointsToOrGids(*info->get(), {0}); - ASSERT_TRUE(or_gids.ok()); - EXPECT_EQ(*or_gids, (GlyphSet{37})); -} - TEST_F(GlyphClosureCacheTest, UnicodeCompDecomp_ExpandClosure) { auto cache = GlyphClosureCache::Create(roboto.get(), *resolver); ASSERT_TRUE(cache.ok()) << cache.status(); diff --git a/ift/encoder/glyph_groupings.cc b/ift/encoder/glyph_groupings.cc index 9a4e0a96..54d5bc16 100644 --- a/ift/encoder/glyph_groupings.cc +++ b/ift/encoder/glyph_groupings.cc @@ -7,15 +7,11 @@ #include "ift/common/int_set.h" #include "ift/common/try.h" #include "ift/encoder/activation_condition.h" -#include "ift/encoder/complex_condition_finder.h" -#include "ift/encoder/glyph_closure_cache.h" #include "ift/encoder/glyph_condition_set.h" #include "ift/encoder/glyph_partition.h" #include "ift/encoder/requested_segmentation_information.h" #include "ift/encoder/types.h" -using ift::config::FIND_CONDITIONS; - using absl::btree_map; using absl::btree_set; using absl::flat_hash_map; @@ -27,9 +23,6 @@ using ift::common::SegmentSet; namespace ift::encoder { void GlyphGroupings::InvalidateGlyphInformation(uint32_t gid) { - unmapped_glyphs_.erase(gid); - found_condition_glyphs_.erase(gid); - std::optional post_combination_condition_or = conditions_and_glyphs_.Invalidate(gid); std::optional pre_combination_condition_or = @@ -99,99 +92,41 @@ Status GlyphGroupings::CombinePatches(const GlyphSet& a, const GlyphSet& b) { return absl::OkStatus(); } -Status GlyphGroupings::AddGlyphsToExclusiveGroup( - segment_index_t exclusive_segment, const GlyphSet& glyphs) { - for (glyph_id_t gid : glyphs) { - InvalidateGlyphInformation(gid); - } - - auto& exc_glyphs = exclusive_glyph_groups_[exclusive_segment]; - exc_glyphs.union_set(glyphs); - - ActivationCondition condition = - ActivationCondition::exclusive_segment(exclusive_segment, 0); - - TRYV(conditions_and_glyphs_.Union(condition, glyphs)); - TRYV(conditions_and_glyphs_pre_combination_.Union(condition, glyphs)); - - // When merging this way we have to check if any of the involved glyphs - // are involved with the combined patches mechanism. If at least one is - // then it's necessary to recompute all combined patches to reflect any - // downstream changes. - return RecomputeCombinedConditionsIfNeeded(glyphs); -} - // Converts this grouping into a finalized GlyphSegmentation. StatusOr GlyphGroupings::ToGlyphSegmentation( const RequestedSegmentationInformation& segmentation_info, bool exclusive_only) const { GlyphSegmentation segmentation( segmentation_info.InitFontSegmentWithoutDefaults(), - segmentation_info.InitFontGlyphs(), unmapped_glyphs_); + segmentation_info.InitFontGlyphs()); segmentation.CopySegments(segmentation_info.SegmentSubsetDefinitions()); - // The fallback patch isn't stored in ConditionAndGlyphs() so add it in - // manually. - SegmentSet fallback_segments; - btree_map conditions_with_fallback; + btree_map filtered_conditions; for (const auto& [condition, glyphs] : ConditionsAndGlyphs()) { if (exclusive_only && !condition.IsExclusive()) { continue; } - conditions_with_fallback.insert({condition, glyphs}); + filtered_conditions.insert({condition, glyphs}); } - if (!exclusive_only && !unmapped_glyphs_.empty()) { - fallback_segments = segmentation_info.NonEmptySegments(); - - ActivationCondition non_fallback_cond = - ActivationCondition::or_segments(fallback_segments, 0, false); - ActivationCondition fallback_cond = - ActivationCondition::or_segments(fallback_segments, 0, true); - - if (conditions_with_fallback.contains(non_fallback_cond)) { - // If an equivalent non-fallback condition exists merge it with the - // fallback one. - conditions_with_fallback[fallback_cond].union_set( - conditions_with_fallback[non_fallback_cond]); - conditions_with_fallback.erase(non_fallback_cond); - } - - conditions_with_fallback[fallback_cond].union_set(unmapped_glyphs_); - } - - TRYV(GlyphSegmentation::ConditionsToSegmentation( - conditions_with_fallback, fallback_segments, segmentation)); + TRYV(GlyphSegmentation::ConditionsToSegmentation(filtered_conditions, + segmentation)); return segmentation; } Status GlyphGroupings::GroupGlyphs( const RequestedSegmentationInformation& segmentation_info, - const GlyphConditionSet& glyph_condition_set, - GlyphClosureCache& closure_cache, - std::optional dependency_closure, GlyphSet glyphs, - const SegmentSet& modified_segments, bool additional_conditions_check) { - const auto& initial_closure = segmentation_info.InitFontGlyphs(); - SegmentSet inscope_fallback_segments; - + const GlyphConditionSet& glyph_condition_set, GlyphSet glyphs, + const SegmentSet& modified_segments) { for (glyph_id_t gid : glyphs) { - CollectSegments(gid, inscope_fallback_segments); InvalidateGlyphInformation(gid); } - if (!inscope_fallback_segments.empty()) { - inscope_fallback_segments.union_set(modified_segments); - } else { - // If no existing conditions exist, all segments are inscope. - inscope_fallback_segments.invert(); - } - // Find any additional glyphs that are affected by changes in // modified_segments GlyphSet additional_glyphs = ModifiedGlyphs(modified_segments); for (glyph_id_t gid : additional_glyphs) { - CollectSegments(gid, inscope_fallback_segments); InvalidateGlyphInformation(gid); } glyphs.union_set(additional_glyphs); @@ -204,11 +139,6 @@ Status GlyphGroupings::GroupGlyphs( const auto& condition = glyph_condition_set.ConditionsFor(gid).activation(); if (condition.IsAlwaysTrue()) { - if (!initial_closure.contains(gid) && - segmentation_info.FullClosure().contains(gid)) { - // no condition so this is an unmapped glyph - unmapped_glyphs_.insert(gid); - } continue; } @@ -239,82 +169,25 @@ Status GlyphGroupings::GroupGlyphs( } } - // Add conditions for everything except for purely disjunctive conditions whic - // need additional processing. + // Add conditions for everything except for purely disjunctive conditions. for (const auto& c : all_other_modified_conditions) { TRYV(AddConditionAndGlyphs(c, new_groupings.at(c))); } - // Any of the or_set conditions we've generated may have some additional - // conditions that were not detected. Therefore we need to rule out the - // presence of these additional conditions if an or group is able to be - // used. for (const auto& or_group : modified_or_groups) { auto& glyphs = or_glyph_groups_[or_group]; ActivationCondition condition = ActivationCondition::or_segments(or_group, 0); - - if (!additional_conditions_check) { - TRYV(AddConditionAndGlyphs(condition, glyphs)); - continue; - } - - SegmentSet all_other_segment_ids; - if (!segmentation_info.Segments().empty()) { - all_other_segment_ids.insert_range( - 0, segmentation_info.Segments().size() - 1); - all_other_segment_ids.subtract(or_group); - } - - GlyphSet or_gids = TRY(closure_cache.CodepointsToOrGids( - segmentation_info, all_other_segment_ids)); - - // Any "OR" glyphs associated with all other codepoints have some - // additional conditions to activate so we can't safely include them into - // this or condition. They are instead moved to the set of unmapped - // glyphs. - for (uint32_t gid : or_gids) { - if (glyphs.erase(gid) > 0) { - unmapped_glyphs_.insert(gid); - } - } - - if (glyphs.empty()) { - // Group has been emptied out, so it's no longer needed. - or_glyph_groups_.erase(or_group); - RemoveConditionAndGlyphs(condition); - continue; - } - TRYV(AddConditionAndGlyphs(condition, glyphs)); } - if (segmentation_info.GetUnmappedGlyphHandling() == FIND_CONDITIONS) { - TRYV(FindFallbackGlyphConditions(segmentation_info, glyph_condition_set, - inscope_fallback_segments, closure_cache, - dependency_closure)); - } - // The combined conditions can't be incrementally updated, so we recompute // them in full if needed. TRYV(RecomputeCombinedConditionsIfNeeded(glyphs)); - // Note: we don't need to include the fallback segment/condition in - // conditions_and_glyphs since all downstream processing which - // utilizes that map ignores the fallback segment. It will be - // manually added to the final segmentation in ToGlyphSegmentation() - return absl::OkStatus(); } -void GlyphGroupings::CollectSegments(glyph_id_t gid, SegmentSet& segments) { - auto it = conditions_and_glyphs_.GlyphToCondition().find(gid); - if (it == conditions_and_glyphs_.GlyphToCondition().end()) { - return; - } - segments.union_set(it->second.TriggeringSegments()); -} - GlyphSet GlyphGroupings::ModifiedGlyphs(const SegmentSet& segments) const { GlyphSet glyphs; for (segment_index_t s : segments) { @@ -326,61 +199,6 @@ GlyphSet GlyphGroupings::ModifiedGlyphs(const SegmentSet& segments) const { return glyphs; } -Status GlyphGroupings::FindFallbackGlyphConditions( - const RequestedSegmentationInformation& segmentation_info, - const GlyphConditionSet& glyph_condition_set, - const SegmentSet& inscope_segments, GlyphClosureCache& closure_cache, - std::optional dependency_closure) { - if (unmapped_glyphs_.empty()) { - return absl::OkStatus(); - } - - SegmentSet inscope = SegmentSet::all(); - if (dependency_closure.has_value()) { - inscope = TRY( - dependency_closure.value()->SegmentsThatInteractWith(unmapped_glyphs_)); - VLOG(0) << "used dep graph to scope complex condition finding to " - << inscope.size() << " segments."; - } - - // Note: inscope_segments is not currently used, the approach needs more - // work. In testing in some cases it caused complex conditions to be larger - // than necessary when a segment which could shorten the condition isn't in - // scope. - // - // For example, with true condition (a or b) AND (b or c), if only {a, c} is - // inscope then we don't have the possibility of finding the superset of {b}. - btree_map complex_conditions = - TRY(FindSupersetDisjunctiveConditionsFor( - segmentation_info, glyph_condition_set, closure_cache, - unmapped_glyphs_, inscope)); - - found_condition_glyphs_.union_set(unmapped_glyphs_); - unmapped_glyphs_.clear(); - for (const auto& [s, g] : complex_conditions) { - if (s.empty()) { - return absl::InternalError("Complex conditions should never be empty."); - } - - ActivationCondition c = ActivationCondition::or_segments(s, 0); - if (s.size() == 1) { - segment_index_t segment = *s.begin(); - exclusive_glyph_groups_[segment].union_set(g); - c = ActivationCondition::exclusive_segment(segment, 0); - } else { - or_glyph_groups_[s].union_set(g); - } - - // There may be existing glyphs at this specific condition, so union into - // it. - TRYV(UnionConditionAndGlyphs(c, g)); - } - VLOG(0) - << "Unmapped glyphs patch removed and replaced with found conditions."; - - return absl::OkStatus(); -} - Status GlyphGroupings::RecomputeCombinedConditions() { // Find all conditions that are affected by combined_patches_. btree_set affected_conditions; diff --git a/ift/encoder/glyph_groupings.h b/ift/encoder/glyph_groupings.h index c83b576b..5a8e8205 100644 --- a/ift/encoder/glyph_groupings.h +++ b/ift/encoder/glyph_groupings.h @@ -8,8 +8,6 @@ #include "ift/common/int_set.h" #include "ift/encoder/activation_condition.h" #include "ift/encoder/condition_to_glyphs_index.h" -#include "ift/encoder/dependency_closure.h" -#include "ift/encoder/glyph_closure_cache.h" #include "ift/encoder/glyph_condition_set.h" #include "ift/encoder/glyph_partition.h" #include "ift/encoder/glyph_segmentation.h" @@ -33,8 +31,7 @@ class GlyphGroupings { simplify_combined_ == other.simplify_combined_ && conditions_and_glyphs_ == other.conditions_and_glyphs_ && conditions_and_glyphs_pre_combination_ == - other.conditions_and_glyphs_pre_combination_ && - unmapped_glyphs_ == other.unmapped_glyphs_; + other.conditions_and_glyphs_pre_combination_; } bool operator!=(const GlyphGroupings& other) { return !(*this == other); } @@ -88,16 +85,6 @@ class GlyphGroupings { return empty; } - // Returns the set of glyphs that are considered unmapped, - // which will be placed in the fallback (always loaded) patch. - ift::common::GlyphSet UnmappedGlyphs() const { return unmapped_glyphs_; } - - // Returns the set of glyphs that were unmapped but had conditions - // found for them. - const ift::common::GlyphSet& FoundConditionGlyphs() const { - return found_condition_glyphs_; - } - // Returns a list of conditions which include segment. const absl::flat_hash_set& TriggeringSegmentToConditions( segment_index_t segment) const { @@ -125,11 +112,6 @@ class GlyphGroupings { return empty; } - // Add a set of glyphs to an existing exclusive group (and_group of one - // segment). - absl::Status AddGlyphsToExclusiveGroup(segment_index_t exclusive_segment, - const ift::common::GlyphSet& glyphs); - // Specify that any patches containing glyphs from either a or b should be // merged into one patch. Only affects exclusive and disjunctive patches. // @@ -157,11 +139,8 @@ class GlyphGroupings { absl::Status GroupGlyphs( const RequestedSegmentationInformation& segmentation_info, const GlyphConditionSet& glyph_condition_set, - GlyphClosureCache& closure_cache, - std::optional dependency_closure, ift::common::GlyphSet glyphs, - const ift::common::SegmentSet& modified_segments, - bool additional_conditions_check = true); + const ift::common::SegmentSet& modified_segments); // Converts this grouping into a finalized GlyphSegmentation. absl::StatusOr ToGlyphSegmentation( @@ -189,21 +168,9 @@ class GlyphGroupings { } private: - void CollectSegments(glyph_id_t gid, ift::common::SegmentSet& segments); - ift::common::GlyphSet ModifiedGlyphs( const ift::common::SegmentSet& segments) const; - // Perform a more detailed analysis to try and find more granular conditions - // for fallback glyphs. Will replace the fallback glyphs with any found - // conditions. - absl::Status FindFallbackGlyphConditions( - const RequestedSegmentationInformation& segmentation_info, - const GlyphConditionSet& glyph_condition_set, - const ift::common::SegmentSet& inscope_segments, - GlyphClosureCache& closure_cache, - std::optional dependency_closure); - // Removes all stored grouping information related to glyph with the specified // condition. void InvalidateGlyphInformation(uint32_t gid); @@ -310,14 +277,6 @@ class GlyphGroupings { // An alternate representation of and/or_glyph_groups_, derived from them. ConditionToGlyphsIndex conditions_and_glyphs_; ConditionToGlyphsIndex conditions_and_glyphs_pre_combination_; - - // These glyphs aren't mapped by any conditions and as a result should be - // included in the fallback patch. - ift::common::GlyphSet unmapped_glyphs_; - - // These glyphs were previously considered unmapped, but have had conditions - // found for them. - ift::common::GlyphSet found_condition_glyphs_; }; } // namespace ift::encoder diff --git a/ift/encoder/glyph_groupings_test.cc b/ift/encoder/glyph_groupings_test.cc index c9698907..08ddaf6a 100644 --- a/ift/encoder/glyph_groupings_test.cc +++ b/ift/encoder/glyph_groupings_test.cc @@ -10,7 +10,6 @@ #include "ift/common/int_set.h" #include "ift/common/test_font_loader.h" #include "ift/encoder/activation_condition.h" -#include "ift/encoder/complex_condition_finder.h" #include "ift/encoder/dependency_closure.h" #include "ift/encoder/glyph_closure_cache.h" #include "ift/encoder/glyph_condition_set.h" @@ -21,9 +20,6 @@ #include "ift/encoder/types.h" #include "ift/freq/probability_bound.h" -using ift::config::FIND_CONDITIONS; -using ift::config::PATCH; - namespace ift::encoder { using absl::btree_map; @@ -65,15 +61,7 @@ class GlyphGroupingsTest : public ::testing::Test { glyph_groupings_(hb_face_get_glyph_count(roboto_.get())), glyph_groupings_simplification_(hb_face_get_glyph_count(roboto_.get()), - true), - segments_complex_({ - Segment({0x54}), // s0 - Segment({0x6C}), // s1 - Segment({0x13C}), // s2 - Segment({0x21A}), // s3 - Segment({0xF6C3}), // s4 - }), - glyph_groupings_complex_(hb_face_get_glyph_count(roboto_.get())) { + true) { uint32_t num_glyphs = hb_face_get_glyph_count(roboto_.get()); SubsetDefinition init_font_segment; @@ -83,7 +71,7 @@ class GlyphGroupingsTest : public ::testing::Test { closure_cache_ = std::move(*GlyphClosureCache::Create(roboto_.get(), *resolver)); requested_segmentation_info_ = *RequestedSegmentationInformation::Create( - segments_, init_font_segment, *closure_cache_, PATCH); + segments_, init_font_segment, *closure_cache_); glyph_conditions_ = std::make_unique(num_glyphs); @@ -125,34 +113,6 @@ class GlyphGroupingsTest : public ::testing::Test { // Disjunctive on segments 2 and 3 glyph_conditions_->AddOrCondition(cp_to_gid_['j'], 2); glyph_conditions_->AddOrCondition(cp_to_gid_['j'], 3); - - SetupComplexCase(); - } - - void SetupComplexCase() { - hb_font_t* font = hb_font_create(roboto_.get()); - std::vector codepoints = {0x54, 0x6C, 0x13C, 0x21A, 0xF6C3}; - - for (hb_codepoint_t cp : codepoints) { - glyph_id_t gid; - hb_font_get_nominal_glyph(font, cp, &gid); - cp_to_gid_[cp] = gid; - glyphs_to_group_complex_.insert(gid); - } - hb_font_destroy(font); - - glyph_conditions_complex_ = std::make_unique( - hb_face_get_glyph_count(roboto_.get())); - glyph_conditions_complex_->AddAndCondition(ToGlyph(0x54), 0); - glyph_conditions_complex_->AddAndCondition(ToGlyph(0x6C), 1); - - SubsetDefinition init_font_segment; - AddInitSubsetDefaults(init_font_segment); - - requested_segmentation_info_complex_ = - *RequestedSegmentationInformation::Create( - segments_complex_, init_font_segment, *closure_cache_, - FIND_CONDITIONS); } hb_face_unique_ptr from_file(const char* filename) { @@ -182,19 +142,11 @@ class GlyphGroupingsTest : public ::testing::Test { GlyphGroupings glyph_groupings_simplification_; GlyphSet glyphs_to_group_; flat_hash_map cp_to_gid_; - - std::vector segments_complex_; - std::unique_ptr glyph_conditions_complex_; - GlyphGroupings glyph_groupings_complex_; - GlyphSet glyphs_to_group_complex_; - std::unique_ptr - requested_segmentation_info_complex_; }; TEST_F(GlyphGroupingsTest, SimpleGrouping) { - auto sc = glyph_groupings_.GroupGlyphs(*requested_segmentation_info_, - *glyph_conditions_, *closure_cache_, - std::nullopt, glyphs_to_group_, {}); + auto sc = glyph_groupings_.GroupGlyphs( + *requested_segmentation_info_, *glyph_conditions_, glyphs_to_group_, {}); ASSERT_TRUE(sc.ok()) << sc; // Condition map: @@ -217,9 +169,8 @@ TEST_F(GlyphGroupingsTest, SimpleGrouping) { } TEST_F(GlyphGroupingsTest, SegmentChange) { - auto sc = glyph_groupings_.GroupGlyphs(*requested_segmentation_info_, - *glyph_conditions_, *closure_cache_, - std::nullopt, glyphs_to_group_, {}); + auto sc = glyph_groupings_.GroupGlyphs( + *requested_segmentation_info_, *glyph_conditions_, glyphs_to_group_, {}); ASSERT_TRUE(sc.ok()) << sc; // Now in the glyph condition set combine segments s1 into s0 @@ -253,8 +204,7 @@ TEST_F(GlyphGroupingsTest, SegmentChange) { // Recompute the grouping sc = glyph_groupings_.GroupGlyphs(*requested_segmentation_info_, - new_conditions, *closure_cache_, - std::nullopt, glyphs_to_group_, {}); + new_conditions, glyphs_to_group_, {}); ASSERT_TRUE(sc.ok()) << sc; // Condition map: @@ -280,8 +230,7 @@ TEST_F(GlyphGroupingsTest, CombinePatches) { ASSERT_TRUE(sc.ok()) << sc; sc = glyph_groupings_.GroupGlyphs(*requested_segmentation_info_, - *glyph_conditions_, *closure_cache_, - std::nullopt, glyphs_to_group_, {}); + *glyph_conditions_, glyphs_to_group_, {}); ASSERT_TRUE(sc.ok()) << sc; // Condition map: @@ -308,8 +257,7 @@ TEST_F(GlyphGroupingsTest, CombinePatches_WithSimpfliciation) { ASSERT_TRUE(sc.ok()) << sc; sc = glyph_groupings_simplification_.GroupGlyphs( - *requested_segmentation_info_, *glyph_conditions_, *closure_cache_, - std::nullopt, glyphs_to_group_, {}); + *requested_segmentation_info_, *glyph_conditions_, glyphs_to_group_, {}); ASSERT_TRUE(sc.ok()) << sc; // Starting Condition map: @@ -346,8 +294,7 @@ TEST_F(GlyphGroupingsTest, CombinePatches_SimplificationCollision) { ASSERT_TRUE(sc.ok()) << sc; sc = glyph_groupings_simplification_.GroupGlyphs( - *requested_segmentation_info_, *glyph_conditions_, *closure_cache_, - std::nullopt, glyphs_to_group_, {}); + *requested_segmentation_info_, *glyph_conditions_, glyphs_to_group_, {}); ASSERT_TRUE(sc.ok()) << sc; flat_hash_map expected = { @@ -375,74 +322,26 @@ TEST_F(GlyphGroupingsTest, CombinePatches_SimplificationCollisionInvalidation) { ASSERT_TRUE(sc.ok()) << sc; sc = glyph_groupings_simplification_.GroupGlyphs( - *requested_segmentation_info_, *glyph_conditions_, *closure_cache_, - std::nullopt, glyphs_to_group_, {}); + *requested_segmentation_info_, *glyph_conditions_, glyphs_to_group_, {}); ASSERT_TRUE(sc.ok()) << sc; // Invalidate 'l' by calling GroupGlyphs with 'l' in the glyphs set. sc = glyph_groupings_simplification_.GroupGlyphs( - *requested_segmentation_info_, *glyph_conditions_, *closure_cache_, - std::nullopt, ToGlyphs({'l'}), {}); - ASSERT_TRUE(sc.ok()) << sc; -} - -TEST_F(GlyphGroupingsTest, CombinePatches_WithInertSpecialCase) { - // Initial Condition map: - // s0 -> {a, b} - // s1 -> {c, d} - // s3 -> {k} - // s2 AND s3 -> {e, f} - // s2 OR s3 -> {j} - // s3 OR s4 -> {g, h} - auto sc = glyph_groupings_.CombinePatches(ToGlyphs({'a'}), ToGlyphs({'c'})); - ASSERT_TRUE(sc.ok()) << sc; - - sc = glyph_groupings_.GroupGlyphs(*requested_segmentation_info_, - *glyph_conditions_, *closure_cache_, - std::nullopt, glyphs_to_group_, {}); + *requested_segmentation_info_, *glyph_conditions_, ToGlyphs({'l'}), {}); ASSERT_TRUE(sc.ok()) << sc; - - // Combined Condition map: - // s0 OR s1 -> {a, b, c, d} - // s3 -> {k} - // s2 AND s3 -> {e, f} - // s2 OR s3 -> {j} - // s3 OR s4 -> {g, h} - - // Create a new exclusive patch without using GroupGlyphs(), merges s1 and s3 - // into s3 - sc = glyph_groupings_.AddGlyphsToExclusiveGroup(3, ToGlyphs({'c', 'd', 'k'})); - ASSERT_TRUE(sc.ok()) << sc; - - // s3 should get pulled into the combined patch of s0 and s1: - // s0 OR s1 OR s3 -> {a, b, c, d, k} - // s2 AND s3 -> {e, f} - // s2 OR s3 -> {j} - // s3 OR s4 -> {g, h} - flat_hash_map expected = { - {ActivationCondition::or_segments({0, 3}, 0), - ToGlyphs({'a', 'b', 'c', 'd', 'k'})}, - {ActivationCondition::and_segments({2, 3}, 0), ToGlyphs({'e', 'f'})}, - {ActivationCondition::or_segments({2, 3}, 0), ToGlyphs({'j'})}, - {ActivationCondition::or_segments({3, 4}, 0), ToGlyphs({'g', 'h'})}, - }; - - ASSERT_EQ(expected, glyph_groupings_.ConditionsAndGlyphs()); } TEST_F(GlyphGroupingsTest, CombinePatches_Invalidates) { // Form grouping without union's - auto sc = glyph_groupings_.GroupGlyphs(*requested_segmentation_info_, - *glyph_conditions_, *closure_cache_, - std::nullopt, glyphs_to_group_, {}); + auto sc = glyph_groupings_.GroupGlyphs( + *requested_segmentation_info_, *glyph_conditions_, glyphs_to_group_, {}); ASSERT_TRUE(sc.ok()) << sc; sc = glyph_groupings_.CombinePatches(ToGlyphs({'g'}), ToGlyphs({'b'})); ASSERT_TRUE(sc.ok()) << sc; sc = glyph_groupings_.GroupGlyphs(*requested_segmentation_info_, - *glyph_conditions_, *closure_cache_, - std::nullopt, {}, {}); + *glyph_conditions_, {}, {}); ASSERT_TRUE(sc.ok()) << sc; // UnionPatches + GroupGlyphs() will automatically invalidate and then fix @@ -466,9 +365,8 @@ TEST_F(GlyphGroupingsTest, CombinePatches_Invalidates) { TEST_F(GlyphGroupingsTest, CombinePatches_PartialUpdate) { // Form grouping without union's - auto sc = glyph_groupings_.GroupGlyphs(*requested_segmentation_info_, - *glyph_conditions_, *closure_cache_, - std::nullopt, glyphs_to_group_, {}); + auto sc = glyph_groupings_.GroupGlyphs( + *requested_segmentation_info_, *glyph_conditions_, glyphs_to_group_, {}); ASSERT_TRUE(sc.ok()) << sc; sc = glyph_groupings_.CombinePatches(ToGlyphs({'g'}), ToGlyphs({'b'})); @@ -478,8 +376,8 @@ TEST_F(GlyphGroupingsTest, CombinePatches_PartialUpdate) { // and see if grouping correctly reforms the full mapping. Invalidates: // s0 -> {a, b} sc = glyph_groupings_.GroupGlyphs(*requested_segmentation_info_, - *glyph_conditions_, *closure_cache_, - std::nullopt, ToGlyphs({'a', 'b'}), {}); + *glyph_conditions_, ToGlyphs({'a', 'b'}), + {}); ASSERT_TRUE(sc.ok()) << sc; // Expected condition map: @@ -501,8 +399,8 @@ TEST_F(GlyphGroupingsTest, CombinePatches_PartialUpdate) { // Do another partial invalidation this time on: s3 OR s4 -> {g, h} sc = glyph_groupings_.GroupGlyphs(*requested_segmentation_info_, - *glyph_conditions_, *closure_cache_, - std::nullopt, ToGlyphs({'g', 'h'}), {}); + *glyph_conditions_, ToGlyphs({'g', 'h'}), + {}); ASSERT_TRUE(sc.ok()) << sc; // Groupings should still be the same. @@ -513,8 +411,7 @@ TEST_F(GlyphGroupingsTest, CombinePatches_Noop) { auto sc = glyph_groupings_.CombinePatches(ToGlyphs({'a'}), ToGlyphs({'b'})); ASSERT_TRUE(sc.ok()) << sc; sc = glyph_groupings_.GroupGlyphs(*requested_segmentation_info_, - *glyph_conditions_, *closure_cache_, - std::nullopt, glyphs_to_group_, {}); + *glyph_conditions_, glyphs_to_group_, {}); ASSERT_TRUE(sc.ok()) << sc; // The combination is a noop since it only combines things already in the @@ -542,8 +439,7 @@ TEST_F(GlyphGroupingsTest, CombinePatches_AffectsConjunction) { ASSERT_TRUE(sc.ok()) << sc; sc = glyph_groupings_.GroupGlyphs(*requested_segmentation_info_, - *glyph_conditions_, *closure_cache_, - std::nullopt, glyphs_to_group_, {}); + *glyph_conditions_, glyphs_to_group_, {}); ASSERT_TRUE(sc.ok()) << sc; // Condition map after merging 'd' and 'e': @@ -571,8 +467,7 @@ TEST_F(GlyphGroupingsTest, CombinePatches_SegmentChanges) { ASSERT_TRUE(sc.ok()) << sc; sc = glyph_groupings_.GroupGlyphs(*requested_segmentation_info_, - *glyph_conditions_, *closure_cache_, - std::nullopt, glyphs_to_group_, {}); + *glyph_conditions_, glyphs_to_group_, {}); ASSERT_TRUE(sc.ok()) << sc; // Now in the glyph condition set combine segments s1 into s0 @@ -606,9 +501,9 @@ TEST_F(GlyphGroupingsTest, CombinePatches_SegmentChanges) { new_conditions.AddOrCondition(cp_to_gid_['j'], 3); // Recompute the grouping - sc = glyph_groupings_.GroupGlyphs( - *requested_segmentation_info_, new_conditions, *closure_cache_, - std::nullopt, ToGlyphs({'a', 'b', 'c', 'd'}), {}); + sc = glyph_groupings_.GroupGlyphs(*requested_segmentation_info_, + new_conditions, + ToGlyphs({'a', 'b', 'c', 'd'}), {}); ASSERT_TRUE(sc.ok()) << sc; // Condition map: @@ -632,13 +527,12 @@ TEST_F(GlyphGroupingsTest, EqualityRespectsPatchCombination) { ASSERT_TRUE(sc.ok()) << sc; sc = glyph_groupings_.GroupGlyphs(*requested_segmentation_info_, - *glyph_conditions_, *closure_cache_, - std::nullopt, glyphs_to_group_, {}); + *glyph_conditions_, glyphs_to_group_, {}); ASSERT_TRUE(sc.ok()) << sc; GlyphGroupings other(hb_face_get_glyph_count(roboto_.get())); sc = other.GroupGlyphs(*requested_segmentation_info_, *glyph_conditions_, - *closure_cache_, std::nullopt, glyphs_to_group_, {}); + glyphs_to_group_, {}); ASSERT_TRUE(sc.ok()) << sc; // other does not have the same patch combinations and so should not be equal @@ -649,7 +543,7 @@ TEST_F(GlyphGroupingsTest, EqualityRespectsPatchCombination) { ASSERT_TRUE(sc.ok()); sc = other.GroupGlyphs(*requested_segmentation_info_, *glyph_conditions_, - *closure_cache_, std::nullopt, glyphs_to_group_, {}); + glyphs_to_group_, {}); ASSERT_TRUE(sc.ok()) << sc; // Now that combined patches matches they should be equal. @@ -661,8 +555,7 @@ TEST_F(GlyphGroupingsTest, ExclusiveGlyphsRespectsPatchCombinations) { ASSERT_TRUE(sc.ok()) << sc; sc = glyph_groupings_.GroupGlyphs(*requested_segmentation_info_, - *glyph_conditions_, *closure_cache_, - std::nullopt, glyphs_to_group_, {}); + *glyph_conditions_, glyphs_to_group_, {}); ASSERT_TRUE(sc.ok()) << sc; // Condition map: @@ -679,228 +572,16 @@ TEST_F(GlyphGroupingsTest, ExclusiveGlyphsRespectsPatchCombinations) { ASSERT_EQ(glyph_groupings_.ExclusiveGlyphs(10), (GlyphSet{})); } -TEST_F(GlyphGroupingsTest, ComplexConditionFinding_LeaveUnmapped) { - SubsetDefinition init_font_segment; - AddInitSubsetDefaults(init_font_segment); - std::unique_ptr segmentation_info = - *RequestedSegmentationInformation::Create( - segments_complex_, init_font_segment, *closure_cache_, PATCH); - - auto sc = glyph_groupings_complex_.GroupGlyphs( - *segmentation_info, *glyph_conditions_complex_, *closure_cache_, - std::nullopt, glyphs_to_group_complex_, {}); - ASSERT_TRUE(sc.ok()) << sc; - - // Condition map: - // if (s0) then p0 => {56} [excl] - // if (s1) then p0 => {80} [excl] - flat_hash_map expected = { - {ActivationCondition::exclusive_segment(0, 0), ToGlyphs({0x54})}, - {ActivationCondition::exclusive_segment(1, 0), ToGlyphs({0x6C})}, - }; - - ASSERT_EQ(expected, glyph_groupings_complex_.ConditionsAndGlyphs()); - ASSERT_EQ(glyph_groupings_complex_.UnmappedGlyphs(), - (GlyphSet{442, 748, 782})); -} - -TEST_F(GlyphGroupingsTest, ComplexConditionFinding_Basic) { - auto sc = glyph_groupings_complex_.GroupGlyphs( - *requested_segmentation_info_complex_, *glyph_conditions_complex_, - *closure_cache_, std::nullopt, glyphs_to_group_complex_, {}); - ASSERT_TRUE(sc.ok()) << sc; - - // Condition map: - // if (s0) then p0 => {56} [excl] - // if (s1) then p0 => {80} [excl] - // if ((s0 OR s3 OR s4)) then p0 => {782} - // if ((s1 OR s2 OR s4)) then p0 => {748} - // if ((s2 OR s3 OR s4)) then p0 => {442} - flat_hash_map expected = { - {ActivationCondition::exclusive_segment(0, 0), ToGlyphs({0x54})}, - {ActivationCondition::exclusive_segment(1, 0), ToGlyphs({0x6C})}, - - {ActivationCondition::or_segments({0, 3, 4}, 0), {782}}, - {ActivationCondition::or_segments({1, 2, 4}, 0), {748}}, - {ActivationCondition::or_segments({2, 3, 4}, 0), {442}}, - }; - - ASSERT_EQ(expected, glyph_groupings_complex_.ConditionsAndGlyphs()); - ASSERT_TRUE(glyph_groupings_complex_.UnmappedGlyphs().empty()); -} - -TEST_F(GlyphGroupingsTest, ComplexConditionFinding_Basic_WithDependencyGraph) { - std::unique_ptr dep_closure = *DependencyClosure::Create( - requested_segmentation_info_complex_.get(), roboto_.get(), *resolver); - - auto sc = glyph_groupings_complex_.GroupGlyphs( - *requested_segmentation_info_complex_, *glyph_conditions_complex_, - *closure_cache_, dep_closure.get(), glyphs_to_group_complex_, {}); - ASSERT_TRUE(sc.ok()) << sc; - - // Condition map: - // if (s0) then p0 => {56} [excl] - // if (s1) then p0 => {80} [excl] - // if ((s0 OR s3 OR s4)) then p0 => {782} - // if ((s1 OR s2 OR s4)) then p0 => {748} - // if ((s2 OR s3 OR s4)) then p0 => {442} - flat_hash_map expected = { - {ActivationCondition::exclusive_segment(0, 0), ToGlyphs({0x54})}, - {ActivationCondition::exclusive_segment(1, 0), ToGlyphs({0x6C})}, - - {ActivationCondition::or_segments({0, 3, 4}, 0), {782}}, - {ActivationCondition::or_segments({1, 2, 4}, 0), {748}}, - {ActivationCondition::or_segments({2, 3, 4}, 0), {442}}, - }; - - ASSERT_EQ(expected, glyph_groupings_complex_.ConditionsAndGlyphs()); - ASSERT_TRUE(glyph_groupings_complex_.UnmappedGlyphs().empty()); - - // Also try incremental - sc = glyph_groupings_complex_.GroupGlyphs( - *requested_segmentation_info_complex_, *glyph_conditions_complex_, - *closure_cache_, dep_closure.get(), {748}, {}); - ASSERT_TRUE(sc.ok()) << sc; - ASSERT_EQ(expected, glyph_groupings_complex_.ConditionsAndGlyphs()); - ASSERT_TRUE(glyph_groupings_complex_.UnmappedGlyphs().empty()); -} - -TEST_F(GlyphGroupingsTest, ComplexConditionFinding_IncrementalUnchanged) { - auto sc = glyph_groupings_complex_.GroupGlyphs( - *requested_segmentation_info_complex_, *glyph_conditions_complex_, - *closure_cache_, std::nullopt, glyphs_to_group_complex_, {}); - ASSERT_TRUE(sc.ok()) << sc; - flat_hash_map expected = - glyph_groupings_complex_.ConditionsAndGlyphs(); - - // Incremental regrouping, should arrive back at the same mapping. - sc = glyph_groupings_complex_.GroupGlyphs( - *requested_segmentation_info_complex_, *glyph_conditions_complex_, - *closure_cache_, std::nullopt, {748}, {}); - ASSERT_TRUE(sc.ok()) << sc; - ASSERT_EQ(expected, glyph_groupings_complex_.ConditionsAndGlyphs()); - ASSERT_TRUE(glyph_groupings_complex_.UnmappedGlyphs().empty()); - - sc = glyph_groupings_complex_.GroupGlyphs( - *requested_segmentation_info_complex_, *glyph_conditions_complex_, - *closure_cache_, std::nullopt, ToGlyphs({0x54}), {2}); - ASSERT_TRUE(sc.ok()) << sc; - ASSERT_EQ(expected, glyph_groupings_complex_.ConditionsAndGlyphs()); - ASSERT_TRUE(glyph_groupings_complex_.UnmappedGlyphs().empty()); -} - -TEST_F(GlyphGroupingsTest, ComplexConditionFinding_IncrementalChanged) { - auto sc = glyph_groupings_complex_.GroupGlyphs( - *requested_segmentation_info_complex_, *glyph_conditions_complex_, - *closure_cache_, std::nullopt, glyphs_to_group_complex_, {}); - ASSERT_TRUE(sc.ok()) << sc; - - // Modify a segment definition and then incremental recompute the groupings. - Segment merged = segments_complex_[0]; - merged.Definition().Union(segments_complex_[3].Definition()); - requested_segmentation_info_complex_->AssignMergedSegment(0, {3}, merged); - - // Incremental regrouping, should arrive back at the same mapping. - sc = glyph_groupings_complex_.GroupGlyphs( - *requested_segmentation_info_complex_, *glyph_conditions_complex_, - *closure_cache_, std::nullopt, ToGlyphs({0x54}), {0, 3}); - ASSERT_TRUE(sc.ok()) << sc; - - flat_hash_map expected = { - {ActivationCondition::exclusive_segment(0, 0), ToGlyphs({0x54})}, - {ActivationCondition::exclusive_segment(1, 0), ToGlyphs({0x6C})}, - {ActivationCondition::or_segments({1, 2, 4}, 0), {748}}, - }; - - auto new_mappings = *FindSupersetDisjunctiveConditionsFor( - *requested_segmentation_info_complex_, *glyph_conditions_complex_, - *closure_cache_, {442, 782}, SegmentSet::all()); - - for (const auto& [s, g] : new_mappings) { - if (s.size() == 1) { - expected[ActivationCondition::exclusive_segment(*s.begin(), 0)].union_set( - g); - } else { - expected[ActivationCondition::or_segments(s, 0)].union_set(g); - } - } - - ASSERT_EQ(expected, glyph_groupings_complex_.ConditionsAndGlyphs()); - ASSERT_TRUE(glyph_groupings_complex_.UnmappedGlyphs().empty()); -} - -TEST_F(GlyphGroupingsTest, ComplexConditionFinding_CombinedPatches) { - auto sc = glyph_groupings_complex_.CombinePatches({ToGlyph(0x6C)}, {782}); - ASSERT_TRUE(sc.ok()) << sc; - - sc = glyph_groupings_complex_.GroupGlyphs( - *requested_segmentation_info_complex_, *glyph_conditions_complex_, - *closure_cache_, std::nullopt, glyphs_to_group_complex_, {}); - ASSERT_TRUE(sc.ok()) << sc; - - // Condition map: - // if (s0) then p0 => {56} [excl] - // if ((s0 OR s1 OR s3 OR s4)) then p0 => {80, 782} - // if ((s1 OR s2 OR s4)) then p0 => {748} - // if ((s2 OR s3 OR s4)) then p0 => {442} - flat_hash_map expected = { - {ActivationCondition::exclusive_segment(0, 0), ToGlyphs({0x54})}, - - {ActivationCondition::or_segments({0, 1, 3, 4}, 0), {ToGlyph(0x6C), 782}}, - {ActivationCondition::or_segments({1, 2, 4}, 0), {748}}, - {ActivationCondition::or_segments({2, 3, 4}, 0), {442}}, - }; - - ASSERT_EQ(expected, glyph_groupings_complex_.ConditionsAndGlyphs()); - ASSERT_TRUE(glyph_groupings_complex_.UnmappedGlyphs().empty()); -} - -TEST_F(GlyphGroupingsTest, - ComplexConditionFinding_IncrementalAndCombinedPatches) { - auto sc = glyph_groupings_complex_.CombinePatches({ToGlyph(0x6C)}, {782}); - ASSERT_TRUE(sc.ok()) << sc; - - sc = glyph_groupings_complex_.GroupGlyphs( - *requested_segmentation_info_complex_, *glyph_conditions_complex_, - *closure_cache_, std::nullopt, glyphs_to_group_complex_, {}); - ASSERT_TRUE(sc.ok()) << sc; - - // Modify a segment definition and then incremental recompute the groupings. - Segment merged = segments_complex_[0]; - merged.Definition().Union(segments_complex_[3].Definition()); - requested_segmentation_info_complex_->AssignMergedSegment(0, {3}, merged); - - sc = glyph_groupings_complex_.GroupGlyphs( - *requested_segmentation_info_complex_, *glyph_conditions_complex_, - *closure_cache_, std::nullopt, ToGlyphs({0x54}), {0, 3}); - ASSERT_TRUE(sc.ok()) << sc; - - // Condition map: - // if ((s0 OR s1)) then p0 => {56, 80, 782} - // if ((s1 OR s2 OR s4)) then p0 => {748} - // if ((s0 OR s2 OR s4)) then p0 => {442} - flat_hash_map expected = { - {ActivationCondition::or_segments({0, 1}, 0), - {ToGlyph(0x54), ToGlyph(0x6C), 782}}, - {ActivationCondition::or_segments({1, 2, 4}, 0), {748}}, - {ActivationCondition::or_segments({0, 2, 4}, 0), {442}}, - }; - - ASSERT_EQ(expected, glyph_groupings_complex_.ConditionsAndGlyphs()); - ASSERT_TRUE(glyph_groupings_complex_.UnmappedGlyphs().empty()); -} - TEST_F(GlyphGroupingsTest, IncrementalExclusiveConflict) { // Initial grouping for both - auto sc = glyph_groupings_.GroupGlyphs( - *requested_segmentation_info_, *glyph_conditions_, *closure_cache_, - std::nullopt, {ToGlyph('a'), ToGlyph('b')}, {}); + auto sc = glyph_groupings_.GroupGlyphs(*requested_segmentation_info_, + *glyph_conditions_, + {ToGlyph('a'), ToGlyph('b')}, {}); ASSERT_TRUE(sc.ok()) << sc; // Incremental grouping for ONLY one of them sc = glyph_groupings_.GroupGlyphs(*requested_segmentation_info_, - *glyph_conditions_, *closure_cache_, - std::nullopt, {ToGlyph('a')}, {}); + *glyph_conditions_, {ToGlyph('a')}, {}); EXPECT_TRUE(sc.ok()) << sc; ActivationCondition expected = ActivationCondition::exclusive_segment(0, 0); @@ -923,9 +604,8 @@ TEST_F(GlyphGroupingsTest, glyph_conditions_->AddOrCondition(gid_z, 1); // Initial grouping - auto sc = glyph_groupings_.GroupGlyphs(*requested_segmentation_info_, - *glyph_conditions_, *closure_cache_, - std::nullopt, glyphs_to_group_, {}); + auto sc = glyph_groupings_.GroupGlyphs( + *requested_segmentation_info_, *glyph_conditions_, glyphs_to_group_, {}); ASSERT_TRUE(sc.ok()) << sc; // Merge s0 and s1. This will create combined condition s0 OR s1. @@ -934,8 +614,7 @@ TEST_F(GlyphGroupingsTest, // Apply the merge. sc = glyph_groupings_.GroupGlyphs(*requested_segmentation_info_, - *glyph_conditions_, *closure_cache_, - std::nullopt, glyphs_to_group_, {}); + *glyph_conditions_, glyphs_to_group_, {}); ASSERT_TRUE(sc.ok()) << sc; // Trigger another merge that does NOT involve s0 or s1, which will cause @@ -944,9 +623,9 @@ TEST_F(GlyphGroupingsTest, ASSERT_TRUE(sc.ok()) << sc; // Apply the second merge. Only pass affected glyphs. - sc = glyph_groupings_.GroupGlyphs( - *requested_segmentation_info_, *glyph_conditions_, *closure_cache_, - std::nullopt, ToGlyphs({'g', 'h', 'j'}), {}); + sc = glyph_groupings_.GroupGlyphs(*requested_segmentation_info_, + *glyph_conditions_, + ToGlyphs({'g', 'h', 'j'}), {}); ASSERT_TRUE(sc.ok()) << sc; // Check if 'z' is still in s0 OR s1. diff --git a/ift/encoder/glyph_segmentation.cc b/ift/encoder/glyph_segmentation.cc index c7a94759..821fa5da 100644 --- a/ift/encoder/glyph_segmentation.cc +++ b/ift/encoder/glyph_segmentation.cc @@ -40,7 +40,7 @@ namespace ift::encoder { Status GlyphSegmentation::ConditionsToSegmentation( const btree_map& conditions, - const SegmentSet& fallback_group, GlyphSegmentation& segmentation) { + GlyphSegmentation& segmentation) { patch_id_t next_id = 0; segmentation.patches_.clear(); segmentation.conditions_.clear(); @@ -58,11 +58,8 @@ Status GlyphSegmentation::ConditionsToSegmentation( segmentation.conditions_.insert(ActivationCondition::exclusive_segment( *condition.TriggeringSegments().begin(), next_id++)); } else { - bool is_fallback = !fallback_group.empty() && - condition.IsPurelyDisjunctive() && - (condition.TriggeringSegments() == fallback_group); segmentation.conditions_.insert(ActivationCondition::composite_condition( - condition.conditions(), next_id++, is_fallback)); + condition.conditions(), next_id++)); } } diff --git a/ift/encoder/glyph_segmentation.h b/ift/encoder/glyph_segmentation.h index 82e29297..6023e297 100644 --- a/ift/encoder/glyph_segmentation.h +++ b/ift/encoder/glyph_segmentation.h @@ -30,16 +30,13 @@ namespace ift::encoder { class GlyphSegmentation { public: GlyphSegmentation(SubsetDefinition init_font_segment, - ift::common::GlyphSet init_font_glyph_closure, - ift::common::GlyphSet unmapped_glyphs) + ift::common::GlyphSet init_font_glyph_closure) : init_font_segment_(init_font_segment), - init_font_glyph_closure_(init_font_glyph_closure), - unmapped_glyphs_(unmapped_glyphs) {} + init_font_glyph_closure_(init_font_glyph_closure) {} bool operator==(const GlyphSegmentation& other) const { return init_font_segment_ == other.init_font_segment_ && init_font_glyph_closure_ == other.init_font_glyph_closure_ && - unmapped_glyphs_ == other.unmapped_glyphs_ && conditions_ == other.conditions_ && segments_ == other.segments_ && patches_ == other.patches_; } @@ -75,17 +72,6 @@ class GlyphSegmentation { return patches_; } - /* - * These glyphs where unable to be grouped into patches due to complex - * interactions. - * - * TODO(garretrieger): instead of treating them separately generate a catch - * all patch that contains the unmapped glyphs. - */ - const ift::common::GlyphSet& UnmappedGlyphs() const { - return unmapped_glyphs_; - }; - /* * These glyphs should be included in the initial font. */ @@ -107,7 +93,6 @@ class GlyphSegmentation { static absl::Status ConditionsToSegmentation( const absl::btree_map& conditions, - const common::SegmentSet& fallback_group, GlyphSegmentation& segmentation); void CopySegments(const std::vector& segments); @@ -115,7 +100,6 @@ class GlyphSegmentation { private: SubsetDefinition init_font_segment_; ift::common::GlyphSet init_font_glyph_closure_; - ift::common::GlyphSet unmapped_glyphs_; absl::btree_set conditions_; std::vector segments_; absl::btree_map patches_; diff --git a/ift/encoder/glyph_segmentation_test.cc b/ift/encoder/glyph_segmentation_test.cc index 9b41948a..ca5f84d2 100644 --- a/ift/encoder/glyph_segmentation_test.cc +++ b/ift/encoder/glyph_segmentation_test.cc @@ -14,8 +14,7 @@ #include "ift/proto/patch_encoding.h" #include "ift/proto/patch_map.h" -using ift::config::CLOSURE_ONLY; -using ift::config::PATCH; +using ift::config::DEP_GRAPH_ONLY; using google::protobuf::TextFormat; using ift::common::BazelDataFileResolver; @@ -54,7 +53,7 @@ class GlyphSegmentationTest : public ::testing::Test { }; TEST_F(GlyphSegmentationTest, SimpleSegmentation_ToConfigProto) { - ClosureGlyphSegmenter segmenter(8, 11, PATCH, CLOSURE_ONLY, resolver); + ClosureGlyphSegmenter segmenter(8, 11, DEP_GRAPH_ONLY, resolver); auto segmentation = segmenter.CodepointToGlyphSegments(roboto.get(), {'a'}, {{'b'}, {'c'}}); ASSERT_TRUE(segmentation.ok()) << segmentation.status(); @@ -125,7 +124,7 @@ initial_glyphs { } TEST_F(GlyphSegmentationTest, SimpleSegmentationWithFeatures_ToConfigProto) { - ClosureGlyphSegmenter segmenter(8, 11, PATCH, CLOSURE_ONLY, resolver); + ClosureGlyphSegmenter segmenter(8, 11, DEP_GRAPH_ONLY, resolver); SubsetDefinition smcp; smcp.feature_tags.insert(HB_TAG('s', 'm', 'c', 'p')); @@ -266,7 +265,7 @@ initial_glyphs { } TEST_F(GlyphSegmentationTest, MixedAndOr_ToConfigProto) { - ClosureGlyphSegmenter segmenter(8, 11, PATCH, CLOSURE_ONLY, resolver); + ClosureGlyphSegmenter segmenter(8, 11, DEP_GRAPH_ONLY, resolver); UnicodeFrequencies freq; auto segmentation = segmenter.CodepointToGlyphSegments( roboto.get(), {'a'}, {{'f', 0xc1}, {'i', 0x106}}); @@ -384,7 +383,7 @@ initial_glyphs { TEST_F(GlyphSegmentationTest, MergeBases_ToConfigProto) { // {e, f} is too smal, since no conditional patches exist it should merge with // the next available base which is {'j', 'k'} - ClosureGlyphSegmenter segmenter(8, 11, PATCH, CLOSURE_ONLY, resolver); + ClosureGlyphSegmenter segmenter(8, 11, DEP_GRAPH_ONLY, resolver); auto segmentation = segmenter.CodepointToGlyphSegments(roboto.get(), {}, { @@ -538,7 +537,4 @@ TEST_F(GlyphSegmentationTest, ActivationConditionOrdering) { ASSERT_FALSE(g < f); } -// TODO(garretrieger): add test where or_set glyphs are moved back to unmapped -// due to found "additional conditions". - } // namespace ift::encoder \ No newline at end of file diff --git a/ift/encoder/merger.cc b/ift/encoder/merger.cc index ba018f93..92ce7076 100644 --- a/ift/encoder/merger.cc +++ b/ift/encoder/merger.cc @@ -190,7 +190,7 @@ StatusOr> Merger::TryNextPatchMerge() { last_exclusive); it != Context().glyph_groupings.OrderedConditions().end(); ++it) { const auto& condition = *it; - if (condition.IsFallback() || condition.IsExclusive()) { + if (condition.IsExclusive()) { continue; } const GlyphSet& glyphs = @@ -758,10 +758,7 @@ Status Merger::CollectCompositeCandidateMerges( last_exclusive); it != context_->glyph_groupings.OrderedConditions().end(); it++) { const auto& next_condition = *it; - if (next_condition.IsFallback() || next_condition.IsExclusive()) { - // Merging the fallback will cause all segments to be merged into one, - // which is undesirable so don't consider the fallback. Also skip - // any non composite conditions. + if (next_condition.IsExclusive()) { continue; } @@ -823,7 +820,7 @@ GlyphSet Merger::ComputeCandidatePatchMergeGlyphs( it != context.glyph_groupings.OrderedConditions().end(); it++) { const auto& condition = *it; - if (condition.IsFallback() || condition.IsExclusive()) { + if (condition.IsExclusive()) { continue; } @@ -1047,10 +1044,9 @@ StatusOr> Merger::TryMergingACompositeCondition( for (const ActivationCondition& next_condition : context_->glyph_groupings.TriggeringSegmentToConditions( base_segment_index)) { - if (next_condition.IsFallback() || next_condition.IsUnitary()) { - // Merging the fallback will cause all segments to be merged into one, - // which is undesirable so don't consider the fallback. Unitary conditions - // only reference a single segment so there is nothing to merge. + if (next_condition.IsUnitary()) { + // Unitary conditions only reference a single segment so there is nothing + // to merge. continue; } diff --git a/ift/encoder/requested_segmentation_information.cc b/ift/encoder/requested_segmentation_information.cc index b33b3d74..e4a13d33 100644 --- a/ift/encoder/requested_segmentation_information.cc +++ b/ift/encoder/requested_segmentation_information.cc @@ -7,8 +7,6 @@ #include "ift/encoder/segment.h" #include "ift/encoder/subset_definition.h" -using ift::config::UnmappedGlyphHandling; - using absl::StatusOr; using ift::common::CodepointSet; using ift::common::SegmentSet; @@ -37,13 +35,11 @@ static bool CheckSegmentsAreDisjoint(const SubsetDefinition& init_segment, } StatusOr> -RequestedSegmentationInformation::Create( - std::vector segments, SubsetDefinition init_font_segment, - GlyphClosureCache& closure_cache, - UnmappedGlyphHandling unmapped_glyph_handling) { +RequestedSegmentationInformation::Create(std::vector segments, + SubsetDefinition init_font_segment, + GlyphClosureCache& closure_cache) { std::unique_ptr info( - new RequestedSegmentationInformation(segments, init_font_segment, - unmapped_glyph_handling)); + new RequestedSegmentationInformation(segments, init_font_segment)); TRYV(info->ReassignInitSubset(closure_cache, init_font_segment)); info->segments_disjoint_ = CheckSegmentsAreDisjoint(init_font_segment, info->segments_); @@ -51,11 +47,8 @@ RequestedSegmentationInformation::Create( } RequestedSegmentationInformation::RequestedSegmentationInformation( - std::vector segments, SubsetDefinition init_font_segment, - UnmappedGlyphHandling unmapped_glyph_handling) - : segments_(std::move(segments)), - init_font_segment_(), - unmapped_glyph_handling_(unmapped_glyph_handling) { + std::vector segments, SubsetDefinition init_font_segment) + : segments_(std::move(segments)), init_font_segment_() { // ReassignInitSubset expects full_definition_ is already populated. full_definition_ = init_font_segment; for (size_t i = 0; i < segments_.size(); ++i) { diff --git a/ift/encoder/requested_segmentation_information.h b/ift/encoder/requested_segmentation_information.h index 7bb5447c..13a9b649 100644 --- a/ift/encoder/requested_segmentation_information.h +++ b/ift/encoder/requested_segmentation_information.h @@ -6,7 +6,6 @@ #include "absl/container/flat_hash_map.h" #include "ift/common/int_set.h" #include "ift/common/try.h" -#include "ift/config/common.pb.h" #include "ift/encoder/glyph_closure_cache.h" #include "ift/encoder/init_subset_defaults.h" #include "ift/encoder/segment.h" @@ -22,13 +21,11 @@ class RequestedSegmentationInformation { public: static absl::StatusOr> Create(std::vector segments, SubsetDefinition init_font_segment, - GlyphClosureCache& closure_cache, - ift::config::UnmappedGlyphHandling unmapped_glyph_handling); + GlyphClosureCache& closure_cache); private: - RequestedSegmentationInformation( - std::vector segments, SubsetDefinition init_font_segment, - ift::config::UnmappedGlyphHandling unmapped_glyph_handling); + RequestedSegmentationInformation(std::vector segments, + SubsetDefinition init_font_segment); public: // Merge all of the segments in to_merge into base, assigned it @@ -83,10 +80,6 @@ class RequestedSegmentationInformation { return absl::OkStatus(); } - ift::config::UnmappedGlyphHandling GetUnmappedGlyphHandling() const { - return unmapped_glyph_handling_; - } - const SubsetDefinition& InitFontSegment() const { return init_font_segment_; } // Returns the init font segment with all default always included items @@ -173,7 +166,6 @@ class RequestedSegmentationInformation { ift::common::CodepointSet full_codepoint_closure_; bool segments_disjoint_; - enum ift::config::UnmappedGlyphHandling unmapped_glyph_handling_; absl::flat_hash_map codepoint_to_segments_; }; diff --git a/ift/encoder/requested_segmentation_information_test.cc b/ift/encoder/requested_segmentation_information_test.cc index 7fc1539d..8ea8035d 100644 --- a/ift/encoder/requested_segmentation_information_test.cc +++ b/ift/encoder/requested_segmentation_information_test.cc @@ -17,7 +17,6 @@ using ift::common::FontData; using ift::common::hb_face_unique_ptr; using ift::common::make_hb_face; using ift::common::SegmentSet; -using ift::config::PATCH; using ift::freq::ProbabilityBound; namespace ift::encoder { @@ -48,8 +47,7 @@ TEST_F(RequestedSegmentationInformationTest, SegmentsForCodepoints) { {{'d'}}, }; - auto info_or = - RequestedSegmentationInformation::Create(segments, {}, *cache, PATCH); + auto info_or = RequestedSegmentationInformation::Create(segments, {}, *cache); ASSERT_TRUE(info_or.ok()); auto& info = *info_or; @@ -71,8 +69,7 @@ TEST_F(RequestedSegmentationInformationTest, IndexUpdatesOnMerge) { {{'c'}}, }; - auto info_or = - RequestedSegmentationInformation::Create(segments, {}, *cache, PATCH); + auto info_or = RequestedSegmentationInformation::Create(segments, {}, *cache); ASSERT_TRUE(info_or.ok()); auto& info = *info_or; @@ -97,8 +94,7 @@ TEST_F(RequestedSegmentationInformationTest, IndexUpdatesOnReassignInit) { {{'b', 'c'}}, }; - auto info_or = - RequestedSegmentationInformation::Create(segments, {}, *cache, PATCH); + auto info_or = RequestedSegmentationInformation::Create(segments, {}, *cache); ASSERT_TRUE(info_or.ok()); auto& info = *info_or; diff --git a/ift/encoder/segmentation_context.cc b/ift/encoder/segmentation_context.cc index 1703161a..e5c97a5d 100644 --- a/ift/encoder/segmentation_context.cc +++ b/ift/encoder/segmentation_context.cc @@ -22,11 +22,8 @@ #include "ift/encoder/subset_definition.h" #include "ift/encoder/types.h" -using ift::config::CLOSURE_AND_VALIDATE_DEP_GRAPH; -using ift::config::CLOSURE_ONLY; -using ift::config::DEP_GRAPH_ONLY_WITH_SIMPLIFICATION; using ift::config::ConditionAnalysisMode; -using ift::config::UnmappedGlyphHandling; +using ift::config::DEP_GRAPH_ONLY_WITH_SIMPLIFICATION; using absl::Status; using absl::StatusOr; @@ -98,8 +95,7 @@ Status SegmentationContext::ValidateSegmentation( // In unicode range mode not all conditions get record into segmentation, so // check disjointness across the non-exclusive patches which were skipped. - TRYV(check_disjoint(glyph_groupings.UnmappedGlyphs())); - GlyphSet unreachable_glyphs = glyph_groupings.UnmappedGlyphs(); + GlyphSet unreachable_glyphs; for (const auto& cond : glyph_groupings.OrderedConditions()) { if (cond.IsExclusive()) { continue; @@ -214,84 +210,20 @@ void SegmentationContext::LogSegmentCodepointCountHistogram( } Status SegmentationContext::ReprocessChanged(InvalidationSet modified) { - segment_index_t last_merged_segment_index = modified.base_segment; - - if (!IsPureDepGraphAnalysisMode()) { - if (!InertSegments().contains(last_merged_segment_index)) { - VLOG(1) << "Re-analyzing segment " << last_merged_segment_index - << " due to merge."; - GlyphSet analysis_modified_gids = - TRY(ReprocessSegment(last_merged_segment_index)); - modified.glyphs.union_set(analysis_modified_gids); - } - } else { - modified.glyphs.union_set( - (*dependency_closure_) - ->SegmentsToAffectedGlyphs({modified.base_segment})); - TransferDependencyGraphGlyphConditions(modified.glyphs); - } + modified.glyphs.union_set( + dependency_closure_->SegmentsToAffectedGlyphs({modified.base_segment})); + TransferDependencyGraphGlyphConditions(modified.glyphs); return GroupGlyphs(modified.glyphs, modified.segments); } Status SegmentationContext::ReprocessAll() { - if (!IsPureDepGraphAnalysisMode()) { - for (segment_index_t segment_index = 0; - segment_index < SegmentationInfo().Segments().size(); - segment_index++) { - TRY(ReprocessSegment(segment_index)); - } - } else { - // Pull conditions directly out of the dep graph instead of running closure - // processing. - TransferDependencyGraphGlyphConditions( - segmentation_info_->NonInitFontGlyphs()); - } - - return GroupGlyphs(SegmentationInfo().NonInitFontGlyphs(), {}); -} - -StatusOr SegmentationContext::ReprocessSegment( - segment_index_t segment_index) { - if (segmentation_info_->Segments()[segment_index].Definition().Empty()) { - // Empty segment is a noop; - return GlyphSet{}; - } - - GlyphSet and_gids; - GlyphSet or_gids; - GlyphSet exclusive_gids; - TRYV(AnalyzeSegment({segment_index}, and_gids, or_gids, exclusive_gids)); - - GlyphSet changed_gids; - changed_gids.union_set(and_gids); - changed_gids.union_set(or_gids); - changed_gids.union_set(exclusive_gids); - - for (uint32_t gid : changed_gids) { - glyph_condition_set.InvalidateGlyphInformation(GlyphSet{gid}, - SegmentSet{segment_index}); - } - - if (and_gids.empty() && or_gids.empty()) { - inert_segments_.insert(segment_index); - } - - for (uint32_t and_gid : exclusive_gids) { - // TODO(garretrieger): if we are assigning an exclusive gid there should - // be no other and segments, check and error if this is violated. - glyph_condition_set.AddAndCondition(and_gid, segment_index); - } - - for (uint32_t and_gid : and_gids) { - glyph_condition_set.AddAndCondition(and_gid, segment_index); - } - - for (uint32_t or_gid : or_gids) { - glyph_condition_set.AddOrCondition(or_gid, segment_index); - } + // Pull conditions directly out of the dep graph instead of running closure + // processing. + TransferDependencyGraphGlyphConditions( + segmentation_info_->NonInitFontGlyphs()); - return changed_gids; + return GroupGlyphs(segmentation_info_->NonInitFontGlyphs(), {}); } /* @@ -319,19 +251,10 @@ Status SegmentationContext::ReassignInitSubset(SubsetDefinition new_def) { s_index++; } - SegmentSet segments_to_reprocess; - if (!segmentation_info_->Segments().empty()) { - segments_to_reprocess.insert_range( - 0, segmentation_info_->Segments().size() - 1); - } - if (dependency_closure_.has_value()) { - SubsetDefinition new_def_delta = new_def; - new_def_delta.Subtract(SegmentationInfo().InitFontSegment()); - // If dep graph is enabled we can use it to narrow the set of segments that - // need reprocessing. - segments_to_reprocess = - TRY((*dependency_closure_)->SegmentsThatInteractWith(new_def_delta)); - } + SubsetDefinition new_def_delta = new_def; + new_def_delta.Subtract(SegmentationInfo().InitFontSegment()); + SegmentSet segments_to_reprocess = + TRY(dependency_closure_->SegmentsThatInteractWith(new_def_delta)); TRYV(segmentation_info_->ReassignInitSubset(*glyph_closure_cache, new_def)); @@ -343,11 +266,9 @@ Status SegmentationContext::ReassignInitSubset(SubsetDefinition new_def) { original_face.get(), init_font_brotli_quality_); } - if (dependency_closure_.has_value()) { - // Dep closure doesn't need the expanded affected set, as it will traverse - // from the input segments to find the affected subgraph. - TRYV((*dependency_closure_)->InitFontChanged(segments_with_changed_defs)); - } + // Dep closure doesn't need the expanded affected set, as it will traverse + // from the input segments to find the affected subgraph. + TRYV(dependency_closure_->InitFontChanged(segments_with_changed_defs)); // All segments depend on the init subset def, so we must reprocess // everything. First reset condition set information: @@ -358,17 +279,11 @@ Status SegmentationContext::ReassignInitSubset(SubsetDefinition new_def) { inert_segments_.subtract(segments_to_reprocess); // Then reprocess segments: - if (!IsPureDepGraphAnalysisMode()) { - for (segment_index_t segment_index : segments_to_reprocess) { - TRY(ReprocessSegment(segment_index)); - } - } else { - GlyphSet gids; - for (segment_index_t s : segments_to_reprocess) { - gids.union_set((*dependency_closure_)->SegmentsToAffectedGlyphs({s})); - } - TransferDependencyGraphGlyphConditions(gids); + GlyphSet gids; + for (segment_index_t s : segments_to_reprocess) { + gids.union_set(dependency_closure_->SegmentsToAffectedGlyphs({s})); } + TransferDependencyGraphGlyphConditions(gids); // the groupings can be incrementally recomputed by looking at what conditions // have changed. @@ -388,9 +303,7 @@ Status SegmentationContext::ReassignInitSubset(SubsetDefinition new_def) { Status SegmentationContext::ResetSegments(std::vector segments) { segmentation_info_->ResetSegments(std::move(segments)); - if (dependency_closure_.has_value()) { - TRYV((*dependency_closure_)->SegmentsReset()); - } + TRYV(dependency_closure_->SegmentsReset()); uint32_t glyph_count = hb_face_get_glyph_count(original_face.get()); glyph_condition_set = GlyphConditionSet(glyph_count); glyph_groupings = GlyphGroupings( @@ -400,85 +313,10 @@ Status SegmentationContext::ResetSegments(std::vector segments) { return ReprocessAll(); } -static void PrintDiff(absl::string_view set_name, const GlyphSet& closure, - const GlyphSet& dep) { - std::string op = " == "; - if (closure != dep) { - op = " != "; - } - - LOG(ERROR) << "Set " << set_name << ": closure glyphs " << closure.ToString() - << op << " dependency glyphs " << dep.ToString(); -} - -Status SegmentationContext::AnalyzeSegment(const SegmentSet& segment_ids, - GlyphSet& and_gids, - GlyphSet& or_gids, - GlyphSet& exclusive_gids) { - ConditionAnalysisMode effective_mode = condition_analysis_mode_; - GlyphSet dep_and_gids = and_gids; - GlyphSet dep_or_gids = or_gids; - GlyphSet dep_exclusive_gids = exclusive_gids; - if (dependency_closure_.has_value()) { - auto accuracy = TRY((*dependency_closure_) - ->AnalyzeSegment(segment_ids, dep_and_gids, - dep_or_gids, dep_exclusive_gids, - effective_mode == DEP_GRAPH_ONLY_WITH_SIMPLIFICATION)); - if (!IsPureDepGraphAnalysisMode() && accuracy == DependencyClosure::INACCURATE) { - // Don't fallback to closure in pure dep graph mode. - effective_mode = CLOSURE_ONLY; - } - } - - if (effective_mode == CLOSURE_ONLY || - effective_mode == CLOSURE_AND_VALIDATE_DEP_GRAPH) { - TRYV(glyph_closure_cache->AnalyzeSegment( - *segmentation_info_, segment_ids, and_gids, or_gids, exclusive_gids)); - } else { - or_gids.union_set(dep_or_gids); - and_gids.union_set(dep_and_gids); - exclusive_gids.union_set(dep_exclusive_gids); - } - - if (effective_mode == CLOSURE_AND_VALIDATE_DEP_GRAPH) { - if (and_gids != dep_and_gids || or_gids != dep_or_gids || - exclusive_gids != dep_exclusive_gids) { - LOG(ERROR) << "Mismatch between closure and dependency analysis " - "conditions for segments " - << segment_ids.ToString(); - for (segment_index_t s : segment_ids) { - LOG(ERROR) << "segment[" << s << "].codepoints = " - << segmentation_info_->Segments() - .at(s) - .Definition() - .codepoints.ToString(); - LOG(ERROR) << "segment[" << s << "].features.size() = " - << segmentation_info_->Segments() - .at(s) - .Definition() - .feature_tags.size(); - } - PrintDiff("AND", and_gids, dep_and_gids); - PrintDiff("OR ", or_gids, dep_or_gids); - PrintDiff("EXC", exclusive_gids, dep_exclusive_gids); - LOG(ERROR) << "init codepoints = " - << segmentation_info_->InitFontSegment().codepoints.ToString(); - LOG(ERROR) << "init glyphs = " - << segmentation_info_->InitFontGlyphs().ToString(); - return absl::InternalError( - "Dependency graph conditions does not match the closure analysis " - "conditions"); - } - } - - return absl::OkStatus(); -} - StatusOr SegmentationContext::InitializeSegmentationContext( hb_face_t* face, SubsetDefinition initial_segment, std::vector segments, - UnmappedGlyphHandling unmapped_glyph_handling, ConditionAnalysisMode condition_analysis_mode, uint32_t brotli_quality, uint32_t init_font_brotli_quality, std::shared_ptr resolver, @@ -493,9 +331,8 @@ SegmentationContext::InitializeSegmentationContext( // No merging is done during init. SegmentationContext context = TRY(SegmentationContext::Create( - face, initial_segment, segments, unmapped_glyph_handling, - condition_analysis_mode, brotli_quality, init_font_brotli_quality, - std::move(resolver), target_mode)); + face, initial_segment, segments, condition_analysis_mode, brotli_quality, + init_font_brotli_quality, std::move(resolver), target_mode)); // ### Generate the initial conditions and groupings by processing all // segments and glyphs. ### @@ -507,7 +344,7 @@ SegmentationContext::InitializeSegmentationContext( void SegmentationContext::TransferDependencyGraphGlyphConditions( const GlyphSet& gids) { - const auto& conditions = (*dependency_closure_)->AllGlyphConditions(); + const auto& conditions = dependency_closure_->AllGlyphConditions(); for (glyph_id_t g : gids) { ActivationCondition condition = conditions.at(g); if (condition.IsUnitary()) { @@ -526,7 +363,7 @@ void SegmentationContext::TransferDependencyGraphGlyphConditions( } glyph_condition_set.SetCondition(g, condition); } - inert_segments_ = (*dependency_closure_)->InertSegments(); + inert_segments_ = dependency_closure_->InertSegments(); } } // namespace ift::encoder \ No newline at end of file diff --git a/ift/encoder/segmentation_context.h b/ift/encoder/segmentation_context.h index 564a1350..e1d705d8 100644 --- a/ift/encoder/segmentation_context.h +++ b/ift/encoder/segmentation_context.h @@ -4,6 +4,7 @@ #include #include #include +#include #include "absl/log/log.h" #include "absl/status/status.h" @@ -51,7 +52,6 @@ class SegmentationContext { static absl::StatusOr Create( hb_face_t* face, const SubsetDefinition& initial_segment, const std::vector& segments, - ift::config::UnmappedGlyphHandling unmapped_glyph_handling, ift::config::ConditionAnalysisMode condition_analysis_mode, uint32_t brotli_quality, uint32_t init_font_brotli_quality, std::shared_ptr resolver, @@ -63,13 +63,12 @@ class SegmentationContext { TRY(GlyphClosureCache::Create(face, *resolver)); std::unique_ptr segmentation_info = TRY(RequestedSegmentationInformation::Create(segments, initial_segment, - *closure_cache, - unmapped_glyph_handling)); + *closure_cache)); SegmentationContext context( - face, unmapped_glyph_handling, condition_analysis_mode, brotli_quality, - init_font_brotli_quality, std::move(closure_cache), - std::move(segmentation_info), std::move(resolver), target_mode); + face, condition_analysis_mode, brotli_quality, init_font_brotli_quality, + std::move(closure_cache), std::move(segmentation_info), + std::move(resolver), target_mode); TRYV(context.InitDependencyClosure()); return context; @@ -82,14 +81,13 @@ class SegmentationContext { std::unique_ptr segmentation_info = TRY(RequestedSegmentationInformation::Create( SegmentationInfo().Segments(), SegmentationInfo().InitFontSegment(), - *closure_cache, SegmentationInfo().GetUnmappedGlyphHandling())); + *closure_cache)); SegmentationContext context( - original_face.get(), SegmentationInfo().GetUnmappedGlyphHandling(), - condition_analysis_mode_, brotli_quality_, init_font_brotli_quality_, - std::move(closure_cache), std::move(segmentation_info), resolver_, - target_mode_, estimated_compression_ratio_, - estimated_woff2_base_overhead_); + original_face.get(), condition_analysis_mode_, brotli_quality_, + init_font_brotli_quality_, std::move(closure_cache), + std::move(segmentation_info), resolver_, target_mode_, + estimated_compression_ratio_, estimated_woff2_base_overhead_); context.initial_glyph_probabilities_ = initial_glyph_probabilities_; TRYV(context.InitDependencyClosure()); @@ -105,7 +103,6 @@ class SegmentationContext { static absl::StatusOr InitializeSegmentationContext( hb_face_t* face, SubsetDefinition initial_segment, std::vector segments, - ift::config::UnmappedGlyphHandling unmapped_glyph_handling, ift::config::ConditionAnalysisMode condition_analysis_mode, uint32_t brotli_quality, uint32_t init_font_brotli_quality, std::shared_ptr resolver, @@ -114,7 +111,6 @@ class SegmentationContext { private: SegmentationContext( hb_face_t* face, - ift::config::UnmappedGlyphHandling unmapped_glyph_handling, ift::config::ConditionAnalysisMode condition_analysis_mode, uint32_t brotli_quality, uint32_t init_font_brotli_quality, std::unique_ptr closure_cache, @@ -128,7 +124,7 @@ class SegmentationContext { glyph_closure_cache(std::move(closure_cache)), original_face(ift::common::make_hb_face(hb_face_reference(face))), segmentation_info_(std::move(segmentation_info)), - dependency_closure_(std::nullopt), + dependency_closure_(nullptr), glyph_condition_set(hb_face_get_glyph_count(face)), glyph_groupings(hb_face_get_glyph_count(face), condition_analysis_mode == @@ -144,11 +140,8 @@ class SegmentationContext { } absl::Status InitDependencyClosure() { - if (UsingDepGraph()) { - dependency_closure_ = TRY(DependencyClosure::Create( - segmentation_info_.get(), original_face.get(), *resolver_, - IsPureDepGraphAnalysisMode())); - } + dependency_closure_ = TRY(DependencyClosure::Create( + segmentation_info_.get(), original_face.get(), *resolver_)); return absl::OkStatus(); } @@ -178,39 +171,26 @@ class SegmentationContext { } void LogClosureStatistics() const { - uint64_t dep_graph_closures = 0; - uint64_t dep_graph_inaccurate = 0; - if (dependency_closure_.has_value()) { - dep_graph_closures = (*dependency_closure_)->AccurateResults() * 2; - dep_graph_inaccurate = (*dependency_closure_)->InaccurateResults(); - } - - uint64_t potential_closures = glyph_closure_cache->CacheHits() + - glyph_closure_cache->CacheMisses() + - dep_graph_closures; + uint64_t total_closures = + glyph_closure_cache->CacheHits() + glyph_closure_cache->CacheMisses(); double hb_subset_rate = - 100.0 * ((double)glyph_closure_cache->CacheMisses() / - (double)potential_closures); - double cache_hit_rate = 100.0 * ((double)glyph_closure_cache->CacheHits() / - (double)potential_closures); - double dep_graph_hit_rate = - 100.0 * ((double)dep_graph_closures / (double)potential_closures); - uint64_t other_closures = glyph_closure_cache->CacheHits() + - glyph_closure_cache->CacheMisses() - - (2 * dep_graph_inaccurate); - - VLOG(0) << ">> Of " << potential_closures - << " potential closure operations:" << std::endl + total_closures == 0 + ? 0.0 + : 100.0 * ((double)glyph_closure_cache->CacheMisses() / + (double)total_closures); + double cache_hit_rate = + total_closures == 0 + ? 0.0 + : 100.0 * ((double)glyph_closure_cache->CacheHits() / + (double)total_closures); + + VLOG(0) << ">> Of " << total_closures << " closure operations:" << std::endl << " " << glyph_closure_cache->CacheMisses() << " (" - << hb_subset_rate << "%) were handled by hb-subset-plan" + << hb_subset_rate << "%) were handled by hb-subset-plan" << std::endl - << " " << dep_graph_closures << " (" << dep_graph_hit_rate - << "%) were handled by dep graph" << std::endl << " " << glyph_closure_cache->CacheHits() << " (" - << cache_hit_rate << "%) were provided by the cache" << std::endl - << " " << other_closures - << " were from something other than AnalyzeSegment()" << std::endl; + << cache_hit_rate << "%) were provided by the cache" << std::endl; } const ift::common::SegmentSet& InertSegments() const { @@ -225,28 +205,6 @@ class SegmentationContext { return condition_analysis_mode_; } - bool IsPureDepGraphAnalysisMode() const { - switch (condition_analysis_mode_) { - case config::DEP_GRAPH_ONLY: - case config::DEP_GRAPH_ONLY_WITH_SIMPLIFICATION: - return true; - default: - return false; - } - } - - bool UsingDepGraph() const { - switch (condition_analysis_mode_) { - case config::CLOSURE_AND_DEP_GRAPH: - case config::CLOSURE_AND_VALIDATE_DEP_GRAPH: - case config::DEP_GRAPH_ONLY: - case config::DEP_GRAPH_ONLY_WITH_SIMPLIFICATION: - return true; - default: - return false; - } - } - // Assign a new merged segment to base and clear all of the segments that // were merged into it. uint32_t AssignMergedSegment(segment_index_t base, @@ -278,10 +236,7 @@ class SegmentationContext { // group glyphs is called. glyph_condition_set.InvalidateGlyphInformation(glyphs, segments); - if (dependency_closure_.has_value()) { - return (*dependency_closure_)->SegmentsMerged(base_segment, segments); - } - return absl::OkStatus(); + return dependency_closure_->SegmentsMerged(base_segment, segments); } /* @@ -305,13 +260,6 @@ class SegmentationContext { // Regenerates glyphs conditions and groupings for all segments. absl::Status ReprocessAll(); - // Performs a closure analysis on codepoints and returns the associated - // and, or, and exclusive glyph sets. - absl::Status AnalyzeSegment(const ift::common::SegmentSet& segment_ids, - ift::common::GlyphSet& and_gids, - ift::common::GlyphSet& or_gids, - ift::common::GlyphSet& exclusive_gids); - private: // Update the glyph groups for 'glyphs'. // @@ -319,23 +267,12 @@ class SegmentationContext { // calling this. absl::Status GroupGlyphs(const ift::common::GlyphSet& glyphs, const ift::common::SegmentSet& modified_segments) { - std::optional maybe_dep_closure = std::nullopt; - if (dependency_closure_.has_value()) { - maybe_dep_closure = (*dependency_closure_).get(); - } return glyph_groupings.GroupGlyphs(*segmentation_info_, glyph_condition_set, - *glyph_closure_cache, maybe_dep_closure, - glyphs, modified_segments, - !IsPureDepGraphAnalysisMode()); + glyphs, modified_segments); } void TransferDependencyGraphGlyphConditions(const common::GlyphSet& gids); - // Generates updated glyph conditions and glyph groupings for segment_index - // which has the provided set of codepoints. - absl::StatusOr ReprocessSegment( - segment_index_t segment_index); - /* * Ensures that the produce segmentation is: * - Disjoint (no duplicated glyphs) and doesn't overlap what's in the initial @@ -416,7 +353,7 @@ class SegmentationContext { private: std::unique_ptr segmentation_info_; - std::optional> dependency_closure_; + std::unique_ptr dependency_closure_; public: // == Phase 1 - derived from segments and init information diff --git a/ift/encoder/unicode_range_compiler_test.cc b/ift/encoder/unicode_range_compiler_test.cc index ea9677a7..36367fb9 100644 --- a/ift/encoder/unicode_range_compiler_test.cc +++ b/ift/encoder/unicode_range_compiler_test.cc @@ -21,9 +21,8 @@ using ift::common::FontData; using ift::common::FontHelper; using ift::common::hb_face_unique_ptr; using ift::common::Woff2; -using ift::config::CLOSURE_AND_VALIDATE_DEP_GRAPH; using ift::config::ConfigCompiler; -using ift::config::FIND_CONDITIONS; +using ift::config::DEP_GRAPH_ONLY; using ift::config::SegmentationPlan; namespace ift::encoder { @@ -64,8 +63,7 @@ TEST_F(UnicodeRangeCompilerTest, ExtractFontFamily) { } TEST_F(UnicodeRangeCompilerTest, CompileFromSegmentationPlan) { - ClosureGlyphSegmenter segmenter(1, 1, FIND_CONDITIONS, - CLOSURE_AND_VALIDATE_DEP_GRAPH, resolver, + ClosureGlyphSegmenter segmenter(1, 1, DEP_GRAPH_ONLY, resolver, ift::config::UNICODE_RANGE); auto segmentation = segmenter.CodepointToGlyphSegments( roboto.get(), {'a'}, {{'f'}, {'i'}, {'b', 'c', 'd'}}); diff --git a/util/fallback_analysis/count-fallback-glyphs-for-font.sh b/util/fallback_analysis/count-fallback-glyphs-for-font.sh deleted file mode 100755 index 2cb78260..00000000 --- a/util/fallback_analysis/count-fallback-glyphs-for-font.sh +++ /dev/null @@ -1,11 +0,0 @@ -#!/bin/bash - -FONT_NAME=$(basename $1) -./bazel-bin/util/gen_ift_segmentation_plan \ - --input_font="$1" \ - --config=util/fallback_analysis/fallback_count_config.txtpb \ - --noinclude_initial_codepoints_in_config \ - --nooutput_segmentation_analysis \ - --nooutput_segmentation_plan \ - --output_fallback_glyph_count 2> /dev/null | grep num_fallback_glyphs | \ - awk "BEGIN {FS=\",\"; OFS=\";\"} {print \"$FONT_NAME\", \$2, \$3, \$4, \$5}" diff --git a/util/fallback_analysis/count-fallback-glyphs.sh b/util/fallback_analysis/count-fallback-glyphs.sh deleted file mode 100755 index 0cae1b3f..00000000 --- a/util/fallback_analysis/count-fallback-glyphs.sh +++ /dev/null @@ -1,19 +0,0 @@ -#!/bin/bash -# -# Usage: ./util/fallback_analysis/count-fallback-glyphs.sh -# -# will be scanned recursively to locate all font files of the form *.ttf or *.otf. -# For each found font a closure analysis will be performed and the number of fallback glyphs -# and the compressed size in bytes of the fallback glyph patch will be output. -# -# sets the number of parallel executions of the closure analysis. -# -# Fallback glyphs are those that have conditions which could not be detected with the current -# closure analysis and as a result will always be loaded. -TARGET_DIR=$1 -CONCURRENCY=$2 - -bazel build -c opt util:gen_ift_segmentation_plan -echo "file; num_fallback_glyphs; total_glyphs; fallback_glyphs_compressed_bytes; all_glyphs_compressed_bytes" -find $TARGET_DIR \( -iname "*.ttf" -or -iname "*.otf" \) -print0 | xargs -0 -n 1 -P $CONCURRENCY \ - ./util/fallback_analysis/count-fallback-glyphs-for-font.sh diff --git a/util/fallback_analysis/fallback_count_config.txtpb b/util/fallback_analysis/fallback_count_config.txtpb deleted file mode 100644 index 07bb073a..00000000 --- a/util/fallback_analysis/fallback_count_config.txtpb +++ /dev/null @@ -1,19 +0,0 @@ -# This config is used to run the closure segmenter to test how many fallback glyphs result from -# segmenting a font with one codepoint per segment. - -# Don't move fallback glyphs to init font -move_fallback_glyphs_into_initial_font: false - -# Add segments for all non-default features -generate_feature_segments: true - -# Brotli shouldn't be used, but just in case set to lowest quality. -brotli_quality: 1 -brotli_quality_for_initial_font_merging: 1 - -# Ensure there is no merging. -preprocess_merging_group_size_for_ungrouped: 1 -ungrouped_config { - min_patch_size: 0 - max_patch_size: 4294967295 -} diff --git a/util/gen_ift_segmentation_plan.cc b/util/gen_ift_segmentation_plan.cc index 5cc9c022..5b628b17 100644 --- a/util/gen_ift_segmentation_plan.cc +++ b/util/gen_ift_segmentation_plan.cc @@ -39,8 +39,7 @@ using ift::common::BazelDataFileResolver; using ift::common::DataFileResolver; -using ift::config::CLOSURE_ONLY; -using ift::config::PATCH; +using ift::config::DEP_GRAPH_ONLY; using ift::config::SegmentationPlan; using ift::config::SegmenterConfig; using ift::freq::ProbabilityCalculator; @@ -76,10 +75,6 @@ ABSL_FLAG(bool, output_segmentation_analysis, false, ABSL_FLAG(bool, segmentation_analysis_split_cjk, false, "If set, then for analysis split \"Script_CJK\" into the individual scripts."); -ABSL_FLAG( - bool, output_fallback_glyph_count, false, - "If set the number of fallback glyphs in the segmentation will be output."); - ABSL_FLAG( int, verbosity, 0, "Log verbosity level from. 0 is least verbose, higher values are more."); @@ -188,7 +183,7 @@ static Status Analysis(hb_face_t* font, } } - ClosureGlyphSegmenter segmenter(11, 11, PATCH, CLOSURE_ONLY, resolver, + ClosureGlyphSegmenter segmenter(11, 11, DEP_GRAPH_ONLY, resolver, target_mode); auto costs = TRY(segmenter.TotalCosts(font, segmentation, strategy_probability_calculators)); @@ -233,28 +228,6 @@ static Status Analysis(hb_face_t* font, return absl::OkStatus(); } -static Status OutputFallbackGlyphCount(hb_face_t* original_face, - const ClosureGlyphSegmenter& segmenter, - const GlyphSegmentation& segmentation) { - uint32_t num_fallback_glyphs = segmentation.UnmappedGlyphs().size(); - uint32_t fallback_glyphs_size = 0; - uint32_t all_glyphs_size = 0; - TRYV(segmenter.FallbackCost(original_face, segmentation, fallback_glyphs_size, - all_glyphs_size)); - - GlyphSet all_glyphs; - for (const auto& [_, gids] : segmentation.GidSegments()) { - all_glyphs.union_set(gids); - } - - uint32_t num_glyphs = all_glyphs.size() + num_fallback_glyphs; - std::cout << "num_fallback_glyphs, " << num_fallback_glyphs << ", " - << num_glyphs << ", " << fallback_glyphs_size << ", " - << all_glyphs_size << std::endl; - - return absl::OkStatus(); -} - static Status Main(const std::vector args) { auto resolver = TRY(BazelDataFileResolver::Create(args[0])); @@ -296,16 +269,6 @@ static Status Main(const std::vector args) { std::cout << segmentation.ToString() << std::endl; } - if (absl::GetFlag(FLAGS_output_fallback_glyph_count)) { - ClosureGlyphSegmenter segmenter( - config.brotli_quality(), - config.brotli_quality_for_initial_font_merging(), - config.unmapped_glyph_handling(), config.condition_analysis_mode(), - resolver, config.target_mode(), - config.disjoint_conditions_probability_threshold()); - TRYV(OutputFallbackGlyphCount(font.get(), segmenter, segmentation)); - } - if (!absl::GetFlag(FLAGS_output_segmentation_analysis)) { return absl::OkStatus(); } From 648e6afce4a46d8b83c5910c2a841702d6d1706c Mon Sep 17 00:00:00 2001 From: Garret Rieger Date: Fri, 13 Nov 2026 22:37:06 +0000 Subject: [PATCH 2/3] Remove now uneeded glyph_set_unchanged in CandidateMerge. --- ift/encoder/candidate_merge.cc | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/ift/encoder/candidate_merge.cc b/ift/encoder/candidate_merge.cc index e950fabf..6ba214dc 100644 --- a/ift/encoder/candidate_merge.cc +++ b/ift/encoder/candidate_merge.cc @@ -698,7 +698,6 @@ StatusOr CandidateMerge::ComputeCostDelta( const ActivationCondition* first_condition = nullptr; const GlyphSet* first_glyphs = nullptr; uint32_t source_count = 0; - bool glyph_set_unchanged = false; bool skip_new_condition = false; GlyphSet glyphs; uint32_t largest_patch_size = 0; @@ -735,7 +734,7 @@ StatusOr CandidateMerge::ComputeCostDelta( << "]"; } - if (!best_case && !info.glyph_set_unchanged) { + if (!best_case && info.source_count > 1) { info.glyphs.union_set(*glyphs); } @@ -772,11 +771,13 @@ StatusOr CandidateMerge::ComputeCostDelta( continue; } - if (info.source_count == 1) { + info.source_count++; + if (info.source_count == 2) { + // The previously collected first condition needs to be removed now + // that it's merging with this condition. TRYV(add_removed_condition(info.first_condition, info.first_glyphs, info)); } - info.source_count++; TRYV(add_removed_condition(condition, glyphs, info)); } @@ -786,7 +787,6 @@ StatusOr CandidateMerge::ComputeCostDelta( } bool was_active_before_merge = (!unicode_range_mode || info.first_condition->IsExclusive()); - info.glyph_set_unchanged = true; if (best_case || (was_active_before_merge && info.first_condition->IsUnchangedByMerge(merged_segments))) { @@ -814,7 +814,7 @@ StatusOr CandidateMerge::ComputeCostDelta( extra = std::max((uint32_t)(extra * best_case_reduction_fraction), Merger::BEST_CASE_MERGE_SIZE_DELTA); size = info.largest_patch_size + extra; - } else if (info.glyph_set_unchanged) { + } else if (info.source_count == 1) { size = info.largest_patch_size; } else { size = TRY(patch_size_cache->GetPatchSize(info.glyphs)); From fbbfe462ae5ce02596f0400bae1324299bf2fafc Mon Sep 17 00:00:00 2001 From: Garret Rieger Date: Fri, 13 Nov 2026 22:42:40 +0000 Subject: [PATCH 3/3] Remove docs on closure based condition analysis. --- .../closure_glyph_segmentation.md | 418 ------------------ ...e_glyph_segmentation_complex_conditions.md | 157 ------- 2 files changed, 575 deletions(-) delete mode 100644 docs/experimental/closure_glyph_segmentation.md delete mode 100644 docs/experimental/closure_glyph_segmentation_complex_conditions.md diff --git a/docs/experimental/closure_glyph_segmentation.md b/docs/experimental/closure_glyph_segmentation.md deleted file mode 100644 index 1f8ec297..00000000 --- a/docs/experimental/closure_glyph_segmentation.md +++ /dev/null @@ -1,418 +0,0 @@ -# Converting Unicode Code Point to Glyph Keyed Segmentations during IFT Encoding using Subsetter Glyph Closure - -Author: Garret Rieger -Date: Jan 27, 2025 -Updated: Dec 17, 2025 - -## Introduction - -*Note: the approach in this document is obsolete and has been replaced by - [dependency graph condition analysis](../dependency_graph_condition_extract.md).* - -A key part of encoding an [IFT](https://w3c.github.io/IFT/Overview.html) font is splitting the -outline (eg. glyf, gvar) data into a set of patches which are loaded by the client as needed. Each -patch contains the data associated with one or more glyph ids. Patch loads are triggered by a client -based on which code points and layout features are being rendered. - -Fonts can contain complex substitution rules that can swap the glyphs in use depending on the -specifics of the text. Thus when generating a segmentation for glyphs across patches it must be done -in a way that ensures the correct glyphs for the client's text are available. - -In an IFT font each patch has an associated activation condition which is a boolean expression using -conjunction and disjunction on the presence of unicode code points and layout features. -(eg. $(a ∧ b ∧ c) ∨ (d ∧ e ∧ f)$ ). - -To illustrate the problem here's a common example: let's say we have a Latin font which contains a -ligature which replaces the individual glyphs for f and i with the fi ligature glyph. We decide -to place the f glyph in one patch and the i glyph in another. When the client is rendering text with -both an f and i character it will load both patches, however this makes it possible for the fi glyph -to be used. Thus we need to make the fi glyph available in this case. One way to do that is to -form a third patch containing the ligature glyph and assign the activation condition (f and i). - -Because the IFT font will use unicode code points in the activation conditions, it will be typical -to express a desired segmentation of the original font using unicode code points. The remainder of -this document describes a possible procedure for converting a desired unicode code point -segmentation into a set of glyph patches with load conditions described in terms of those provided -unicode segments. - -Notably, this document does not aim to describe a solution to producing the unicode code point -segmentation which is of high importance to the production of performant overall glyph -segmentations. - -The code that implements the procedures in this document can be found in -[closure_glyph_segmenter.cc](../../ift/encoder/closure_glyph_segmenter.cc) - -There is also a command line utility to generate segmentations: -[util/gen_ift_segmentation_plan.cc](../../util/gen_ift_segmentation_plan.cc) - -At the end of this document you can find a couple of [examples](#examples) which illustrate how the -procedures work in practice. - -## Status - -The current prototype implementation in -[closure_glyph_segmenter.cc](../../ift/encoder/closure_glyph_segmenter.cc) can produce segmentations -that satisfy the closure requirement and are performant (via merging). The approach laid -out in this document is just one possible approach to solving the problem. This document aims -primarily to describe how the prototype implementation in -[closure_glyph_segmenter.cc](../../ift/encoder/closure_glyph_segmenter.cc) functions, and is not -intended to present the final (or only) solution to the problem. There are several unsolved problems -and remaining areas for development in this particular approach: - -* Much of the ongoing work is on the "merger" which is a sub-problem of producing segmentations. - That's discussed in the separate - [closure_glyph_segmentation_merging.md](closure_glyph_segmentation_merging.md) document. - See the implementation status and areas for further development sections for more specifics. - -* Running the segmenter currently requires manual configuration to get good results. Configuration - is needed to select appropriate frequency data and settings for parameters controlling merger - behaviour. The goal is to get to the point where good results can be produced with zero - configuration. - -* Support for merging segmentations involving multiple overlapping scripts is not yet implemented - (for example creating a segmentation that supports Chinese and Japanese simultaneously). - -* [Multi segment analysis](#multi-segment-dependencies): the current implementation utilizes an approach which - approximates multi segment analysis by finding superset disjunctive conditions for multi segment - conditions. See: - [closure_glyph_segmentation_complex_conditions.md](./closure_glyph_segmentation_complex_conditions.md). - -* Input segmentation generation: the glyph segmentation process starts with an existing - codepoint/feature based segmentation. Good results can be achieved by starting with one input - segment per codepoint/feature and letting merging join segments as needed. However, there is still - value in starting with a good quality input segmentation that places commonly used codepoints - together. This can significantly reduce the amount of work the merger needs to do. Therefore it - may be useful to develop functionality that creates a first pass input segmentation based on - codepoint frequency data. - -* Incorporating dependency information: whatever produces the input code point segments will likely - have discovered dependency information related to those code points. That information can be - reused in this process to narrow selections during patch merging and multi segment - analysis. Future work will look at adding dependency information as an optional input to this - procedure. - -* One of the main down sides to this approach is it's reliance on a subsetting closure function which - are computationally costly. Complex fonts which can require hundreds of closure operation which as a - result can be slow to process. So another area of open research is if a non closure based approach - could be developed that is computationally cheaper (for example by producing a segmentation by - working directly with the substitution and dependencies encoded in a font). - -## Goals - -The segmentation procedure described in this document aims to achieve the following goals: - -* Given a desired segmentation of the font expressed in terms of Unicode code points find a - corresponding grouping of glyphs to be made into a set of patches, where each patch adds the data - for that set of glyphs. - -* Determine the activation conditions for each of those patches. An activation condition is a boolean - expression using conjunction and disjunction with the presence of Unicode code points being the boolean - values. The input unicode code point segmentations are used to form the conditions. - -* Optimize for minimal data transfer by avoiding duplicating glyphs across patches where possible. - -* Support optimization of a generated segmentation via merging to reduce network overhead. - -* The chosen glyph segmentation and activation conditions must satisfy the closure requirement: - - The set of glyphs contained in patches loaded for a font subset definition (a set of Unicode - code points and a set of layout feature tags) through the patch map tables must be a superset of - those in the glyph closure of the font subset definition. - - -## Subsetter Glyph Closures - -Font subsetters run into a similar problem faced here and solve it with an operation called the glyph -closure. This takes a subset definition (set of code points and set of feature tags) and calculates -all glyphs that code be reached by any text using any combination of the input code points. - -Closure is a significantly complex process so to save ourselves significant effort we can re-use -existing subsetter closure implementations to help form segmentations in this process. - -For the purposes of this document we define a function: - -``` -closure(font, subset_definition) -> {glyph id set} -``` - -Where subset_definition is a set of unicode code points and layout feature tags. Subset definitions may -also alternately be referred to as input segments. - -## Segment Closure Analysis - -This section defines a procedure to analyze an input segment and determine how it interacts with -glyphs in the font. It is an extension of the chunking algorithm in Skef’s [binned IFT -prototype](https://github.com/adobe/binned-ift-reference). Notably it incorporates new capabilities -of the current IFT spec such as the ability to have conjunction and disjunction in activation -conditions for patches. - -Given: - -* An input font, $F$. -* A list of segments $s_0, …, s_n$. Where: - * $s_0$ is the subset definition for the initial IFT font. - * $s_i \neq s_0$ is the segment to be analyzed. - -It generates three sets of glyph ids: - -1. Exclusive Glyph Set: these glyphs are needed exclusively by this segment. That is they are only - needed if and only if the $s_i$ is present. - -2. Conjunctive Glyph Set: the presence of this input segment is a requirement for these glyphs to be - needed, but there may be additional segments required as well. The condition for the - set of glyphs to be needed is: ($s_i ∧ …$). - -3. Disjunctive Glyph Set: these glyphs are needed when the input segment is present, but the input - segment is not a requirement for the glyphs to be needed. The condition for the - set of glyphs to be needed is: ($s_i ∨ …$) - -The process utilizes a subsetter glyph closure computation to group glyphs together based on how -they are activated by the closure. - -First compute the following sets: - -* Set $A = \text{closure} (F, \bigcup_{0}^{n} s_j)$ -* Set $B = \text{closure} (F, \bigcup_{j \neq i} s_j)$ -* Set $I = \text{closure} (F, s_0 \cup s_i) - \text{closure} (F, s_0)$, the set of glyphs needed for - the closure of $s_i$. -* Set $D = A - B$, the set of glyphs that are dropped when $s_i$ is removed. - -Then we know the following: - -* Glyphs in $I$ should be included whenever $s_i$ is activated. -* $s_i$ is necessary for glyphs in $D$ to be required, but other segments may be required aswell. - -Furthermore we can intersect $I$ and $D$ to produce three sets: - -1. $D - I$: the activation condition for these glyphs is ($s_i ∧ …)$ where … is one or more additional segments. -2. $I - D$: the activation conditions for these glyphs is $(s_i ∨ …)$ where … is one or more additional segments. -3. $D \cap I$: the activation conditions for these glyphs is only $s_i$. - -## Segmenting Glyphs Based on Closure Analysis - -For each glyph in the input font we track the following information: - -* The glyph is either exclusive to a segment if it appears in the Exclusive Glyph Set of a segment, or; -* The glyph has an associated set of segments which can be either conjunctive or disjunctive. - -If the set is conjunctive, then this glyph is needed when all segments in the set are present, and -possibly some additional undiscovered conditions. However, we can ignore these additional conditions -since they are conjunctive and will only further narrow the activation condition. Otherwise if the -set is disjunctive, then the glyph is needed when at least one segment in the set is present or -possibly due to some additional undiscovered conditions. These potential undiscovered conditions are -important, and later on the process will attempt to rule them out. - -Run the Segment Closure Analysis on each segment in $s_1, …, s_n$. For each glyph in the: -* Exclusive Glyph Set of $s_i$ - mark that glyph as exclusive to $s_i$. -* Conjunctive Glyph Set of $s_i$ - add $s_i$ to that glyph's conjunctive set of segments. -* Disjunctive Glyph Set of $s_i$ - add $s_i$ to that glyph's conjunctive set of segments. - -After performing this analysis for all segments there may be some glyphs that were not associated -with any segments, these have more complicated activation conditions and may need further analysis -which is discussed later on. - -Next, we can use the per glyph information to form the glyph groupings: - -1. For each unique exclusive segment, $s_i$, collect the associated set of glyphs into a group. This forms - an exclusive patch whose activation condition is $s_i$. - -2. For each unique conjunctive segment set, $C$, collect the group of glyphs that is marked with that set. This - group forms a conditional patch whose activation condition is $\bigwedge s_j \in C$. - -3. For each unique disjunctive segment set, $C$, collect the group of glyphs that is marked with that set. This - group forms a conditional patch whose activation condition is $(\bigvee s_j \in C) \vee \text{additional conditions}$. As - noted above we will need to rule out the additional conditions, the process for this follows. - - Compute the Segment Closure Analysis for a new composite segment $\bigcup s_i \notin C$. Remove any glyphs from the - group that are found in $I - D$. These glyphs may appear as a result of additional conditions and so - need to be considered unmapped. If the group has no remaining glyphs don’t make a patch for it. - -4. Lastly, collect the set of glyphs that are not part of any patch formed in steps 1 through 3. These form a fallback patch - whose activation condition is $\bigvee_{1}^{n} s_j$. - -## Fallback Patch - -An output of the segmentation process above will include a fallback patch which is activated on the -presence of any input segment. In practice it will be slightly more efficient to instead move all of -the glyphs in the fallback patch into the initial font. This has the same overall effect as using a -fallback patch. In some cases the encoder may be purposely trying to minimize and/or eliminate glyph -data in the initial font. In these cases leaving them in the fallback patch may make the most sense. - -## Merging - -When starting with an input segmentation that is fine grained (for example using one segment per -code point) the resulting glyph segmentation may involve a large number of patches. This results in -excessive network overhead when loading the patches. Performance can be increased by selectively merging -patches together to reduce overhead. This is a complex problem as it needs to be done in a way that -avoids excessive transfers of glyph data that isn't needed. - -The segmenter currently implements a cost based merging algorithm which selects merges that minimize -an overall cost function. This process is documented in detail in -[closure_glyph_segmentation_merging.md](closure_glyph_segmentation_merging.md). - -The best segmentation results so far have been obtained by starting with one input segment per -code point in the font and then letting the merger figure out how to best place them together in a way -that minimizes overall cost. - -## Multi Segment Dependencies - -Note: this section is somewhat speculative as this functionality has not yet been fully implemented. -More research and exploration is definitely needed. - -The Segmenting Glyphs Based on Closure Analysis procedure places any glyphs whose conditions aren't -discovered into a fallback patch. Glyphs will end up in the fallback patch if there exists more -complex activation conditions that involve more than one segment at a time. For example consider a -glyph which is added to the closure when code points from ($s_1 \wedge s_2) \vee (s_3 \wedge s_4$) -are present. This glyph would not show up in either sets $I$ or $D$ for any single segment since -that would require at the union of at least two segments (for example $s_1$ and $s_2$) to be closure -tested. If the fallback patch is large than it may be necessary to analyze the dependencies further -to try and determine what the more complex activation conditions are. - -To discover these multi segment dependencies it’s necessary to test all combinations of two or more -segments with the closure analysis. Each combination of segments is unioned together to produce a -new segment and then the closure test is performed and new groups and activation conditions are -formed as described above (these are added to the previously determined patches). - -Unfortunately this approach will likely lead to a combinatorial explosion, so mitigations will be -needed to reduce the amount of combinations to test. Some suggestions: - -* It's not necessary to test all combinations. Once a sufficient number of fallback glyphs have been - categorized the combination testing can stop at any point. With remaining glyphs left in the - fallback patch. Given this, the remaining points provide suggestions for better targeting the - combinations to test in order to focus on combinations that are likely interacting. - -* In most cases complex dependencies should be isolated within a script, and for most scripts (other - than CJK) there should be a reasonably small number of segments per script. So if we limit the - multi segment combinations to within a script it should produce a reasonable number of - combinations to test while still being able to capture most dependencies. - -* Analysis of the font’s underlying GSUB rules could yield groups of code points which may possibly - interact. These could be used to prune segment combinations that don’t share interacting - code points. Alternatively these groups could be used to guide initial segment selection where - code points that appear to interact are placed in the same segment, thus getting a higher hit rate - in the single segment analysis. - -* The performance of a segmentation is likely driven solely by the high frequency code points. So - divide the font into a high frequency set and low frequency set of code points. Where a more - extensive multi segment dependency check is done for only the high frequency segments. - -As an alternative a simpler approach to the problem is to limit the scope to just finding conditions which are a superset -of the true condition. This superset condition can be used in place of the true condition without violating the closure -requirement. This is the approach currently used in the segmenter implementation. This procedure is discussed in more -details in [closure_glyph_segmentation_complex_conditions.md](./closure_glyph_segmentation_complex_conditions.md). The -advantage to this approach is it's much less computationally costly then multi segment analysis. The downside is these -superset conditions will activate more frequently then the true conditions and thus may be loaded in cases where they -are not actually needed. - -## Examples - -### UVS Selectors - -This example demonstrates how the proposed algorithm can detect the appropriate segmentations and activation conditions for a font that has closure glyphs which activate when two segments are simultaneously present. - -The original font contains: - -* Codepoints `{a, …, z}` and variation selector `VS1`. -* Each of a through z has an alternate form when combined with `VS1`. eg. `a + VS1` subs in glyph `a.alt` -* Four input segments are defined: - * `s_0 = {A} # initial font` - * `s_1 = {a, …, m}` - * `s_2 = {n, …, z}` - * `s_3 = {VS1}` - -Performing the per segment closure test will have the following results: - -| `S` | `A` | `B` | `I` | `D = A - B` | -| :---- | :---- | :---- | :---- | :---- | -| `s1` | `{A, a..z, a.alt..z.alt}` | `{A, n..z, n.alt..z.alt}` | `{a..m}` | `{a..m, a.alt..m.alt}` | -| `s2` | `{A, a..z, a.alt..z.alt}` | `{A, a..m, a.alt..m.alt}` | `{n..z}` | `{n..z, n.alt..z.alt}` | -| `s3` | `{A, a..z, a.alt..z.alt}` | `{A, a..z}` | `{}` | `{a.alt..z.alt}` | - -And the derived sets: - -| `S` | `D - I` | `I - D` | `D n I` | -| :---- | :---- | :---- | :---- | -| `s1` | `{a.alt..m.alt}` | `{}` | `{a..m}` | -| `s2` | `{n.alt..z.alt}` | `{}` | `{n..z}` | -| `s3` | `{a.alt..z.alt}` | `{}` | `{}` | - -Based on these we then we assign conditions to each glyph: - -| `glyph` | `Exclusive` | `Conjunctive Set` | `DisjunctiveR Set` | -| :---- | :---- | :---- | :---- | -| `a` | `{s1}` | `{}` | `{}` | -| `…` | `…` | `…` | `…` | -| `m` | `{s1}` | `{}` | `{}` | -| `n` | `{s2}` | `{}` | `{}` | -| `…` | `…` | `…` | `…` | -| `z` | `{s2}` | `{}` | `{}` | -| `a.alt` | `{}` | `{s1, s3}` | `{}` | -| `…` | `…` | `…` | `…` | -| `m.alt` | `{}` | `{s1, s3}` | `{}` | -| `n.alt` | `{}` | `{s2, s3}` | `{}` | -| `…` | `…` | `…` | `…` | -| `z.alt` | `{}` | `{s2, s3}` | `…` | - -Lastly, we group glyphs by condition sets to produce the final mappings and glyph keyed patches: - -| `Segment Set` | `Mapping Key (codepoints)` | `Patch Contains Glyphs` | -| :---- | :---- | :---- | -| `{s1}` | `{a..m}` | `{a..m}` | -| `{s2}` | `{n..z}` | `{a..z}` | -| `{s1, s3}` | `{a..m} AND {VS1}` | `{a.alt..m.alt}` | -| `{s2, s3}` | `{n..z} AND {VS1}` | `{n.alt..z.alt}` | - -### Shared Component - -This example demonstrates how the proposed algorithm can detect the appropriate segmentations and -activation conditions for a font that has closure glyphs which activate when at least one of several -segments are present. - -The original font contains: - -* Codepoints `{a, o, 0xe2, 0xf4}`. -* Simple glyphs 1, 2, and 5. -* Glyph 3 is a composite glyph using 1 + 5. -* Glyph 4 is a composite glyph using 2 + 5. -* With cmap: - * a: 1 - * o: 2 - * 0xe2: 3 - * 0xf4: 4 -* Two input segments are defined: - * `s0 = {} # initial font` - * `s1 = {a, 0xe2}` - * `s2 = {o, 0xf4}` - -Performing the per segment closure test will have the following results: - -| `S` | `A` | `B` | `I` | `D = A - B` | -| :---- | :---- | :---- | :---- | :---- | -| `s1` | `{1, 2, 3, 4, 5}` | `{2, 4, 5}` | `{1, 3, 5}` | `{1, 3}` | -| `s2` | `{1, 2, 3, 4, 5}` | `{1, 3, 5}` | `{2, 4, 5}` | `{2, 4}` | - -And the derived sets: - -| `S` | `D - I` | `I - D` | `D n I` | -| :---- | :---- | :---- | :---- | -| `s1` | `{}` | `{5}` | `{1, 3}` | -| `s2` | `{}` | `{5}` | `{2, 4}` | - -Based on these we then we assign conditions to each glyph: - -| `glyph` | `Exclusive Set` | `Conjunctive Set` | `Disjunctive Set` | -| :---- | :---- | :---- | :---- | -| `1` | `{s1}` | `{}` | `{}` | -| `2` | `{s2}` | `{}` | `{}` | -| `3` | `{s1}` | `{}` | `{}` | -| `4` | `{s2}` | `{}` | `{}` | -| `5` | `{}` | `{}` | `{s1, s2}` | - -Lastly, we group glyphs by unique set to produce the final mappings and glyph keyed patches: - -| `Exclusive Segment Set` | `Mapping Key (codepoints)` | `Patch Contains Glyphs` | -| :---- | :---- | :---- | -| `{s1}` | `{a, 0xe2}` | `{1, 3}` | -| `{s2}` | `{o, 0xf4}` | `{2, 4}` | -| `Disjunctive Set` | `Mapping Key (codepoints)` | `Patch Contains Glyphs` | -| `{s1, s2}` | `{a, o, 0xe2, 0xf4}` | `{5}` | diff --git a/docs/experimental/closure_glyph_segmentation_complex_conditions.md b/docs/experimental/closure_glyph_segmentation_complex_conditions.md deleted file mode 100644 index 868916ec..00000000 --- a/docs/experimental/closure_glyph_segmentation_complex_conditions.md +++ /dev/null @@ -1,157 +0,0 @@ - -# Complex Condition Finding in Closure Glyph Keyed Segmenter - -Author: Garret Rieger__ -Date: Dec 17, 2025 - -## Introduction - -*Note: the approach in this document is obsolete and has been replaced by - [dependency graph condition analysis](../dependency_graph_condition_extract.md).* - -Before reading this document is recommended to first review the -[closure glyph segmentation](./closure_glyph_segmentation.md) document. This document borrows concepts and terms from it. - -In closure glyph segmentation the closure analysis step is capable of locating glyph activation conditions that are -either fully disjunctive or fully conjunctive (eg. `(A or B or C)`). It is not capable of finding conditions that are a mix -of conjunction and disjunction (eg. `(A and B) or (B and C)`). These are referred to as complex conditions. By default -glyphs with complex conditions are assigned to a patch that is always loaded, since the true conditions are not known. - -This document describes an algorithm which can be used to find purely disjunctive conditions which are supersets of -complex conditions. A superset condition is one that will activate at least whenever the true condition would. This -property allows the superset condition to be used for a patch in place of the true condition without violating the -closure requirement. - -For a given complex condition there typically exists more than one possible superset disjunctive condition. The -algorithm will find one of them, but not necessarily the smallest one. The found superset condition will always only -contain only segments which appear in the original condition. - -For example if we had a glyph with a activation condition of `((A and B) or (B and C))` then this process will find one -of the possible superset conditions such as `(A or C)`, `(A or B)`, `(B or C)`, or `(B)`. In a segmentation we could -then have a patch with the found condition which loads the glyph and this would satisfy the closure requirement. - -## Foundations - -The algorithm is based on the following assertions: - -1. Given some fully disjunctive condition for a glyph, we can verify that the condition is a superset of the true - condition for the glyph and meets the closure requirement by the following procedure: compute a glyph closure of the - union of all segments except for those in the condition. If the glyph does not appear in this closure, then the - condition satisfies the closure requirement for that glyph and is a superset of the true condition. This is called - the “additional conditions” check. - -2. The glyph closure of all segments includes the glyph that we are analyzing. - -3. We have a glyph which has some true activation condition. If we compute a glyph closure of some combination of - segments, then adding or removing a segment, which is not part of the activation condition, to the glyph closure input - will have no affect on whether or not the glyph appears in the closure output. - -4. The closure of no segments contains only glyphs from the initial font. - -## The Algorithm - -For a glyph with a complex condition we can use the above to find a superset disjunctive condition for that -glyph's complex condition. These conditions will satisfy the closure requirement for each glyph. - -### Finding a Sub Condition - -The algorithm works by identifying a single sub condition at a time, this section describes the algorithm for -finding a single sub condition. - -Inputs: - -* Segments to exclude from the analysis. -* `glyph` to analyze. - -Algorithm: - -1. Start with a set of all segments except those to be excluded, called `to_test`. -2. Initialize a second set of segments, `sub_condition`, to the empty set. -3. Remove a segment `s` from `to_test` and compute the glyph closure of `to_test U sub_condition`. -4. If `glyph` is not found in the closure then add `s` to `sub_condition`. -5. If `to_test` is empty, then return the sub condition `sub_condition`. -6. Otherwise, go back to step 3. - -### Finding the Complete Condition - -This section describes the algorithm which finds the complete condition, it utilizes `Finding a Sub Condition`. - -Inputs: - -* `glyph` to analyze. - -Algorithm: - -1. Initialize a set of segments `condition` to the empty set. -2. Execute the `Finding a Sub Condition` algorithm with `condition` as the excluded set. -3. Union the returned set into `condition`. -4. Compute the glyph closure of all segments except those in `condition`. -5. If `glyph` is found in the closure, then more sub conditions still exist. Go back to step 2. -6. Return the complete condition, `condition`. - -### Initial Font - -Any time a closure operation is executed by the above two algorithms it's necessary to union the subset definition for -the initial font into the closure input. This is required because the closure of the initial font affects what's -reachable by the segments. - -### Why this works - -Here we show this procedure is guaranteed to find a disjunctive superset of a glyph's true condition which includes -only segments from the true condition, when the glyph is not already in the initial font: - -* For each call to `Finding a Sub Condition` glyph will be in the closure of all non-excluded segments. For the first - call this is guaranteed by assertion (2). For subsequent calls this is guaranteed by the "additional conditions" check - which gates execution. - -* Any segments which are not part of the true condition will not impact the glyph's presence in the closure (assertion - (3)). Further by the previous point we know that at the start of `Finding a Sub Condition` the closure of all - non-excluded segments will contain glyph. Thus testing a segment which is not part of the true condition will never - result in glyph missing from the closure, and won't be added to `sub_condition`. Therefore `Finding a Sub Condition` - will only ever return segments that are part of the true condition. - -* `Finding a Sub Condition` will always return at least one segment: if when the last segment is tested `sub_condition` - is still the empty set, then the closure will be on no segments and will not have glyph in it. This is a result of - assertion (4) and the premise that glyph is not already in the initial font. As a consequence the returned - `sub_condition` will always have at least one segment in it. - -* Since all returned segments from `Finding a Sub Condition` are excluded from future calls, there will be a finite - number of `Finding a Sub Condition` executions which return only segments part of the true condition. - -* Lastly, the algorithm terminates only once the additional conditions check finds no additional conditions, - guaranteeing we have found a superset disjunctive condition (assertion (1)). - -## Making it More Performant - -As described above this approach can be slow since it processes glyphs one at a time. Improvements to performance -can be made by processing glyphs in a batch. This can be done with a recursive approach where after each segment test -the set of input glyphs gets split into those that require the tested segment and those that don't. Each of the splits spawns -a new recursion (if there is at least one glyph in the split). - -Also from the closure analysis run by the segmenter we may have discovered some partial conditions for glyphs. These -can be incorporated as a starting point into the complex condition analysis. - -More substantial performance improvements could be realized by using a dependency graph generated from the font. This -could come in two forms: - -1. Determine what segments interact with GSUB in some way and use that to scope the analysis. Segments that don't - interact with GSUB can be discovered via regular closure analysis as they will only ever have disjunctive conditions. - After completing a scoped analysis, a final additional conditions check against all segments could be used to ensure - we have actually arrived at a superset condition. This is more speculative and would need more research to validate the - approach. - -2. Using a actual dependency graph, even if it's no fully accurate, to generate a set of initial activation - conditions. Then as needed when additional conditions check fails, we could find any additional segments for the - superset condition using this process. - -## Integrating into the Segmentation Algorithm - -The complex condition analysis has been added as a post processing step during the glyph grouping phase. After the -grouping phase finds the initial condition set, complex condition finding is run to find conditions for any remaining -unmapped glyphs. The found complex conditions are added to the condition set and thus are able to participate -in both initial font analysis and patch merging. To make this performant complex conditions are only recomputed for -glyphs/segments that are modified after a merge is performed. The glyph grouping phase has an existing invalidation -mechanism which is utilized to determine which glyphs need to be re-analyzed. - - -