diff --git a/docs/experimental/closure_glyph_segmentation.md b/docs/experimental/closure_glyph_segmentation.md deleted file mode 100644 index 1f8ec297..00000000 --- a/docs/experimental/closure_glyph_segmentation.md +++ /dev/null @@ -1,418 +0,0 @@ -# Converting Unicode Code Point to Glyph Keyed Segmentations during IFT Encoding using Subsetter Glyph Closure - -Author: Garret Rieger -Date: Jan 27, 2025 -Updated: Dec 17, 2025 - -## Introduction - -*Note: the approach in this document is obsolete and has been replaced by - [dependency graph condition analysis](../dependency_graph_condition_extract.md).* - -A key part of encoding an [IFT](https://w3c.github.io/IFT/Overview.html) font is splitting the -outline (eg. glyf, gvar) data into a set of patches which are loaded by the client as needed. Each -patch contains the data associated with one or more glyph ids. Patch loads are triggered by a client -based on which code points and layout features are being rendered. - -Fonts can contain complex substitution rules that can swap the glyphs in use depending on the -specifics of the text. Thus when generating a segmentation for glyphs across patches it must be done -in a way that ensures the correct glyphs for the client's text are available. - -In an IFT font each patch has an associated activation condition which is a boolean expression using -conjunction and disjunction on the presence of unicode code points and layout features. -(eg. $(a ∧ b ∧ c) ∨ (d ∧ e ∧ f)$ ). - -To illustrate the problem here's a common example: let's say we have a Latin font which contains a -ligature which replaces the individual glyphs for f and i with the fi ligature glyph. We decide -to place the f glyph in one patch and the i glyph in another. When the client is rendering text with -both an f and i character it will load both patches, however this makes it possible for the fi glyph -to be used. Thus we need to make the fi glyph available in this case. One way to do that is to -form a third patch containing the ligature glyph and assign the activation condition (f and i). - -Because the IFT font will use unicode code points in the activation conditions, it will be typical -to express a desired segmentation of the original font using unicode code points. The remainder of -this document describes a possible procedure for converting a desired unicode code point -segmentation into a set of glyph patches with load conditions described in terms of those provided -unicode segments. - -Notably, this document does not aim to describe a solution to producing the unicode code point -segmentation which is of high importance to the production of performant overall glyph -segmentations. - -The code that implements the procedures in this document can be found in -[closure_glyph_segmenter.cc](../../ift/encoder/closure_glyph_segmenter.cc) - -There is also a command line utility to generate segmentations: -[util/gen_ift_segmentation_plan.cc](../../util/gen_ift_segmentation_plan.cc) - -At the end of this document you can find a couple of [examples](#examples) which illustrate how the -procedures work in practice. - -## Status - -The current prototype implementation in -[closure_glyph_segmenter.cc](../../ift/encoder/closure_glyph_segmenter.cc) can produce segmentations -that satisfy the closure requirement and are performant (via merging). The approach laid -out in this document is just one possible approach to solving the problem. This document aims -primarily to describe how the prototype implementation in -[closure_glyph_segmenter.cc](../../ift/encoder/closure_glyph_segmenter.cc) functions, and is not -intended to present the final (or only) solution to the problem. There are several unsolved problems -and remaining areas for development in this particular approach: - -* Much of the ongoing work is on the "merger" which is a sub-problem of producing segmentations. - That's discussed in the separate - [closure_glyph_segmentation_merging.md](closure_glyph_segmentation_merging.md) document. - See the implementation status and areas for further development sections for more specifics. - -* Running the segmenter currently requires manual configuration to get good results. Configuration - is needed to select appropriate frequency data and settings for parameters controlling merger - behaviour. The goal is to get to the point where good results can be produced with zero - configuration. - -* Support for merging segmentations involving multiple overlapping scripts is not yet implemented - (for example creating a segmentation that supports Chinese and Japanese simultaneously). - -* [Multi segment analysis](#multi-segment-dependencies): the current implementation utilizes an approach which - approximates multi segment analysis by finding superset disjunctive conditions for multi segment - conditions. See: - [closure_glyph_segmentation_complex_conditions.md](./closure_glyph_segmentation_complex_conditions.md). - -* Input segmentation generation: the glyph segmentation process starts with an existing - codepoint/feature based segmentation. Good results can be achieved by starting with one input - segment per codepoint/feature and letting merging join segments as needed. However, there is still - value in starting with a good quality input segmentation that places commonly used codepoints - together. This can significantly reduce the amount of work the merger needs to do. Therefore it - may be useful to develop functionality that creates a first pass input segmentation based on - codepoint frequency data. - -* Incorporating dependency information: whatever produces the input code point segments will likely - have discovered dependency information related to those code points. That information can be - reused in this process to narrow selections during patch merging and multi segment - analysis. Future work will look at adding dependency information as an optional input to this - procedure. - -* One of the main down sides to this approach is it's reliance on a subsetting closure function which - are computationally costly. Complex fonts which can require hundreds of closure operation which as a - result can be slow to process. So another area of open research is if a non closure based approach - could be developed that is computationally cheaper (for example by producing a segmentation by - working directly with the substitution and dependencies encoded in a font). - -## Goals - -The segmentation procedure described in this document aims to achieve the following goals: - -* Given a desired segmentation of the font expressed in terms of Unicode code points find a - corresponding grouping of glyphs to be made into a set of patches, where each patch adds the data - for that set of glyphs. - -* Determine the activation conditions for each of those patches. An activation condition is a boolean - expression using conjunction and disjunction with the presence of Unicode code points being the boolean - values. The input unicode code point segmentations are used to form the conditions. - -* Optimize for minimal data transfer by avoiding duplicating glyphs across patches where possible. - -* Support optimization of a generated segmentation via merging to reduce network overhead. - -* The chosen glyph segmentation and activation conditions must satisfy the closure requirement: - - The set of glyphs contained in patches loaded for a font subset definition (a set of Unicode - code points and a set of layout feature tags) through the patch map tables must be a superset of - those in the glyph closure of the font subset definition. - - -## Subsetter Glyph Closures - -Font subsetters run into a similar problem faced here and solve it with an operation called the glyph -closure. This takes a subset definition (set of code points and set of feature tags) and calculates -all glyphs that code be reached by any text using any combination of the input code points. - -Closure is a significantly complex process so to save ourselves significant effort we can re-use -existing subsetter closure implementations to help form segmentations in this process. - -For the purposes of this document we define a function: - -``` -closure(font, subset_definition) -> {glyph id set} -``` - -Where subset_definition is a set of unicode code points and layout feature tags. Subset definitions may -also alternately be referred to as input segments. - -## Segment Closure Analysis - -This section defines a procedure to analyze an input segment and determine how it interacts with -glyphs in the font. It is an extension of the chunking algorithm in Skef’s [binned IFT -prototype](https://github.com/adobe/binned-ift-reference). Notably it incorporates new capabilities -of the current IFT spec such as the ability to have conjunction and disjunction in activation -conditions for patches. - -Given: - -* An input font, $F$. -* A list of segments $s_0, …, s_n$. Where: - * $s_0$ is the subset definition for the initial IFT font. - * $s_i \neq s_0$ is the segment to be analyzed. - -It generates three sets of glyph ids: - -1. Exclusive Glyph Set: these glyphs are needed exclusively by this segment. That is they are only - needed if and only if the $s_i$ is present. - -2. Conjunctive Glyph Set: the presence of this input segment is a requirement for these glyphs to be - needed, but there may be additional segments required as well. The condition for the - set of glyphs to be needed is: ($s_i ∧ …$). - -3. Disjunctive Glyph Set: these glyphs are needed when the input segment is present, but the input - segment is not a requirement for the glyphs to be needed. The condition for the - set of glyphs to be needed is: ($s_i ∨ …$) - -The process utilizes a subsetter glyph closure computation to group glyphs together based on how -they are activated by the closure. - -First compute the following sets: - -* Set $A = \text{closure} (F, \bigcup_{0}^{n} s_j)$ -* Set $B = \text{closure} (F, \bigcup_{j \neq i} s_j)$ -* Set $I = \text{closure} (F, s_0 \cup s_i) - \text{closure} (F, s_0)$, the set of glyphs needed for - the closure of $s_i$. -* Set $D = A - B$, the set of glyphs that are dropped when $s_i$ is removed. - -Then we know the following: - -* Glyphs in $I$ should be included whenever $s_i$ is activated. -* $s_i$ is necessary for glyphs in $D$ to be required, but other segments may be required aswell. - -Furthermore we can intersect $I$ and $D$ to produce three sets: - -1. $D - I$: the activation condition for these glyphs is ($s_i ∧ …)$ where … is one or more additional segments. -2. $I - D$: the activation conditions for these glyphs is $(s_i ∨ …)$ where … is one or more additional segments. -3. $D \cap I$: the activation conditions for these glyphs is only $s_i$. - -## Segmenting Glyphs Based on Closure Analysis - -For each glyph in the input font we track the following information: - -* The glyph is either exclusive to a segment if it appears in the Exclusive Glyph Set of a segment, or; -* The glyph has an associated set of segments which can be either conjunctive or disjunctive. - -If the set is conjunctive, then this glyph is needed when all segments in the set are present, and -possibly some additional undiscovered conditions. However, we can ignore these additional conditions -since they are conjunctive and will only further narrow the activation condition. Otherwise if the -set is disjunctive, then the glyph is needed when at least one segment in the set is present or -possibly due to some additional undiscovered conditions. These potential undiscovered conditions are -important, and later on the process will attempt to rule them out. - -Run the Segment Closure Analysis on each segment in $s_1, …, s_n$. For each glyph in the: -* Exclusive Glyph Set of $s_i$ - mark that glyph as exclusive to $s_i$. -* Conjunctive Glyph Set of $s_i$ - add $s_i$ to that glyph's conjunctive set of segments. -* Disjunctive Glyph Set of $s_i$ - add $s_i$ to that glyph's conjunctive set of segments. - -After performing this analysis for all segments there may be some glyphs that were not associated -with any segments, these have more complicated activation conditions and may need further analysis -which is discussed later on. - -Next, we can use the per glyph information to form the glyph groupings: - -1. For each unique exclusive segment, $s_i$, collect the associated set of glyphs into a group. This forms - an exclusive patch whose activation condition is $s_i$. - -2. For each unique conjunctive segment set, $C$, collect the group of glyphs that is marked with that set. This - group forms a conditional patch whose activation condition is $\bigwedge s_j \in C$. - -3. For each unique disjunctive segment set, $C$, collect the group of glyphs that is marked with that set. This - group forms a conditional patch whose activation condition is $(\bigvee s_j \in C) \vee \text{additional conditions}$. As - noted above we will need to rule out the additional conditions, the process for this follows. - - Compute the Segment Closure Analysis for a new composite segment $\bigcup s_i \notin C$. Remove any glyphs from the - group that are found in $I - D$. These glyphs may appear as a result of additional conditions and so - need to be considered unmapped. If the group has no remaining glyphs don’t make a patch for it. - -4. Lastly, collect the set of glyphs that are not part of any patch formed in steps 1 through 3. These form a fallback patch - whose activation condition is $\bigvee_{1}^{n} s_j$. - -## Fallback Patch - -An output of the segmentation process above will include a fallback patch which is activated on the -presence of any input segment. In practice it will be slightly more efficient to instead move all of -the glyphs in the fallback patch into the initial font. This has the same overall effect as using a -fallback patch. In some cases the encoder may be purposely trying to minimize and/or eliminate glyph -data in the initial font. In these cases leaving them in the fallback patch may make the most sense. - -## Merging - -When starting with an input segmentation that is fine grained (for example using one segment per -code point) the resulting glyph segmentation may involve a large number of patches. This results in -excessive network overhead when loading the patches. Performance can be increased by selectively merging -patches together to reduce overhead. This is a complex problem as it needs to be done in a way that -avoids excessive transfers of glyph data that isn't needed. - -The segmenter currently implements a cost based merging algorithm which selects merges that minimize -an overall cost function. This process is documented in detail in -[closure_glyph_segmentation_merging.md](closure_glyph_segmentation_merging.md). - -The best segmentation results so far have been obtained by starting with one input segment per -code point in the font and then letting the merger figure out how to best place them together in a way -that minimizes overall cost. - -## Multi Segment Dependencies - -Note: this section is somewhat speculative as this functionality has not yet been fully implemented. -More research and exploration is definitely needed. - -The Segmenting Glyphs Based on Closure Analysis procedure places any glyphs whose conditions aren't -discovered into a fallback patch. Glyphs will end up in the fallback patch if there exists more -complex activation conditions that involve more than one segment at a time. For example consider a -glyph which is added to the closure when code points from ($s_1 \wedge s_2) \vee (s_3 \wedge s_4$) -are present. This glyph would not show up in either sets $I$ or $D$ for any single segment since -that would require at the union of at least two segments (for example $s_1$ and $s_2$) to be closure -tested. If the fallback patch is large than it may be necessary to analyze the dependencies further -to try and determine what the more complex activation conditions are. - -To discover these multi segment dependencies it’s necessary to test all combinations of two or more -segments with the closure analysis. Each combination of segments is unioned together to produce a -new segment and then the closure test is performed and new groups and activation conditions are -formed as described above (these are added to the previously determined patches). - -Unfortunately this approach will likely lead to a combinatorial explosion, so mitigations will be -needed to reduce the amount of combinations to test. Some suggestions: - -* It's not necessary to test all combinations. Once a sufficient number of fallback glyphs have been - categorized the combination testing can stop at any point. With remaining glyphs left in the - fallback patch. Given this, the remaining points provide suggestions for better targeting the - combinations to test in order to focus on combinations that are likely interacting. - -* In most cases complex dependencies should be isolated within a script, and for most scripts (other - than CJK) there should be a reasonably small number of segments per script. So if we limit the - multi segment combinations to within a script it should produce a reasonable number of - combinations to test while still being able to capture most dependencies. - -* Analysis of the font’s underlying GSUB rules could yield groups of code points which may possibly - interact. These could be used to prune segment combinations that don’t share interacting - code points. Alternatively these groups could be used to guide initial segment selection where - code points that appear to interact are placed in the same segment, thus getting a higher hit rate - in the single segment analysis. - -* The performance of a segmentation is likely driven solely by the high frequency code points. So - divide the font into a high frequency set and low frequency set of code points. Where a more - extensive multi segment dependency check is done for only the high frequency segments. - -As an alternative a simpler approach to the problem is to limit the scope to just finding conditions which are a superset -of the true condition. This superset condition can be used in place of the true condition without violating the closure -requirement. This is the approach currently used in the segmenter implementation. This procedure is discussed in more -details in [closure_glyph_segmentation_complex_conditions.md](./closure_glyph_segmentation_complex_conditions.md). The -advantage to this approach is it's much less computationally costly then multi segment analysis. The downside is these -superset conditions will activate more frequently then the true conditions and thus may be loaded in cases where they -are not actually needed. - -## Examples - -### UVS Selectors - -This example demonstrates how the proposed algorithm can detect the appropriate segmentations and activation conditions for a font that has closure glyphs which activate when two segments are simultaneously present. - -The original font contains: - -* Codepoints `{a, …, z}` and variation selector `VS1`. -* Each of a through z has an alternate form when combined with `VS1`. eg. `a + VS1` subs in glyph `a.alt` -* Four input segments are defined: - * `s_0 = {A} # initial font` - * `s_1 = {a, …, m}` - * `s_2 = {n, …, z}` - * `s_3 = {VS1}` - -Performing the per segment closure test will have the following results: - -| `S` | `A` | `B` | `I` | `D = A - B` | -| :---- | :---- | :---- | :---- | :---- | -| `s1` | `{A, a..z, a.alt..z.alt}` | `{A, n..z, n.alt..z.alt}` | `{a..m}` | `{a..m, a.alt..m.alt}` | -| `s2` | `{A, a..z, a.alt..z.alt}` | `{A, a..m, a.alt..m.alt}` | `{n..z}` | `{n..z, n.alt..z.alt}` | -| `s3` | `{A, a..z, a.alt..z.alt}` | `{A, a..z}` | `{}` | `{a.alt..z.alt}` | - -And the derived sets: - -| `S` | `D - I` | `I - D` | `D n I` | -| :---- | :---- | :---- | :---- | -| `s1` | `{a.alt..m.alt}` | `{}` | `{a..m}` | -| `s2` | `{n.alt..z.alt}` | `{}` | `{n..z}` | -| `s3` | `{a.alt..z.alt}` | `{}` | `{}` | - -Based on these we then we assign conditions to each glyph: - -| `glyph` | `Exclusive` | `Conjunctive Set` | `DisjunctiveR Set` | -| :---- | :---- | :---- | :---- | -| `a` | `{s1}` | `{}` | `{}` | -| `…` | `…` | `…` | `…` | -| `m` | `{s1}` | `{}` | `{}` | -| `n` | `{s2}` | `{}` | `{}` | -| `…` | `…` | `…` | `…` | -| `z` | `{s2}` | `{}` | `{}` | -| `a.alt` | `{}` | `{s1, s3}` | `{}` | -| `…` | `…` | `…` | `…` | -| `m.alt` | `{}` | `{s1, s3}` | `{}` | -| `n.alt` | `{}` | `{s2, s3}` | `{}` | -| `…` | `…` | `…` | `…` | -| `z.alt` | `{}` | `{s2, s3}` | `…` | - -Lastly, we group glyphs by condition sets to produce the final mappings and glyph keyed patches: - -| `Segment Set` | `Mapping Key (codepoints)` | `Patch Contains Glyphs` | -| :---- | :---- | :---- | -| `{s1}` | `{a..m}` | `{a..m}` | -| `{s2}` | `{n..z}` | `{a..z}` | -| `{s1, s3}` | `{a..m} AND {VS1}` | `{a.alt..m.alt}` | -| `{s2, s3}` | `{n..z} AND {VS1}` | `{n.alt..z.alt}` | - -### Shared Component - -This example demonstrates how the proposed algorithm can detect the appropriate segmentations and -activation conditions for a font that has closure glyphs which activate when at least one of several -segments are present. - -The original font contains: - -* Codepoints `{a, o, 0xe2, 0xf4}`. -* Simple glyphs 1, 2, and 5. -* Glyph 3 is a composite glyph using 1 + 5. -* Glyph 4 is a composite glyph using 2 + 5. -* With cmap: - * a: 1 - * o: 2 - * 0xe2: 3 - * 0xf4: 4 -* Two input segments are defined: - * `s0 = {} # initial font` - * `s1 = {a, 0xe2}` - * `s2 = {o, 0xf4}` - -Performing the per segment closure test will have the following results: - -| `S` | `A` | `B` | `I` | `D = A - B` | -| :---- | :---- | :---- | :---- | :---- | -| `s1` | `{1, 2, 3, 4, 5}` | `{2, 4, 5}` | `{1, 3, 5}` | `{1, 3}` | -| `s2` | `{1, 2, 3, 4, 5}` | `{1, 3, 5}` | `{2, 4, 5}` | `{2, 4}` | - -And the derived sets: - -| `S` | `D - I` | `I - D` | `D n I` | -| :---- | :---- | :---- | :---- | -| `s1` | `{}` | `{5}` | `{1, 3}` | -| `s2` | `{}` | `{5}` | `{2, 4}` | - -Based on these we then we assign conditions to each glyph: - -| `glyph` | `Exclusive Set` | `Conjunctive Set` | `Disjunctive Set` | -| :---- | :---- | :---- | :---- | -| `1` | `{s1}` | `{}` | `{}` | -| `2` | `{s2}` | `{}` | `{}` | -| `3` | `{s1}` | `{}` | `{}` | -| `4` | `{s2}` | `{}` | `{}` | -| `5` | `{}` | `{}` | `{s1, s2}` | - -Lastly, we group glyphs by unique set to produce the final mappings and glyph keyed patches: - -| `Exclusive Segment Set` | `Mapping Key (codepoints)` | `Patch Contains Glyphs` | -| :---- | :---- | :---- | -| `{s1}` | `{a, 0xe2}` | `{1, 3}` | -| `{s2}` | `{o, 0xf4}` | `{2, 4}` | -| `Disjunctive Set` | `Mapping Key (codepoints)` | `Patch Contains Glyphs` | -| `{s1, s2}` | `{a, o, 0xe2, 0xf4}` | `{5}` | diff --git a/docs/experimental/closure_glyph_segmentation_complex_conditions.md b/docs/experimental/closure_glyph_segmentation_complex_conditions.md deleted file mode 100644 index 868916ec..00000000 --- a/docs/experimental/closure_glyph_segmentation_complex_conditions.md +++ /dev/null @@ -1,157 +0,0 @@ - -# Complex Condition Finding in Closure Glyph Keyed Segmenter - -Author: Garret Rieger__ -Date: Dec 17, 2025 - -## Introduction - -*Note: the approach in this document is obsolete and has been replaced by - [dependency graph condition analysis](../dependency_graph_condition_extract.md).* - -Before reading this document is recommended to first review the -[closure glyph segmentation](./closure_glyph_segmentation.md) document. This document borrows concepts and terms from it. - -In closure glyph segmentation the closure analysis step is capable of locating glyph activation conditions that are -either fully disjunctive or fully conjunctive (eg. `(A or B or C)`). It is not capable of finding conditions that are a mix -of conjunction and disjunction (eg. `(A and B) or (B and C)`). These are referred to as complex conditions. By default -glyphs with complex conditions are assigned to a patch that is always loaded, since the true conditions are not known. - -This document describes an algorithm which can be used to find purely disjunctive conditions which are supersets of -complex conditions. A superset condition is one that will activate at least whenever the true condition would. This -property allows the superset condition to be used for a patch in place of the true condition without violating the -closure requirement. - -For a given complex condition there typically exists more than one possible superset disjunctive condition. The -algorithm will find one of them, but not necessarily the smallest one. The found superset condition will always only -contain only segments which appear in the original condition. - -For example if we had a glyph with a activation condition of `((A and B) or (B and C))` then this process will find one -of the possible superset conditions such as `(A or C)`, `(A or B)`, `(B or C)`, or `(B)`. In a segmentation we could -then have a patch with the found condition which loads the glyph and this would satisfy the closure requirement. - -## Foundations - -The algorithm is based on the following assertions: - -1. Given some fully disjunctive condition for a glyph, we can verify that the condition is a superset of the true - condition for the glyph and meets the closure requirement by the following procedure: compute a glyph closure of the - union of all segments except for those in the condition. If the glyph does not appear in this closure, then the - condition satisfies the closure requirement for that glyph and is a superset of the true condition. This is called - the “additional conditions” check. - -2. The glyph closure of all segments includes the glyph that we are analyzing. - -3. We have a glyph which has some true activation condition. If we compute a glyph closure of some combination of - segments, then adding or removing a segment, which is not part of the activation condition, to the glyph closure input - will have no affect on whether or not the glyph appears in the closure output. - -4. The closure of no segments contains only glyphs from the initial font. - -## The Algorithm - -For a glyph with a complex condition we can use the above to find a superset disjunctive condition for that -glyph's complex condition. These conditions will satisfy the closure requirement for each glyph. - -### Finding a Sub Condition - -The algorithm works by identifying a single sub condition at a time, this section describes the algorithm for -finding a single sub condition. - -Inputs: - -* Segments to exclude from the analysis. -* `glyph` to analyze. - -Algorithm: - -1. Start with a set of all segments except those to be excluded, called `to_test`. -2. Initialize a second set of segments, `sub_condition`, to the empty set. -3. Remove a segment `s` from `to_test` and compute the glyph closure of `to_test U sub_condition`. -4. If `glyph` is not found in the closure then add `s` to `sub_condition`. -5. If `to_test` is empty, then return the sub condition `sub_condition`. -6. Otherwise, go back to step 3. - -### Finding the Complete Condition - -This section describes the algorithm which finds the complete condition, it utilizes `Finding a Sub Condition`. - -Inputs: - -* `glyph` to analyze. - -Algorithm: - -1. Initialize a set of segments `condition` to the empty set. -2. Execute the `Finding a Sub Condition` algorithm with `condition` as the excluded set. -3. Union the returned set into `condition`. -4. Compute the glyph closure of all segments except those in `condition`. -5. If `glyph` is found in the closure, then more sub conditions still exist. Go back to step 2. -6. Return the complete condition, `condition`. - -### Initial Font - -Any time a closure operation is executed by the above two algorithms it's necessary to union the subset definition for -the initial font into the closure input. This is required because the closure of the initial font affects what's -reachable by the segments. - -### Why this works - -Here we show this procedure is guaranteed to find a disjunctive superset of a glyph's true condition which includes -only segments from the true condition, when the glyph is not already in the initial font: - -* For each call to `Finding a Sub Condition` glyph will be in the closure of all non-excluded segments. For the first - call this is guaranteed by assertion (2). For subsequent calls this is guaranteed by the "additional conditions" check - which gates execution. - -* Any segments which are not part of the true condition will not impact the glyph's presence in the closure (assertion - (3)). Further by the previous point we know that at the start of `Finding a Sub Condition` the closure of all - non-excluded segments will contain glyph. Thus testing a segment which is not part of the true condition will never - result in glyph missing from the closure, and won't be added to `sub_condition`. Therefore `Finding a Sub Condition` - will only ever return segments that are part of the true condition. - -* `Finding a Sub Condition` will always return at least one segment: if when the last segment is tested `sub_condition` - is still the empty set, then the closure will be on no segments and will not have glyph in it. This is a result of - assertion (4) and the premise that glyph is not already in the initial font. As a consequence the returned - `sub_condition` will always have at least one segment in it. - -* Since all returned segments from `Finding a Sub Condition` are excluded from future calls, there will be a finite - number of `Finding a Sub Condition` executions which return only segments part of the true condition. - -* Lastly, the algorithm terminates only once the additional conditions check finds no additional conditions, - guaranteeing we have found a superset disjunctive condition (assertion (1)). - -## Making it More Performant - -As described above this approach can be slow since it processes glyphs one at a time. Improvements to performance -can be made by processing glyphs in a batch. This can be done with a recursive approach where after each segment test -the set of input glyphs gets split into those that require the tested segment and those that don't. Each of the splits spawns -a new recursion (if there is at least one glyph in the split). - -Also from the closure analysis run by the segmenter we may have discovered some partial conditions for glyphs. These -can be incorporated as a starting point into the complex condition analysis. - -More substantial performance improvements could be realized by using a dependency graph generated from the font. This -could come in two forms: - -1. Determine what segments interact with GSUB in some way and use that to scope the analysis. Segments that don't - interact with GSUB can be discovered via regular closure analysis as they will only ever have disjunctive conditions. - After completing a scoped analysis, a final additional conditions check against all segments could be used to ensure - we have actually arrived at a superset condition. This is more speculative and would need more research to validate the - approach. - -2. Using a actual dependency graph, even if it's no fully accurate, to generate a set of initial activation - conditions. Then as needed when additional conditions check fails, we could find any additional segments for the - superset condition using this process. - -## Integrating into the Segmentation Algorithm - -The complex condition analysis has been added as a post processing step during the glyph grouping phase. After the -grouping phase finds the initial condition set, complex condition finding is run to find conditions for any remaining -unmapped glyphs. The found complex conditions are added to the condition set and thus are able to participate -in both initial font analysis and patch merging. To make this performant complex conditions are only recomputed for -glyphs/segments that are modified after a merge is performed. The glyph grouping phase has an existing invalidation -mechanism which is utilized to determine which glyphs need to be re-analyzed. - - - diff --git a/docs/experimental/unicode_range_segmentation.md b/docs/experimental/unicode_range_segmentation.md index 565bc91a..a26d9d16 100644 --- a/docs/experimental/unicode_range_segmentation.md +++ b/docs/experimental/unicode_range_segmentation.md @@ -16,7 +16,6 @@ With `@font-face` `unicode-range`, a font is split into a collection of standalo | **Client Loading Model** | Single font instance incrementally extended by patches | Independent `@font-face` WOFF2 font files matched per character by `unicode-range` | | **Initial Segment (`config.initial_segment`)** | Loaded once as the base font; initial font merging moves high-probability patches into it | Included in **every** subset (union of `initial_segment` codepoints/glyphs/features + default features and the patch's glyphs; useful for UVS codepoints, etc.). Initial font merging is disabled | | **Layout Features** | Optional feature patches + table-keyed patches | Codepoints only; each subset statically retains the default layout features plus any explicitly requested in `config.initial_segment.features`. Optional feature segments and table-keyed segments are disallowed | -| **Unmapped Glyphs** | Configurable (defaults to `MOVE_TO_INIT_FONT`) | Must be `FIND_CONDITIONS` or `PATCH` (`AutoSegmenterConfig` sets `FIND_CONDITIONS`; `MOVE_TO_INIT_FONT` is rejected) | | **Patch Merging** | Optional (`experimental_use_patch_merges`) | Disabled; only segment merges are performed | | **Interacting Glyphs & Disjoint Conditions** | Can be placed in separate patches (e.g. `fi` in patch `f AND i`, shared component `A` in `A OR À`) | Determined via dependency graph / closure analysis; `MakeActivationConditionsDisjoint` merges triggering segments of non-exclusive conditions whose probability $\ge$ `disjoint_conditions_probability_threshold`. Conditions below the threshold are skipped | | **Final Subsets & Activation Conditions** | CNF expressions (`(s1 OR s2) AND s3`) | Only **exclusive** patches (`(s_i)`) are emitted in the final plan (any remaining non-exclusive patches are ignored), yielding strictly disjunctive and mutually disjoint `unicode-range` sets | @@ -29,16 +28,14 @@ With `@font-face` `unicode-range`, a font is split into a collection of standalo A `TargetMode` enum (`IFT`, `UNICODE_RANGE`) in [segmenter_config.proto](../ift/config/segmenter_config.proto) selects the segmentation target mode (`target_mode` on `SegmenterConfig`, defaulting to `IFT`), alongside a `disjoint_conditions_probability_threshold` setting. The auto-configuration generator (`AutoSegmenterConfig::GenerateConfig`) also accepts `TargetMode` and automatically configures appropriate defaults when `UNICODE_RANGE` is selected: * Disables optional feature segment generation (`generate_feature_segments = false`) and table-keyed segment generation (`generate_table_keyed_segments = false`). -* Sets `unmapped_glyph_handling = FIND_CONDITIONS` so all glyphs in the full closure receive activation conditions. * Leaves initial font merging thresholds unset and skips IFT-specific base segmentation plan settings (`jump_ahead`, `use_prefetch_lists`). To ensure invalid configurations are caught regardless of whether a caller goes through `SegmenterConfigUtil` or invokes `ClosureGlyphSegmenter` directly, `ClosureGlyphSegmenter` validates that when `target_mode == UNICODE_RANGE`: -1. `unmapped_glyph_handling` is set to either `FIND_CONDITIONS` or `PATCH` (`MOVE_TO_INIT_FONT` is rejected). -2. No input segment definitions contain layout features (optional feature loading is unsupported). -3. Initial font merging is not enabled on any merge group. -4. Patch merging is disabled on all merge groups. -5. `generate_table_keyed_segments` is not enabled. +1. No input segment definitions contain layout features (optional feature loading is unsupported). +2. Initial font merging is not enabled on any merge group. +3. Patch merging is disabled on all merge groups. +4. `generate_table_keyed_segments` is not enabled. ### Initial Segment Handling @@ -57,7 +54,7 @@ In CSS `@font-face`, each subset is treated as a separate font face during text ### `MakeActivationConditionsDisjoint` -After initial condition analysis (and unmapped glyph condition finding when `FIND_CONDITIONS` is used), the segmenter runs a `MakeActivationConditionsDisjoint` pass before starting general cost-based or heuristic merging: +After initial condition analysis, the segmenter runs a `MakeActivationConditionsDisjoint` pass before starting general cost-based or heuristic merging: 1. Inspect all non-exclusive patch conditions (conditions involving more than one triggering segment, whether conjunctive or disjunctive). 2. Group segments into connected components by unioning the triggering segments of each non-exclusive condition. diff --git a/docs/segmenter.md b/docs/segmenter.md index 34baefb9..f085ba57 100644 --- a/docs/segmenter.md +++ b/docs/segmenter.md @@ -85,15 +85,9 @@ In the proceeding sections each of these steps will be discussed in more details ## Condition Analysis -Detecting the conditions under which a glyph is needed can be done in a couple of different -ways. The segmentation implementation currently has two main approaches: +Condition analysis infers the activation conditions under which each glyph in the font is needed by analyzing a [glyph dependency graph](dependency_graph.md) and running [dependency graph condition extraction](dependency_graph_condition_extract.md) (previously, an experimental font-subsetting closure approach was also supported; that approach has been removed). -1. Infer glyph conditions via [font subsetting closure](experimental/closure_glyph_segmentation.md). -2. Infer glyph conditions via a [glyph dependency graph](dependency_graph.md) and [dependency graph condition extraction](dependency_graph_condition_extract.md). - -Both approaches ultimately aim to detect conditions that each glyph is needed which will satisfy the glyph closure -requirement. The dependency graph is the default and recommended method for finding conditions. It's generally faster -and produces more detailed conditions. +Using the dependency graph generates detailed activation conditions that satisfy the glyph closure requirement, and the final segmentation is validated against HarfBuzz subsetting closure. ## Merging @@ -373,8 +367,6 @@ In the merger implementation, actual cost delta computation is optimized by firs *aggregate probability* $P_{\text{agg}}(c_i) = \sum_{m=1}^M P_m(c_i)$ of a condition across all frequency data sets in the merge group, and then multiplying that sum once by $(\text{size}(p_i) + k)$ for each affected patch. This avoids redundant patch size evaluations and per-data-set delta loops. -Similarly, because the fallback patch is needed with probability $1.0$ under every frequency data set, -its size delta is multiplied by the number of data sets $M$. ### Prioritizing a Primary Script and Language diff --git a/ift/config/auto_segmenter_config.cc b/ift/config/auto_segmenter_config.cc index f44a597b..3bb13534 100644 --- a/ift/config/auto_segmenter_config.cc +++ b/ift/config/auto_segmenter_config.cc @@ -114,10 +114,6 @@ enum Quality { // To start, the list of parameters we can use to make quality/performance // tradeoffs: // -// - unmapped_glyph_handling (global) -// Lower quality is to not find conditions (so use patch or init font), high -// quality is to find conditions. -// // - generate_feature_segments (global): high quality generate segment per // feature, low quality put all optional features // in one segment. @@ -135,10 +131,10 @@ enum Quality { // Would be reasonable to always have this set to at least the minimum group // size. // -// - condition_analysis_mode: use DEP_GRAPH_ONLY if dependency API is available, -// otherwise CLOSURE_ONLY. DEP_GRAPH_ONLY_WITH_SIMPLIFICATION can significantly -// speed up cases with high condition complexity, at the cost of less specific -// final conditions. +// - condition_analysis_mode: use DEP_GRAPH_ONLY or +// DEP_GRAPH_ONLY_WITH_SIMPLIFICATION. DEP_GRAPH_ONLY_WITH_SIMPLIFICATION can +// significantly speed up cases with high condition complexity, at the cost of +// less specific final conditions. // // Merge group settings: // @@ -403,10 +399,9 @@ static Status GroupInteractingScripts( std::unique_ptr segmentation_info = TRY(RequestedSegmentationInformation::Create( - std::move(segments), init_segment, *closure_cache, FIND_CONDITIONS)); + std::move(segments), init_segment, *closure_cache)); std::unique_ptr dependency_closure = - TRY(DependencyClosure::Create(segmentation_info.get(), face, resolver, - /*allow_context_glyph_analysis=*/true)); + TRY(DependencyClosure::Create(segmentation_info.get(), face, resolver)); GlyphPartition ungrouped_partition(ungrouped_list.size()); flat_hash_map ungrouped_to_scripts; @@ -958,12 +953,6 @@ static void ApplyQualityLevelTo(Quality quality, MergeGroup& merge_group) { static void ApplyQualityLevelTo(Quality quality, SegmenterConfig& config) { config.set_preprocess_merging_group_size_for_ungrouped(kMinimumGroupSize * 3); - if (config.target_mode() == UNICODE_RANGE) { - config.set_unmapped_glyph_handling(FIND_CONDITIONS); - } else { - config.set_unmapped_glyph_handling(MOVE_TO_INIT_FONT); - } - if (quality == ONE) { config.set_condition_analysis_mode(DEP_GRAPH_ONLY_WITH_SIMPLIFICATION); } else { diff --git a/ift/config/auto_segmenter_config_test.cc b/ift/config/auto_segmenter_config_test.cc index 068b890a..6b53be1f 100644 --- a/ift/config/auto_segmenter_config_test.cc +++ b/ift/config/auto_segmenter_config_test.cc @@ -20,9 +20,7 @@ #include "ift/common/test_font_loader.h" #include "ift/config/load_codepoints.h" -using ift::config::CLOSURE_AND_DEP_GRAPH; using ift::config::FIND_CONDITIONS; -using ift::config::MOVE_TO_INIT_FONT; using ift::config::SegmenterConfig; namespace ift::config { @@ -173,8 +171,7 @@ TEST_F(AutoSegmenterConfigTest, Roboto_UnspecifiedPrimary) { std::string config_string; TextFormat::PrintToString(*config_or, &config_string); - ASSERT_EQ(config_string, R"(unmapped_glyph_handling: MOVE_TO_INIT_FONT -brotli_quality: 11 + ASSERT_EQ(config_string, R"(brotli_quality: 11 brotli_quality_for_initial_font_merging: 9 base_heuristic_config { min_patch_size: 2500 @@ -549,12 +546,23 @@ static std::vector GetUseBigrams(const SegmenterConfig& config) { return result; } +#pragma GCC diagnostic push +#pragma GCC diagnostic ignored "-Wdeprecated-declarations" +bool HasUnmappedGlyphHandling(const SegmenterConfig& config) { + return config.has_unmapped_glyph_handling(); +} +UnmappedGlyphHandling GetUnmappedGlyphHandling(const SegmenterConfig& config) { + return config.unmapped_glyph_handling(); +} +#pragma GCC diagnostic pop + TEST_F(AutoSegmenterConfigTest, QualityLevelForcing) { auto config_or = AutoSegmenterConfig::GenerateConfig(face_.get(), *resolver, std::nullopt, 1); ASSERT_TRUE(config_or.ok()) << config_or.status(); EXPECT_EQ(config_or->brotli_quality(), 0); - EXPECT_EQ(config_or->unmapped_glyph_handling(), MOVE_TO_INIT_FONT); + EXPECT_FALSE(HasUnmappedGlyphHandling(*config_or)); + EXPECT_EQ(GetUnmappedGlyphHandling(*config_or), FIND_CONDITIONS); EXPECT_THAT(GetUseBigrams(*config_or), Each(false)); EXPECT_EQ(config_or->brotli_quality_for_initial_font_merging(), 0); EXPECT_EQ(config_or->base_cost_config().optimization_cutoff_fraction(), 0.05); @@ -563,7 +571,8 @@ TEST_F(AutoSegmenterConfigTest, QualityLevelForcing) { std::nullopt, 7); ASSERT_TRUE(config_or_8.ok()) << config_or_8.status(); EXPECT_EQ(config_or_8->brotli_quality(), 11); - EXPECT_EQ(config_or_8->unmapped_glyph_handling(), MOVE_TO_INIT_FONT); + EXPECT_FALSE(HasUnmappedGlyphHandling(*config_or_8)); + EXPECT_EQ(GetUnmappedGlyphHandling(*config_or_8), FIND_CONDITIONS); EXPECT_THAT(GetUseBigrams(*config_or_8), Each(true)); EXPECT_EQ(config_or_8->brotli_quality_for_initial_font_merging(), 11); EXPECT_EQ(config_or_8->base_cost_config().optimization_cutoff_fraction(), @@ -707,8 +716,7 @@ TEST_F(AutoSegmenterConfigTest, Roboto_UnspecifiedPrimary_UnicodeRange) { // - Ignored for overlap codepoints present in multiple groups (ie. 0x20) are moved to exactly one group. // - This in turn requires explicit segment definitions for affected groups. // - Combining marks are not ignored for overlap detection so latin + cyrillic are grouped. - ASSERT_EQ(config_string, R"(unmapped_glyph_handling: FIND_CONDITIONS -brotli_quality: 11 + ASSERT_EQ(config_string, R"(brotli_quality: 11 base_heuristic_config { min_patch_size: 2500 max_patch_size: 20000 @@ -776,7 +784,8 @@ TEST_F(AutoSegmenterConfigTest, UnicodeRange_QualityLevels) { ASSERT_TRUE(config_q1.ok()) << config_q1.status(); EXPECT_EQ(config_q1->target_mode(), UNICODE_RANGE); EXPECT_EQ(config_q1->brotli_quality(), 0); - EXPECT_EQ(config_q1->unmapped_glyph_handling(), FIND_CONDITIONS); + EXPECT_FALSE(HasUnmappedGlyphHandling(*config_q1)); + EXPECT_EQ(GetUnmappedGlyphHandling(*config_q1), FIND_CONDITIONS); EXPECT_FALSE(config_q1->has_brotli_quality_for_initial_font_merging()); EXPECT_FALSE(config_q1->base_cost_config().experimental_use_patch_merges()); EXPECT_THAT(GetUseBigrams(*config_q1), Each(false)); @@ -787,7 +796,8 @@ TEST_F(AutoSegmenterConfigTest, UnicodeRange_QualityLevels) { ASSERT_TRUE(config_q7.ok()) << config_q7.status(); EXPECT_EQ(config_q7->target_mode(), UNICODE_RANGE); EXPECT_EQ(config_q7->brotli_quality(), 11); - EXPECT_EQ(config_q7->unmapped_glyph_handling(), FIND_CONDITIONS); + EXPECT_FALSE(HasUnmappedGlyphHandling(*config_q7)); + EXPECT_EQ(GetUnmappedGlyphHandling(*config_q7), FIND_CONDITIONS); EXPECT_FALSE(config_q7->has_brotli_quality_for_initial_font_merging()); EXPECT_FALSE(config_q7->base_cost_config().experimental_use_patch_merges()); EXPECT_THAT(GetUseBigrams(*config_q7), Each(true)); diff --git a/ift/config/common.proto b/ift/config/common.proto index 0d51abcb..a86b0466 100644 --- a/ift/config/common.proto +++ b/ift/config/common.proto @@ -38,15 +38,17 @@ message AxisRangeProto { float end = 2; } -// This enum is used to specify how unmapped glyphs are handled in the produced segmentation. +// Deprecated: unmapped glyph handling is only used by removed closure analysis +// modes. Specifying anything other than FIND_CONDITIONS will result in an +// error. enum UnmappedGlyphHandling { - // All unmapped glyphs will be placed into a patch which is always loaded. - PATCH = 0; + // Deprecated: specifying this value will result in an error. + PATCH = 0 [deprecated = true]; - // All unmapped glyphs will be moved into the initial font. - MOVE_TO_INIT_FONT = 1; + // Deprecated: specifying this value will result in an error. + MOVE_TO_INIT_FONT = 1 [deprecated = true]; - // Advanced condition analysis will be used (expensive) to discover the conditions - // for unmapped glyphs. + // Default value; triggers no additional behavior in dependency graph + // condition analysis modes. FIND_CONDITIONS = 2; } \ No newline at end of file diff --git a/ift/config/segmenter_config.proto b/ift/config/segmenter_config.proto index d5df9900..f15361a8 100644 --- a/ift/config/segmenter_config.proto +++ b/ift/config/segmenter_config.proto @@ -6,23 +6,14 @@ import "ift/config/segmentation_plan.proto"; import "ift/config/common.proto"; enum ConditionAnalysisMode { - // [DEPRECATED] - // Analyzes glyph conditions using closure analysis only. - // This is a legacy mode, it's much slower and generally produces - // worse results then the dependency graph approach. - CLOSURE_ONLY = 0; - - // Attempts to analyze glyph conditions using a dependency - // graph where possible. When dep graph is expected not to be accurate - // it falls back to closure analysis. This is for many cases much faster - // then pure closure based analysis. - CLOSURE_AND_DEP_GRAPH = 1; - - // Always runs both closure and dependency graph analysis, the - // results are compared and an error is raised if they differ. - // Useful primarily for testing the accuracy of the dependency - // graph approach. - CLOSURE_AND_VALIDATE_DEP_GRAPH = 2; + // Deprecated: specifying this mode will result in an error. + CLOSURE_ONLY = 0 [deprecated = true]; + + // Deprecated: specifying this mode will result in an error. + CLOSURE_AND_DEP_GRAPH = 1 [deprecated = true]; + + // Deprecated: specifying this mode will result in an error. + CLOSURE_AND_VALIDATE_DEP_GRAPH = 2 [deprecated = true]; // [RECOMMENDED] // Analyze glyph conditions using dep graph only. Closure is @@ -112,12 +103,10 @@ message SegmenterConfig { // are included in all subsets. SegmentProto initial_segment = 1; - // Specifies what to do with unmapped glyphs. Unmapped glyphs are glyphs in which the analysis - // is unable to determine activation conditions for. - // - // When target_mode is UNICODE_RANGE, this must be set to either FIND_CONDITIONS or PATCH - // (MOVE_TO_INIT_FONT is invalid). - UnmappedGlyphHandling unmapped_glyph_handling = 2 [default = MOVE_TO_INIT_FONT]; + // Deprecated: only FIND_CONDITIONS is valid; specifying any other value will + // result in an error. + UnmappedGlyphHandling unmapped_glyph_handling = 2 + [default = FIND_CONDITIONS, deprecated = true]; // Deprecated: use generate_table_keyed_segments_mode instead. // diff --git a/ift/config/segmenter_config_util.cc b/ift/config/segmenter_config_util.cc index fec939dc..1caf89aa 100644 --- a/ift/config/segmenter_config_util.cc +++ b/ift/config/segmenter_config_util.cc @@ -176,6 +176,22 @@ static StatusOr ResolveTableKeyedSegmentMode( return config.generate_table_keyed_segments_mode(); } +static absl::Status ValidateConditionAnalysisAndUnmappedGlyphs( + const SegmenterConfig& config) { + if (config.condition_analysis_mode() != DEP_GRAPH_ONLY && + config.condition_analysis_mode() != DEP_GRAPH_ONLY_WITH_SIMPLIFICATION) { + return absl::InvalidArgumentError( + "Closure-based condition_analysis_mode values are deprecated and " + "unsupported; use DEP_GRAPH_ONLY or " + "DEP_GRAPH_ONLY_WITH_SIMPLIFICATION."); + } + if (config.unmapped_glyph_handling() != FIND_CONDITIONS) { + return absl::InvalidArgumentError( + "unmapped_glyph_handling is deprecated and must be FIND_CONDITIONS."); + } + return absl::OkStatus(); +} + #pragma GCC diagnostic pop StatusOr @@ -355,6 +371,8 @@ SegmenterConfigUtil::ConfigToMergeGroups( const SegmenterConfig& config, const CodepointSet& font_codepoints, const btree_set& font_features, std::vector& segments) { + TRYV(ValidateConditionAnalysisAndUnmappedGlyphs(config)); + if (config.target_mode() == UNICODE_RANGE && (!config.feature_segments().empty() || config.generate_feature_segments())) { @@ -410,6 +428,8 @@ SegmenterConfigUtil::ConfigToMergeGroups( StatusOr SegmenterConfigUtil::RunSegmenter( hb_face_t* face, const SegmenterConfig& config) { + TRYV(ValidateConditionAnalysisAndUnmappedGlyphs(config)); + TableKeyedSegmentMode table_keyed_mode = TRY(ResolveTableKeyedSegmentMode(config)); if (config.target_mode() == UNICODE_RANGE && table_keyed_mode != NONE) { @@ -429,8 +449,7 @@ StatusOr SegmenterConfigUtil::RunSegmenter( ClosureGlyphSegmenter segmenter( config.brotli_quality(), config.brotli_quality_for_initial_font_merging(), - config.unmapped_glyph_handling(), config.condition_analysis_mode(), - resolver_, config.target_mode(), + config.condition_analysis_mode(), resolver_, config.target_mode(), config.disjoint_conditions_probability_threshold()); GlyphSegmentation segmentation = TRY(segmenter.CodepointToGlyphSegments( diff --git a/ift/config/segmenter_config_util_test.cc b/ift/config/segmenter_config_util_test.cc index 9814a2de..e6e104ca 100644 --- a/ift/config/segmenter_config_util_test.cc +++ b/ift/config/segmenter_config_util_test.cc @@ -84,6 +84,16 @@ void SetLegacyGenerateTableKeyedSegments(SegmenterConfig& config, bool value) { config.set_generate_table_keyed_segments(value); } +void SetLegacyUnmappedGlyphHandling(SegmenterConfig& config, + ift::config::UnmappedGlyphHandling value) { + config.set_unmapped_glyph_handling(value); +} + +void SetLegacyConditionAnalysisMode(SegmenterConfig& config, + ift::config::ConditionAnalysisMode value) { + config.set_condition_analysis_mode(value); +} + #pragma GCC diagnostic pop MergeStrategy ExpectedCostStrategy( @@ -960,7 +970,6 @@ TEST_F(SegmenterConfigUtilTest, SegmenterConfig config; config.set_target_mode(ift::config::UNICODE_RANGE); - config.set_unmapped_glyph_handling(ift::config::FIND_CONDITIONS); config.set_generate_table_keyed_segments_mode(FROM_FREQ_DATA); AddSegment(config, 1, {0x43}); @@ -983,7 +992,6 @@ TEST_F(SegmenterConfigUtilTest, { SegmenterConfig config; config.set_target_mode(ift::config::UNICODE_RANGE); - config.set_unmapped_glyph_handling(ift::config::FIND_CONDITIONS); config.set_generate_feature_segments(true); AddSegment(config, 1, {0x43}); @@ -999,7 +1007,6 @@ TEST_F(SegmenterConfigUtilTest, { SegmenterConfig config; config.set_target_mode(ift::config::UNICODE_RANGE); - config.set_unmapped_glyph_handling(ift::config::FIND_CONDITIONS); Features features; features.add_values("smcp"); (*config.mutable_feature_segments())[1] = features; @@ -1012,7 +1019,6 @@ TEST_F(SegmenterConfigUtilTest, { SegmenterConfig config; config.set_target_mode(ift::config::UNICODE_RANGE); - config.set_unmapped_glyph_handling(ift::config::FIND_CONDITIONS); AddSegment(config, 1, {0x43}); (*config.mutable_segments())[1].mutable_features()->add_values("smcp"); @@ -1028,21 +1034,26 @@ TEST_F(SegmenterConfigUtilTest, loader->LoadFace("ift/common/testdata/Roboto-Regular.ttf").value(); SegmenterConfigUtil util("util/testdata/config.txtpb", resolver); - // Default unmapped_glyph_handling (MOVE_TO_INIT_FONT) is rejected. + // Explicit MOVE_TO_INIT_FONT or PATCH is rejected. { - SegmenterConfig config; - config.set_target_mode(ift::config::UNICODE_RANGE); - AddSegment(config, 1, {0x43}); - - auto result = util.RunSegmenter(face.get(), config); - EXPECT_TRUE(absl::IsInvalidArgument(result.status())) << result.status(); +#pragma GCC diagnostic push +#pragma GCC diagnostic ignored "-Wdeprecated-declarations" + for (auto handling : {ift::config::MOVE_TO_INIT_FONT, ift::config::PATCH}) { +#pragma GCC diagnostic pop + SegmenterConfig config; + config.set_target_mode(ift::config::UNICODE_RANGE); + SetLegacyUnmappedGlyphHandling(config, handling); + AddSegment(config, 1, {0x43}); + + auto result = util.RunSegmenter(face.get(), config); + EXPECT_TRUE(absl::IsInvalidArgument(result.status())) << result.status(); + } } // Initial font merging is rejected. { SegmenterConfig config; config.set_target_mode(ift::config::UNICODE_RANGE); - config.set_unmapped_glyph_handling(ift::config::FIND_CONDITIONS); AddSegment(config, 1, {0x43}); auto* group = config.add_merge_groups(); AddFreqData(*group, "test_freq_data.riegeli"); @@ -1058,7 +1069,6 @@ TEST_F(SegmenterConfigUtilTest, { SegmenterConfig config; config.set_target_mode(ift::config::UNICODE_RANGE); - config.set_unmapped_glyph_handling(ift::config::FIND_CONDITIONS); AddSegment(config, 1, {0x43}); auto* group = config.add_merge_groups(); AddFreqData(*group, "test_freq_data.riegeli"); @@ -1072,7 +1082,6 @@ TEST_F(SegmenterConfigUtilTest, { SegmenterConfig config; config.set_target_mode(ift::config::UNICODE_RANGE); - config.set_unmapped_glyph_handling(ift::config::FIND_CONDITIONS); config.set_brotli_quality(0); config.set_brotli_quality_for_initial_font_merging(0); AddSegment(config, 1, {0x43}); @@ -1085,4 +1094,67 @@ TEST_F(SegmenterConfigUtilTest, } } +TEST_F(SegmenterConfigUtilTest, + RunSegmenter_RejectsDeprecatedConditionAnalysisModes) { + auto loader = TestFontLoader::Default().value(); + auto face = + loader->LoadFace("ift/common/testdata/Roboto-Regular.ttf").value(); + SegmenterConfigUtil util("util/testdata/config.txtpb", resolver); + +#pragma GCC diagnostic push +#pragma GCC diagnostic ignored "-Wdeprecated-declarations" + for (auto mode : + {ift::config::CLOSURE_ONLY, ift::config::CLOSURE_AND_DEP_GRAPH, + ift::config::CLOSURE_AND_VALIDATE_DEP_GRAPH}) { +#pragma GCC diagnostic pop + SegmenterConfig config; + SetLegacyConditionAnalysisMode(config, mode); + AddSegment(config, 1, {0x43}); + + std::vector segments_out; + auto groups = util.ConfigToMergeGroups(config, {0x43}, {}, segments_out); + EXPECT_TRUE(absl::IsInvalidArgument(groups.status())) << groups.status(); + + auto result = util.RunSegmenter(face.get(), config); + EXPECT_TRUE(absl::IsInvalidArgument(result.status())) << result.status(); + } +} + +TEST_F(SegmenterConfigUtilTest, + RunSegmenter_RejectsInvalidUnmappedGlyphHandling) { + auto loader = TestFontLoader::Default().value(); + auto face = + loader->LoadFace("ift/common/testdata/Roboto-Regular.ttf").value(); + SegmenterConfigUtil util("util/testdata/config.txtpb", resolver); + +#pragma GCC diagnostic push +#pragma GCC diagnostic ignored "-Wdeprecated-declarations" + for (auto handling : {ift::config::PATCH, ift::config::MOVE_TO_INIT_FONT}) { +#pragma GCC diagnostic pop + SegmenterConfig config; + SetLegacyUnmappedGlyphHandling(config, handling); + AddSegment(config, 1, {0x43}); + + std::vector segments_out; + auto groups = util.ConfigToMergeGroups(config, {0x43}, {}, segments_out); + EXPECT_TRUE(absl::IsInvalidArgument(groups.status())) << groups.status(); + + auto result = util.RunSegmenter(face.get(), config); + EXPECT_TRUE(absl::IsInvalidArgument(result.status())) << result.status(); + } + + // Default (unset) and explicit FIND_CONDITIONS succeed. + { + SegmenterConfig config; + config.set_brotli_quality(0); + AddSegment(config, 1, {0x43}); + auto result = util.RunSegmenter(face.get(), config); + ASSERT_TRUE(result.ok()) << result.status(); + + SetLegacyUnmappedGlyphHandling(config, ift::config::FIND_CONDITIONS); + result = util.RunSegmenter(face.get(), config); + ASSERT_TRUE(result.ok()) << result.status(); + } +} + // TODO test for feature segment auto generation. \ No newline at end of file diff --git a/ift/dep_graph/dependency_graph_test.cc b/ift/dep_graph/dependency_graph_test.cc index 01721b64..d7d7d7f0 100644 --- a/ift/dep_graph/dependency_graph_test.cc +++ b/ift/dep_graph/dependency_graph_test.cc @@ -17,8 +17,6 @@ #include "ift/encoder/types.h" #include "ift/freq/probability_bound.h" -using ift::config::PATCH; - using absl::btree_set; using absl::flat_hash_map; using absl::flat_hash_set; @@ -62,7 +60,7 @@ class DependencyGraphTest : public ::testing::Test { loader->LoadFace("ift/common/testdata/NotoSansKR[wght].subset.ttf") .value()), segmentation_info(*RequestedSegmentationInformation::Create( - segments, WithDefaultFeatures({}), *closure_cache, PATCH)), + segments, WithDefaultFeatures({}), *closure_cache)), graph(*DependencyGraph::Create(segmentation_info.get(), face.get(), *resolver)) {} @@ -75,7 +73,7 @@ class DependencyGraphTest : public ::testing::Test { void Reconfigure(SubsetDefinition new_init, std::vector new_segments) { segmentation_info = *RequestedSegmentationInformation::Create( - new_segments, new_init, *closure_cache, PATCH); + new_segments, new_init, *closure_cache); graph = *DependencyGraph::Create(segmentation_info.get(), face.get(), *resolver); } @@ -84,7 +82,7 @@ class DependencyGraphTest : public ::testing::Test { std::vector new_segments) { closure_cache = std::move(*GlyphClosureCache::Create(new_face, *resolver)); segmentation_info = *RequestedSegmentationInformation::Create( - new_segments, new_init, *closure_cache, PATCH); + new_segments, new_init, *closure_cache); graph = *DependencyGraph::Create(segmentation_info.get(), new_face, *resolver); } diff --git a/ift/encoder/BUILD b/ift/encoder/BUILD index 4698b15f..2eb8668b 100644 --- a/ift/encoder/BUILD +++ b/ift/encoder/BUILD @@ -80,7 +80,6 @@ cc_library( srcs = [ "all_pairs_merger.cc", "candidate_merge.cc", - "complex_condition_finder.cc", "dependency_closure.cc", "estimated_patch_size_cache.cc", "glyph_condition_set.cc", @@ -92,7 +91,6 @@ cc_library( hdrs = [ "all_pairs_merger.h", "candidate_merge.h", - "complex_condition_finder.h", "condition_to_glyphs_index.h", "dependency_closure.h", "estimated_patch_size_cache.h", @@ -158,7 +156,6 @@ cc_library( "//ift/common", "//ift/common:data_file_resolver", "//ift/common:try", - "//ift/config:common_cc_proto", "//ift/dep_graph:unicode_edges", "//ift/feature_registry", "@abseil-cpp//absl/container:flat_hash_map", @@ -380,31 +377,6 @@ cc_test( ], ) -cc_test( - name = "complex_condition_finder_test", - size = "medium", - srcs = [ - "complex_condition_finder_test.cc", - ], - data = [ - "//ift:testdata", - "//ift/common:testdata", - ], - deps = [ - ":common", - ":encoder", - ":segmentation_context", - ":segmentation_info", - "//ift/common", - "//ift/common:data_file_resolver", - "//ift/common:test_font_loader", - "//ift/freq:common", - "@abseil-cpp//absl/container:btree", - "@googletest//:gtest", - "@googletest//:gtest_main", - ], -) - cc_test( name = "closure_glyph_segmenter_test", size = "medium", @@ -538,7 +510,6 @@ cc_test( "//ift/common", "//ift/common:data_file_resolver", "//ift/common:test_font_loader", - "//ift/config:common_cc_proto", "//ift/dep_graph", "//ift/freq:common", "@bazel_tools//tools/cpp/runfiles", diff --git a/ift/encoder/activation_condition.cc b/ift/encoder/activation_condition.cc index 0c96c770..f7949b57 100644 --- a/ift/encoder/activation_condition.cc +++ b/ift/encoder/activation_condition.cc @@ -74,19 +74,16 @@ ActivationCondition ActivationCondition::and_segments( } ActivationCondition ActivationCondition::or_segments(const SegmentSet& segments, - patch_id_t activated, - bool is_fallback) { + patch_id_t activated) { ActivationCondition conditions; conditions.activated_ = {activated}; conditions.conditions_.push_back(segments); - conditions.is_fallback_ = is_fallback; return conditions; } ActivationCondition ActivationCondition::composite_condition( - absl::Span groups, patch_id_t activated, - bool is_fallback) { + absl::Span groups, patch_id_t activated) { ActivationCondition conditions; conditions.activated_ = {activated}; for (const auto& group : groups) { @@ -95,7 +92,6 @@ ActivationCondition ActivationCondition::composite_condition( Simplify(conditions.conditions_); - conditions.is_fallback_ = is_fallback; return conditions; } @@ -174,7 +170,6 @@ ActivationCondition ActivationCondition::Or(const ActivationCondition& a, // they must be combined in a commutative and associative way. Otherwise // folding Or() across a group of conditions would produce different results // depending on the order the conditions are visited in. - condition.is_fallback_ = a.is_fallback_ || b.is_fallback_; condition.is_exclusive_ = false; condition.activated_ = std::max(a.activated_, b.activated_); condition.encoding_ = std::max(a.encoding_, b.encoding_); @@ -297,7 +292,6 @@ ActivationCondition ActivationCondition::ReplaceSegments( } new_condition.activated_ = activated_; - new_condition.is_fallback_ = is_fallback_; new_condition.is_exclusive_ = is_exclusive_ || new_condition.conditions_.size() == 1; new_condition.encoding_ = encoding_; @@ -321,7 +315,6 @@ ActivationCondition ActivationCondition::RemoveIntersectingSubgroups( ActivationCondition new_condition = ActivationCondition::True(0); new_condition.activated_ = activated_; new_condition.encoding_ = encoding_; - new_condition.is_fallback_ = is_fallback_; new_condition.is_exclusive_ = is_exclusive_; for (const auto& condition_group : conditions_) { @@ -378,7 +371,6 @@ ActivationCondition ActivationCondition::NonCompositeSuperset() const { } ActivationCondition out; - out.is_fallback_ = is_fallback_; out.is_exclusive_ = is_exclusive_; out.conditions_ = {}; out.activated_ = activated_; @@ -472,10 +464,6 @@ bool ActivationCondition::operator<(const ActivationCondition& other) const { return is_exclusive_; } - if (is_fallback_ != other.is_fallback_) { - return !is_fallback_; - } - return encoding_ < other.encoding_; } diff --git a/ift/encoder/activation_condition.h b/ift/encoder/activation_condition.h index bd2713a5..9830fb6b 100644 --- a/ift/encoder/activation_condition.h +++ b/ift/encoder/activation_condition.h @@ -53,16 +53,14 @@ class ActivationCondition { * (segment_1) OR ... OR inersects(segment_n). */ static ActivationCondition or_segments(const ift::common::SegmentSet& ids, - patch_id_t activated, - bool is_fallback = false); + patch_id_t activated); /* * Constructs a condition that activates when the input intersects: * (s1 OR ..) AND (si OR ...) AND ... */ static ActivationCondition composite_condition( - absl::Span groups, patch_id_t activated, - bool is_fallback = false); + absl::Span groups, patch_id_t activated); /* * Constructs a new condition equal to 'condition' except with the is @@ -105,8 +103,6 @@ class ActivationCondition { bool IsAlwaysTrue() const { return conditions().size() == 0; } - bool IsFallback() const { return is_fallback_; } - /* * Populates out with the set of patch ids that are part of this condition * (excluding the activated patch) @@ -229,8 +225,8 @@ class ActivationCondition { size_t Hash() const { if (!cached_hash_.has_value()) { - cached_hash_ = absl::HashOf(conditions_, activated_, is_fallback_, - is_exclusive_, encoding_); + cached_hash_ = + absl::HashOf(conditions_, activated_, is_exclusive_, encoding_); } return *cached_hash_; } @@ -248,7 +244,6 @@ class ActivationCondition { return false; } return conditions_ == other.conditions_ && activated_ == other.activated_ && - is_fallback_ == other.is_fallback_ && is_exclusive_ == other.is_exclusive_ && encoding_ == other.encoding_; } @@ -270,7 +265,6 @@ class ActivationCondition { ActivationCondition() : conditions_(), activated_({0}), encoding_(proto::GLYPH_KEYED) {} - bool is_fallback_ = false; bool is_exclusive_ = false; // Represents: // (s_1_1 OR s_1_2 OR ...) AND (s_2_1 OR ...) ... diff --git a/ift/encoder/activation_condition_test.cc b/ift/encoder/activation_condition_test.cc index d7580094..42a232e7 100644 --- a/ift/encoder/activation_condition_test.cc +++ b/ift/encoder/activation_condition_test.cc @@ -417,7 +417,6 @@ TEST(ActivationConditionTest, And) { EXPECT_EQ(combined_ab.ToString(), "if ((s1 OR s2) AND s3) then p10"); EXPECT_EQ(combined_ab.activated(), 10); EXPECT_FALSE(combined_ab.IsExclusive()); - EXPECT_FALSE(combined_ab.IsFallback()); auto combined_ba = ActivationCondition::And(b, a); EXPECT_EQ(combined_ab.conditions(), combined_ba.conditions()); @@ -468,7 +467,6 @@ TEST(ActivationConditionTest, Or) { "then p11"); EXPECT_EQ(combined_ab.activated(), 11); EXPECT_FALSE(combined_ab.IsExclusive()); - EXPECT_FALSE(combined_ab.IsFallback()); auto combined_ba = ActivationCondition::Or(b, a); EXPECT_EQ(combined_ab, combined_ba); @@ -477,14 +475,13 @@ TEST(ActivationConditionTest, Or) { TEST(ActivationConditionTest, Or_IsOrderIndependent) { auto a = ActivationCondition::or_segments({1, 2}, 10); a.SetEncoding(ift::proto::DEFAULT_ENCODING); - auto b = ActivationCondition::or_segments({3}, 11, /*is_fallback=*/true); + auto b = ActivationCondition::or_segments({3}, 11); b.SetEncoding(ift::proto::DEFAULT_ENCODING); auto c = ActivationCondition::and_segments({4, 5}, 9); c.SetEncoding(ift::proto::TABLE_KEYED_PARTIAL); auto ab = ActivationCondition::Or(a, b); EXPECT_EQ(ab, ActivationCondition::Or(b, a)); - EXPECT_TRUE(ab.IsFallback()); EXPECT_EQ(ab.activated(), 11); auto ac = ActivationCondition::Or(a, c); diff --git a/ift/encoder/all_pairs_merger.cc b/ift/encoder/all_pairs_merger.cc index f608e975..219f1baa 100644 --- a/ift/encoder/all_pairs_merger.cc +++ b/ift/encoder/all_pairs_merger.cc @@ -44,20 +44,13 @@ void AllPairsMerger::AssignMergedSegment(const Merger& merger, // let us locate any conditions involved with those glyphs post-merge. for (const auto& c : groupings.TriggeringSegmentToConditionsPreCombination(s)) { - if (c.IsFallback()) { - continue; - } if (auto rep = pre_comb_map.at(c).min(); rep.has_value()) { dirty_glyphs_.insert(*rep); } } // Also record any segments involved in conditions directly impacted by the merge. - for (const auto& c : - groupings.TriggeringSegmentToConditions(s)) { - if (c.IsFallback()) { - continue; - } + for (const auto& c : groupings.TriggeringSegmentToConditions(s)) { if (!c.IsExclusive()) { dirty_segments_.union_set(c.TriggeringSegments()); } @@ -138,7 +131,7 @@ StatusOr AllPairsMerger::NormalizationDivisor( uint64_t total_bytes = 0; for (segment_index_t s : merged_segments) { for (const auto& c : groupings.TriggeringSegmentToConditions(s)) { - if (c.IsFallback() || (unicode_range_mode && !c.IsExclusive())) { + if (unicode_range_mode && !c.IsExclusive()) { continue; } auto it = conditions_and_glyphs.find(c); @@ -168,8 +161,8 @@ Status AllPairsMerger::QueueBestCaseCandidate( return absl::OkStatus(); } - auto best_case_delta = CandidateMerge::ComputeCostDelta( - merger, merged_segments, std::nullopt, 0.0); + auto best_case_delta = + CandidateMerge::ComputeCostDelta(merger, merged_segments, 0.0); if (absl::IsCancelled(best_case_delta.status())) { return absl::OkStatus(); } @@ -227,7 +220,7 @@ Status AllPairsMerger::QueueCompositeCandidates( last_exclusive); it != merger.Context().glyph_groupings.OrderedConditions().end(); ++it) { const auto& next_condition = *it; - if (next_condition.IsFallback() || next_condition.IsExclusive()) { + if (next_condition.IsExclusive()) { continue; } @@ -349,8 +342,7 @@ Status AllPairsMerger::Initialize(Merger& merger) { Status AllPairsMerger::RefreshDirtySegments(Merger& merger) { const auto& groupings = merger.Context().glyph_groupings; for (glyph_id_t gid : dirty_glyphs_) { - if (auto cond = groupings.GlyphToCondition(gid); - cond.has_value() && !cond->IsFallback()) { + if (auto cond = groupings.GlyphToCondition(gid); cond.has_value()) { dirty_segments_.union_set(cond->TriggeringSegments()); } } @@ -360,7 +352,7 @@ Status AllPairsMerger::RefreshDirtySegments(Merger& merger) { for (segment_index_t base : dirty_bases_) { dirty_segments_.insert(base); for (const auto& c : groupings.TriggeringSegmentToConditions(base)) { - if (!c.IsFallback() && !c.IsExclusive()) { + if (!c.IsExclusive()) { dirty_segments_.union_set(c.TriggeringSegments()); } } diff --git a/ift/encoder/candidate_merge.cc b/ift/encoder/candidate_merge.cc index b3ff016e..6ba214dc 100644 --- a/ift/encoder/candidate_merge.cc +++ b/ift/encoder/candidate_merge.cc @@ -106,39 +106,15 @@ StatusOr CandidateMerge::Apply(Merger& merger) { return InvalidationSet(base_segment_index_); } - // Upon application of this merger if all of the input segments were inert - // then it's likely that the new segment will also be inert (though not - // gauranteed). We can check for this case by running a closure analysis - // and checking there are no non-exclusive gids. If the input segments are - // inert and the new segment is inert then we can directly compute the result - // of the merge without needing to invalidate and reprocess. SegmentSet segments_to_merge_with_base = segments_to_merge_; segments_to_merge_with_base.insert(base_segment_index_); - bool new_segment_is_inert = false; - if (input_segments_are_inert_ && - !merger.Context().IsPureDepGraphAnalysisMode()) { - // Only need to compute the new segment inertness in non pure dep graph - // mode. In pure dep graph mode this will be recomputed directly from the - // graph after the merge is applied. - GlyphSet and_gids, or_gids, exclusive_gids; - TRYV(merger.Context().AnalyzeSegment(segments_to_merge_with_base, and_gids, - or_gids, exclusive_gids)); - new_segment_is_inert = (and_gids.empty() && or_gids.empty()); - if (new_segment_is_inert) { - // When the new segment is inert then invalidated glyphs is - // used to shortcut closure analysis and directly construct - // the new condition and glyph mappings, save all of the - // closure glyphs to it. - invalidated_glyphs_.union_set(exclusive_gids); - } - } const auto& segments = merger.Context().SegmentationInfo().Segments(); uint32_t size_before = segments[base_segment_index_].Definition().codepoints.size(); uint32_t size_after = merger.AssignMergedSegment(base_segment_index_, segments_to_merge_, - *merged_segment_, new_segment_is_inert); + *merged_segment_, input_segments_are_inert_); VLOG(0) << " Merged " << size_before << " codepoints up to " << size_after << " codepoints for segment " << base_segment_index_ << "." @@ -154,24 +130,6 @@ StatusOr CandidateMerge::Apply(Merger& merger) { TRYV(merger.Context().InvalidateGlyphInformationForMerge( invalidated_glyphs_, segments_to_merge_with_base, base_segment_index_)); - if (new_segment_is_inert) { - // The newly formed segment will be inert which means we can construct the - // new condition sets and glyph groupings here instead of using the - // closure analysis to do it. The new segment is simply the union of all - // glyphs associated with each segment that is part of the merge. - // (gid_conditons_to_update) - for (glyph_id_t gid : invalidated_glyphs_) { - merger.Context().glyph_condition_set.AddAndCondition(gid, - base_segment_index_); - } - TRYV(merger.Context().glyph_groupings.AddGlyphsToExclusiveGroup( - base_segment_index_, invalidated_glyphs_)); - - // We've now fully updated information for these glyphs so don't need to - // return them. - invalidated_glyphs_.clear(); - } - return InvalidationSet(invalidated_glyphs_, segments_to_merge_, base_segment_index_); } @@ -251,11 +209,6 @@ static Status FindModifiedConditions( for (auto s : merged_segments) { for (const auto& c : groupings.TriggeringSegmentToConditions(s)) { - if (c.IsFallback()) { - // Ignore fallback for this analysis. - continue; - } - auto glyph_it = conditions_and_glyphs.find(c); if (glyph_it == conditions_and_glyphs.end()) { return absl::InternalError( @@ -296,16 +249,8 @@ static flat_hash_map PatchesWithGlyphsOrSegments( const SegmentSet& segments) { // To more efficiently target our search we can use the glyph_condition_set to // locate conditions that intersect with gids. - GlyphSet fallback_glyphs = context.glyph_groupings.UnmappedGlyphs(); flat_hash_set conditions_of_interest; for (glyph_id_t gid : gids) { - if (fallback_glyphs.contains(gid)) { - // Fallback glyphs are handled separately at the end since the - // conditions in the glyph condition set associated with a fallback - // glyph are not accurate. - continue; - } - auto result = context.glyph_groupings.GlyphToCondition(gid); if (!result.has_value()) { continue; @@ -327,13 +272,6 @@ static flat_hash_map PatchesWithGlyphsOrSegments( context.glyph_groupings.ConditionsAndGlyphs().at(condition))); } - // We also need to check if there's a fallback patch and it intersects gids. - if (!fallback_glyphs.empty() && fallback_glyphs.intersects(gids)) { - ActivationCondition condition = - ActivationCondition::or_segments({}, 0, true); - result.insert(std::make_pair(condition, fallback_glyphs)); - } - return result; } @@ -389,10 +327,8 @@ static StatusOr CostFor(Merger& merger, const freq::ProbabilityCalculator* calculator, const ActivationCondition& condition, const GlyphSet& glyphs) { - double probability = 1.0; - if (!condition.IsFallback()) { - probability = TRY(condition.Probability(merger.Context().SegmentationInfo().Segments(), *calculator)); - } + double probability = TRY(condition.Probability( + merger.Context().SegmentationInfo().Segments(), *calculator)); double patch_size = TRY(merger.Context().patch_size_cache_for_init_font->GetPatchSize( @@ -524,13 +460,10 @@ StatusOr> CandidateMerge::ComputeInitFontCostDelta( } for (const auto& [condition, glyphs] : new_conditions) { - double patch_probability_after = 1.0; - if (!condition.IsFallback()) { - // TODO(garretrieger): XXXX also include the effect of modified segments - // in this calc. Start with finding a test case. - patch_probability_after = - TRY(condition.Probability(merger.Context().SegmentationInfo().Segments(), *calculator)); - } + // TODO(garretrieger): XXXX also include the effect of modified segments + // in this calc. Start with finding a test case. + double patch_probability_after = TRY(condition.Probability( + merger.Context().SegmentationInfo().Segments(), *calculator)); double patch_size_after = TRY(merger.Context().patch_size_cache_for_init_font->GetPatchSize( @@ -576,11 +509,8 @@ StatusOr CandidateMerge::ComputeBestCaseInitFontCostDelta( merger.Context(), glyph_closure_delta, SegmentSet{}); deltas.reserve(affected_conditions.size()); for (const auto& [condition, glyphs] : affected_conditions) { - double patch_probability = 1.0; - if (!condition.IsFallback()) { - patch_probability = - TRY(condition.Probability(merger.Context().SegmentationInfo().Segments(), *calculator)); - } + double patch_probability = TRY(condition.Probability( + merger.Context().SegmentationInfo().Segments(), *calculator)); GlyphSet new_glyphs = glyphs; new_glyphs.subtract(glyph_closure_delta); @@ -641,7 +571,6 @@ template static StatusOr ComputeInertPairCostDelta( Merger& merger, const SegmentSet& merged_segments, segment_index_t u, segment_index_t v, uint32_t size_u, uint32_t size_v, - const std::optional& exclusive_gids, std::optional delta_to_beat) { const auto& context = merger.Context(); const auto& patch_size_cache = context.patch_size_cache; @@ -667,26 +596,13 @@ static StatusOr ComputeInertPairCostDelta( Merger::BEST_CASE_MERGE_SIZE_DELTA); size = largest_patch_size + extra; } else { - const GlyphSet* glyphs_for_size = nullptr; - GlyphSet combined_glyphs; - const auto& glyphs_u = context.glyph_groupings.ExclusiveGlyphs(u); - const auto& glyphs_v = context.glyph_groupings.ExclusiveGlyphs(v); - if (exclusive_gids.has_value() && glyphs_u.is_subset_of(*exclusive_gids) && - glyphs_v.is_subset_of(*exclusive_gids)) { - glyphs_for_size = &*exclusive_gids; - } else { - combined_glyphs = glyphs_u; - combined_glyphs.union_set(glyphs_v); - if (exclusive_gids.has_value()) { - combined_glyphs.union_set(*exclusive_gids); - } - glyphs_for_size = &combined_glyphs; - } - if (glyphs_for_size->empty()) { + GlyphSet combined_glyphs = context.glyph_groupings.ExclusiveGlyphs(u); + combined_glyphs.union_set(context.glyph_groupings.ExclusiveGlyphs(v)); + if (combined_glyphs.empty()) { std::array deltas = {-d_u, -d_v}; return SumDeltas(absl::MakeSpan(deltas)); } - size = TRY(patch_size_cache->GetPatchSize(*glyphs_for_size)); + size = TRY(patch_size_cache->GetPatchSize(combined_glyphs)); if (merger.ShouldRecordMergedSizeReductions() && extra_raw != 0) { int32_t extra_actual = ((int32_t)size) - largest_patch_size; merger.RecordMergedSizeReduction((double)extra_actual / @@ -738,7 +654,7 @@ static StatusOr ComputeInertPairCostDelta( template StatusOr CandidateMerge::ComputeCostDelta( Merger& merger, const SegmentSet& merged_segments, - std::optional exclusive_gids, std::optional delta_to_beat) { + std::optional delta_to_beat) { if (merged_segments.size() <= 1) { return 0; } @@ -756,13 +672,9 @@ StatusOr CandidateMerge::ComputeCostDelta( context.InertSegments().contains(v)) { uint32_t size_u = TRY(merger.ExclusivePatchSize(u)); uint32_t size_v = TRY(merger.ExclusivePatchSize(v)); - if (size_u > 0 && size_v > 0 && - (best_case || unicode_range_mode || !exclusive_gids.has_value() || - !context.glyph_groupings.UnmappedGlyphs().intersects( - *exclusive_gids))) { + if (size_u > 0 && size_v > 0) { return ComputeInertPairCostDelta( - merger, merged_segments, u, v, size_u, size_v, exclusive_gids, - delta_to_beat); + merger, merged_segments, u, v, size_u, size_v, delta_to_beat); } } } @@ -786,7 +698,6 @@ StatusOr CandidateMerge::ComputeCostDelta( const ActivationCondition* first_condition = nullptr; const GlyphSet* first_glyphs = nullptr; uint32_t source_count = 0; - bool glyph_set_unchanged = false; bool skip_new_condition = false; GlyphSet glyphs; uint32_t largest_patch_size = 0; @@ -807,7 +718,6 @@ StatusOr CandidateMerge::ComputeCostDelta( deltas.reserve(2 * modified_conditions.size() + 1); double current_delta = 0.0; - bool has_unitary = false; const uint32_t per_request_overhead = merger.Strategy().NetworkOverheadCost(); auto add_removed_condition = [&](const ActivationCondition* condition, @@ -824,7 +734,7 @@ StatusOr CandidateMerge::ComputeCostDelta( << "]"; } - if (!best_case && !info.glyph_set_unchanged) { + if (!best_case && info.source_count > 1) { info.glyphs.union_set(*glyphs); } @@ -851,9 +761,6 @@ StatusOr CandidateMerge::ComputeCostDelta( continue; } - if (!best_case && updated.IsUnitary()) { - has_unitary = true; - } auto [it, did_insert] = new_conditions.try_emplace(std::move(updated)); auto& info = it->second; @@ -864,41 +771,25 @@ StatusOr CandidateMerge::ComputeCostDelta( continue; } - if (info.source_count == 1) { + info.source_count++; + if (info.source_count == 2) { + // The previously collected first condition needs to be removed now + // that it's merging with this condition. TRYV(add_removed_condition(info.first_condition, info.first_glyphs, info)); } - info.source_count++; TRYV(add_removed_condition(condition, glyphs, info)); } - if (unicode_range_mode && !best_case && exclusive_gids.has_value() && - !exclusive_gids->empty()) { - new_conditions.try_emplace(ActivationCondition::or_segments({base}, 0)); - } - for (auto& [c, info] : new_conditions) { if (info.source_count != 1) { continue; } bool was_active_before_merge = (!unicode_range_mode || info.first_condition->IsExclusive()); - bool glyph_set_unchanged = true; - if (!best_case && exclusive_gids.has_value()) { - // Check if the provided exclusive gids will cause a change to this condition's glyphs. - // exclusive_gids are added to the unitary condition for this merge, and subtracted - // from everything else. - if (c.IsUnitary()) { - glyph_set_unchanged = exclusive_gids->is_subset_of(*info.first_glyphs); - } else { - glyph_set_unchanged = !info.first_glyphs->intersects(*exclusive_gids); - } - } - info.glyph_set_unchanged = glyph_set_unchanged; - if (glyph_set_unchanged && - (best_case || - (was_active_before_merge && - info.first_condition->IsUnchangedByMerge(merged_segments)))) { + if (best_case || + (was_active_before_merge && + info.first_condition->IsUnchangedByMerge(merged_segments))) { info.skip_new_condition = true; continue; } @@ -906,30 +797,6 @@ StatusOr CandidateMerge::ComputeCostDelta( } double initial_magnitude_sum = -current_delta; - - bool fallback_changed = - !unicode_range_mode && !best_case && has_unitary && - exclusive_gids.has_value() && - context.glyph_groupings.UnmappedGlyphs().intersects(*exclusive_gids); - if (fallback_changed) { - GlyphSet new_fallback = context.glyph_groupings.UnmappedGlyphs(); - new_fallback.subtract(*exclusive_gids); - - // Fallback is always needed with 100% probability so delta is just the size - // difference (new - old). Since the fallback is needed with 100% - // probability under every calculator, it's aggregate probability is the - // number of probability profiles. - double fallback_probability = merger.MaxAggregateProbability(); - double diff = (((double)TRY(patch_size_cache->GetPatchSize(new_fallback))) - - ((double)TRY(patch_size_cache->GetPatchSize( - context.glyph_groupings.UnmappedGlyphs())))) * - fallback_probability; - VLOG(1) << " + " << diff << " [fallback delta]"; - deltas.push_back(diff); - current_delta += diff; - initial_magnitude_sum += std::abs(diff); - } - double initial_delta = current_delta; // Add in cost associated within the new conditions patch pairs. @@ -947,24 +814,9 @@ StatusOr CandidateMerge::ComputeCostDelta( extra = std::max((uint32_t)(extra * best_case_reduction_fraction), Merger::BEST_CASE_MERGE_SIZE_DELTA); size = info.largest_patch_size + extra; - } else if (info.glyph_set_unchanged) { + } else if (info.source_count == 1) { size = info.largest_patch_size; } else { - if (exclusive_gids.has_value()) { - // When we do not have complete glyph conditions then exclusive_gids - // provides a hint at which glyphs will end up in the exclusive patch - // after the merge, incorporate them into the size calculation here. - // Remove them from all other patches and place them into the exclusive - // patch. - info.glyphs.subtract(*exclusive_gids); - if (c.IsUnitary()) { - info.glyphs.union_set(*exclusive_gids); - } - if (info.glyphs.empty()) { - // Patch has no remaining glyphs, so don't add back any cost for it. - continue; - } - } size = TRY(patch_size_cache->GetPatchSize(info.glyphs)); if (merger.ShouldRecordMergedSizeReductions()) { int32_t extra_raw = @@ -1004,11 +856,9 @@ StatusOr CandidateMerge::ComputeCostDelta( template StatusOr CandidateMerge::ComputeCostDelta( Merger& merger, const SegmentSet& merged_segments, - std::optional exclusive_gids, std::optional delta_to_beat); template StatusOr CandidateMerge::ComputeCostDelta( Merger& merger, const SegmentSet& merged_segments, - std::optional exclusive_gids, std::optional delta_to_beat); StatusOr @@ -1179,17 +1029,11 @@ StatusOr CandidateMerge::PatchMergeDetails::ComputePatchMergeCostDelta( return cost_delta; } -static StatusOr ExceedsMaxPatchSize( - const Merger& merger, segment_index_t base_segment_index, - const SegmentSet& merged_segments, - const std::optional& exclusive_gids, uint32_t& new_patch_size) { +static StatusOr ExceedsMaxPatchSize(const Merger& merger, + segment_index_t base_segment_index, + const SegmentSet& merged_segments, + uint32_t& new_patch_size) { uint32_t max_patch_size = merger.Strategy().PatchSizeMaxBytes(); - if (exclusive_gids.has_value() && new_patch_size > max_patch_size) { - VLOG(0) << " Merged exclusive patch for " << base_segment_index - << " is too big (" << new_patch_size << " > " << max_patch_size - << "), skipping."; - return true; - } flat_hash_map modified_conditions; @@ -1215,18 +1059,16 @@ static StatusOr ExceedsMaxPatchSize( } const auto& patch_size_cache = merger.Context().patch_size_cache; - if (!exclusive_gids.has_value()) { - auto unitary_it = new_conditions.find( - ActivationCondition::or_segments({base_segment_index}, 0)); - if (unitary_it != new_conditions.end()) { - new_patch_size = - TRY(patch_size_cache->GetPatchSize(unitary_it->second.glyphs)); - if (new_patch_size > max_patch_size) { - VLOG(0) << " Merged exclusive patch for " << base_segment_index - << " is too big (" << new_patch_size << " > " << max_patch_size - << "), skipping."; - return true; - } + auto unitary_it = new_conditions.find( + ActivationCondition::or_segments({base_segment_index}, 0)); + if (unitary_it != new_conditions.end()) { + new_patch_size = + TRY(patch_size_cache->GetPatchSize(unitary_it->second.glyphs)); + if (new_patch_size > max_patch_size) { + VLOG(0) << " Merged exclusive patch for " << base_segment_index + << " is too big (" << new_patch_size << " > " << max_patch_size + << "), skipping."; + return true; } } @@ -1235,10 +1077,6 @@ static StatusOr ExceedsMaxPatchSize( continue; } - if (exclusive_gids.has_value()) { - entry.glyphs.subtract(*exclusive_gids); - } - if (entry.glyphs.size() <= entry.max_input_glyph_count) { // This non-exclusive condition did not grow in glyph count as a result // of the merge. @@ -1303,7 +1141,7 @@ StatusOr> CandidateMerge::AssessSegmentMerge( // If that doesn't beat the current smallest there's no need to // do more indepth analysis. auto best_case_delta_status = ComputeCostDelta( - merger, segments_to_merge_with_base, std::nullopt, delta_to_beat); + merger, segments_to_merge_with_base, delta_to_beat); if (absl::IsCancelled(best_case_delta_status.status())) { // Delta computation was ended early since it couldn't beat // best_merge_candidate. @@ -1322,44 +1160,16 @@ StatusOr> CandidateMerge::AssessSegmentMerge( // we need to update those (and the down stream and/or glyph groups) to // reflect the removal/change and allow recalculation during the // GroupGlyphs steps - // - // Changes caused by adding new segments into the base segment will be - // handled by the next AnalyzeSegment step. gid_conditions_to_update.union_set( merger.Context().glyph_condition_set.GlyphsWithSegment(segment_index)); } uint32_t new_patch_size = 0; - std::optional exclusive_gids; - if (!merger.Context().IsPureDepGraphAnalysisMode()) { - if (!segments_to_merge_are_inert) { - // When we're not in pure depgraph mode then glyph conditions are - // incomplete. To help improve accuracy run a closure to find the new set - // of exclusive glyphs associated with the merge and incorporate that into - // the cost calculations. - GlyphSet and_gids, or_gids; - exclusive_gids = GlyphSet{}; - TRYV(merger.Context().AnalyzeSegment(segments_to_merge_with_base, - and_gids, or_gids, *exclusive_gids)); - } else { - // When the segments being added to base are all inert then we can assume - // the merged patch is just a combination of the glyphs from the base and - // the input segments. Since the segments being added are intert we know - // that they won't interact with base and will just bring along their own - // glyphs. - exclusive_gids = gid_conditions_to_update; - exclusive_gids->union_set( - merger.Context().glyph_groupings.ExclusiveGlyphs(base_segment_index)); - } - new_patch_size = - TRY(merger.Context().patch_size_cache->GetPatchSize(*exclusive_gids)); - } - double cost_delta = 0.0; if (merger.Strategy().UseCosts()) { // Cost delta values are only needed when using cost based merge strategy. auto cost_delta_status = ComputeCostDelta( - merger, segments_to_merge_with_base, exclusive_gids, delta_to_beat); + merger, segments_to_merge_with_base, delta_to_beat); if (absl::IsCancelled(cost_delta_status.status())) { // Delta computation was ended early since it couldn't beat // best_merge_candidate. @@ -1367,8 +1177,8 @@ StatusOr> CandidateMerge::AssessSegmentMerge( } cost_delta = TRY(cost_delta_status); } else if (TRY(ExceedsMaxPatchSize(merger, base_segment_index, - segments_to_merge_with_base, exclusive_gids, - new_patch_size))) { + segments_to_merge_with_base, + new_patch_size))) { return std::nullopt; } diff --git a/ift/encoder/candidate_merge.h b/ift/encoder/candidate_merge.h index d1d623c9..617dca1a 100644 --- a/ift/encoder/candidate_merge.h +++ b/ift/encoder/candidate_merge.h @@ -138,10 +138,6 @@ struct CandidateMerge { // Computes the predicted change to the total cost if merged_segments // are joined together into a new segment. // - // exclusive_gids is an optional hint that specifies which glyphs - // will end up in the exclusive patch after merged_segments are merged - // together. - // // If best_case is true then this will compute an estimated best possible cost // delta from the merge (computationally cheap) instead of the real delta. // @@ -150,7 +146,6 @@ struct CandidateMerge { template static absl::StatusOr ComputeCostDelta( Merger& merger, const ift::common::SegmentSet& merged_segments, - std::optional exclusive_gids, std::optional delta_to_beat); // Computes the predicted change to the toal cost if moved_glyphs are diff --git a/ift/encoder/candidate_merge_test.cc b/ift/encoder/candidate_merge_test.cc index a925c785..d9919de5 100644 --- a/ift/encoder/candidate_merge_test.cc +++ b/ift/encoder/candidate_merge_test.cc @@ -22,10 +22,8 @@ using absl::flat_hash_map; -using ift::config::CLOSURE_ONLY; using ift::config::DEP_GRAPH_ONLY; using ift::config::DEP_GRAPH_ONLY_WITH_SIMPLIFICATION; -using ift::config::PATCH; using ift::common::BazelDataFileResolver; using ift::common::CodepointSet; @@ -134,11 +132,11 @@ TEST_F(CandidateMergeTest, AssessMerge_CostDeltas) { auto probability_calculator = std::make_unique(segment_probabilities); - ClosureGlyphSegmenter segmenter(8, 8, PATCH, CLOSURE_ONLY, resolver); + ClosureGlyphSegmenter segmenter(8, 8, DEP_GRAPH_ONLY, resolver); auto context = SegmentationContext::InitializeSegmentationContext( - roboto.get(), {}, segments, segmenter.unmapped_glyph_handling(), - segmenter.condition_analysis_mode(), segmenter.brotli_quality(), - segmenter.init_font_merging_brotli_quality(), resolver); + roboto.get(), {}, segments, segmenter.condition_analysis_mode(), + segmenter.brotli_quality(), segmenter.init_font_merging_brotli_quality(), + resolver); ASSERT_TRUE(context.ok()) << context.status(); Merger merger = *Merger::New( @@ -203,11 +201,11 @@ TEST_F(CandidateMergeTest, AssessMerge_WithBestCandidate) { auto probability_calculator = std::make_unique(segment_probabilities); - ClosureGlyphSegmenter segmenter(8, 8, PATCH, CLOSURE_ONLY, resolver); + ClosureGlyphSegmenter segmenter(8, 8, DEP_GRAPH_ONLY, resolver); auto context = SegmentationContext::InitializeSegmentationContext( - roboto.get(), {}, segments, segmenter.unmapped_glyph_handling(), - segmenter.condition_analysis_mode(), segmenter.brotli_quality(), - segmenter.init_font_merging_brotli_quality(), resolver); + roboto.get(), {}, segments, segmenter.condition_analysis_mode(), + segmenter.brotli_quality(), segmenter.init_font_merging_brotli_quality(), + resolver); ASSERT_TRUE(context.ok()) << context.status(); Merger merger = *Merger::New( @@ -263,11 +261,11 @@ TEST_F(CandidateMergeTest, AssessMerge_CostDeltas_Complex) { {{'i'}}, }; - ClosureGlyphSegmenter segmenter(8, 8, PATCH, CLOSURE_ONLY, resolver); + ClosureGlyphSegmenter segmenter(8, 8, DEP_GRAPH_ONLY, resolver); auto context = SegmentationContext::InitializeSegmentationContext( - roboto.get(), {}, segments, segmenter.unmapped_glyph_handling(), - segmenter.condition_analysis_mode(), segmenter.brotli_quality(), - segmenter.init_font_merging_brotli_quality(), resolver); + roboto.get(), {}, segments, segmenter.condition_analysis_mode(), + segmenter.brotli_quality(), segmenter.init_font_merging_brotli_quality(), + resolver); ASSERT_TRUE(context.ok()) << context.status(); Merger merger = *Merger::New( @@ -319,11 +317,11 @@ TEST_F(CandidateMergeTest, AssessMerge_CostDeltas_Complex_ModifiedConditions) { freq::UnicodeFrequencies frequencies{ {{' ', ' '}, 100}, {{'a', 'a'}, 50}, {{'f', 'f'}, 75}, {{'i', 'i'}, 95}}; - ClosureGlyphSegmenter segmenter(8, 8, PATCH, CLOSURE_ONLY, resolver); + ClosureGlyphSegmenter segmenter(8, 8, DEP_GRAPH_ONLY, resolver); auto context = SegmentationContext::InitializeSegmentationContext( - roboto.get(), {}, segments, segmenter.unmapped_glyph_handling(), - segmenter.condition_analysis_mode(), segmenter.brotli_quality(), - segmenter.init_font_merging_brotli_quality(), resolver); + roboto.get(), {}, segments, segmenter.condition_analysis_mode(), + segmenter.brotli_quality(), segmenter.init_font_merging_brotli_quality(), + resolver); ASSERT_TRUE(context.ok()) << context.status(); Merger merger = *Merger::New( @@ -394,11 +392,11 @@ TEST_F(CandidateMergeTest, AssessMerge_CostDeltas_MultipleCalculators) { strategy.AddProbabilityProfile(ProbabilityProfile( std::make_shared(probabilities_2))); - ClosureGlyphSegmenter segmenter(8, 8, PATCH, CLOSURE_ONLY, resolver); + ClosureGlyphSegmenter segmenter(8, 8, DEP_GRAPH_ONLY, resolver); auto context = SegmentationContext::InitializeSegmentationContext( - roboto.get(), {}, segments, segmenter.unmapped_glyph_handling(), - segmenter.condition_analysis_mode(), segmenter.brotli_quality(), - segmenter.init_font_merging_brotli_quality(), resolver); + roboto.get(), {}, segments, segmenter.condition_analysis_mode(), + segmenter.brotli_quality(), segmenter.init_font_merging_brotli_quality(), + resolver); ASSERT_TRUE(context.ok()) << context.status(); Merger merger = *Merger::New(*context, strategy, all, all); @@ -461,11 +459,11 @@ TEST_F(CandidateMergeTest, AssessMerge_CostDeltas_UnrelatedCalculator) { strategy.AddProbabilityProfile(ProbabilityProfile( std::make_shared(probabilities_2))); - ClosureGlyphSegmenter segmenter(8, 8, PATCH, CLOSURE_ONLY, resolver); + ClosureGlyphSegmenter segmenter(8, 8, DEP_GRAPH_ONLY, resolver); auto context = SegmentationContext::InitializeSegmentationContext( - roboto.get(), {}, segments, segmenter.unmapped_glyph_handling(), - segmenter.condition_analysis_mode(), segmenter.brotli_quality(), - segmenter.init_font_merging_brotli_quality(), resolver); + roboto.get(), {}, segments, segmenter.condition_analysis_mode(), + segmenter.brotli_quality(), segmenter.init_font_merging_brotli_quality(), + resolver); ASSERT_TRUE(context.ok()) << context.status(); Merger merger = *Merger::New(*context, strategy, all, all); @@ -532,11 +530,11 @@ TEST_F(CandidateMergeTest, AssessPatchMerge) { auto probability_calculator = std::make_unique(segment_probabilities); - ClosureGlyphSegmenter segmenter(8, 8, PATCH, CLOSURE_ONLY, resolver); + ClosureGlyphSegmenter segmenter(8, 8, DEP_GRAPH_ONLY, resolver); auto context = SegmentationContext::InitializeSegmentationContext( - roboto.get(), {}, segments, segmenter.unmapped_glyph_handling(), - segmenter.condition_analysis_mode(), segmenter.brotli_quality(), - segmenter.init_font_merging_brotli_quality(), resolver); + roboto.get(), {}, segments, segmenter.condition_analysis_mode(), + segmenter.brotli_quality(), segmenter.init_font_merging_brotli_quality(), + resolver); ASSERT_TRUE(context.ok()) << context.status(); Merger merger = *Merger::New( @@ -590,11 +588,11 @@ TEST_F(CandidateMergeTest, AssessPatchMerge_RequiresPatches) { auto probability_calculator = std::make_unique(segment_probabilities); - ClosureGlyphSegmenter segmenter(8, 8, PATCH, CLOSURE_ONLY, resolver); + ClosureGlyphSegmenter segmenter(8, 8, DEP_GRAPH_ONLY, resolver); auto context = SegmentationContext::InitializeSegmentationContext( - roboto.get(), {}, segments, segmenter.unmapped_glyph_handling(), - segmenter.condition_analysis_mode(), segmenter.brotli_quality(), - segmenter.init_font_merging_brotli_quality(), resolver); + roboto.get(), {}, segments, segmenter.condition_analysis_mode(), + segmenter.brotli_quality(), segmenter.init_font_merging_brotli_quality(), + resolver); ASSERT_TRUE(context.ok()) << context.status(); Merger merger = *Merger::New( @@ -635,11 +633,11 @@ TEST_F(CandidateMergeTest, AssessPatchMerge_NonDisjunctive) { MockPatchSizeCache* size_cache = new MockPatchSizeCache(); - ClosureGlyphSegmenter segmenter(8, 8, PATCH, CLOSURE_ONLY, resolver); + ClosureGlyphSegmenter segmenter(8, 8, DEP_GRAPH_ONLY, resolver); auto context = SegmentationContext::InitializeSegmentationContext( - roboto.get(), {}, segments, segmenter.unmapped_glyph_handling(), - segmenter.condition_analysis_mode(), segmenter.brotli_quality(), - segmenter.init_font_merging_brotli_quality(), resolver); + roboto.get(), {}, segments, segmenter.condition_analysis_mode(), + segmenter.brotli_quality(), segmenter.init_font_merging_brotli_quality(), + resolver); ASSERT_TRUE(context.ok()) << context.status(); Merger merger = *Merger::New( @@ -664,10 +662,9 @@ TEST_F(CandidateMergeTest, AssessPatchMerge_NonDisjunctive) { context->glyph_condition_set.AddAndCondition(gid_A, 0); // Update glyph groupings. - auto sc = context->glyph_groupings.GroupGlyphs( - context->SegmentationInfo(), context->glyph_condition_set, - *context->glyph_closure_cache, std::nullopt, {gid_A, gid_B}, {0, 1, 2}, - false); + auto sc = context->glyph_groupings.GroupGlyphs(context->SegmentationInfo(), + context->glyph_condition_set, + {gid_A, gid_B}, {0, 1, 2}); ASSERT_TRUE(sc.ok()) << sc; context->patch_size_cache.reset(size_cache); @@ -716,11 +713,11 @@ TEST_F(CandidateMergeTest, AssessPatchMerge_MultipleCalculators) { MockPatchSizeCache* size_cache = new MockPatchSizeCache(); - ClosureGlyphSegmenter segmenter(8, 8, PATCH, CLOSURE_ONLY, resolver); + ClosureGlyphSegmenter segmenter(8, 8, DEP_GRAPH_ONLY, resolver); auto context = SegmentationContext::InitializeSegmentationContext( - roboto.get(), {}, segments, segmenter.unmapped_glyph_handling(), - segmenter.condition_analysis_mode(), segmenter.brotli_quality(), - segmenter.init_font_merging_brotli_quality(), resolver); + roboto.get(), {}, segments, segmenter.condition_analysis_mode(), + segmenter.brotli_quality(), segmenter.init_font_merging_brotli_quality(), + resolver); ASSERT_TRUE(context.ok()) << context.status(); Merger merger = *Merger::New(*context, strategy, all, all); @@ -741,10 +738,9 @@ TEST_F(CandidateMergeTest, AssessPatchMerge_MultipleCalculators) { context->glyph_condition_set.AddAndCondition(gid_A, 0); // Update glyph groupings. - auto sc = context->glyph_groupings.GroupGlyphs( - context->SegmentationInfo(), context->glyph_condition_set, - *context->glyph_closure_cache, std::nullopt, {gid_A, gid_B}, {0, 1, 2}, - false); + auto sc = context->glyph_groupings.GroupGlyphs(context->SegmentationInfo(), + context->glyph_condition_set, + {gid_A, gid_B}, {0, 1, 2}); ASSERT_TRUE(sc.ok()) << sc; context->patch_size_cache.reset(size_cache); @@ -786,12 +782,12 @@ TEST_F(CandidateMergeTest, AssessPatchMerge_NonDisjunctive_WithSimplification) { MockPatchSizeCache* size_cache = new MockPatchSizeCache(); - ClosureGlyphSegmenter segmenter(8, 8, PATCH, - DEP_GRAPH_ONLY_WITH_SIMPLIFICATION, resolver); + ClosureGlyphSegmenter segmenter(8, 8, DEP_GRAPH_ONLY_WITH_SIMPLIFICATION, + resolver); auto context = SegmentationContext::InitializeSegmentationContext( - roboto.get(), {}, segments, segmenter.unmapped_glyph_handling(), - segmenter.condition_analysis_mode(), segmenter.brotli_quality(), - segmenter.init_font_merging_brotli_quality(), resolver); + roboto.get(), {}, segments, segmenter.condition_analysis_mode(), + segmenter.brotli_quality(), segmenter.init_font_merging_brotli_quality(), + resolver); ASSERT_TRUE(context.ok()) << context.status(); Merger merger = *Merger::New( @@ -816,10 +812,9 @@ TEST_F(CandidateMergeTest, AssessPatchMerge_NonDisjunctive_WithSimplification) { context->glyph_condition_set.AddAndCondition(gid_A, 0); // Update glyph groupings. - auto sc = context->glyph_groupings.GroupGlyphs( - context->SegmentationInfo(), context->glyph_condition_set, - *context->glyph_closure_cache, std::nullopt, {gid_A, gid_B}, {0, 1, 2}, - false); + auto sc = context->glyph_groupings.GroupGlyphs(context->SegmentationInfo(), + context->glyph_condition_set, + {gid_A, gid_B}, {0, 1, 2}); ASSERT_TRUE(sc.ok()) << sc; context->patch_size_cache.reset(size_cache); @@ -866,11 +861,11 @@ TEST_F(CandidateMergeTest, ComputeInitFontCostDelta) { auto probability_calculator = std::make_unique(segment_probabilities); - ClosureGlyphSegmenter segmenter(8, 8, PATCH, CLOSURE_ONLY, resolver); + ClosureGlyphSegmenter segmenter(8, 8, DEP_GRAPH_ONLY, resolver); auto context = SegmentationContext::InitializeSegmentationContext( - roboto.get(), {'a'}, segments, segmenter.unmapped_glyph_handling(), - segmenter.condition_analysis_mode(), segmenter.brotli_quality(), - segmenter.init_font_merging_brotli_quality(), resolver); + roboto.get(), {'a'}, segments, segmenter.condition_analysis_mode(), + segmenter.brotli_quality(), segmenter.init_font_merging_brotli_quality(), + resolver); ASSERT_TRUE(context.ok()) << context.status(); Merger merger = *Merger::New( @@ -1010,11 +1005,11 @@ TEST_F(CandidateMergeTest, ComputeInitFontCostDelta_TracksSmallestDelta) { auto probability_calculator = std::make_unique(segment_probabilities); - ClosureGlyphSegmenter segmenter(8, 8, PATCH, CLOSURE_ONLY, resolver); + ClosureGlyphSegmenter segmenter(8, 8, DEP_GRAPH_ONLY, resolver); auto context = SegmentationContext::InitializeSegmentationContext( - roboto.get(), {'a'}, segments, segmenter.unmapped_glyph_handling(), - segmenter.condition_analysis_mode(), segmenter.brotli_quality(), - segmenter.init_font_merging_brotli_quality(), resolver); + roboto.get(), {'a'}, segments, segmenter.condition_analysis_mode(), + segmenter.brotli_quality(), segmenter.init_font_merging_brotli_quality(), + resolver); ASSERT_TRUE(context.ok()) << context.status(); Merger merger = *Merger::New( @@ -1076,11 +1071,11 @@ TEST_F(CandidateMergeTest, ComputeCostDelta_EarlyExitAndBoundary) { freq::UnicodeFrequencies frequencies{ {{' ', ' '}, 100}, {{'a', 'a'}, 50}, {{'f', 'f'}, 75}, {{'i', 'i'}, 95}}; - ClosureGlyphSegmenter segmenter(8, 8, PATCH, CLOSURE_ONLY, resolver); + ClosureGlyphSegmenter segmenter(8, 8, DEP_GRAPH_ONLY, resolver); auto context = SegmentationContext::InitializeSegmentationContext( - roboto.get(), {}, segments, segmenter.unmapped_glyph_handling(), - segmenter.condition_analysis_mode(), segmenter.brotli_quality(), - segmenter.init_font_merging_brotli_quality(), resolver); + roboto.get(), {}, segments, segmenter.condition_analysis_mode(), + segmenter.brotli_quality(), segmenter.init_font_merging_brotli_quality(), + resolver); ASSERT_TRUE(context.ok()) << context.status(); Merger merger = *Merger::New( @@ -1096,32 +1091,30 @@ TEST_F(CandidateMergeTest, ComputeCostDelta_EarlyExitAndBoundary) { size_cache->SetPatchSize({69, 74}, 450); context->patch_size_cache.reset(size_cache); - GlyphSet exclusive_gids = {69, 74}; - auto exact_false = CandidateMerge::ComputeCostDelta( - merger, {0, 1}, exclusive_gids, std::nullopt); + auto exact_false = + CandidateMerge::ComputeCostDelta(merger, {0, 1}, std::nullopt); ASSERT_TRUE(exact_false.ok()) << exact_false.status(); - auto exact_true = CandidateMerge::ComputeCostDelta( - merger, {0, 1}, std::nullopt, std::nullopt); + auto exact_true = + CandidateMerge::ComputeCostDelta(merger, {0, 1}, std::nullopt); ASSERT_TRUE(exact_true.ok()) << exact_true.status(); // Well below the actual delta: should cancel early. auto cancelled_false = CandidateMerge::ComputeCostDelta( - merger, {0, 1}, exclusive_gids, *exact_false - 10.0); + merger, {0, 1}, *exact_false - 10.0); EXPECT_TRUE(absl::IsCancelled(cancelled_false.status())); auto cancelled_true = CandidateMerge::ComputeCostDelta( - merger, {0, 1}, std::nullopt, *exact_true - 10.0); + merger, {0, 1}, *exact_true - 10.0); EXPECT_TRUE(absl::IsCancelled(cancelled_true.status())); // Within the FP rounding margin of the actual delta (e.g. equal to or 1 ULP // below): should NOT cancel early, and should return the exact sorted sum. auto boundary_false = CandidateMerge::ComputeCostDelta( - merger, {0, 1}, exclusive_gids, - std::nextafter(*exact_false, -INFINITY)); + merger, {0, 1}, std::nextafter(*exact_false, -INFINITY)); ASSERT_TRUE(boundary_false.ok()) << boundary_false.status(); EXPECT_EQ(*boundary_false, *exact_false); auto boundary_true = CandidateMerge::ComputeCostDelta( - merger, {0, 1}, std::nullopt, std::nextafter(*exact_true, -INFINITY)); + merger, {0, 1}, std::nextafter(*exact_true, -INFINITY)); ASSERT_TRUE(boundary_true.ok()) << boundary_true.status(); EXPECT_EQ(*boundary_true, *exact_true); @@ -1140,61 +1133,6 @@ TEST_F(CandidateMergeTest, ComputeCostDelta_EarlyExitAndBoundary) { EXPECT_EQ((*better_by_1_ulp)->CostDelta(), *exact_false); } -TEST_F(CandidateMergeTest, ComputeCostDelta_FallbackChangedBeforeEarlyExit) { - std::vector segments = { - {{0x54}}, // s0 -> gid 56 - {{0x6C}}, // s1 -> gid 80 - {{0x13C}}, // s2 - {{0x21A}}, // s3 - {{0xF6C3}}, // s4 - }; - freq::UnicodeFrequencies frequencies{ - {{' ', ' '}, 100}, {{0x54, 0x54}, 50}, {{0x6C, 0x6C}, 75}, - {{0x13C, 0x13C}, 10}, {{0x21A, 0x21A}, 10}, {{0xF6C3, 0xF6C3}, 10}}; - - ClosureGlyphSegmenter segmenter(8, 8, PATCH, CLOSURE_ONLY, resolver); - auto context = SegmentationContext::InitializeSegmentationContext( - roboto.get(), {}, segments, segmenter.unmapped_glyph_handling(), - segmenter.condition_analysis_mode(), segmenter.brotli_quality(), - segmenter.init_font_merging_brotli_quality(), resolver); - ASSERT_TRUE(context.ok()) << context.status(); - - Merger merger = *Merger::New( - *context, - MergeStrategy::CostBased( - *ProbabilityProfile::Unigram(std::move(frequencies)), 75, 4), - all, all); - - // Include an unmapped glyph in exclusive_gids so fallback_changed is - // triggered and reduces the fallback patch size by 1000 bytes. - GlyphSet old_fallback = context->glyph_groupings.UnmappedGlyphs(); - ASSERT_FALSE(old_fallback.empty()); - glyph_id_t unmapped_gid = *old_fallback.min(); - GlyphSet exclusive_gids = {56, 80, unmapped_gid}; - GlyphSet new_fallback = old_fallback; - new_fallback.subtract(exclusive_gids); - - MockPatchSizeCache* size_cache = new MockPatchSizeCache(); - size_cache->SetPatchSize({56}, 200); - size_cache->SetPatchSize({80}, 300); - size_cache->SetPatchSize(exclusive_gids, 450); - size_cache->SetPatchSize(old_fallback, 2000); - size_cache->SetPatchSize(new_fallback, 1000); - context->patch_size_cache.reset(size_cache); - - auto exact_with_fallback = CandidateMerge::ComputeCostDelta( - merger, {0, 1}, exclusive_gids, std::nullopt); - ASSERT_TRUE(exact_with_fallback.ok()) << exact_with_fallback.status(); - - // Without the -1000 fallback delta applied prior to the second loop, the - // running delta during the second loop would be +1000 higher and would - // falsely exceed delta_to_beat = *exact_with_fallback + 100. - auto with_threshold = CandidateMerge::ComputeCostDelta( - merger, {0, 1}, exclusive_gids, *exact_with_fallback + 100.0); - ASSERT_TRUE(with_threshold.ok()) << with_threshold.status(); - EXPECT_EQ(*with_threshold, *exact_with_fallback); -} - TEST_F(CandidateMergeTest, AssessMerge_Heuristic_NonExclusiveMaxPatchSize) { // s0: 'a' -> {69} // s1: 'i' -> {77}, s1 AND s3 -> {444, 446} @@ -1207,11 +1145,11 @@ TEST_F(CandidateMergeTest, AssessMerge_Heuristic_NonExclusiveMaxPatchSize) { {{'f'}}, }; - ClosureGlyphSegmenter segmenter(8, 8, PATCH, CLOSURE_ONLY, resolver); + ClosureGlyphSegmenter segmenter(8, 8, DEP_GRAPH_ONLY, resolver); auto context = SegmentationContext::InitializeSegmentationContext( - roboto.get(), {}, segments, segmenter.unmapped_glyph_handling(), - segmenter.condition_analysis_mode(), segmenter.brotli_quality(), - segmenter.init_font_merging_brotli_quality(), resolver); + roboto.get(), {}, segments, segmenter.condition_analysis_mode(), + segmenter.brotli_quality(), segmenter.init_font_merging_brotli_quality(), + resolver); ASSERT_TRUE(context.ok()) << context.status(); Merger merger = @@ -1260,11 +1198,11 @@ TEST_F(CandidateMergeTest, {{'f'}}, }; - ClosureGlyphSegmenter segmenter(8, 8, PATCH, DEP_GRAPH_ONLY, resolver); + ClosureGlyphSegmenter segmenter(8, 8, DEP_GRAPH_ONLY, resolver); auto context = SegmentationContext::InitializeSegmentationContext( - roboto.get(), {}, segments, segmenter.unmapped_glyph_handling(), - segmenter.condition_analysis_mode(), segmenter.brotli_quality(), - segmenter.init_font_merging_brotli_quality(), resolver); + roboto.get(), {}, segments, segmenter.condition_analysis_mode(), + segmenter.brotli_quality(), segmenter.init_font_merging_brotli_quality(), + resolver); ASSERT_TRUE(context.ok()) << context.status(); Merger merger = @@ -1316,13 +1254,12 @@ TEST_F(CandidateMergeTest, AssessMerge_CostDeltas_UnicodeRangeIgnoresNonExclusiv freq::UnicodeFrequencies frequencies{ {{' ', ' '}, 100}, {{'a', 'a'}, 50}, {{'f', 'f'}, 75}, {{'i', 'i'}, 95}}; - ClosureGlyphSegmenter segmenter(8, 8, PATCH, CLOSURE_ONLY, resolver, + ClosureGlyphSegmenter segmenter(8, 8, DEP_GRAPH_ONLY, resolver, ift::config::UNICODE_RANGE); auto context = SegmentationContext::InitializeSegmentationContext( - roboto.get(), {}, segments, segmenter.unmapped_glyph_handling(), - segmenter.condition_analysis_mode(), segmenter.brotli_quality(), - segmenter.init_font_merging_brotli_quality(), resolver, - ift::config::UNICODE_RANGE); + roboto.get(), {}, segments, segmenter.condition_analysis_mode(), + segmenter.brotli_quality(), segmenter.init_font_merging_brotli_quality(), + resolver, ift::config::UNICODE_RANGE); ASSERT_TRUE(context.ok()) << context.status(); Merger merger = *Merger::New( @@ -1418,18 +1355,17 @@ TEST_F(CandidateMergeTest, {{'l', 'l'}, 40}, }; - ClosureGlyphSegmenter segmenter(8, 8, PATCH, CLOSURE_ONLY, resolver); + ClosureGlyphSegmenter segmenter(8, 8, DEP_GRAPH_ONLY, resolver); auto context = SegmentationContext::InitializeSegmentationContext( - roboto.get(), {}, segments, segmenter.unmapped_glyph_handling(), - segmenter.condition_analysis_mode(), segmenter.brotli_quality(), - segmenter.init_font_merging_brotli_quality(), resolver); + roboto.get(), {}, segments, segmenter.condition_analysis_mode(), + segmenter.brotli_quality(), segmenter.init_font_merging_brotli_quality(), + resolver); ASSERT_TRUE(context.ok()) << context.status(); GlyphSet g0 = context->glyph_groupings.ExclusiveGlyphs(0); GlyphSet g1 = context->glyph_groupings.ExclusiveGlyphs(1); ASSERT_FALSE(g0.empty()); ASSERT_FALSE(g1.empty()); - ASSERT_TRUE(context->glyph_groupings.UnmappedGlyphs().empty()); ASSERT_TRUE(context->glyph_groupings.ConditionsAndGlyphs().contains( ActivationCondition::or_segments({0, 1}, 0))); @@ -1499,27 +1435,27 @@ TEST_F(CandidateMergeTest, // g_acute has now collapsed into the exclusive patch for segment 0. EXPECT_EQ(context->glyph_groupings.ExclusiveGlyphs(0), g01); - // Step 2 & 3: Patch merging combines composite ligature patches. + // Step 2: Patch merging combines composite ligature patches (s2 AND s3) and + // (s2 AND s4) into s2 AND (s3 OR s4). auto step2 = merger.TryNextMerge(); ASSERT_TRUE(step2.ok()) << step2.status(); ASSERT_TRUE(step2->has_value()); ASSERT_TRUE(context->ReprocessChanged(**step2).ok()); + // Step 3: No further merges remain. The exclusive patch for segment 0 must + // NOT be merged into any composite patch. auto step3 = merger.TryNextMerge(); ASSERT_TRUE(step3.ok()) << step3.status(); - ASSERT_TRUE(step3->has_value()); - ASSERT_TRUE(context->ReprocessChanged(**step3).ok()); - - // Step 4: No further merges remain. The exclusive patch for segment 0 must - // NOT be merged into any composite patch. - auto step4 = merger.TryNextMerge(); - ASSERT_TRUE(step4.ok()) << step4.status(); - EXPECT_FALSE(step4->has_value()); + EXPECT_FALSE(step3->has_value()); EXPECT_EQ(context->glyph_groupings.ExclusiveGlyphs(0), g01); EXPECT_EQ(context->glyph_groupings.ConditionsAndGlyphs().at( ActivationCondition::exclusive_segment(0, 0)), g01); + EXPECT_EQ(context->glyph_groupings.ConditionsAndGlyphs().at( + ActivationCondition::composite_condition( + {SegmentSet{2}, SegmentSet{3, 4}}, 0)), + g_fi_fl); } TEST_F(CandidateMergeTest, @@ -1541,12 +1477,12 @@ TEST_F(CandidateMergeTest, {{0x00F9, 0x00F9}, 40}, }; - ClosureGlyphSegmenter segmenter( - 8, 8, PATCH, DEP_GRAPH_ONLY_WITH_SIMPLIFICATION, resolver); + ClosureGlyphSegmenter segmenter(8, 8, DEP_GRAPH_ONLY_WITH_SIMPLIFICATION, + resolver); auto context = SegmentationContext::InitializeSegmentationContext( - roboto.get(), {}, segments, segmenter.unmapped_glyph_handling(), - segmenter.condition_analysis_mode(), segmenter.brotli_quality(), - segmenter.init_font_merging_brotli_quality(), resolver); + roboto.get(), {}, segments, segmenter.condition_analysis_mode(), + segmenter.brotli_quality(), segmenter.init_font_merging_brotli_quality(), + resolver); ASSERT_TRUE(context.ok()) << context.status(); GlyphSet g0 = context->glyph_groupings.ExclusiveGlyphs(0); @@ -1638,11 +1574,11 @@ TEST_F(CandidateMergeTest, ComputeCostDelta_InertPairFastPathAndPreCheck) { freq::UnicodeFrequencies frequencies{ {{' ', ' '}, 100}, {{'a', 'a'}, 80}, {{'b', 'b'}, 5}, {{'c', 'c'}, 75}}; - ClosureGlyphSegmenter segmenter(8, 8, PATCH, CLOSURE_ONLY, resolver); + ClosureGlyphSegmenter segmenter(8, 8, DEP_GRAPH_ONLY, resolver); auto context = SegmentationContext::InitializeSegmentationContext( - roboto.get(), {}, segments, segmenter.unmapped_glyph_handling(), - segmenter.condition_analysis_mode(), segmenter.brotli_quality(), - segmenter.init_font_merging_brotli_quality(), resolver); + roboto.get(), {}, segments, segmenter.condition_analysis_mode(), + segmenter.brotli_quality(), segmenter.init_font_merging_brotli_quality(), + resolver); ASSERT_TRUE(context.ok()) << context.status(); ASSERT_TRUE(context->InertSegments().contains(0)); ASSERT_TRUE(context->InertSegments().contains(1)); @@ -1664,8 +1600,8 @@ TEST_F(CandidateMergeTest, ComputeCostDelta_InertPairFastPathAndPreCheck) { // First evaluation warms the per-segment exclusive patch size cache for s0 // and s2. size_cache->ResetCallCount(); - auto best_case_0_2 = CandidateMerge::ComputeCostDelta( - merger, {0, 2}, std::nullopt, std::nullopt); + auto best_case_0_2 = + CandidateMerge::ComputeCostDelta(merger, {0, 2}, std::nullopt); ASSERT_TRUE(best_case_0_2.ok()) << best_case_0_2.status(); EXPECT_EQ(size_cache->GetPatchSizeCallCount(), 2); @@ -1679,8 +1615,8 @@ TEST_F(CandidateMergeTest, ComputeCostDelta_InertPairFastPathAndPreCheck) { // Subsequent best-case evaluation of {0, 2} hits the per-segment size cache // with zero calls to PatchSizeCache::GetPatchSize. size_cache->ResetCallCount(); - auto best_case_0_2_cached = CandidateMerge::ComputeCostDelta( - merger, {0, 2}, std::nullopt, std::nullopt); + auto best_case_0_2_cached = + CandidateMerge::ComputeCostDelta(merger, {0, 2}, std::nullopt); ASSERT_TRUE(best_case_0_2_cached.ok()) << best_case_0_2_cached.status(); EXPECT_EQ(*best_case_0_2_cached, *best_case_0_2); EXPECT_EQ(size_cache->GetPatchSizeCallCount(), 0); @@ -1691,13 +1627,12 @@ TEST_F(CandidateMergeTest, ComputeCostDelta_InertPairFastPathAndPreCheck) { // 0.80*(400+75) - 0.80*(200+75) - 0.05*(300+75) = 160 - 18.75 = +141.25 > 0, // so the O(1) pre-check cancels immediately. auto cancelled_precheck = - CandidateMerge::ComputeCostDelta(merger, {0, 1}, std::nullopt, 0.0); + CandidateMerge::ComputeCostDelta(merger, {0, 1}, 0.0); EXPECT_TRUE(absl::IsCancelled(cancelled_precheck.status())); // Exact evaluation of {0, 2} via the inert fast-path. - GlyphSet exclusive_0_2 = {69, 71}; - auto exact_0_2 = CandidateMerge::ComputeCostDelta( - merger, {0, 2}, exclusive_0_2, std::nullopt); + auto exact_0_2 = + CandidateMerge::ComputeCostDelta(merger, {0, 2}, std::nullopt); ASSERT_TRUE(exact_0_2.ok()) << exact_0_2.status(); double expected_exact_0_2 = -0.80 * (200 + 75) - 0.75 * (240 + 75) + p_0_or_2 * (320 + 75); @@ -1705,7 +1640,7 @@ TEST_F(CandidateMergeTest, ComputeCostDelta_InertPairFastPathAndPreCheck) { // Boundary check: 1 ULP below exact_0_2 does not cancel early. auto boundary_0_2 = CandidateMerge::ComputeCostDelta( - merger, {0, 2}, exclusive_0_2, std::nextafter(*exact_0_2, -INFINITY)); + merger, {0, 2}, std::nextafter(*exact_0_2, -INFINITY)); ASSERT_TRUE(boundary_0_2.ok()) << boundary_0_2.status(); EXPECT_EQ(*boundary_0_2, *exact_0_2); @@ -1715,7 +1650,9 @@ TEST_F(CandidateMergeTest, ComputeCostDelta_InertPairFastPathAndPreCheck) { CandidateMerge::AssessSegmentMerge(merger, 0, {2}, std::nullopt); ASSERT_TRUE(candidate.ok()) << candidate.status(); ASSERT_TRUE(candidate->has_value()); - ASSERT_TRUE((*candidate)->Apply(merger).ok()); + auto applied = (*candidate)->Apply(merger); + ASSERT_TRUE(applied.ok()) << applied.status(); + ASSERT_TRUE(context->ReprocessChanged(*applied).ok()); size_cache->ResetCallCount(); EXPECT_EQ(*merger.ExclusivePatchSize(1), 300u); @@ -1734,13 +1671,12 @@ TEST_F(CandidateMergeTest, freq::UnicodeFrequencies frequencies{ {{' ', ' '}, 100}, {{'a', 'a'}, 80}, {{'c', 'c'}, 75}}; - ClosureGlyphSegmenter segmenter(8, 8, PATCH, CLOSURE_ONLY, resolver, + ClosureGlyphSegmenter segmenter(8, 8, DEP_GRAPH_ONLY, resolver, ift::config::UNICODE_RANGE); auto context = SegmentationContext::InitializeSegmentationContext( - roboto.get(), {}, segments, segmenter.unmapped_glyph_handling(), - segmenter.condition_analysis_mode(), segmenter.brotli_quality(), - segmenter.init_font_merging_brotli_quality(), resolver, - ift::config::UNICODE_RANGE); + roboto.get(), {}, segments, segmenter.condition_analysis_mode(), + segmenter.brotli_quality(), segmenter.init_font_merging_brotli_quality(), + resolver, ift::config::UNICODE_RANGE); ASSERT_TRUE(context.ok()) << context.status(); Merger merger = *Merger::New( @@ -1762,16 +1698,15 @@ TEST_F(CandidateMergeTest, double p_0_or_1 = 0.80 + 0.75 - 0.80 * 0.75; double expected_best = -0.80 * (200 + 75) - 0.75 * (240 + 75) + p_0_or_1 * (290 + 75); - auto best_delta = CandidateMerge::ComputeCostDelta( - merger, {0, 1}, std::nullopt, std::nullopt); + auto best_delta = + CandidateMerge::ComputeCostDelta(merger, {0, 1}, std::nullopt); ASSERT_TRUE(best_delta.ok()) << best_delta.status(); EXPECT_NEAR(*best_delta, expected_best, 1e-9); - GlyphSet exclusive_gids = {69, 71}; double expected_exact = -0.80 * (200 + 75) - 0.75 * (240 + 75) + p_0_or_1 * (320 + 75); - auto exact_delta = CandidateMerge::ComputeCostDelta( - merger, {0, 1}, exclusive_gids, std::nullopt); + auto exact_delta = + CandidateMerge::ComputeCostDelta(merger, {0, 1}, std::nullopt); ASSERT_TRUE(exact_delta.ok()) << exact_delta.status(); EXPECT_NEAR(*exact_delta, expected_exact, 1e-9); } @@ -1795,11 +1730,11 @@ TEST_F(CandidateMergeTest, AllPairsMerging_AvoidsSingleBaseGreedyTrap) { {{{'a', 'b', 'c'}}, 0.99}, }; - ClosureGlyphSegmenter segmenter(8, 8, PATCH, CLOSURE_ONLY, resolver); + ClosureGlyphSegmenter segmenter(8, 8, DEP_GRAPH_ONLY, resolver); auto context = SegmentationContext::InitializeSegmentationContext( - roboto.get(), {}, segments, segmenter.unmapped_glyph_handling(), - segmenter.condition_analysis_mode(), segmenter.brotli_quality(), - segmenter.init_font_merging_brotli_quality(), resolver); + roboto.get(), {}, segments, segmenter.condition_analysis_mode(), + segmenter.brotli_quality(), segmenter.init_font_merging_brotli_quality(), + resolver); ASSERT_TRUE(context.ok()) << context.status(); auto* size_cache = new MockPatchSizeCache(); @@ -1857,11 +1792,11 @@ TEST_F(CandidateMergeTest, {{{'a', 'b', 'c', 'd'}}, 0.80}, }; - ClosureGlyphSegmenter segmenter(8, 8, PATCH, CLOSURE_ONLY, resolver); + ClosureGlyphSegmenter segmenter(8, 8, DEP_GRAPH_ONLY, resolver); auto context = SegmentationContext::InitializeSegmentationContext( - roboto.get(), {}, segments, segmenter.unmapped_glyph_handling(), - segmenter.condition_analysis_mode(), segmenter.brotli_quality(), - segmenter.init_font_merging_brotli_quality(), resolver); + roboto.get(), {}, segments, segmenter.condition_analysis_mode(), + segmenter.brotli_quality(), segmenter.init_font_merging_brotli_quality(), + resolver); ASSERT_TRUE(context.ok()) << context.status(); auto* size_cache = new MockPatchSizeCache(); @@ -1935,11 +1870,11 @@ TEST_F(CandidateMergeTest, {{{'c', 'd', 'e'}}, 0.661}, }; - ClosureGlyphSegmenter segmenter(8, 8, PATCH, CLOSURE_ONLY, resolver); + ClosureGlyphSegmenter segmenter(8, 8, DEP_GRAPH_ONLY, resolver); auto context = SegmentationContext::InitializeSegmentationContext( - roboto.get(), {}, segments, segmenter.unmapped_glyph_handling(), - segmenter.condition_analysis_mode(), segmenter.brotli_quality(), - segmenter.init_font_merging_brotli_quality(), resolver); + roboto.get(), {}, segments, segmenter.condition_analysis_mode(), + segmenter.brotli_quality(), segmenter.init_font_merging_brotli_quality(), + resolver); ASSERT_TRUE(context.ok()) << context.status(); // Glyphs: 'a'=69, 'b'=70, 'c'=71, 'd'=72, 'e'=73. @@ -2038,11 +1973,11 @@ TEST_F(CandidateMergeTest, AllPairsMerging_ByteNormalization) { probs.push_back({Segment(def), 1.0}); } - ClosureGlyphSegmenter segmenter(8, 8, PATCH, CLOSURE_ONLY, resolver); + ClosureGlyphSegmenter segmenter(8, 8, DEP_GRAPH_ONLY, resolver); auto context = SegmentationContext::InitializeSegmentationContext( - roboto.get(), {}, segments, segmenter.unmapped_glyph_handling(), - segmenter.condition_analysis_mode(), segmenter.brotli_quality(), - segmenter.init_font_merging_brotli_quality(), resolver); + roboto.get(), {}, segments, segmenter.condition_analysis_mode(), + segmenter.brotli_quality(), segmenter.init_font_merging_brotli_quality(), + resolver); ASSERT_TRUE(context.ok()) << context.status(); const auto& groupings = context->glyph_groupings; @@ -2118,11 +2053,11 @@ TEST_F(CandidateMergeTest, ComputeCostDelta_BestCaseSkipsUnchangedGlyphSetCondit freq::UnicodeFrequencies frequencies{ {{' ', ' '}, 100}, {{'a', 'a'}, 50}, {{'f', 'f'}, 75}, {{'i', 'i'}, 95}}; - ClosureGlyphSegmenter segmenter(8, 8, PATCH, CLOSURE_ONLY, resolver); + ClosureGlyphSegmenter segmenter(8, 8, DEP_GRAPH_ONLY, resolver); auto context = SegmentationContext::InitializeSegmentationContext( - roboto.get(), {}, segments, segmenter.unmapped_glyph_handling(), - segmenter.condition_analysis_mode(), segmenter.brotli_quality(), - segmenter.init_font_merging_brotli_quality(), resolver); + roboto.get(), {}, segments, segmenter.condition_analysis_mode(), + segmenter.brotli_quality(), segmenter.init_font_merging_brotli_quality(), + resolver); ASSERT_TRUE(context.ok()) << context.status(); Merger merger = *Merger::New( @@ -2143,8 +2078,8 @@ TEST_F(CandidateMergeTest, ComputeCostDelta_BestCaseSkipsUnchangedGlyphSetCondit // In best-case mode, the delta contribution for (s1 AND s2) is treated as 0, // and its patch size {444, 446} is never queried. size_cache->ResetCallCount(); - auto best_case_delta = CandidateMerge::ComputeCostDelta( - merger, {0, 1}, std::nullopt, std::nullopt); + auto best_case_delta = + CandidateMerge::ComputeCostDelta(merger, {0, 1}, std::nullopt); ASSERT_TRUE(best_case_delta.ok()) << best_case_delta.status(); // Only the two merging exclusive patches {69} and {74} are queried for size. @@ -2177,11 +2112,11 @@ TEST_F(CandidateMergeTest, ComputeCostDelta_SkipsUnchangedEffectiveConditions) { {{0x00E9, 0x00E9}, 75}, {{0x0107, 0x0107}, 95}}; - ClosureGlyphSegmenter segmenter(8, 8, PATCH, CLOSURE_ONLY, resolver); + ClosureGlyphSegmenter segmenter(8, 8, DEP_GRAPH_ONLY, resolver); auto context = SegmentationContext::InitializeSegmentationContext( - roboto.get(), {}, segments, segmenter.unmapped_glyph_handling(), - segmenter.condition_analysis_mode(), segmenter.brotli_quality(), - segmenter.init_font_merging_brotli_quality(), resolver); + roboto.get(), {}, segments, segmenter.condition_analysis_mode(), + segmenter.brotli_quality(), segmenter.init_font_merging_brotli_quality(), + resolver); ASSERT_TRUE(context.ok()) << context.status(); ASSERT_FALSE(context->InertSegments().contains(0)); ASSERT_FALSE(context->InertSegments().contains(1)); @@ -2213,8 +2148,8 @@ TEST_F(CandidateMergeTest, ComputeCostDelta_SkipsUnchangedEffectiveConditions) { // change. Its cost delta is 0 in both best-case and exact mode, and g_acute's // patch size is never queried. size_cache->ResetCallCount(); - auto best_delta = CandidateMerge::ComputeCostDelta( - merger, {0, 1}, std::nullopt, std::nullopt); + auto best_delta = + CandidateMerge::ComputeCostDelta(merger, {0, 1}, std::nullopt); ASSERT_TRUE(best_delta.ok()) << best_delta.status(); EXPECT_EQ(size_cache->GetPatchSizeCallCount(), 2); // only g0 and g1 @@ -2225,8 +2160,8 @@ TEST_F(CandidateMergeTest, ComputeCostDelta_SkipsUnchangedEffectiveConditions) { EXPECT_NEAR(*best_delta, expected_best, 1e-9); size_cache->ResetCallCount(); - auto exact_delta = CandidateMerge::ComputeCostDelta( - merger, {0, 1}, g01, std::nullopt); + auto exact_delta = + CandidateMerge::ComputeCostDelta(merger, {0, 1}, std::nullopt); ASSERT_TRUE(exact_delta.ok()) << exact_delta.status(); // In exact mode: g0, g1, and g01 are queried; g_acute is NOT. EXPECT_EQ(size_cache->GetPatchSizeCallCount(), 3); @@ -2252,11 +2187,11 @@ TEST_F(CandidateMergeTest, {{0x00E9, 0x00E9}, 75}, {{0x0107, 0x0107}, 95}}; - ClosureGlyphSegmenter segmenter(8, 8, PATCH, CLOSURE_ONLY, resolver); + ClosureGlyphSegmenter segmenter(8, 8, DEP_GRAPH_ONLY, resolver); auto context = SegmentationContext::InitializeSegmentationContext( - roboto.get(), {}, segments, segmenter.unmapped_glyph_handling(), - segmenter.condition_analysis_mode(), segmenter.brotli_quality(), - segmenter.init_font_merging_brotli_quality(), resolver); + roboto.get(), {}, segments, segmenter.condition_analysis_mode(), + segmenter.brotli_quality(), segmenter.init_font_merging_brotli_quality(), + resolver); ASSERT_TRUE(context.ok()) << context.status(); Merger merger = *Merger::New( @@ -2277,9 +2212,9 @@ TEST_F(CandidateMergeTest, context->glyph_condition_set.AddOrCondition(gid_d, 0); context->glyph_condition_set.AddOrCondition(gid_d, 2); - auto sc = context->glyph_groupings.GroupGlyphs( - context->SegmentationInfo(), context->glyph_condition_set, - *context->glyph_closure_cache, std::nullopt, {gid_d}, {0, 2}, false); + auto sc = context->glyph_groupings.GroupGlyphs(context->SegmentationInfo(), + context->glyph_condition_set, + {gid_d}, {0, 2}); ASSERT_TRUE(sc.ok()) << sc; GlyphSet g_acute_and_d = g_acute; @@ -2298,8 +2233,8 @@ TEST_F(CandidateMergeTest, // Even though (s0 OR s1 OR s2) individually has an unchanged effective // condition, it merges into (s0 OR s2) where (s0 OR s2) already exists! // So their glyph set becomes g_acute U {gid_d}. Both patches must be evaluated. - auto exact_delta = CandidateMerge::ComputeCostDelta( - merger, {0, 1}, g01, std::nullopt); + auto exact_delta = + CandidateMerge::ComputeCostDelta(merger, {0, 1}, std::nullopt); ASSERT_TRUE(exact_delta.ok()) << exact_delta.status(); double p_0_or_1 = 0.50 + 0.75 - 0.50 * 0.75; diff --git a/ift/encoder/closure_glyph_segmenter.cc b/ift/encoder/closure_glyph_segmenter.cc index e4c6ccc4..8bca135f 100644 --- a/ift/encoder/closure_glyph_segmenter.cc +++ b/ift/encoder/closure_glyph_segmenter.cc @@ -23,6 +23,7 @@ #include "ift/common/try.h" #include "ift/common/woff2.h" #include "ift/config/auto_segmenter_config.h" +#include "ift/config/common.pb.h" #include "ift/encoder/activation_condition.h" #include "ift/encoder/glyph_groupings.h" #include "ift/encoder/glyph_segmentation.h" @@ -40,11 +41,9 @@ using ift::config::FROM_FREQ_DATA; using ift::config::FROM_MERGE_GROUPS; using ift::config::kDefaultNetworkCost; -using ift::config::MOVE_TO_INIT_FONT; using ift::config::SegmentationPlan; using ift::config::SegmentsProto; using ift::config::TableKeyedSegmentMode; -using ift::config::UnmappedGlyphHandling; using absl::btree_map; using absl::btree_set; @@ -128,69 +127,6 @@ static void PrintDiff(const GlyphGroupings& a, const GlyphGroupings& b) { } } -// Returns true if the glyph groupings a and b are equivalent (not necessarily -// equal). -// -// Complex condition finding can return multiple possible valid conditions for a -// glyph so for these we allowe differences as long as the closure condition is -// satisfied (that is there are no additional conditions.) -static StatusOr GlyphGroupingsAreEquivalent( - const SegmentationContext& context, const GlyphGroupings& a, - const GlyphGroupings& b) { - if (a.ConditionsAndGlyphs() == b.ConditionsAndGlyphs()) { - return true; - } - - for (glyph_id_t gid = 0; - gid < hb_face_get_glyph_count(context.original_face.get()); gid++) { - auto maybe_cond_a = a.GlyphToCondition(gid); - auto maybe_cond_b = b.GlyphToCondition(gid); - - if (maybe_cond_a.has_value() != maybe_cond_b.has_value()) { - return false; - } - - if (!maybe_cond_a.has_value()) { - continue; - } - - ActivationCondition cond_a = *maybe_cond_a; - ActivationCondition cond_b = *maybe_cond_b; - - if (cond_a == cond_b) { - continue; - } - - if (!a.FoundConditionGlyphs().contains(gid) || - !b.FoundConditionGlyphs().contains(gid)) { - // diffs only allowed if gid is a found conditions gid in both a and b. - return false; - } - - if (cond_a.conditions().size() > 1 || cond_b.conditions().size() > 1) { - // diffs only allowed for purely disjunctive conditions - return false; - } - - if (TRY(context.glyph_closure_cache->HasAdditionalConditions( - &context.SegmentationInfo(), cond_a.TriggeringSegments(), - GlyphSet{gid}))) { - return false; - } - - if (TRY(context.glyph_closure_cache->HasAdditionalConditions( - &context.SegmentationInfo(), cond_b.TriggeringSegments(), - GlyphSet{gid}))) { - return false; - } - - // If no additional conditions are present on either side then we allow the - // diff. - } - - return true; -} - /* * Checks that the incrementally generated glyph conditions and groupings in * context match what would have been produced by a non incremental process. @@ -211,20 +147,13 @@ Status ValidateIncrementalGroupings(hb_face_t* face, } TRYV(non_incremental_context.ReprocessAll()); - bool glyph_groupings_diffs_allowed = false; if (non_incremental_context.glyph_groupings.ConditionsAndGlyphs() != context.glyph_groupings.ConditionsAndGlyphs()) { - if (!TRY(GlyphGroupingsAreEquivalent( - context, context.glyph_groupings, - non_incremental_context.glyph_groupings))) { - VLOG(0) << "-- incremental grouping"; - VLOG(0) << "++ non-incremental grouping"; - PrintDiff(context.glyph_groupings, - non_incremental_context.glyph_groupings); - return absl::FailedPreconditionError( - "conditions_and_glyphs aren't correct."); - } - glyph_groupings_diffs_allowed = true; + VLOG(0) << "-- incremental grouping"; + VLOG(0) << "++ non-incremental grouping"; + PrintDiff(context.glyph_groupings, non_incremental_context.glyph_groupings); + return absl::FailedPreconditionError( + "conditions_and_glyphs aren't correct."); } if (non_incremental_context.glyph_condition_set != @@ -232,8 +161,7 @@ Status ValidateIncrementalGroupings(hb_face_t* face, return absl::FailedPreconditionError("glyph_condition_set isn't correct."); } - if (!glyph_groupings_diffs_allowed && - non_incremental_context.glyph_groupings != context.glyph_groupings) { + if (non_incremental_context.glyph_groupings != context.glyph_groupings) { return absl::FailedPreconditionError("glyph groups aren't correct."); } @@ -609,6 +537,15 @@ static StatusOr ToFinalSegmentation( Status ClosureGlyphSegmenter::ValidateInput( const std::vector& subset_definitions, const btree_map& merge_groups) const { + if (condition_analysis_mode_ != ift::config::DEP_GRAPH_ONLY && + condition_analysis_mode_ != + ift::config::DEP_GRAPH_ONLY_WITH_SIMPLIFICATION) { + return absl::InvalidArgumentError( + "Closure-based condition_analysis_mode values are deprecated and " + "unsupported; use DEP_GRAPH_ONLY or " + "DEP_GRAPH_ONLY_WITH_SIMPLIFICATION."); + } + if (disjoint_conditions_probability_threshold_ < 0.0 || disjoint_conditions_probability_threshold_ > 1.0) { return absl::InvalidArgumentError( @@ -616,13 +553,6 @@ Status ClosureGlyphSegmenter::ValidateInput( } if (target_mode_ == ift::config::UNICODE_RANGE) { - if (unmapped_glyph_handling_ != ift::config::FIND_CONDITIONS && - unmapped_glyph_handling_ != ift::config::PATCH) { - return absl::InvalidArgumentError( - "unmapped_glyph_handling must be FIND_CONDITIONS or PATCH when " - "target_mode is UNICODE_RANGE."); - } - for (const auto& def : subset_definitions) { if (!def.feature_tags.empty()) { return absl::InvalidArgumentError( @@ -671,7 +601,7 @@ static StatusOr ShouldMergeForDisjointConditions( const SegmentationContext& context, const ActivationCondition& condition, double probability_threshold, const btree_map& merge_groups) { - if (condition.IsExclusive() || condition.IsFallback()) { + if (condition.IsExclusive()) { return false; } if (merge_groups.empty()) { @@ -726,7 +656,7 @@ static StatusOr PartitionInteractingSegments( GlyphPartition segment_partition(num_segments); for (const auto& condition : context.glyph_groupings.OrderedConditions()) { SegmentSet triggering = condition.TriggeringSegments(); - if (condition.IsExclusive() || condition.IsFallback() || triggering.size() <= 1) { + if (condition.IsExclusive() || triggering.size() <= 1) { continue; } @@ -836,9 +766,9 @@ StatusOr ClosureGlyphSegmenter::CodepointToGlyphSegments( TRY(ToOrderedSegments(subset_definitions, merge_groups, with_shared)); SegmentationContext context = TRY(SegmentationContext::InitializeSegmentationContext( - face, initial_segment, std::move(segments), unmapped_glyph_handling_, - condition_analysis_mode_, brotli_quality_, - init_font_merging_brotli_quality_, resolver_, target_mode_)); + face, initial_segment, std::move(segments), condition_analysis_mode_, + brotli_quality_, init_font_merging_brotli_quality_, resolver_, + target_mode_)); if (target_mode_ == ift::config::UNICODE_RANGE) { std::vector calculators; @@ -880,20 +810,6 @@ StatusOr ClosureGlyphSegmenter::CodepointToGlyphSegments( } } - // Once we've gotten standard segments placed into the initial font as needed, - // if requested any remaining fallback glyphs are also moved into the init - // font. - GlyphSet fallback_glyphs = context.glyph_groupings.UnmappedGlyphs(); - if (unmapped_glyph_handling_ == MOVE_TO_INIT_FONT && - !fallback_glyphs.empty()) { - VLOG(0) << "Moving " << fallback_glyphs.size() - << " fallback glyphs into the initial font." << std::endl; - SubsetDefinition new_def = context.SegmentationInfo().InitFontSegment(); - new_def.gids.union_set(fallback_glyphs); - TRYV(context.ReassignInitSubset(new_def)); - init_font_changed = true; - } - if (init_font_changed) { for (Merger& merger : mergers) { // Any init font moves above can cause segment removals that affect other @@ -1072,23 +988,6 @@ StatusOr> ClosureGlyphSegmenter::TotalCosts( return out; } -Status ClosureGlyphSegmenter::FallbackCost( - hb_face_t* original_face, const GlyphSegmentation& segmentation, - uint32_t& fallback_glyphs_size, uint32_t& all_glyphs_size) const { - GlyphSet all_glyphs = segmentation.InitialFontGlyphClosure(); - for (const auto& [_, gids] : segmentation.GidSegments()) { - all_glyphs.union_set(gids); - } - - GlyphSet fallback_glyphs = segmentation.UnmappedGlyphs(); - - PatchSizeCacheImpl patch_sizer(original_face, 11); - all_glyphs_size = TRY(patch_sizer.GetPatchSize(all_glyphs)); - fallback_glyphs_size = TRY(patch_sizer.GetPatchSize(fallback_glyphs)); - - return absl::OkStatus(); -} - static std::vector TableKeyedSegmentsFromMergeGroups( const btree_map& merge_groups, const std::vector& segments, diff --git a/ift/encoder/closure_glyph_segmenter.h b/ift/encoder/closure_glyph_segmenter.h index 40314e20..a96984f7 100644 --- a/ift/encoder/closure_glyph_segmenter.h +++ b/ift/encoder/closure_glyph_segmenter.h @@ -8,7 +8,6 @@ #include "absl/container/btree_map.h" #include "absl/status/statusor.h" #include "ift/common/data_file_resolver.h" -#include "ift/config/common.pb.h" #include "ift/config/segmentation_plan.pb.h" #include "ift/config/segmenter_config.pb.h" #include "ift/encoder/glyph_segmentation.h" @@ -40,14 +39,12 @@ class ClosureGlyphSegmenter { public: ClosureGlyphSegmenter( uint32_t brotli_quality, uint32_t init_font_merging_brotli_quality, - ift::config::UnmappedGlyphHandling unmapped_glyph_handling, ift::config::ConditionAnalysisMode condition_analysis_mode, std::shared_ptr resolver, ift::config::TargetMode target_mode = ift::config::IFT, double disjoint_conditions_probability_threshold = 0.0) : brotli_quality_(brotli_quality), init_font_merging_brotli_quality_(init_font_merging_brotli_quality), - unmapped_glyph_handling_(unmapped_glyph_handling), condition_analysis_mode_(condition_analysis_mode), resolver_(std::move(resolver)), target_mode_(target_mode), @@ -82,15 +79,6 @@ class ClosureGlyphSegmenter { absl::Span probability_calculators) const; - /* - * Computes the total cost of the fallback patch (expected number of bytes - * transferred) - */ - absl::Status FallbackCost(hb_face_t* original_face, - const GlyphSegmentation& segmentation, - uint32_t& fallback_glyphs_size, - uint32_t& all_glyphs_size) const; - static absl::Status AddTableKeyedSegments( ift::config::SegmentationPlan& plan, const absl::btree_map& @@ -104,9 +92,6 @@ class ClosureGlyphSegmenter { uint32_t init_font_merging_brotli_quality() const { return init_font_merging_brotli_quality_; } - ift::config::UnmappedGlyphHandling unmapped_glyph_handling() const { - return unmapped_glyph_handling_; - } ift::config::ConditionAnalysisMode condition_analysis_mode() const { return condition_analysis_mode_; } @@ -123,7 +108,6 @@ class ClosureGlyphSegmenter { uint32_t brotli_quality_; uint32_t init_font_merging_brotli_quality_; - ift::config::UnmappedGlyphHandling unmapped_glyph_handling_; ift::config::ConditionAnalysisMode condition_analysis_mode_; std::shared_ptr resolver_; ift::config::TargetMode target_mode_; diff --git a/ift/encoder/closure_glyph_segmenter_test.cc b/ift/encoder/closure_glyph_segmenter_test.cc index 7002ab0b..c47d2af5 100644 --- a/ift/encoder/closure_glyph_segmenter_test.cc +++ b/ift/encoder/closure_glyph_segmenter_test.cc @@ -22,14 +22,9 @@ #include "ift/freq/unicode_frequencies.h" #include "ift/freq/unigram_probability_calculator.h" -using ift::config::CLOSURE_AND_VALIDATE_DEP_GRAPH; -using ift::config::CLOSURE_ONLY; using ift::config::DEP_GRAPH_ONLY; using ift::config::DEP_GRAPH_ONLY_WITH_SIMPLIFICATION; -using ift::config::FIND_CONDITIONS; using ift::config::kDefaultNetworkCost; -using ift::config::MOVE_TO_INIT_FONT; -using ift::config::PATCH; using absl::btree_map; using absl::StatusOr; @@ -58,20 +53,9 @@ class ClosureGlyphSegmenterTest : public ::testing::Test { noto_sans_jp(make_hb_face(nullptr)), resolver(*BazelDataFileResolver::CreateForTest()), - segmenter(8, 8, PATCH, CLOSURE_ONLY, resolver), - segmenter_find_conditions(8, 8, FIND_CONDITIONS, CLOSURE_ONLY, - resolver), - segmenter_move_to_init_font(8, 8, MOVE_TO_INIT_FONT, CLOSURE_ONLY, - resolver), - segmenter_dep_graph(8, 8, PATCH, CLOSURE_AND_VALIDATE_DEP_GRAPH, - resolver), - segmenter_dep_graph_only(8, 8, PATCH, DEP_GRAPH_ONLY, resolver), + segmenter(8, 8, DEP_GRAPH_ONLY, resolver), segmenter_dep_graph_only_with_simplification( - 8, 8, PATCH, DEP_GRAPH_ONLY_WITH_SIMPLIFICATION, resolver), - segmenter_find_conditions_dep_graph( - 8, 8, FIND_CONDITIONS, CLOSURE_AND_VALIDATE_DEP_GRAPH, resolver), - segmenter_move_to_init_font_dep_graph( - 8, 8, MOVE_TO_INIT_FONT, CLOSURE_AND_VALIDATE_DEP_GRAPH, resolver) { + 8, 8, DEP_GRAPH_ONLY_WITH_SIMPLIFICATION, resolver) { roboto = from_file("ift/common/testdata/Roboto-Regular.ttf"); noto_nastaliq_urdu = from_file("ift/common/testdata/NotoNastaliqUrdu.subset.ttf"); @@ -87,98 +71,16 @@ class ClosureGlyphSegmenterTest : public ::testing::Test { hb_face_t* face, SubsetDefinition initial_segment, const std::vector& subset_definitions, std::optional strategy = std::nullopt) const { - auto segmentation = TRY(segmenter.CodepointToGlyphSegments( - face, initial_segment, subset_definitions, strategy)); - auto dep_graph_segmentation = - TRY(segmenter_dep_graph.CodepointToGlyphSegments( - face, initial_segment, subset_definitions, strategy)); - auto dep_graph_only_segmentation = - TRY(segmenter_dep_graph_only.CodepointToGlyphSegments( - face, initial_segment, subset_definitions, strategy)); - - if (segmentation != dep_graph_segmentation) { - return absl::InternalError( - "Basic segmenter: segmentation from dep graph is not equivalent to " - "the non-dep graph version."); - } - if (segmentation != dep_graph_only_segmentation) { - std::cerr << "Closure segmentation: " << segmentation.ToString() - << std::endl; - std::cerr << "Dep graph only segmentation: " - << dep_graph_only_segmentation.ToString() << std::endl; - return absl::InternalError( - "Basic segmenter: segmentation from dep graph only is not equivalent " - "to " - "the non-dep graph version."); - } - - return segmentation; + return segmenter.CodepointToGlyphSegments(face, initial_segment, + subset_definitions, strategy); } StatusOr CodepointToGlyphSegments( hb_face_t* face, SubsetDefinition initial_segment, const std::vector& subset_definitions, btree_map merge_groups) const { - auto segmentation = TRY(segmenter.CodepointToGlyphSegments( - face, initial_segment, subset_definitions, merge_groups)); - auto dep_graph_segmentation = - TRY(segmenter_dep_graph.CodepointToGlyphSegments( - face, initial_segment, subset_definitions, merge_groups)); - auto dep_graph_only_segmentation = - TRY(segmenter_dep_graph_only.CodepointToGlyphSegments( - face, initial_segment, subset_definitions, merge_groups)); - - if (segmentation != dep_graph_segmentation) { - return absl::InternalError( - "Basic segmenter: segmentation from dep graph is not equivalent to " - "the non-dep graph version."); - } - if (segmentation != dep_graph_only_segmentation) { - std::cerr << "Closure segmentation: " << segmentation.ToString() - << std::endl; - std::cerr << "Dep graph only segmentation: " - << dep_graph_only_segmentation.ToString() << std::endl; - return absl::InternalError( - "Basic segmenter: segmentation from dep graph only is not equivalent " - "to " - "the non-dep graph version."); - } - - return segmentation; - } - - StatusOr FindConditionsCodepointToGlyphSegments( - hb_face_t* face, SubsetDefinition initial_segment, - const std::vector& subset_definitions, - std::optional strategy = std::nullopt) const { - auto segmentation = segmenter_find_conditions.CodepointToGlyphSegments( - face, initial_segment, subset_definitions, strategy); - auto dep_graph_segmentation = - segmenter_find_conditions_dep_graph.CodepointToGlyphSegments( - face, initial_segment, subset_definitions, strategy); - if (segmentation != dep_graph_segmentation) { - return absl::InternalError( - "Find conditions segmenter: segmentation from dep graph is not " - "equivalent to the non-dep graph version."); - } - return segmentation; - } - - StatusOr MoveToInitFontCodepointToGlyphSegments( - hb_face_t* face, SubsetDefinition initial_segment, - const std::vector& subset_definitions, - absl::btree_map merge_groups) const { - auto segmentation = segmenter_move_to_init_font.CodepointToGlyphSegments( - face, initial_segment, subset_definitions, merge_groups); - auto dep_graph_segmentation = - segmenter_move_to_init_font_dep_graph.CodepointToGlyphSegments( - face, initial_segment, subset_definitions, merge_groups); - if (segmentation != dep_graph_segmentation) { - return absl::InternalError( - "Move to init font segmenter: segmentation from dep graph is not " - "equivalent to the non-dep graph version."); - } - return segmentation; + return segmenter.CodepointToGlyphSegments(face, initial_segment, + subset_definitions, merge_groups); } hb_face_unique_ptr roboto; @@ -187,13 +89,7 @@ class ClosureGlyphSegmenterTest : public ::testing::Test { std::shared_ptr resolver; ClosureGlyphSegmenter segmenter; - ClosureGlyphSegmenter segmenter_find_conditions; - ClosureGlyphSegmenter segmenter_move_to_init_font; - ClosureGlyphSegmenter segmenter_dep_graph; - ClosureGlyphSegmenter segmenter_dep_graph_only; ClosureGlyphSegmenter segmenter_dep_graph_only_with_simplification; - ClosureGlyphSegmenter segmenter_find_conditions_dep_graph; - ClosureGlyphSegmenter segmenter_move_to_init_font_dep_graph; }; TEST_F(ClosureGlyphSegmenterTest, SimpleSegmentation) { @@ -335,16 +231,9 @@ if ((s0 OR s1)) then p1 } TEST_F(ClosureGlyphSegmenterTest, SegmentationWithAdditionalConditionOverlap) { - auto segmentation = segmenter.CodepointToGlyphSegments( - roboto.get(), {'a'}, {{'f'}, {'i'}, {'f', 'i'}}); - auto dep_graph_segmentation = segmenter_dep_graph.CodepointToGlyphSegments( - roboto.get(), {'a'}, {{'f'}, {'i'}, {'f', 'i'}}); - auto dep_graph_only_segmentation = - segmenter_dep_graph_only.CodepointToGlyphSegments( - roboto.get(), {'a'}, {{'f'}, {'i'}, {'f', 'i'}}); + auto segmentation = + CodepointToGlyphSegments(roboto.get(), {'a'}, {{'f'}, {'i'}, {'f', 'i'}}); ASSERT_TRUE(segmentation.ok()) << segmentation.status(); - ASSERT_TRUE(dep_graph_segmentation.ok()) << segmentation.status(); - ASSERT_TRUE(dep_graph_only_segmentation.ok()) << segmentation.status(); std::vector expected_segments = {{'f'}, {'i'}, {'f', 'i'}}; ASSERT_EQ(segmentation->Segments(), expected_segments); @@ -356,17 +245,6 @@ p1: { gid77 } p2: { gid444, gid446 } if ((s0 OR s2)) then p0 if ((s1 OR s2)) then p1 -if ((s0 OR s1 OR s2)) then p2 -)"); - ASSERT_EQ(segmentation->ToString(), dep_graph_segmentation->ToString()); - - ASSERT_EQ(dep_graph_only_segmentation->ToString(), - R"(initial font: { gid0, gid69 } -p0: { gid74 } -p1: { gid77 } -p2: { gid444, gid446 } -if ((s0 OR s2)) then p0 -if ((s1 OR s2)) then p1 if ((s0 OR s2) AND (s1 OR s2)) then p2 )"); } @@ -589,101 +467,12 @@ if (s0 AND s1) then p3 )"); } -TEST_F(ClosureGlyphSegmenterTest, UnmappedGlyphs_FallbackSegment) { - auto segmentation = segmenter.CodepointToGlyphSegments( - noto_nastaliq_urdu.get(), {}, {{0x62a}, {0x62b}, {0x62c}, {0x62d}}); - auto dep_graph_segmentation = segmenter_dep_graph.CodepointToGlyphSegments( - noto_nastaliq_urdu.get(), {}, {{0x62a}, {0x62b}, {0x62c}, {0x62d}}); - auto dep_graph_only_segmentation = - segmenter_dep_graph_only.CodepointToGlyphSegments( - noto_nastaliq_urdu.get(), {}, {{0x62a}, {0x62b}, {0x62c}, {0x62d}}); - ASSERT_TRUE(segmentation.ok()) << segmentation.status(); - ASSERT_TRUE(dep_graph_segmentation.ok()) << segmentation.status(); - ASSERT_TRUE(dep_graph_only_segmentation.ok()) << segmentation.status(); - - ASSERT_EQ(segmentation->UnmappedGlyphs().size(), 12); - - ASSERT_EQ(segmentation->ToString(), - R"(initial font: { gid0 } -p0: { gid3, gid9, gid155 } -p1: { gid4, gid10, gid156 } -p2: { gid5, gid6, gid11, gid157 } -p3: { gid158 } -p4: { gid12, gid13, gid24, gid30, gid38, gid39, gid57, gid59, gid62, gid68, gid139, gid140, gid153, gid172 } -p5: { gid14, gid33, gid60, gid91, gid112, gid145, gid152 } -p6: { gid47, gid64, gid73, gid74, gid75, gid76, gid77, gid83, gid111, gid149, gid174, gid190, gid191 } -if (s0) then p0 -if (s1) then p1 -if (s2) then p2 -if (s3) then p3 -if ((s0 OR s1)) then p4 -if ((s2 OR s3)) then p5 -if ((s0 OR s1 OR s2 OR s3)) then p6 -)"); - ASSERT_EQ(segmentation->ToString(), dep_graph_segmentation->ToString()); - - ASSERT_EQ(dep_graph_only_segmentation->ToString(), - R"(initial font: { gid0 } -p0: { gid3, gid9, gid155 } -p1: { gid4, gid10, gid156 } -p2: { gid5, gid6, gid157 } -p3: { gid158 } -p4: { gid12, gid13, gid24, gid30, gid38, gid39, gid57, gid59, gid62, gid64, gid68, gid77, gid139, gid140, gid153, gid172 } -p5: { gid14, gid33, gid60, gid91, gid112, gid145, gid152, gid190, gid191 } -p6: { gid174 } -p7: { gid11 } -p8: { gid47, gid73, gid74, gid75, gid76, gid83, gid111, gid149 } -if (s0) then p0 -if (s1) then p1 -if (s2) then p2 -if (s3) then p3 -if ((s0 OR s1)) then p4 -if ((s2 OR s3)) then p5 -if ((s0 OR s1 OR s2 OR s3)) then p6 -if ((s0 OR s1) AND s2) then p7 -if ((s0 OR s1) AND (s2 OR s3)) then p8 -)"); -} - -TEST_F(ClosureGlyphSegmenterTest, UnmappedGlyphs_FindConditions) { - auto segmentation = FindConditionsCodepointToGlyphSegments( - noto_nastaliq_urdu.get(), {}, - {{0x20}, {0x62a}, {0x62b}, {0x62c}, {0x62d}}, std::nullopt); - ASSERT_TRUE(segmentation.ok()) << segmentation.status(); - - ASSERT_TRUE(segmentation->UnmappedGlyphs().empty()) - << segmentation->UnmappedGlyphs().ToString(); - - ASSERT_EQ(segmentation->ToString(), - R"(initial font: { gid0 } -p0: { gid1 } -p1: { gid3, gid9, gid155 } -p2: { gid4, gid10, gid156 } -p3: { gid5, gid6, gid11, gid157 } -p4: { gid158 } -p5: { gid12, gid13, gid24, gid30, gid38, gid39, gid57, gid59, gid62, gid68, gid139, gid140, gid153, gid172 } -p6: { gid14, gid33, gid60, gid91, gid112, gid145, gid152 } -p7: { gid47, gid64, gid73, gid74, gid75, gid76, gid77, gid83, gid111, gid149, gid174, gid190, gid191 } -if (s0) then p0 -if (s1) then p1 -if (s2) then p2 -if (s3) then p3 -if (s4) then p4 -if ((s1 OR s2)) then p5 -if ((s3 OR s4)) then p6 -if ((s1 OR s2 OR s3 OR s4)) then p7 -)"); -} - TEST_F(ClosureGlyphSegmenterTest, DepGraphOnly_FindConditions) { - auto segmentation = segmenter_dep_graph_only.CodepointToGlyphSegments( + auto segmentation = segmenter.CodepointToGlyphSegments( noto_nastaliq_urdu.get(), {}, {{0x20}, {0x62a}, {0x62b}, {0x62c}, {0x62d}}); ASSERT_TRUE(segmentation.ok()) << segmentation.status(); - ASSERT_TRUE(segmentation->UnmappedGlyphs().empty()) - << segmentation->UnmappedGlyphs().ToString(); - ASSERT_EQ(segmentation->ToString(), R"(initial font: { gid0 } p0: { gid1 } @@ -716,9 +505,6 @@ TEST_F(ClosureGlyphSegmenterTest, DepGraphOnly_WithSimplification) { {{0x20}, {0x62a}, {0x62b}, {0x62c}, {0x62d}}); ASSERT_TRUE(segmentation.ok()) << segmentation.status(); - ASSERT_TRUE(segmentation->UnmappedGlyphs().empty()) - << segmentation->UnmappedGlyphs().ToString(); - ASSERT_EQ(segmentation->ToString(), R"(initial font: { gid0 } p0: { gid1 } @@ -740,61 +526,6 @@ if ((s1 OR s2 OR s3 OR s4)) then p7 )"); } -TEST_F(ClosureGlyphSegmenterTest, UnmappedGlyphs_FindConditions_IsFallback) { - // Here the found conditions are equal to the fallback segment, this ensures - // everything works properly in this case. - auto segmentation = FindConditionsCodepointToGlyphSegments( - noto_nastaliq_urdu.get(), {}, {{0x62a}, {0x62b}, {0x62c}, {0x62d}}, - std::nullopt); - ASSERT_TRUE(segmentation.ok()) << segmentation.status(); - ASSERT_TRUE(segmentation->UnmappedGlyphs().empty()) - << segmentation->UnmappedGlyphs().ToString(); - ASSERT_EQ(segmentation->ToString(), - R"(initial font: { gid0 } -p0: { gid3, gid9, gid155 } -p1: { gid4, gid10, gid156 } -p2: { gid5, gid6, gid11, gid157 } -p3: { gid158 } -p4: { gid12, gid13, gid24, gid30, gid38, gid39, gid57, gid59, gid62, gid68, gid139, gid140, gid153, gid172 } -p5: { gid14, gid33, gid60, gid91, gid112, gid145, gid152 } -p6: { gid47, gid64, gid73, gid74, gid75, gid76, gid77, gid83, gid111, gid149, gid174, gid190, gid191 } -if (s0) then p0 -if (s1) then p1 -if (s2) then p2 -if (s3) then p3 -if ((s0 OR s1)) then p4 -if ((s2 OR s3)) then p5 -if ((s0 OR s1 OR s2 OR s3)) then p6 -)"); -} - -TEST_F(ClosureGlyphSegmenterTest, - UnmappedGlyphs_FallbackSegmentMovedToInitFont) { - auto segmentation = MoveToInitFontCodepointToGlyphSegments( - noto_nastaliq_urdu.get(), {}, {{0x62a}, {0x62b}, {0x62c}, {0x62d}}, - btree_map{}); - ASSERT_TRUE(segmentation.ok()) << segmentation.status(); - - ASSERT_EQ(segmentation->UnmappedGlyphs().size(), 0); - - ASSERT_EQ( - segmentation->ToString(), - R"(initial font: { gid0, gid47, gid64, gid73, gid74, gid75, gid76, gid77, gid83, gid111, gid149, gid174, gid190, gid191 } -p0: { gid3, gid9, gid155 } -p1: { gid4, gid10, gid156 } -p2: { gid5, gid6, gid11, gid157 } -p3: { gid158 } -p4: { gid12, gid13, gid24, gid30, gid38, gid39, gid57, gid59, gid62, gid68, gid139, gid140, gid153, gid172 } -p5: { gid14, gid33, gid60, gid91, gid112, gid145, gid152 } -if (s0) then p0 -if (s1) then p1 -if (s2) then p2 -if (s3) then p3 -if ((s0 OR s1)) then p4 -if ((s2 OR s3)) then p5 -)"); -} - TEST_F(ClosureGlyphSegmenterTest, FullRoboto_WithFeaturesAndDepGraph) { auto codepoints = FontHelper::ToCodepointsSet(roboto.get()); @@ -822,9 +553,8 @@ TEST_F(ClosureGlyphSegmenterTest, FullRoboto_WithFeaturesAndDepGraph) { smcp.feature_tags.insert(HB_TAG('s', 'm', 'c', 'p')); segments.push_back(smcp); - auto dep_graph_only_segmentation = - segmenter_dep_graph_only.CodepointToGlyphSegments( - roboto.get(), {}, segments, MergeStrategy::Heuristic(4000, 12000)); + auto dep_graph_only_segmentation = segmenter.CodepointToGlyphSegments( + roboto.get(), {}, segments, MergeStrategy::Heuristic(4000, 12000)); ASSERT_TRUE(dep_graph_only_segmentation.ok()) << dep_graph_only_segmentation.status(); } @@ -983,15 +713,15 @@ TEST_F(ClosureGlyphSegmenterTest, SimpleSegmentation_PatchMerge) { strategy.SetOptimizationCutoffFraction(0.0); strategy.SetUsePatchMerges(true); - auto segmentation = segmenter_dep_graph_only.CodepointToGlyphSegments( - roboto.get(), {}, - { - {'A'}, - {'C'}, - {0xC1}, /* Aacute */ - {0x106}, /* Cacute */ - }, - strategy); + auto segmentation = + segmenter.CodepointToGlyphSegments(roboto.get(), {}, + { + {'A'}, + {'C'}, + {0xC1}, /* Aacute */ + {0x106}, /* Cacute */ + }, + strategy); ASSERT_TRUE(segmentation.ok()) << segmentation.status(); std::vector expected_segments = { @@ -1024,15 +754,15 @@ TEST_F(ClosureGlyphSegmenterTest, SimpleSegmentation_NoPatchMerge) { strategy.SetOptimizationCutoffFraction(0.0); strategy.SetUsePatchMerges(false); - auto segmentation = segmenter_dep_graph_only.CodepointToGlyphSegments( - roboto.get(), {}, - { - {'A'}, - {'C'}, - {0xC1}, /* Aacute */ - {0x106}, /* Cacute */ - }, - strategy); + auto segmentation = + segmenter.CodepointToGlyphSegments(roboto.get(), {}, + { + {'A'}, + {'C'}, + {0xC1}, /* Aacute */ + {0x106}, /* Cacute */ + }, + strategy); ASSERT_TRUE(segmentation.ok()) << segmentation.status(); std::vector expected_segments = { @@ -1066,15 +796,15 @@ TEST_F(ClosureGlyphSegmenterTest, SimpleSegmentation_PatchMerge_MinGroupSize) { strategy.SetOptimizationCutoffFraction(0.0); strategy.SetUsePatchMerges(true); - auto segmentation = segmenter_dep_graph_only.CodepointToGlyphSegments( - roboto.get(), {}, - { - {'A'}, - {'C'}, - {0xC1}, /* Aacute */ - {0x106}, /* Cacute */ - }, - strategy); + auto segmentation = + segmenter.CodepointToGlyphSegments(roboto.get(), {}, + { + {'A'}, + {'C'}, + {0xC1}, /* Aacute */ + {0x106}, /* Cacute */ + }, + strategy); ASSERT_TRUE(segmentation.ok()) << segmentation.status(); std::vector expected_segments = { @@ -1103,15 +833,14 @@ if ((s1 OR s3) AND (s2 OR s3)) then p4 // Now with min group size larger merges will be done to reach min group size strategy.SetMinimumGroupSize(3); - segmentation = segmenter_dep_graph_only.CodepointToGlyphSegments( - roboto.get(), {}, - { - {'A'}, - {'C'}, - {0xC1}, /* Aacute */ - {0x106}, /* Cacute */ - }, - strategy); + segmentation = segmenter.CodepointToGlyphSegments(roboto.get(), {}, + { + {'A'}, + {'C'}, + {0xC1}, /* Aacute */ + {0x106}, /* Cacute */ + }, + strategy); ASSERT_TRUE(segmentation.ok()) << segmentation.status(); ASSERT_EQ(segmentation->Segments(), expected_segments); // Group size minimum is met for everything other than the last condition @@ -1139,7 +868,7 @@ TEST_F(ClosureGlyphSegmenterTest, PatchMerge_WithSimplification) { strategy.SetOptimizationCutoffFraction(0.0); strategy.SetUsePatchMerges(true); - auto segmentation = segmenter_dep_graph_only.CodepointToGlyphSegments( + auto segmentation = segmenter.CodepointToGlyphSegments( noto_nastaliq_urdu.get(), {}, {{0x62a}, {0x62b}, {0x62c}, {0x62d}}, strategy); auto segmentation_with_simplification = @@ -1311,11 +1040,11 @@ TEST_F(ClosureGlyphSegmenterTest, TotalCost) { UnigramProbabilityCalculator calculator(std::move(frequencies)); // Basic no segment case. - GlyphSegmentation segmentation1({'a', 'b', 'c'}, {}, {}); - auto sc = GlyphSegmentation::ConditionsToSegmentation({}, {}, segmentation1); + GlyphSegmentation segmentation1({'a', 'b', 'c'}, {}); + auto sc = GlyphSegmentation::ConditionsToSegmentation({}, segmentation1); ASSERT_TRUE(sc.ok()) << sc; - ClosureGlyphSegmenter segmenter(8, 8, PATCH, CLOSURE_ONLY, resolver); + ClosureGlyphSegmenter segmenter(8, 8, DEP_GRAPH_ONLY, resolver); std::vector base_cost = *segmenter.TotalCosts(roboto.get(), segmentation1, {&calculator}); ASSERT_GT(base_cost[0].ift_init_cost, 1000); @@ -1324,13 +1053,13 @@ TEST_F(ClosureGlyphSegmenterTest, TotalCost) { ASSERT_EQ(base_cost[0].uncovered_ift_patch_cost, 0); // Add some patches - GlyphSegmentation segmentation2({'a', 'b', 'c'}, {}, {}); + GlyphSegmentation segmentation2({'a', 'b', 'c'}, {}); sc = GlyphSegmentation::ConditionsToSegmentation( { {ActivationCondition::exclusive_segment(0, 0), {100, 101, 102}}, {ActivationCondition::exclusive_segment(1, 0), {103, 104, 105}}, }, - {}, segmentation2); + segmentation2); ASSERT_TRUE(sc.ok()) << sc; std::vector segments{ @@ -1347,7 +1076,7 @@ TEST_F(ClosureGlyphSegmenterTest, TotalCost) { ASSERT_EQ(with_patches_cost[0].uncovered_ift_patch_cost, 0); // Add uncovered segments ('f' and 'g' are not in frequencies). - GlyphSegmentation segmentation3({'a', 'b', 'c'}, {}, {}); + GlyphSegmentation segmentation3({'a', 'b', 'c'}, {}); sc = GlyphSegmentation::ConditionsToSegmentation( { {ActivationCondition::exclusive_segment(0, 0), {100, 101, 102}}, @@ -1355,7 +1084,7 @@ TEST_F(ClosureGlyphSegmenterTest, TotalCost) { {ActivationCondition::and_segments({0, 2}, 0), {105}}, {ActivationCondition::and_segments({2, 3}, 0), {106, 107}}, }, - {}, segmentation3); + segmentation3); ASSERT_TRUE(sc.ok()) << sc; std::vector segments3{ @@ -2153,62 +1882,20 @@ TEST_F(ClosureGlyphSegmenterTest, ConjunctiveAdditionalConditions) { smcp.feature_tags.insert(HB_TAG('s', 'm', 'c', 'p')); smcp.feature_tags.insert(HB_TAG('c', '2', 's', 'c')); - auto segmentation = segmenter.CodepointToGlyphSegments(roboto.get(), {0xE4}, - { - {0xD6}, // Ö - {0xF6}, // ö - smcp, - }, - MergeStrategy::None()); - auto dep_graph_segmentation = - segmenter_dep_graph.CodepointToGlyphSegments(roboto.get(), {0xE4}, - { - {0xD6}, // Ö - {0xF6}, // ö - smcp, - }, - MergeStrategy::None()); - auto dep_graph_only_segmentation = - segmenter_dep_graph_only.CodepointToGlyphSegments(roboto.get(), {0xE4}, - { - {0xD6}, // Ö - {0xF6}, // ö - smcp, - }, - MergeStrategy::None()); + auto segmentation = CodepointToGlyphSegments(roboto.get(), {0xE4}, + { + {0xD6}, // Ö + {0xF6}, // ö + smcp, + }, + MergeStrategy::None()); ASSERT_TRUE(segmentation.ok()) << segmentation.status(); - ASSERT_TRUE(dep_graph_segmentation.ok()) << dep_graph_segmentation.status(); - ASSERT_TRUE(dep_graph_only_segmentation.ok()) - << dep_graph_only_segmentation.status(); - // Note: gid839 is the small caps O dieresis, it's currently mapped with - // condition 'if {smcp}', the true condition is 'if {smcp} AND {Ö or ö}' - // but the closure segmenter does not currently discover composite conditions. ASSERT_EQ(segmentation->ToString(), R"(initial font: { gid0, gid69, gid106, gid670 } p0: { gid51, gid660 } p1: { gid83, gid687 } -p2: { gid477, gid563, gid822, gid839 } -if (s0) then p0 -if (s1) then p1 -if (s2) then p2 -)"); - - ASSERT_EQ(dep_graph_segmentation->ToString(), - R"(initial font: { gid0, gid69, gid106, gid670 } -p0: { gid51, gid660 } -p1: { gid83, gid687 } -p2: { gid477, gid563, gid822, gid839 } -if (s0) then p0 -if (s1) then p1 -if (s2) then p2 -)"); - - ASSERT_EQ(dep_graph_only_segmentation->ToString(), - R"(initial font: { gid0, gid69, gid106, gid670 } -p0: { gid51, gid660 } -p1: { gid83, gid687 } p2: { gid563, gid822 } p3: { gid477, gid839 } if (s0) then p0 @@ -2328,7 +2015,7 @@ TEST_F(ClosureGlyphSegmenterTest, InitFontMergingAndFindConditions) { profile.init_font_merge_threshold = -75; MergeStrategy strategy = MergeStrategy::CostBased(std::move(profile)); - auto segmentation = FindConditionsCodepointToGlyphSegments( + auto segmentation = CodepointToGlyphSegments( noto_nastaliq_urdu.get(), {}, {{0x20}, {0x62a}, {0x62b}, {0x62c}, {0x62d}}, strategy); ASSERT_TRUE(segmentation.ok()) << segmentation.status(); @@ -2339,12 +2026,10 @@ TEST_F(ClosureGlyphSegmenterTest, InitFontMergingAndFindConditions) { }; ASSERT_EQ(segmentation->Segments(), expected_segments); - // Complex condition if ((s1 OR s2 OR s3 OR s4)) then ... is moved to the init - // font. ASSERT_EQ( segmentation->ToString(), - R"(initial font: { gid0, gid1, gid3, gid4, gid9, gid10, gid12, gid13, gid24, gid30, gid38, gid39, gid57, gid59, gid62, gid68, gid139, gid140, gid153, gid155, gid156, gid172, gid174 } -p0: { gid5, gid6, gid11, gid14, gid33, gid47, gid60, gid64, gid73, gid74, gid75, gid76, gid77, gid83, gid91, gid111, gid112, gid145, gid149, gid152, gid157, gid158, gid190, gid191 } + R"(initial font: { gid0, gid1, gid3, gid4, gid9, gid10, gid12, gid13, gid24, gid30, gid38, gid39, gid57, gid59, gid62, gid64, gid68, gid77, gid139, gid140, gid153, gid155, gid156, gid172, gid174 } +p0: { gid5, gid6, gid11, gid14, gid33, gid47, gid60, gid73, gid74, gid75, gid76, gid83, gid91, gid111, gid112, gid145, gid149, gid152, gid157, gid158, gid190, gid191 } if (s3) then p0 )"); } @@ -2704,7 +2389,7 @@ TEST_F(ClosureGlyphSegmenterTest, // When max_patch_size allows the merged exclusive patch {77, 80} but is // smaller than the combined non-exclusive patch {444, 445, 446, 447}, s0 and // s1 must not be merged. - for (ClosureGlyphSegmenter* seg : {&segmenter, &segmenter_dep_graph_only}) { + for (ClosureGlyphSegmenter* seg : {&segmenter}) { auto not_merged = seg->CodepointToGlyphSegments( roboto.get(), {}, segments, MergeStrategy::Heuristic(merged_exclusive_size, @@ -2745,7 +2430,7 @@ TEST_F(ClosureGlyphSegmenterTest, PatchSizeCacheImpl patch_sizer(roboto.get(), segmenter.brotli_quality()); uint32_t i_j_size = *patch_sizer.GetPatchSize({77, 78}); - for (ClosureGlyphSegmenter* seg : {&segmenter, &segmenter_dep_graph_only}) { + for (ClosureGlyphSegmenter* seg : {&segmenter}) { auto result = seg->CodepointToGlyphSegments( roboto.get(), {}, segments, MergeStrategy::Heuristic(i_j_size)); ASSERT_TRUE(result.ok()) << result.status(); @@ -2780,7 +2465,7 @@ TEST_F(ClosureGlyphSegmenterTest, // gid77 ('i'), gid444 ('fi'), gid446 ('ffi'), gid640 (Aacute). uint32_t s0_s3_size = *patch_sizer.GetPatchSize({37, 74, 77, 444, 446, 640}); - for (ClosureGlyphSegmenter* seg : {&segmenter, &segmenter_dep_graph_only}) { + for (ClosureGlyphSegmenter* seg : {&segmenter}) { auto result = seg->CodepointToGlyphSegments( roboto.get(), {}, segments, MergeStrategy::Heuristic(s0_s3_size)); ASSERT_TRUE(result.ok()) << result.status(); @@ -2814,7 +2499,7 @@ TEST_F(ClosureGlyphSegmenterTest, PatchSizeCacheImpl patch_sizer(roboto.get(), segmenter.brotli_quality()); uint32_t s_t_size = *patch_sizer.GetPatchSize({87, 88}); - for (ClosureGlyphSegmenter* seg : {&segmenter, &segmenter_dep_graph_only}) { + for (ClosureGlyphSegmenter* seg : {&segmenter}) { auto result = seg->CodepointToGlyphSegments( roboto.get(), {'f', 'F'}, segments, MergeStrategy::Heuristic(s_t_size, s_t_size)); @@ -2853,7 +2538,7 @@ TEST_F(ClosureGlyphSegmenterTest, {{0, 1, 2}, MergeStrategy::Heuristic(a_b_size)}, }; - for (ClosureGlyphSegmenter* seg : {&segmenter, &segmenter_dep_graph_only}) { + for (ClosureGlyphSegmenter* seg : {&segmenter}) { auto result = seg->CodepointToGlyphSegments(roboto.get(), {}, segments, merge_groups); ASSERT_TRUE(result.ok()) << result.status(); @@ -2880,10 +2565,9 @@ TEST_F(ClosureGlyphSegmenterTest, HeuristicStrategy_BatchesInertBaseSegments) { *patch_sizer.GetPatchSize({69, 70, 71, 72, 73, 75}); auto context = SegmentationContext::InitializeSegmentationContext( - roboto.get(), {}, segments, segmenter_dep_graph_only.unmapped_glyph_handling(), - segmenter_dep_graph_only.condition_analysis_mode(), - segmenter_dep_graph_only.brotli_quality(), - segmenter_dep_graph_only.init_font_merging_brotli_quality(), resolver); + roboto.get(), {}, segments, segmenter.condition_analysis_mode(), + segmenter.brotli_quality(), segmenter.init_font_merging_brotli_quality(), + resolver); ASSERT_TRUE(context.ok()) << context.status(); SegmentSet all_segments{0, 1, 2, 3, 4, 5, 6, 7}; @@ -2933,7 +2617,7 @@ TEST_F(ClosureGlyphSegmenterTest, // Glyphs for 'a', 'b', 'c', 'd' are 69, 70, 71, 72. uint32_t a_b_c_d_size = *patch_sizer.GetPatchSize({69, 70, 71, 72}); - for (ClosureGlyphSegmenter* seg : {&segmenter, &segmenter_dep_graph_only}) { + for (ClosureGlyphSegmenter* seg : {&segmenter}) { auto result = seg->CodepointToGlyphSegments( roboto.get(), {}, segments, MergeStrategy::Heuristic(a_b_c_d_size, a_b_c_d_size)); @@ -2950,17 +2634,28 @@ TEST_F(ClosureGlyphSegmenterTest, } } +#pragma GCC diagnostic push +#pragma GCC diagnostic ignored "-Wdeprecated-declarations" + TEST_F(ClosureGlyphSegmenterTest, - UnicodeRangeValidation_RejectsMoveToInitFont) { - ClosureGlyphSegmenter seg(8, 8, MOVE_TO_INIT_FONT, CLOSURE_ONLY, resolver, - ift::config::UNICODE_RANGE); - auto result = seg.CodepointToGlyphSegments(roboto.get(), {'a'}, {{'b'}, {'c'}}); - EXPECT_TRUE(absl::IsInvalidArgument(result.status())) << result.status(); + Validation_RejectsDeprecatedConditionAnalysisModes) { + for (auto target_mode : {ift::config::IFT, ift::config::UNICODE_RANGE}) { + for (auto mode : + {ift::config::CLOSURE_ONLY, ift::config::CLOSURE_AND_DEP_GRAPH, + ift::config::CLOSURE_AND_VALIDATE_DEP_GRAPH}) { + ClosureGlyphSegmenter seg(8, 8, mode, resolver, target_mode); + auto result = + seg.CodepointToGlyphSegments(roboto.get(), {'a'}, {{'b'}, {'c'}}); + EXPECT_TRUE(absl::IsInvalidArgument(result.status())) << result.status(); + } + } } +#pragma GCC diagnostic pop + TEST_F(ClosureGlyphSegmenterTest, UnicodeRangeValidation_RejectsFeatureSegments) { - ClosureGlyphSegmenter seg(8, 8, FIND_CONDITIONS, CLOSURE_ONLY, resolver, + ClosureGlyphSegmenter seg(8, 8, DEP_GRAPH_ONLY, resolver, ift::config::UNICODE_RANGE); SubsetDefinition smcp; smcp.feature_tags.insert(HB_TAG('s', 'm', 'c', 'p')); @@ -2971,7 +2666,7 @@ TEST_F(ClosureGlyphSegmenterTest, TEST_F(ClosureGlyphSegmenterTest, UnicodeRangeValidation_RejectsInitFontMerging) { - ClosureGlyphSegmenter seg(8, 8, FIND_CONDITIONS, CLOSURE_ONLY, resolver, + ClosureGlyphSegmenter seg(8, 8, DEP_GRAPH_ONLY, resolver, ift::config::UNICODE_RANGE); MergeStrategy strategy = MergeStrategy::CostBased(ProbabilityProfile( std::make_shared( @@ -2983,7 +2678,7 @@ TEST_F(ClosureGlyphSegmenterTest, } TEST_F(ClosureGlyphSegmenterTest, UnicodeRangeValidation_RejectsPatchMerging) { - ClosureGlyphSegmenter seg(8, 8, FIND_CONDITIONS, CLOSURE_ONLY, resolver, + ClosureGlyphSegmenter seg(8, 8, DEP_GRAPH_ONLY, resolver, ift::config::UNICODE_RANGE); MergeStrategy strategy = MergeStrategy::CostBased( ProbabilityProfile(std::make_shared( @@ -2996,13 +2691,13 @@ TEST_F(ClosureGlyphSegmenterTest, UnicodeRangeValidation_RejectsPatchMerging) { TEST_F(ClosureGlyphSegmenterTest, UnicodeRangeValidation_RejectsInvalidThreshold) { - ClosureGlyphSegmenter seg_neg(8, 8, FIND_CONDITIONS, CLOSURE_ONLY, resolver, + ClosureGlyphSegmenter seg_neg(8, 8, DEP_GRAPH_ONLY, resolver, ift::config::UNICODE_RANGE, -0.1); auto result = seg_neg.CodepointToGlyphSegments(roboto.get(), {'a'}, {{'b'}, {'c'}}); EXPECT_TRUE(absl::IsInvalidArgument(result.status())) << result.status(); - ClosureGlyphSegmenter seg_gt1(8, 8, FIND_CONDITIONS, CLOSURE_ONLY, resolver, + ClosureGlyphSegmenter seg_gt1(8, 8, DEP_GRAPH_ONLY, resolver, ift::config::UNICODE_RANGE, 1.1); result = seg_gt1.CodepointToGlyphSegments(roboto.get(), {'a'}, {{'b'}, {'c'}}); @@ -3015,9 +2710,8 @@ TEST_F(ClosureGlyphSegmenterTest, {'b'}, {'f'}, {'i'}, {'c', 'd'}, {'d', 'e'}}; for (auto analysis_mode : - {CLOSURE_ONLY, CLOSURE_AND_VALIDATE_DEP_GRAPH, DEP_GRAPH_ONLY, - DEP_GRAPH_ONLY_WITH_SIMPLIFICATION}) { - ClosureGlyphSegmenter seg(8, 8, FIND_CONDITIONS, analysis_mode, resolver, + {DEP_GRAPH_ONLY, DEP_GRAPH_ONLY_WITH_SIMPLIFICATION}) { + ClosureGlyphSegmenter seg(8, 8, analysis_mode, resolver, ift::config::UNICODE_RANGE); auto segmentation = seg.CodepointToGlyphSegments(roboto.get(), {'a'}, segments); @@ -3048,9 +2742,8 @@ if (s3) then p2 TEST_F(ClosureGlyphSegmenterTest, UnicodeRange_MakeActivationConditionsDisjoint_ComplexConditions) { for (auto analysis_mode : - {CLOSURE_ONLY, CLOSURE_AND_VALIDATE_DEP_GRAPH, DEP_GRAPH_ONLY, - DEP_GRAPH_ONLY_WITH_SIMPLIFICATION}) { - ClosureGlyphSegmenter seg(8, 8, FIND_CONDITIONS, analysis_mode, resolver, + {DEP_GRAPH_ONLY, DEP_GRAPH_ONLY_WITH_SIMPLIFICATION}) { + ClosureGlyphSegmenter seg(8, 8, analysis_mode, resolver, ift::config::UNICODE_RANGE); auto segmentation = seg.CodepointToGlyphSegments( noto_nastaliq_urdu.get(), {}, @@ -3089,8 +2782,7 @@ TEST_F(ClosureGlyphSegmenterTest, strategy.SetOptimizationCutoffFraction(1.0); strategy.SetUsePatchMerges(false); - ClosureGlyphSegmenter seg(8, 8, FIND_CONDITIONS, - CLOSURE_AND_VALIDATE_DEP_GRAPH, resolver, + ClosureGlyphSegmenter seg(8, 8, DEP_GRAPH_ONLY, resolver, ift::config::UNICODE_RANGE, /*disjoint_conditions_probability_threshold=*/0.2); @@ -3099,7 +2791,8 @@ TEST_F(ClosureGlyphSegmenterTest, ASSERT_TRUE(segmentation.ok()) << segmentation.status(); // 'f' and 'i' were merged into s0; 0xc1 (s1) and 0x106 (s2) were not merged, - // and their non-exclusive shared patch {gid117, gid169, gid667} was omitted. + // and non-exclusive patches ({gid117, gid169, gid667} and {gid141, gid609, + // gid679}) below the probability threshold were omitted. std::vector expected_segments = { {'f', 'i'}, {0xc1}, {0x106}}; EXPECT_EQ(segmentation->Segments(), expected_segments); @@ -3110,7 +2803,7 @@ TEST_F(ClosureGlyphSegmenterTest, EXPECT_EQ(segmentation->ToString(), R"(initial font: { gid0, gid69 } -p0: { gid74, gid77, gid141, gid444, gid446, gid609, gid679 } +p0: { gid74, gid77, gid444, gid446 } p1: { gid37, gid640 } p2: { gid39, gid700 } if (s0) then p0 @@ -3135,8 +2828,8 @@ TEST_F(ClosureGlyphSegmenterTest, Segment({'f'}), Segment({'i'}), Segment({0xc1}), Segment({0x106})}; auto context = SegmentationContext::InitializeSegmentationContext( - roboto.get(), {'a'}, segments, FIND_CONDITIONS, CLOSURE_ONLY, 8, 8, - resolver, ift::config::UNICODE_RANGE); + roboto.get(), {'a'}, segments, DEP_GRAPH_ONLY, 8, 8, resolver, + ift::config::UNICODE_RANGE); ASSERT_TRUE(context.ok()) << context.status(); ASSERT_TRUE(context->CacheInitialGlyphProbabilities({calc.get()}).ok()); @@ -3161,8 +2854,7 @@ TEST_F(ClosureGlyphSegmenterTest, TEST_F(ClosureGlyphSegmenterTest, UnicodeRange_InitialSegmentCodepointsAndFeatures) { - ClosureGlyphSegmenter seg(8, 8, FIND_CONDITIONS, - CLOSURE_AND_VALIDATE_DEP_GRAPH, resolver, + ClosureGlyphSegmenter seg(8, 8, DEP_GRAPH_ONLY, resolver, ift::config::UNICODE_RANGE); SubsetDefinition init; init.codepoints = {'f'}; @@ -3193,8 +2885,7 @@ TEST_F(ClosureGlyphSegmenterTest, TEST_F(ClosureGlyphSegmenterTest, UnicodeRange_MakeActivationConditionsDisjoint_SpansMultipleMergeGroups) { - ClosureGlyphSegmenter seg(8, 8, FIND_CONDITIONS, - CLOSURE_AND_VALIDATE_DEP_GRAPH, resolver, + ClosureGlyphSegmenter seg(8, 8, DEP_GRAPH_ONLY, resolver, ift::config::UNICODE_RANGE); // s0 ('f') and s1 ('i') are non-shared in Group 0 and interact via 'fi'. @@ -3249,8 +2940,7 @@ TEST_F(ClosureGlyphSegmenterTest, TEST_F(ClosureGlyphSegmenterTest, UnicodeRange_MakeActivationConditionsDisjoint_ReordersByNewProbability) { - ClosureGlyphSegmenter seg(8, 8, FIND_CONDITIONS, - CLOSURE_AND_VALIDATE_DEP_GRAPH, resolver, + ClosureGlyphSegmenter seg(8, 8, DEP_GRAPH_ONLY, resolver, ift::config::UNICODE_RANGE); // Initially {'b'} (0.5) > {'f'} (0.4) > {'i'} (0.3) > {'c'} (0.1). @@ -3306,8 +2996,7 @@ TEST_F(ClosureGlyphSegmenterTest, UnicodeRange_Woff2PatchSizeCostFunction) { strategy.SetUsePatchMerges(false); { - ClosureGlyphSegmenter ift_seg(1, 1, FIND_CONDITIONS, - CLOSURE_AND_VALIDATE_DEP_GRAPH, resolver, + ClosureGlyphSegmenter ift_seg(1, 1, DEP_GRAPH_ONLY, resolver, ift::config::IFT); auto ift_segmentation = ift_seg.CodepointToGlyphSegments( roboto.get(), {'a'}, {{'b'}, {'c'}}, strategy); @@ -3317,8 +3006,7 @@ TEST_F(ClosureGlyphSegmenterTest, UnicodeRange_Woff2PatchSizeCostFunction) { } for (uint32_t quality : {0u, 1u}) { - ClosureGlyphSegmenter ur_seg(quality, quality, FIND_CONDITIONS, - CLOSURE_AND_VALIDATE_DEP_GRAPH, resolver, + ClosureGlyphSegmenter ur_seg(quality, quality, DEP_GRAPH_ONLY, resolver, ift::config::UNICODE_RANGE); auto ur_segmentation = ur_seg.CodepointToGlyphSegments( roboto.get(), {'a'}, {{'b'}, {'c'}}, strategy); @@ -3335,22 +3023,22 @@ TEST_F(ClosureGlyphSegmenterTest, UnicodeRange_TotalCosts) { }; UnigramProbabilityCalculator calculator(std::move(frequencies)); - GlyphSegmentation segmentation({'a'}, {}, {}); + GlyphSegmentation segmentation({'a'}, {}); auto sc = GlyphSegmentation::ConditionsToSegmentation( { {ActivationCondition::exclusive_segment(0, 0), {70}}, {ActivationCondition::exclusive_segment(1, 0), {71}}, }, - {}, segmentation); + segmentation); ASSERT_TRUE(sc.ok()) << sc; segmentation.CopySegments({{'b'}, {'c'}}); - ClosureGlyphSegmenter ift_seg(8, 8, FIND_CONDITIONS, CLOSURE_ONLY, resolver, + ClosureGlyphSegmenter ift_seg(8, 8, DEP_GRAPH_ONLY, resolver, ift::config::IFT); std::vector ift_costs = *ift_seg.TotalCosts(roboto.get(), segmentation, {&calculator}); - ClosureGlyphSegmenter ur_seg(8, 8, FIND_CONDITIONS, CLOSURE_ONLY, resolver, + ClosureGlyphSegmenter ur_seg(8, 8, DEP_GRAPH_ONLY, resolver, ift::config::UNICODE_RANGE); std::vector ur_costs = *ur_seg.TotalCosts(roboto.get(), segmentation, {&calculator}); @@ -3512,8 +3200,7 @@ TEST_F(ClosureGlyphSegmenterTest, AllPairsMerging_UnicodeRangeMode) { strategy.SetUsePatchMerges(false); for (uint32_t quality : {0u, 1u}) { - ClosureGlyphSegmenter ur_seg(quality, quality, FIND_CONDITIONS, - CLOSURE_AND_VALIDATE_DEP_GRAPH, resolver, + ClosureGlyphSegmenter ur_seg(quality, quality, DEP_GRAPH_ONLY, resolver, ift::config::UNICODE_RANGE); auto ur_segmentation = ur_seg.CodepointToGlyphSegments( roboto.get(), {'a'}, {{'b'}, {'c'}, {'f'}, {'i'}}, strategy); diff --git a/ift/encoder/complex_condition_finder.cc b/ift/encoder/complex_condition_finder.cc deleted file mode 100644 index ad3b6c12..00000000 --- a/ift/encoder/complex_condition_finder.cc +++ /dev/null @@ -1,324 +0,0 @@ -#include "ift/encoder/complex_condition_finder.h" - -#include - -#include "absl/log/log.h" -#include "absl/status/status.h" -#include "ift/common/int_set.h" -#include "ift/encoder/glyph_closure_cache.h" -#include "ift/encoder/requested_segmentation_information.h" -#include "ift/encoder/subset_definition.h" -#include "ift/encoder/types.h" - -using absl::btree_map; -using absl::Status; -using absl::StatusOr; -using ift::common::GlyphSet; -using ift::common::SegmentSet; - -// For more information on this process see the explanation in: -// ../../docs/experimental/closure_glyph_segmentation_complex_conditions.md - -namespace ift::encoder { - -// One unit of work for the analysis. One from to_be_tested will be checked. -struct Task { - // These are the segments of the full condition found so far. Should be - // excluded from the analysis now, does not include elements of sub_condition. - SegmentSet full_condition; - - // These segments have been determined to be part of a sub condition. - SegmentSet sub_condition; - - // These segments have not yet been tested. - SegmentSet to_be_tested; - - // The subset definition of the union of segments in sub_condition, - // to_be_tested, and the initial font subset definition. - SubsetDefinition subset_definition; - - // The set of glyphs in scope for analysis. - GlyphSet glyphs; -}; - -struct Context { - const SegmentSet all_segments; - const SegmentSet inscope_segments; - const RequestedSegmentationInformation* segmentation_info; - GlyphClosureCache* glyph_closure_cache; - std::vector queue; - - public: - Status ScheduleInitialTasks( - GlyphSet glyphs, btree_map existing_conditions) { - if (glyphs.intersects(segmentation_info->InitFontGlyphs())) { - return absl::InvalidArgumentError( - "Can't analyze glyphs that are in the init font."); - } - - // Each existing condition will map to one initial task that excludes the - // existing condition from the analysis. - for (const auto& [segments, glyph_sub_group] : existing_conditions) { - if (!TRY(InClosure(segments, glyph_sub_group))) { - return absl::InvalidArgumentError( - "The glyphs of existing conditions must be in the closure of " - "condition segments."); - } - TRYV(ScheduleExistingConditionTask(segments, glyph_sub_group, glyphs)); - } - - if (glyphs.empty()) { - return absl::OkStatus(); - } - - if (!TRY(InClosure(all_segments, glyphs))) { - return absl::InvalidArgumentError( - "glyphs to analyze must be in the closure of all segments."); - } - - SegmentSet to_be_tested = all_segments; - to_be_tested.intersect(inscope_segments); - - // If any glyphs remain that do not have existing conditions these are - // covered by a task with no excluded segments. - queue.push_back(CreateTask({}, {}, to_be_tested, glyphs)); - - return absl::OkStatus(); - } - - Status ProcessQueue(btree_map& glyph_to_conditions) { - // TODO(garretrieger): to reduce runtime of this analysis the processing of - // the queue could be parallelized by using a threadpool to run tasks. The - // tasks are fully independent so this should be straightforward. - while (!queue.empty()) { - Task next = std::move(queue.back()); - queue.pop_back(); - TRYV(RunAnalysisTask(next, glyph_to_conditions)); - } - return absl::OkStatus(); - } - - private: - Task CreateTask(SegmentSet full_condition, SegmentSet sub_condition, - SegmentSet to_be_tested, GlyphSet glyphs) { - SegmentSet all = sub_condition; - all.union_set(to_be_tested); - SubsetDefinition task_definition = - segmentation_info->CombinedDefinition(all); - return Task{ - .full_condition = full_condition, - .sub_condition = sub_condition, - .to_be_tested = to_be_tested, - .subset_definition = task_definition, - .glyphs = glyphs, - }; - } - - Task CreateSubTask(const Task& task, GlyphSet new_glyphs, - segment_index_t tested, bool keep) { - SegmentSet new_to_be_tested = task.to_be_tested; - new_to_be_tested.erase(tested); - - SegmentSet new_sub_condition = task.sub_condition; - SubsetDefinition new_sub_definition = task.subset_definition; - if (keep) { - new_sub_condition.insert(tested); - } else { - new_sub_definition.Subtract( - segmentation_info->Segments().at(tested).Definition()); - } - - return Task{ - .full_condition = task.full_condition, - .sub_condition = new_sub_condition, - .to_be_tested = new_to_be_tested, - .subset_definition = new_sub_definition, - .glyphs = new_glyphs, - }; - } - - // Returns true if all glyphs are in the closure of segments. - StatusOr InClosure(const SegmentSet& segments, const GlyphSet& glyphs) { - GlyphSet closure = - TRY(glyph_closure_cache->SegmentClosure(segmentation_info, segments)); - return glyphs.is_subset_of(closure); - } - - StatusOr> HasAdditionalConditions( - const SegmentSet& segments, const GlyphSet& glyphs) { - SegmentSet except = all_segments; - except.subtract(segments); - GlyphSet closure_glyphs = - TRY(glyph_closure_cache->SegmentClosure(segmentation_info, except)); - closure_glyphs.intersect(glyphs); - - except.intersect(inscope_segments); - return std::make_pair(closure_glyphs, std::move(except)); - } - - Status ScheduleExistingConditionTask(const SegmentSet& condition, - const GlyphSet& condition_glyphs, - GlyphSet& all_glyphs) { - // We need to check if there are any additional conditions, - // if there aren't there is no need to schedule the analysis. - auto [glyphs_with_additional_conditions, except] = - TRY(HasAdditionalConditions(condition, condition_glyphs)); - - if (glyphs_with_additional_conditions.empty()) { - return absl::OkStatus(); - } - - queue.push_back( - CreateTask(condition, {}, except, glyphs_with_additional_conditions)); - all_glyphs.subtract(condition_glyphs); - - return absl::OkStatus(); - } - - // Each analysis step checks one segment to see for which glyphs that segment - // is relevant. The supplied task data structure gives the specific state - // around which the segment is tested. - // - // To test a segment a closure is run without the segment being tested: - // - For inscope glyphs which appear in the closure the test segment is not - // relevant for these glyphs - // - For inscope glyphs which do not appear in the closure the test segment is - // relevant for these glyphs. - // - // Based on the anlysis results up to two more analysis steps are spawned (one - // for glyphs where segment is relevant, the other where it is not relevant) - // to test the next segment. - // - // Once all segments are tested the resulting sub condition segments - // is recorded in out. Lastly, the non-relevant segments are checked to see - // if additional conditions are present, if they are another analysis task is - // queued to discover the additional conditions. - Status RunAnalysisTask( - Task task, btree_map& glyph_to_conditions) { - if (task.glyphs.empty()) { - // Nothing left to check. - return absl::OkStatus(); - } - - if (task.to_be_tested.empty()) { - return RecordSubCondition(task, glyph_to_conditions); - } - - segment_index_t test_segment = *task.to_be_tested.min(); - - SubsetDefinition subset_definition = task.subset_definition; - subset_definition.Subtract( - segmentation_info->Segments().at(test_segment).Definition()); - GlyphSet closure_glyphs = - TRY(glyph_closure_cache->GlyphClosure(subset_definition)); - - GlyphSet needs_test_segment = task.glyphs; - needs_test_segment.subtract(closure_glyphs); - GlyphSet doesnt_need_test_segment = task.glyphs; - doesnt_need_test_segment.intersect(closure_glyphs); - - queue.push_back( - CreateSubTask(task, doesnt_need_test_segment, test_segment, false)); - queue.push_back( - CreateSubTask(task, needs_test_segment, test_segment, true)); - - return absl::OkStatus(); - } - - // A sub condition has been found, record it and kick off any - // further analysis needed for additional conditions. - Status RecordSubCondition( - Task task, btree_map& glyph_to_conditions) { - for (glyph_id_t gid : task.glyphs) { - glyph_to_conditions[gid].union_set(task.sub_condition); - } - - // We have identified a sub condition for glyphs, however as usual - // there may be remaining additional conditions which we need to - // check for - task.full_condition.union_set(task.sub_condition); - auto [additional_condition_glyphs, remaining] = - TRY(HasAdditionalConditions(task.full_condition, task.glyphs)); - - if (!additional_condition_glyphs.empty() && remaining.empty()) { - return absl::InternalError( - "Additional conditions check has failed, but there are no more " - "non-excluded segments to utilize."); - } - - // Anything left in glyphs has additional conditions, recurse again to - // analyze them further - queue.push_back(CreateTask(task.full_condition, {}, remaining, - additional_condition_glyphs)); - return absl::OkStatus(); - } -}; - -static btree_map ExistingConditions( - const GlyphConditionSet& glyph_condition_set, const GlyphSet& glyphs, - btree_map& glyph_to_conditions) { - btree_map existing_conditions; - for (glyph_id_t gid : glyphs) { - const auto& condition = glyph_condition_set.ConditionsFor(gid).activation(); - if (condition.IsExclusive() || !condition.IsPurelyDisjunctive()) { - continue; - } - - SegmentSet or_segments = condition.TriggeringSegments(); - existing_conditions[or_segments].insert(gid); - glyph_to_conditions[gid].union_set(or_segments); - } - return existing_conditions; -} - -StatusOr> FindSupersetDisjunctiveConditionsFor( - const RequestedSegmentationInformation& segmentation_info, - const GlyphConditionSet& glyph_condition_set, - GlyphClosureCache& closure_cache, GlyphSet glyphs, - SegmentSet inscope_segments) { - if (!segmentation_info.SegmentsAreDisjoint()) { - return absl::InvalidArgumentError( - "Complex condition finding requires disjoint segments."); - } - - VLOG(0) << "Analyzing " << glyphs.size() - << " unmapped glyphs with the complex condition detector."; - - // TODO(garretrieger): we should see which unicodes (and thus which segments) - // may interact with the GSUB table. Any segments which don't interact with - // GSUB will already have relavent conditions discovered via the standard - // closure analysis. Only segments which interact with GSUB may be part of - // complex conditions (since complex conditions require at least one 'AND' - // which only GSUB can introduce). As a result we can exclude any segments - // with no GSUB interaction from this analysis which should significantly - // speed things up. - Context context{ - .all_segments = segmentation_info.NonEmptySegments(), - .inscope_segments = inscope_segments, - .segmentation_info = &segmentation_info, - .glyph_closure_cache = &closure_cache, - .queue = {}, - }; - - // We may already have some partial conditions generated for the fallback - // glyphs, preload these into the output and schedule the initial tasks - // excluding those segments. - btree_map glyph_to_conditions; - btree_map existing_conditions = - ExistingConditions(glyph_condition_set, glyphs, glyph_to_conditions); - TRYV(context.ScheduleInitialTasks(std::move(glyphs), existing_conditions)); - - TRYV(context.ProcessQueue(glyph_to_conditions)); - - btree_map grouped_out; - for (const auto& [gid, segments] : glyph_to_conditions) { - grouped_out[segments].insert(gid); - } - - VLOG(0) << "Found " << grouped_out.size() - << " new conditions for the unmapped glyphs."; - - return grouped_out; -} - -} // namespace ift::encoder \ No newline at end of file diff --git a/ift/encoder/complex_condition_finder.h b/ift/encoder/complex_condition_finder.h deleted file mode 100644 index 057ab76d..00000000 --- a/ift/encoder/complex_condition_finder.h +++ /dev/null @@ -1,34 +0,0 @@ -#ifndef IFT_ENCODER_COMPLEX_CONDITION_FINDER_H_ -#define IFT_ENCODER_COMPLEX_CONDITION_FINDER_H_ - -#include "absl/status/statusor.h" -#include "ift/common/int_set.h" -#include "ift/encoder/glyph_closure_cache.h" -#include "ift/encoder/glyph_condition_set.h" -#include "ift/encoder/requested_segmentation_information.h" - -namespace ift::encoder { - -// Finds superset purely disjunctive conditions that activate each -// provided glyph. Returns a map from each condition to the activated -// glyphs. -// -// Takes a glyph condition set which will be used as a starting point. -// -// A superset purely disjunctive condition will activate at least -// whenever the true condition would. It will only ever include segments -// that appear in the true condition. There are typically multiple -// possible superset conditions. This will find one of them. -// -// For example if a glyph has the true condition (a and b) or (b and c) -// this could find the condition (a or c). -absl::StatusOr> -FindSupersetDisjunctiveConditionsFor( - const RequestedSegmentationInformation& segmentation_info, - const GlyphConditionSet& glyph_condition_set, - GlyphClosureCache& closure_cache, ift::common::GlyphSet glyphs, - ift::common::SegmentSet inscope_segments); - -} // namespace ift::encoder - -#endif // IFT_ENCODER_COMPLEX_CONDITION_FINDER_H_ \ No newline at end of file diff --git a/ift/encoder/complex_condition_finder_test.cc b/ift/encoder/complex_condition_finder_test.cc deleted file mode 100644 index f6a22922..00000000 --- a/ift/encoder/complex_condition_finder_test.cc +++ /dev/null @@ -1,301 +0,0 @@ -#include "ift/encoder/complex_condition_finder.h" - -#include "absl/container/btree_map.h" -#include "gtest/gtest.h" -#include "ift/common/bazel_data_file_resolver.h" -#include "ift/common/font_data.h" -#include "ift/common/int_set.h" -#include "ift/common/test_font_loader.h" -#include "ift/encoder/closure_glyph_segmenter.h" -#include "ift/encoder/requested_segmentation_information.h" -#include "ift/encoder/segmentation_context.h" -#include "ift/encoder/types.h" -#include "ift/freq/probability_bound.h" - -using ift::config::CLOSURE_ONLY; -using ift::config::PATCH; - -using absl::btree_map; -using absl::StatusOr; -using ift::common::BazelDataFileResolver; -using ift::common::DataFileResolver; -using ift::common::FontData; -using ift::common::GlyphSet; -using ift::common::hb_face_unique_ptr; -using ift::common::make_hb_face; -using ift::common::SegmentSet; -using ift::freq::ProbabilityBound; - -namespace ift::encoder { - -class ComplexConditionFinderTest : public ::testing::Test { - protected: - ComplexConditionFinderTest() - : roboto(make_hb_face(nullptr)), - resolver(*BazelDataFileResolver::CreateForTest()), - segmenter(1, 1, PATCH, CLOSURE_ONLY, resolver) { - roboto = from_file("ift/common/testdata/Roboto-Regular.ttf"); - } - - hb_face_unique_ptr from_file(const char* filename) { - auto loader = ift::common::TestFontLoader::Default().value(); - return loader->LoadFace(filename).value(); - } - - SubsetDefinition CombinedDefinition(const SegmentationContext& context, - const SegmentSet& segments) { - SubsetDefinition def; - for (segment_index_t s : segments) { - def.Union(context.SegmentationInfo().Segments().at(s).Definition()); - } - return def; - } - - StatusOr SegmentClosure(SegmentationContext& context, - const SegmentSet& segments) { - SubsetDefinition closure_def = CombinedDefinition(context, segments); - return context.glyph_closure_cache->GlyphClosure(closure_def); - } - - SegmentationContext TestContext(bool basic_closure_analysis) { - auto context = *SegmentationContext::InitializeSegmentationContext( - roboto.get(), {'f'}, - { - /* 0 */ {{0x54}}, - /* 1 */ {{0x6C}}, - /* 2 */ {{0x6E}}, - /* 3 */ {{0x13C}}, - /* 4 */ {{0x146}}, - /* 5 */ {{0x21A}}, - /* 6 */ {{0xF6C3}}, - /* 7 */ {{0x69}}, - }, - segmenter.unmapped_glyph_handling(), - segmenter.condition_analysis_mode(), segmenter.brotli_quality(), - segmenter.init_font_merging_brotli_quality(), resolver); - - if (!basic_closure_analysis) { - // initialaztion populates the basic conditions, clear those - // out so we can control them. - context.glyph_condition_set.InvalidateGlyphInformation( - {748, 756, 782}, {0, 1, 2, 3, 4, 5, 6}); - } - - return context; - } - - hb_face_unique_ptr roboto; - std::shared_ptr resolver; - ClosureGlyphSegmenter segmenter; - - // Expected complex conditions: - // - // 0xF6C3, 0x54, 0x21A => g782 - // 0xF6C3, 0x6C, 0x13C => g748 - // 0xF6C3, 0x6E, 0x146 => g756 - btree_map expected = { - {{6, 1, 3}, {748}}, - {{6, 2, 4}, {756}}, - {{6, 0, 5}, {782}}, - }; -}; - -TEST_F(ComplexConditionFinderTest, FindConditions) { - SegmentationContext context = TestContext(false); - - auto r = FindSupersetDisjunctiveConditionsFor(context.SegmentationInfo(), - context.glyph_condition_set, - *context.glyph_closure_cache, - { - 748, - 756, - 782, - }, - SegmentSet::all()); - ASSERT_TRUE(r.ok()) << r.status(); - ASSERT_EQ(expected, *r); - - // Verify that the closure requirement is met. If all segments from - // the minimal condition are excluded then the mapped gid should not - // appear in the closure. - const SegmentSet all = {0, 1, 2, 3, 4, 5, 6}; - for (const auto& [segments, gids] : *r) { - SegmentSet except = all; - except.subtract(segments); - - GlyphSet closure = *SegmentClosure(context, except); - ASSERT_FALSE(closure.intersects(gids)); - - closure = *SegmentClosure(context, segments); - ASSERT_TRUE(gids.is_subset_of(closure)); - } -} - -TEST_F(ComplexConditionFinderTest, FindConditions_Partial) { - SegmentationContext context = TestContext(false); - - auto r = FindSupersetDisjunctiveConditionsFor(context.SegmentationInfo(), - context.glyph_condition_set, - *context.glyph_closure_cache, - { - 748, - }, - SegmentSet::all()); - ASSERT_TRUE(r.ok()) << r.status(); - expected.erase(SegmentSet{6, 0, 5}); - expected.erase(SegmentSet{6, 2, 4}); - ASSERT_EQ(expected, *r); -} - -TEST_F(ComplexConditionFinderTest, FindConditions_IncompleteExistingCondition) { - SegmentationContext context = TestContext(false); - - context.glyph_condition_set.AddOrCondition(748, 6); - auto r = FindSupersetDisjunctiveConditionsFor(context.SegmentationInfo(), - context.glyph_condition_set, - *context.glyph_closure_cache, - { - 748, - }, - SegmentSet::all()); - ASSERT_TRUE(absl::IsInvalidArgument(r.status())) << r.status(); -} - -TEST_F(ComplexConditionFinderTest, FindConditions_GlyphsNotInClosure) { - SegmentationContext context = TestContext(false); - - auto r = FindSupersetDisjunctiveConditionsFor( - context.SegmentationInfo(), context.glyph_condition_set, - *context.glyph_closure_cache, - { - 748, - 40 // this is not in the full closure. - }, - SegmentSet::all()); - ASSERT_TRUE(absl::IsInvalidArgument(r.status())) << r.status(); -} - -TEST_F(ComplexConditionFinderTest, - FindConditions_WithExistingConditions_FromClosureAnalysis) { - SegmentationContext context = TestContext(true); - - auto r = FindSupersetDisjunctiveConditionsFor(context.SegmentationInfo(), - context.glyph_condition_set, - *context.glyph_closure_cache, - { - 748, - 756, - 782, - }, - SegmentSet::all()); - ASSERT_TRUE(r.ok()) << r.status(); - ASSERT_EQ(expected, *r); -} - -TEST_F(ComplexConditionFinderTest, FindConditions_WithExistingConditions) { - SegmentationContext context = TestContext(false); - - context.glyph_condition_set.AddOrCondition(748, 1); - context.glyph_condition_set.AddOrCondition(748, 6); - - auto r = FindSupersetDisjunctiveConditionsFor(context.SegmentationInfo(), - context.glyph_condition_set, - *context.glyph_closure_cache, - { - 748, - 756, - 782, - }, - SegmentSet::all()); - ASSERT_TRUE(r.ok()) << r.status(); - ASSERT_EQ(expected, *r); -} - -TEST_F(ComplexConditionFinderTest, - FindConditions_WithExistingConditions_NoAdditionalConditions) { - SegmentationContext context = TestContext(false); - - context.glyph_condition_set.AddOrCondition(748, 1); - context.glyph_condition_set.AddOrCondition(748, 3); - context.glyph_condition_set.AddOrCondition(748, 6); - - auto r = FindSupersetDisjunctiveConditionsFor(context.SegmentationInfo(), - context.glyph_condition_set, - *context.glyph_closure_cache, - { - 748, - 756, - 782, - }, - SegmentSet::all()); - ASSERT_TRUE(r.ok()) << r.status(); - ASSERT_EQ(expected, *r); -} - -TEST_F(ComplexConditionFinderTest, FindConditions_RejectsInitFontGlyphs) { - SegmentationContext context = TestContext(false); - - auto r = FindSupersetDisjunctiveConditionsFor( - context.SegmentationInfo(), context.glyph_condition_set, - *context.glyph_closure_cache, - { - 748, - 74, // f - in the init closure - }, - SegmentSet::all()); - ASSERT_TRUE(absl::IsInvalidArgument(r.status())) << r.status(); -} - -TEST_F(ComplexConditionFinderTest, FindConditions_ClosureRespectsInitFont) { - SegmentationContext context = TestContext(false); - - auto r = FindSupersetDisjunctiveConditionsFor( - context.SegmentationInfo(), context.glyph_condition_set, - *context.glyph_closure_cache, - { - 446, // fi ligature - combines i with f from the init font - }, - SegmentSet::all()); - ASSERT_TRUE(r.ok()) << r.status(); - expected = { - {{7}, {446}}, - }; - ASSERT_EQ(expected, *r); -} - -TEST_F(ComplexConditionFinderTest, FindConditions_WithInscope) { - SegmentationContext context = TestContext(false); - - auto r = FindSupersetDisjunctiveConditionsFor(context.SegmentationInfo(), - context.glyph_condition_set, - *context.glyph_closure_cache, - { - 748, - 756, - }, - {1, 2, 3, 4, 6}); - ASSERT_TRUE(r.ok()) << r.status(); - - expected.erase({6, 0, 5}); - ASSERT_EQ(expected, *r); -} - -TEST_F(ComplexConditionFinderTest, FindConditions_WithBadInscope) { - SegmentationContext context = TestContext(false); - - auto r = FindSupersetDisjunctiveConditionsFor(context.SegmentationInfo(), - context.glyph_condition_set, - *context.glyph_closure_cache, - { - 748, - 756, - }, - { - 2, - 3, - 4, // missing 1, 6 - }); - ASSERT_TRUE(absl::IsInternal(r.status())); -} - -} // namespace ift::encoder \ No newline at end of file diff --git a/ift/encoder/dependency_closure.cc b/ift/encoder/dependency_closure.cc index 18fdf65f..caa72471 100644 --- a/ift/encoder/dependency_closure.cc +++ b/ift/encoder/dependency_closure.cc @@ -37,39 +37,7 @@ namespace ift::encoder { Status DependencyClosure::InitFontChanged(const SegmentSet& segments) { VLOG(1) << "DependencyClosure::InitFontChanged()"; - TRYV(InitNodeConditionsCache(segments)); - - if (!context_glyphs_.has_value() || !init_font_context_glyphs_.has_value()) { - // Context glyphs aren't need for this instance, so skip generating them. - return absl::OkStatus(); - } - - SegmentSet start_segments; - for (segment_index_t s = 0; s < segmentation_info_->Segments().size(); s++) { - if (segmentation_info_->Segments().at(s).Definition().Empty()) { - continue; - } - start_segments.insert(s); - } - - Traversal traversal = TRY(graph_.ClosureTraversal(start_segments, DependencyGraph::COLLECT_CONTEXT)); - context_glyphs_ = traversal.ContextGlyphs(); - - // The init font may have reachable glyphs which are not in the init font - // closure, we need to record the context glyphs from these as they are - // potential interaction points. - init_font_context_glyphs_ = GlyphSet{}; - Traversal init_font_traversal = TRY(graph_.ClosureTraversal( - {Node::InitFont()}, &segmentation_info_->FullClosure(), - &segmentation_info_->FullDefinition().codepoints, DependencyGraph::COLLECT_CONTEXT)); - for (const auto& [g, context] : init_font_traversal.ContextPerGlyph()) { - if (segmentation_info_->NonInitFontGlyphs().contains(g)) { - context_glyphs_->union_set(context); - init_font_context_glyphs_->union_set(context); - } - } - - return absl::OkStatus(); + return InitNodeConditionsCache(segments); } Status DependencyClosure::SegmentsMerged(segment_index_t base_segment, @@ -188,156 +156,6 @@ flat_hash_set DependencyClosure::SegmentsToAffectedNodeConditions( return nodes; } -StatusOr DependencyClosure::AnalyzeSegment( - const SegmentSet& segments, GlyphSet& and_gids, GlyphSet& or_gids, - GlyphSet& exclusive_gids, bool with_simplification) { - AnalysisResult result = TRY(AnalyzeSegmentInternal(segments, with_simplification)); - if (result.accuracy == INACCURATE) { - inaccurate_results_++; - } else { - accurate_results_++; - } - - and_gids.union_set(result.and_gids); - or_gids.union_set(result.or_gids); - exclusive_gids.union_set(result.exclusive_gids); - return result.accuracy; -} - -// Filters out invalid and empty segment ids. -StatusOr DependencyClosure::FilterSegments( - const SegmentSet& segments) const { - SegmentSet out; - for (segment_index_t segment_id : segments) { - if (segment_id >= segmentation_info_->Segments().size()) { - return absl::InvalidArgumentError( - absl::StrCat("Segment index ", segment_id, " is out of bounds.")); - } - - if (segmentation_info_->Segments().at(segment_id).Definition().Empty()) { - // Empty segments are ignored. - continue; - } - - out.insert(segment_id); - } - return out; -} - -StatusOr -DependencyClosure::AnalyzeSegmentInternal( - const SegmentSet& segments_input, bool with_simplification) const { - // This uses a dependency graph (from harfbuzz) to infer how 'segment_id' - // appears in the activation conditions of any glyphs reachable from it. - // This aims to have identical output to GlyphClosureCache::AnalyzeSegment() - // which uses harfbuzz glyph closure to infer conditions. - // - // Condition analysis using the dep graph is not always able to accurately - // reproduce the closure based conditions, so this method returns an - // indication of the accuracy of the analysis. AnalysisAccuracy::INACCURATE is - // returned when something in the dep graph is encountered which may cause - // results to diverge from the closure approach. - - AnalysisResult result; - result.accuracy = ACCURATE; - - SegmentSet segments = TRY(FilterSegments(segments_input)); - if (segments.empty()) { - return result; - } - - auto inscope_nodes = SegmentsToAffectedNodeConditions(segments); - GlyphSet inscope_glyphs; - for (Node node : inscope_nodes) { - if (node.IsGlyph() && - segmentation_info_->NonInitFontGlyphs().contains(node.Id())) { - inscope_glyphs.insert(node.Id()); - } - } - result.reached_glyphs = inscope_glyphs; - - if ((context_glyphs_.has_value() && - inscope_glyphs.intersects(*context_glyphs_)) || - (init_font_context_glyphs_.has_value() && - inscope_glyphs.intersects(*init_font_context_glyphs_))) { - // context glyphs may result in an inaccurate analysis, we can - // still do the analysis but mark it as inaccurate. - result.accuracy = INACCURATE; - } - - // Classify each glyph based on it's conditions - for (glyph_id_t gid : inscope_glyphs) { - auto it = glyph_condition_cache_.find(gid); - if (it == glyph_condition_cache_.end()) { - continue; - } - ActivationCondition condition = it->second; - segment_index_t segment = *segments.min(); - if (segments.size() > 1) { - // Pretend the input segments are merged into one for this analysis. - condition = condition.ReplaceSegments(segment, segments); - } - - if (!condition.TriggeringSegments().contains(segment)) { - // Shouldn't happen, something is wrong with the glyph condition cache. - return absl::InternalError(absl::StrCat("condition in cache for g", gid, - " does not include s", segment)); - } - - if (with_simplification) { - condition = condition.NonCompositeSuperset(); - } - - // Possible cases: - // 1. condition is unitary (only one segment), then glyph is exclusive to - // the input. - if (condition.IsUnitary()) { - result.exclusive_gids.insert(gid); - continue; - } - - // 2. condition is purely disjunctive, glyph is disjunctive to the input. - if (condition.IsPurelyDisjunctive()) { - result.or_gids.insert(gid); - } - - if (condition.conditions().size() > 1) { - bool in_all = true; - bool mixed = false; - for (const auto& disjunctive_group : condition.conditions()) { - if (!disjunctive_group.contains(segment)) { - in_all = false; - continue; - } - - if (disjunctive_group.size() > 1) { - // input segment is involved in a mix of conjunction and disjunction. - mixed = true; - } - } - - // 2. condition has conjunction (two or more sub-groups), and input - // segment appears in every disjunctive sub group, glyph is disjunctive to - // the input. - if (in_all) { - result.or_gids.insert(gid); - } - - // 3. condition has conjunction (two or more sub-groups), and the - // sub-group that contains the input segment(s) equals the input - // segment(s), then glyph is conjunctive to the input. - if (!mixed) { - result.and_gids.insert(gid); - } - - // 4. otherwise gid does not appear in the analysis results due to more - // complicated conditions. - } - } - - return result; -} - StatusOr DependencyClosure::SegmentsThatInteractWith( const SubsetDefinition& def) const { flat_hash_set nodes; diff --git a/ift/encoder/dependency_closure.h b/ift/encoder/dependency_closure.h index 1cfe27c7..62cbfd74 100644 --- a/ift/encoder/dependency_closure.h +++ b/ift/encoder/dependency_closure.h @@ -23,73 +23,26 @@ namespace ift::encoder { class RequestedSegmentationInformation; /* - * Performs closure analysis (like GlyphClosureCache) using a dependency graph - * instead of closure. The dependency graph is not always accurate - * (overestimating the true closure in some cases) so this returns a signal on - * the accuracy of the analysis. + * Extracts glyph activation conditions and interaction sets using a dependency + * graph instead of repeated subsetting closure. In some cases may overestimate + * activation conditions versus real subsetting closure due to reliance on the + * dependency graph. */ class DependencyClosure { public: - // If allow_context_glyph_analysis is true then AnalyzeSegment() will not - // report INNACCURATE for cases that involve context glyphs. static absl::StatusOr> Create( const RequestedSegmentationInformation* segmentation_info, - hb_face_t* face, const ift::common::DataFileResolver& resolver, - bool allow_context_glyph_analysis = false) { + hb_face_t* face, const ift::common::DataFileResolver& resolver) { dep_graph::DependencyGraph graph = TRY( dep_graph::DependencyGraph::Create(segmentation_info, face, resolver)); auto result = std::unique_ptr( new DependencyClosure(std::move(graph), segmentation_info, face)); - if (!allow_context_glyph_analysis) { - result->context_glyphs_ = common::GlyphSet{}; - result->init_font_context_glyphs_ = common::GlyphSet{}; - } TRYV(result->InitFontChanged(ift::common::SegmentSet::all())); return result; } - enum AnalysisAccuracy { - // The analysis is accurate and should match true glyph closure. - ACCURATE, - - // The analysis may not be accurate and as a result may overestimate - // the true glyph closure. - INACCURATE, - }; - - // Attempts to analyze the given segment using a glyph dependency graph - // from harfbuzz. The return value signals if an analysis was able to be - // performed or not. - // - // If ACCURATE is returned then the dep graph is able to produce an analysis - // which should match GlyphClosureCache::AnalyzeSegment(). In this case - // the three output sets (and_gids, or_gids, and exclusive_gids) will be - // populated with the anaysis results. - // - // Otherwise if INACCURATE is returned then the dep graph analysis was found - // to possibly not match GlyphClosureCache::AnalyzeSegment(). The three - // output sets will not be modified in this case. - // - // The three output sets have the following interpretation: - // and_gids: these gids have the union of input segments as a conjunctive - // condition. - // ie. (s_1 U ... U s_n) AND ... -> and_gids - // - // or_gids: these gids have the union of input segments as a disjunctive - // condition. - // ie. (s_1 U ... U s_n) OR ... -> or_gids - // - // exclusive_gids: these gids are exclusively needed by the union of input - // segments. - // ie. (s_1 U ... U s_n) -> exclusive_gids - absl::StatusOr AnalyzeSegment( - const ift::common::SegmentSet& segments, ift::common::GlyphSet& and_gids, - ift::common::GlyphSet& or_gids, ift::common::GlyphSet& exclusive_gids, - bool with_simplification - ); - // Extracts the full activations conditions (as specified by the dependency // graph) for all glyphs. In some cases may overestimate activation conditions // versus real subsetting closure due to reliance on the dependency graph. @@ -137,9 +90,6 @@ class DependencyClosure { absl::StatusOr SegmentsThatInteractWith( const absl::flat_hash_set nodes) const; - uint64_t AccurateResults() const { return accurate_results_; } - uint64_t InaccurateResults() const { return inaccurate_results_; } - private: // Extracts the full activations conditions (as specified by the dependency @@ -194,20 +144,6 @@ class DependencyClosure { original_face_(ift::common::make_hb_face(hb_face_reference(face))), graph_(std::move(graph)) {} - struct AnalysisResult { - AnalysisAccuracy accuracy; - ift::common::GlyphSet reached_glyphs; - ift::common::GlyphSet and_gids; - ift::common::GlyphSet or_gids; - ift::common::GlyphSet exclusive_gids; - }; - - absl::StatusOr AnalyzeSegmentInternal( - const ift::common::SegmentSet& segments, bool with_simplification) const; - - absl::StatusOr FilterSegments( - const common::SegmentSet& segments) const; - absl::Status InitNodeConditionsCache( const common::SegmentSet& changed_segments); void UpdateNodeConditionsCache(segment_index_t base_segment, @@ -217,9 +153,6 @@ class DependencyClosure { ift::common::hb_face_unique_ptr original_face_; dep_graph::DependencyGraph graph_; - std::optional context_glyphs_ = std::nullopt; - std::optional init_font_context_glyphs_ = std::nullopt; - // Inert segments are segments that: // - Appear only in a single unitary condition, that is // they do not interact with any other segments. @@ -246,9 +179,6 @@ class DependencyClosure { // Cache of previously processed InitFont nodes, used to determine newly added // init items. absl::flat_hash_set init_font_nodes_; - - uint64_t accurate_results_ = 0; - uint64_t inaccurate_results_ = 0; }; } // namespace ift::encoder diff --git a/ift/encoder/dependency_closure_test.cc b/ift/encoder/dependency_closure_test.cc index 30476d15..de767156 100644 --- a/ift/encoder/dependency_closure_test.cc +++ b/ift/encoder/dependency_closure_test.cc @@ -9,7 +9,6 @@ #include "ift/common/font_helper.h" #include "ift/common/int_set.h" #include "ift/common/test_font_loader.h" -#include "ift/config/common.pb.h" #include "ift/dep_graph/node.h" #include "ift/encoder/glyph_closure_cache.h" #include "ift/encoder/init_subset_defaults.h" @@ -20,8 +19,6 @@ #include "ift/freq/probability_bound.h" #include "tools/cpp/runfiles/runfiles.h" -using ift::config::PATCH; - using absl::btree_set; using absl::flat_hash_map; using absl::flat_hash_set; @@ -54,7 +51,7 @@ class DependencyClosureTest : public ::testing::Test { resolver(*BazelDataFileResolver::CreateForTest()), closure_cache(*GlyphClosureCache::Create(face.get(), *resolver)), segmentation_info(*RequestedSegmentationInformation::Create( - segments, WithDefaultFeatures(), *closure_cache, PATCH)), + segments, WithDefaultFeatures(), *closure_cache)), dependency_closure(*DependencyClosure::Create(segmentation_info.get(), face.get(), *resolver)) {} @@ -85,67 +82,64 @@ class DependencyClosureTest : public ::testing::Test { /* 5 */ {{0xC1 /* Aacute */}}, }; - void Reconfigure(SubsetDefinition new_init, std::vector new_segments, - bool context_analysis_allowed = false) { + void Reconfigure(SubsetDefinition new_init, + std::vector new_segments) { segmentation_info = *RequestedSegmentationInformation::Create( - new_segments, new_init, *closure_cache, PATCH); - dependency_closure = - *DependencyClosure::Create(segmentation_info.get(), face.get(), - *resolver, context_analysis_allowed); + new_segments, new_init, *closure_cache); + dependency_closure = *DependencyClosure::Create(segmentation_info.get(), + face.get(), *resolver); } void Reconfigure(hb_face_t* new_face, SubsetDefinition new_init, std::vector new_segments) { closure_cache = *GlyphClosureCache::Create(new_face, *resolver); segmentation_info = *RequestedSegmentationInformation::Create( - new_segments, new_init, *closure_cache, PATCH); + new_segments, new_init, *closure_cache); dependency_closure = *DependencyClosure::Create(segmentation_info.get(), new_face, *resolver); } - Status RejectedAnalysis(segment_index_t segment) { - GlyphSet and_gids; - GlyphSet or_gids; - GlyphSet exclusive_gids; - - if (TRY(dependency_closure->AnalyzeSegment({segment}, and_gids, or_gids, - exclusive_gids, false)) == - DependencyClosure::ACCURATE) { - return absl::InternalError( - "Dependency closure analysis should have been rejected."); + GlyphSet ActivatedGlyphs(const SegmentSet& segments) const { + GlyphSet out; + for (const auto& [gid, condition] : + dependency_closure->AllGlyphConditions()) { + bool matches = true; + for (const auto& group : condition.conditions()) { + if (!group.intersects(segments)) { + matches = false; + break; + } + } + if (matches) { + out.insert(gid); + } } - return absl::OkStatus(); + return out; + } + + absl::StatusOr ExpectedClosure(const SegmentSet& segments) const { + GlyphSet gids = + TRY(closure_cache->SegmentClosure(segmentation_info.get(), segments)); + gids.subtract(segmentation_info->InitFontGlyphs()); + return gids; } Status CompareAnalysis(SegmentSet segments) { - GlyphSet and_gids; - GlyphSet or_gids; - GlyphSet exclusive_gids; - - if (TRY(dependency_closure->AnalyzeSegment(segments, and_gids, or_gids, - exclusive_gids, false)) != - DependencyClosure::ACCURATE) { - return absl::InternalError( - "Dependency closure analysis rejected unexpectedly."); + SegmentSet other; + for (segment_index_t i = 0; i < segmentation_info->Segments().size(); ++i) { + if (!segments.contains(i)) { + other.insert(i); + } } - GlyphSet expected_and_gids; - GlyphSet expected_or_gids; - GlyphSet expected_exclusive_gids; - TRYV(closure_cache->AnalyzeSegment(*segmentation_info, segments, - expected_and_gids, expected_or_gids, - expected_exclusive_gids)); - std::string message; bool success = true; - if (!CompareGids("AND", and_gids, expected_and_gids, message)) { - success = false; - } - if (!CompareGids("OR", or_gids, expected_or_gids, message)) { + if (!CompareGids("SEGMENTS", ActivatedGlyphs(segments), + TRY(ExpectedClosure(segments)), message)) { success = false; } - if (!CompareGids("EXCLUSIVE", exclusive_gids, expected_exclusive_gids, - message)) { + if (!CompareGids("OTHER", ActivatedGlyphs(other), + TRY(ExpectedClosure(other)), message)) { success = false; } @@ -180,98 +174,6 @@ class DependencyClosureTest : public ::testing::Test { std::unique_ptr dependency_closure; }; -TEST_F(DependencyClosureTest, AnalyzeSegment_InaccurateAnalysisPopulatesSets) { - Reconfigure(WithDefaultFeatures({'i'}), - { - /* 0 */ {{0x300 /* gravecomb */}}, - /* 1 */ {{0x485}}, - }); - - GlyphSet and_gids; - GlyphSet or_gids; - GlyphSet exclusive_gids; - - // Segment 0 contains gravecomb (0x300), which intersects with - // init_font_context_glyphs_ ('i' in init font + 0x300 form a context lookup). - // Therefore, AnalyzeSegment will return INACCURATE. - auto accuracy = dependency_closure->AnalyzeSegment( - {0}, and_gids, or_gids, exclusive_gids, /*with_simplification=*/false); - ASSERT_TRUE(accuracy.ok()) << accuracy.status(); - EXPECT_EQ(*accuracy, DependencyClosure::INACCURATE); - - // Verify that the analysis was still performed and populated the glyph sets - // despite being INACCURATE. - auto font = make_hb_font(hb_font_create(face.get())); - hb_codepoint_t gravecomb_gid = 0; - ASSERT_TRUE(hb_font_get_nominal_glyph(font.get(), 0x300, &gravecomb_gid)); - EXPECT_TRUE(exclusive_gids.contains(gravecomb_gid)); -} - -TEST_F(DependencyClosureTest, AnalyzeSegment_WithSimplification) { - SubsetDefinition aalt; - aalt.feature_tags.insert(HB_TAG('a', 'a', 'l', 't')); - SubsetDefinition jp78; - jp78.feature_tags.insert(HB_TAG('j', 'p', '7', '8')); - - Reconfigure(noto_sans_jp_vf.get(), {}, - { - /* 0 */ {{0x6717}}, - /* 1 */ {{0x7891}}, - /* 2 */ {{0x798f}}, - /* 3 */ {{0x6406}}, - /* 4 */ {{0xe0100}}, - /* 5 */ {{0xfe00}}, - /* 6 */ {aalt}, - /* 7 */ {jp78}, - }); - - // g8 (glyph 8) has condition: s1 AND (s4 OR s5). - // Without simplification, s1 is part of a mixed condition for g8, so g8 is - // not classified as exclusive. - { - GlyphSet and_gids; - GlyphSet or_gids; - GlyphSet exclusive_gids; - - auto r = dependency_closure->AnalyzeSegment( - {1}, and_gids, or_gids, exclusive_gids, /*with_simplification=*/false); - ASSERT_TRUE(r.ok()) << r.status(); - EXPECT_EQ(*r, DependencyClosure::ACCURATE); - - EXPECT_FALSE(exclusive_gids.contains(8)); - } - - // With simplification, the condition for g8 simplifies to s1, so g8 is - // classified as exclusive. - { - GlyphSet and_gids; - GlyphSet or_gids; - GlyphSet exclusive_gids; - - auto r = dependency_closure->AnalyzeSegment( - {1}, and_gids, or_gids, exclusive_gids, /*with_simplification=*/true); - ASSERT_TRUE(r.ok()) << r.status(); - EXPECT_EQ(*r, DependencyClosure::ACCURATE); - - EXPECT_TRUE(exclusive_gids.contains(8)); - } -} - -TEST_F(DependencyClosureTest, AddsToSets) { - GlyphSet and_gids{101}; - GlyphSet or_gids{102}; - GlyphSet exclusive_gids{103}; - - auto r = dependency_closure->AnalyzeSegment({0}, and_gids, or_gids, - exclusive_gids, false); - ASSERT_TRUE(r.ok()) << r.status(); - ASSERT_EQ(*r, DependencyClosure::ACCURATE); - - ASSERT_EQ(and_gids, (GlyphSet{101, 667})); - ASSERT_EQ(or_gids, (GlyphSet{102})); - ASSERT_EQ(exclusive_gids, (GlyphSet{69, 103})); -} - TEST_F(DependencyClosureTest, ExtractAllGlyphConditions) { Reconfigure(WithDefaultFeatures(), { @@ -682,57 +584,6 @@ TEST_F(DependencyClosureTest, AlreadyInInitFont) { ASSERT_TRUE(s.ok()) << s; } -TEST_F(DependencyClosureTest, SegmentOutOfBounds) { - Status s = CompareAnalysis({10}); - ASSERT_TRUE(absl::IsInvalidArgument(s)) << s; -} - -TEST_F(DependencyClosureTest, Rejected_LookAheadGlyphs) { - Reconfigure(WithDefaultFeatures({}), { - {{'i'}}, - {{0x485}}, - }); - - Status s = CompareAnalysis({0}); - ASSERT_TRUE(s.ok()) << s; - - // glyph for 0x485 is part of a lookahead on a chain context sub - // so analysis should be rejected. - s = RejectedAnalysis(1); - ASSERT_TRUE(s.ok()) << s; -} - -TEST_F(DependencyClosureTest, Allowed_LookAheadGlyphs) { - Reconfigure(WithDefaultFeatures({}), - { - {{'i'}}, - {{0x485}}, - }, - true); - - Status s = CompareAnalysis({0}); - ASSERT_TRUE(s.ok()) << s; - // glyph for 0x485 is part of a lookahead on a chain context sub - // since context analysis has been turned on then analysis will - // not be rejected. - s = CompareAnalysis({1}); - ASSERT_TRUE(s.ok()) << s; -} - -TEST_F(DependencyClosureTest, Rejected_InitFontContext) { - Reconfigure(WithDefaultFeatures({'i'}), - { - {{0x300 /* gravecomb */}}, - {{0x485}}, - }); - - // Gravecomb doesn't pass through any contextual dependencies, but - // it shows up as a context glyph for the init font subset. - // Thus it should be rejected. - Status s = RejectedAnalysis(0); - ASSERT_TRUE(s.ok()) << s; -} - TEST_F(DependencyClosureTest, Noop_Features) { SubsetDefinition features; features.feature_tags.insert(HB_TAG('a', 'b', 'c', 'd')); @@ -822,19 +673,6 @@ TEST_F(DependencyClosureTest, UvsAndFeatures_ConflictingConjunctiveConditions) { ASSERT_TRUE(s.ok()) << s; } -TEST_F(DependencyClosureTest, Rejected_FullySatisfiedContext) { - Reconfigure(WithDefaultFeatures({}), - { - {{'i', 0x300 /* gravecomb */}}, - }); - - // The segment contains everything needed to activated the i + gravecomb - // contextual lookup, but should still be rejected on account of passing - // through a contextual lookup. - Status s = RejectedAnalysis(0); - ASSERT_TRUE(s.ok()) << s; -} - TEST_F(DependencyClosureTest, BidiMirroring) { // Test that the dep graph analysis accounts for bidi mirroring in the // harfbuzz closure diff --git a/ift/encoder/glyph_closure_cache.cc b/ift/encoder/glyph_closure_cache.cc index 1c33683e..2e0c93ac 100644 --- a/ift/encoder/glyph_closure_cache.cc +++ b/ift/encoder/glyph_closure_cache.cc @@ -10,8 +10,6 @@ #include "ift/encoder/subset_definition.h" #include "ift/encoder/types.h" -using ift::config::Glyphs; - using absl::Status; using absl::StatusOr; using ift::common::CodepointSet; @@ -170,82 +168,6 @@ StatusOr GlyphClosureCache::GlyphClosure( return GlyphSet(gids); } -StatusOr GlyphClosureCache::CodepointsToOrGids( - const RequestedSegmentationInformation& segmentation_info, - const SegmentSet& segment_ids) { - GlyphSet and_gids; - GlyphSet or_gids; - GlyphSet exclusive_gids; - TRYV(AnalyzeSegment(segmentation_info, segment_ids, and_gids, or_gids, - exclusive_gids)); - - return or_gids; -} - -Status GlyphClosureCache::AnalyzeSegment( - const RequestedSegmentationInformation& segmentation_info, - const SegmentSet& segment_ids, GlyphSet& and_gids, GlyphSet& or_gids, - GlyphSet& exclusive_gids) { - if (segment_ids.empty()) { - return absl::OkStatus(); - } - - // This function tests various closures using the segment codepoints to - // determine what conditions are present for the inclusion of closure glyphs. - // - // At a high level we do the following (where s_i is the segment being - // tested): - // - // * Set A: glyph closure on original font of the union of all segments. - // * Set B: glyph closure on original font of the union of all segments except - // for s_i - // * Set I: (glyph closure on original font of s_0 union s_i) - (glyph closure - // on original font of s_0) - // * Set D: A - B, the set of glyphs that are dropped when s_i is removed. - // - // Then we know the following: - // * Glyphs in I should be included whenever s_i is activated. - // * s_i is necessary for glyphs in D to be required, but other segments may - // be needed too. - // - // Furthermore we can intersect I and D to produce three sets: - // * D - I: the activation condition for these glyphs is s_i AND … - // Where … is one or more additional segments. - // * I - D: the activation conditions for these glyphs is s_i OR … - // Where … is one or more additional segments. - // * D intersection I: the activation conditions for these glyphs is only s_i - - SubsetDefinition - combined; // This is the subset definition of the unions of segment_ids. - for (segment_index_t s_id : segment_ids) { - combined.Union(segmentation_info.Segments()[s_id].Definition()); - } - - SubsetDefinition except_segment = - ComputeExceptSegment(segmentation_info, segment_ids, combined); - auto B_except_segment_closure = TRY(GlyphClosure(except_segment)); - - SubsetDefinition only_segment = combined; - only_segment.Union(segmentation_info.InitFontSegment()); - - auto I_only_segment_closure = TRY(GlyphClosure(only_segment)); - I_only_segment_closure.subtract(segmentation_info.InitFontGlyphs()); - - GlyphSet D_dropped = segmentation_info.FullClosure(); - D_dropped.subtract(B_except_segment_closure); - - and_gids.union_set(D_dropped); - and_gids.subtract(I_only_segment_closure); - - or_gids.union_set(I_only_segment_closure); - or_gids.subtract(D_dropped); - - exclusive_gids.union_set(I_only_segment_closure); - exclusive_gids.intersect(D_dropped); - - return absl::OkStatus(); -} - CodepointSet GlyphClosureCache::CodepointsForGlyphs( const GlyphSet& glyphs) const { return unicode_edges_.CodepointsForGlyphs(glyphs); diff --git a/ift/encoder/glyph_closure_cache.h b/ift/encoder/glyph_closure_cache.h index 070aefa8..0f097613 100644 --- a/ift/encoder/glyph_closure_cache.h +++ b/ift/encoder/glyph_closure_cache.h @@ -46,18 +46,6 @@ class GlyphClosureCache { const ift::common::SegmentSet& segments, const ift::common::GlyphSet& glyphs); - // Analyzes the provided segments with AnalyzeSegment() and returns just the - // or_gids - absl::StatusOr CodepointsToOrGids( - const RequestedSegmentationInformation& segmentation_info, - const ift::common::SegmentSet& segment_ids); - - absl::Status AnalyzeSegment( - const RequestedSegmentationInformation& segmentation_info, - const ift::common::SegmentSet& segment_ids, - ift::common::GlyphSet& and_gids, ift::common::GlyphSet& or_gids, - ift::common::GlyphSet& exclusive_gids); - absl::StatusOr ExpandClosure( const SubsetDefinition& definition); diff --git a/ift/encoder/glyph_closure_cache_test.cc b/ift/encoder/glyph_closure_cache_test.cc index 93b83a26..a9958958 100644 --- a/ift/encoder/glyph_closure_cache_test.cc +++ b/ift/encoder/glyph_closure_cache_test.cc @@ -12,8 +12,6 @@ #include "ift/encoder/subset_definition.h" #include "ift/freq/probability_bound.h" -using ift::config::PATCH; - using absl::StatusOr; using ift::common::BazelDataFileResolver; using ift::common::CodepointSet; @@ -63,8 +61,7 @@ TEST_F(GlyphClosureCacheTest, SegmentClosure) { SubsetDefinition init; init.feature_tags = {HB_TAG('l', 'i', 'g', 'a')}; - auto info = - RequestedSegmentationInformation::Create(segments, init, **cache, PATCH); + auto info = RequestedSegmentationInformation::Create(segments, init, **cache); ASSERT_TRUE(info.ok()); auto closure = (*cache)->SegmentClosure(info->get(), {0}); @@ -90,8 +87,7 @@ TEST_F(GlyphClosureCacheTest, HasAdditionalConditions) { // s0 or s1 -> g37 (A) - auto info = - RequestedSegmentationInformation::Create(segments, {}, **cache, PATCH); + auto info = RequestedSegmentationInformation::Create(segments, {}, **cache); ASSERT_TRUE(info.ok()); ASSERT_TRUE( @@ -118,45 +114,15 @@ TEST_F(GlyphClosureCacheTest, HasAdditionalConditions_IncludesInitFont) { SubsetDefinition init; init.feature_tags = {HB_TAG('c', '2', 's', 'c')}; - auto info = - RequestedSegmentationInformation::Create(segments, init, **cache, PATCH); + auto info = RequestedSegmentationInformation::Create(segments, init, **cache); ASSERT_TRUE(info.ok()); // s0 or s1 -> g563 (smcap A) - EXPECT_EQ(*(*cache)->CodepointsToOrGids(*info->get(), {0}), - (GlyphSet{37, 563})); - EXPECT_EQ(*(*cache)->CodepointsToOrGids(*info->get(), {1}), - (GlyphSet{37, 563})); - EXPECT_TRUE(*(*cache)->HasAdditionalConditions(info->get(), {0}, {563})); EXPECT_TRUE(*(*cache)->HasAdditionalConditions(info->get(), {1}, {563})); EXPECT_FALSE(*(*cache)->HasAdditionalConditions(info->get(), {0, 1}, {563})); } -TEST_F(GlyphClosureCacheTest, AnalyzeSegment) { - auto cache = GlyphClosureCache::Create(roboto.get(), *resolver); - ASSERT_TRUE(cache.ok()) << cache.status(); - std::vector segments{ - {{'f'}}, - {{'i'}}, - }; - - SubsetDefinition init; - init.feature_tags = {HB_TAG('l', 'i', 'g', 'a')}; - auto info = - RequestedSegmentationInformation::Create(segments, init, **cache, PATCH); - ASSERT_TRUE(info.ok()); - - GlyphSet and_gids, or_gids, exclusive_gids; - auto status = (*cache)->AnalyzeSegment(*info->get(), {0}, and_gids, or_gids, - exclusive_gids); - ASSERT_TRUE(status.ok()); - - EXPECT_EQ(exclusive_gids, (GlyphSet{74 /* f */})); - EXPECT_EQ(or_gids, (GlyphSet{})); - EXPECT_EQ(and_gids, (GlyphSet{444 /* fi */, 446 /* ffi */})); -} - TEST_F(GlyphClosureCacheTest, ExpandClosure) { auto cache = GlyphClosureCache::Create(roboto.get(), *resolver); ASSERT_TRUE(cache.ok()) << cache.status(); @@ -174,27 +140,6 @@ TEST_F(GlyphClosureCacheTest, ExpandClosure) { ASSERT_EQ(*expanded, expected); } -TEST_F(GlyphClosureCacheTest, CodepointsToOrGids) { - auto cache = GlyphClosureCache::Create(roboto.get(), *resolver); - ASSERT_TRUE(cache.ok()) << cache.status(); - std::vector segments{ - {{'A'}}, - {{0xC1 /* Aacute */}}, - }; - - SubsetDefinition init_font; - auto info = RequestedSegmentationInformation::Create(segments, init_font, - **cache, PATCH); - ASSERT_TRUE(info.ok()); - - SegmentSet segs; - segs.insert(0); - - auto or_gids = (*cache)->CodepointsToOrGids(*info->get(), {0}); - ASSERT_TRUE(or_gids.ok()); - EXPECT_EQ(*or_gids, (GlyphSet{37})); -} - TEST_F(GlyphClosureCacheTest, UnicodeCompDecomp_ExpandClosure) { auto cache = GlyphClosureCache::Create(roboto.get(), *resolver); ASSERT_TRUE(cache.ok()) << cache.status(); diff --git a/ift/encoder/glyph_groupings.cc b/ift/encoder/glyph_groupings.cc index 9a4e0a96..54d5bc16 100644 --- a/ift/encoder/glyph_groupings.cc +++ b/ift/encoder/glyph_groupings.cc @@ -7,15 +7,11 @@ #include "ift/common/int_set.h" #include "ift/common/try.h" #include "ift/encoder/activation_condition.h" -#include "ift/encoder/complex_condition_finder.h" -#include "ift/encoder/glyph_closure_cache.h" #include "ift/encoder/glyph_condition_set.h" #include "ift/encoder/glyph_partition.h" #include "ift/encoder/requested_segmentation_information.h" #include "ift/encoder/types.h" -using ift::config::FIND_CONDITIONS; - using absl::btree_map; using absl::btree_set; using absl::flat_hash_map; @@ -27,9 +23,6 @@ using ift::common::SegmentSet; namespace ift::encoder { void GlyphGroupings::InvalidateGlyphInformation(uint32_t gid) { - unmapped_glyphs_.erase(gid); - found_condition_glyphs_.erase(gid); - std::optional post_combination_condition_or = conditions_and_glyphs_.Invalidate(gid); std::optional pre_combination_condition_or = @@ -99,99 +92,41 @@ Status GlyphGroupings::CombinePatches(const GlyphSet& a, const GlyphSet& b) { return absl::OkStatus(); } -Status GlyphGroupings::AddGlyphsToExclusiveGroup( - segment_index_t exclusive_segment, const GlyphSet& glyphs) { - for (glyph_id_t gid : glyphs) { - InvalidateGlyphInformation(gid); - } - - auto& exc_glyphs = exclusive_glyph_groups_[exclusive_segment]; - exc_glyphs.union_set(glyphs); - - ActivationCondition condition = - ActivationCondition::exclusive_segment(exclusive_segment, 0); - - TRYV(conditions_and_glyphs_.Union(condition, glyphs)); - TRYV(conditions_and_glyphs_pre_combination_.Union(condition, glyphs)); - - // When merging this way we have to check if any of the involved glyphs - // are involved with the combined patches mechanism. If at least one is - // then it's necessary to recompute all combined patches to reflect any - // downstream changes. - return RecomputeCombinedConditionsIfNeeded(glyphs); -} - // Converts this grouping into a finalized GlyphSegmentation. StatusOr GlyphGroupings::ToGlyphSegmentation( const RequestedSegmentationInformation& segmentation_info, bool exclusive_only) const { GlyphSegmentation segmentation( segmentation_info.InitFontSegmentWithoutDefaults(), - segmentation_info.InitFontGlyphs(), unmapped_glyphs_); + segmentation_info.InitFontGlyphs()); segmentation.CopySegments(segmentation_info.SegmentSubsetDefinitions()); - // The fallback patch isn't stored in ConditionAndGlyphs() so add it in - // manually. - SegmentSet fallback_segments; - btree_map conditions_with_fallback; + btree_map filtered_conditions; for (const auto& [condition, glyphs] : ConditionsAndGlyphs()) { if (exclusive_only && !condition.IsExclusive()) { continue; } - conditions_with_fallback.insert({condition, glyphs}); + filtered_conditions.insert({condition, glyphs}); } - if (!exclusive_only && !unmapped_glyphs_.empty()) { - fallback_segments = segmentation_info.NonEmptySegments(); - - ActivationCondition non_fallback_cond = - ActivationCondition::or_segments(fallback_segments, 0, false); - ActivationCondition fallback_cond = - ActivationCondition::or_segments(fallback_segments, 0, true); - - if (conditions_with_fallback.contains(non_fallback_cond)) { - // If an equivalent non-fallback condition exists merge it with the - // fallback one. - conditions_with_fallback[fallback_cond].union_set( - conditions_with_fallback[non_fallback_cond]); - conditions_with_fallback.erase(non_fallback_cond); - } - - conditions_with_fallback[fallback_cond].union_set(unmapped_glyphs_); - } - - TRYV(GlyphSegmentation::ConditionsToSegmentation( - conditions_with_fallback, fallback_segments, segmentation)); + TRYV(GlyphSegmentation::ConditionsToSegmentation(filtered_conditions, + segmentation)); return segmentation; } Status GlyphGroupings::GroupGlyphs( const RequestedSegmentationInformation& segmentation_info, - const GlyphConditionSet& glyph_condition_set, - GlyphClosureCache& closure_cache, - std::optional dependency_closure, GlyphSet glyphs, - const SegmentSet& modified_segments, bool additional_conditions_check) { - const auto& initial_closure = segmentation_info.InitFontGlyphs(); - SegmentSet inscope_fallback_segments; - + const GlyphConditionSet& glyph_condition_set, GlyphSet glyphs, + const SegmentSet& modified_segments) { for (glyph_id_t gid : glyphs) { - CollectSegments(gid, inscope_fallback_segments); InvalidateGlyphInformation(gid); } - if (!inscope_fallback_segments.empty()) { - inscope_fallback_segments.union_set(modified_segments); - } else { - // If no existing conditions exist, all segments are inscope. - inscope_fallback_segments.invert(); - } - // Find any additional glyphs that are affected by changes in // modified_segments GlyphSet additional_glyphs = ModifiedGlyphs(modified_segments); for (glyph_id_t gid : additional_glyphs) { - CollectSegments(gid, inscope_fallback_segments); InvalidateGlyphInformation(gid); } glyphs.union_set(additional_glyphs); @@ -204,11 +139,6 @@ Status GlyphGroupings::GroupGlyphs( const auto& condition = glyph_condition_set.ConditionsFor(gid).activation(); if (condition.IsAlwaysTrue()) { - if (!initial_closure.contains(gid) && - segmentation_info.FullClosure().contains(gid)) { - // no condition so this is an unmapped glyph - unmapped_glyphs_.insert(gid); - } continue; } @@ -239,82 +169,25 @@ Status GlyphGroupings::GroupGlyphs( } } - // Add conditions for everything except for purely disjunctive conditions whic - // need additional processing. + // Add conditions for everything except for purely disjunctive conditions. for (const auto& c : all_other_modified_conditions) { TRYV(AddConditionAndGlyphs(c, new_groupings.at(c))); } - // Any of the or_set conditions we've generated may have some additional - // conditions that were not detected. Therefore we need to rule out the - // presence of these additional conditions if an or group is able to be - // used. for (const auto& or_group : modified_or_groups) { auto& glyphs = or_glyph_groups_[or_group]; ActivationCondition condition = ActivationCondition::or_segments(or_group, 0); - - if (!additional_conditions_check) { - TRYV(AddConditionAndGlyphs(condition, glyphs)); - continue; - } - - SegmentSet all_other_segment_ids; - if (!segmentation_info.Segments().empty()) { - all_other_segment_ids.insert_range( - 0, segmentation_info.Segments().size() - 1); - all_other_segment_ids.subtract(or_group); - } - - GlyphSet or_gids = TRY(closure_cache.CodepointsToOrGids( - segmentation_info, all_other_segment_ids)); - - // Any "OR" glyphs associated with all other codepoints have some - // additional conditions to activate so we can't safely include them into - // this or condition. They are instead moved to the set of unmapped - // glyphs. - for (uint32_t gid : or_gids) { - if (glyphs.erase(gid) > 0) { - unmapped_glyphs_.insert(gid); - } - } - - if (glyphs.empty()) { - // Group has been emptied out, so it's no longer needed. - or_glyph_groups_.erase(or_group); - RemoveConditionAndGlyphs(condition); - continue; - } - TRYV(AddConditionAndGlyphs(condition, glyphs)); } - if (segmentation_info.GetUnmappedGlyphHandling() == FIND_CONDITIONS) { - TRYV(FindFallbackGlyphConditions(segmentation_info, glyph_condition_set, - inscope_fallback_segments, closure_cache, - dependency_closure)); - } - // The combined conditions can't be incrementally updated, so we recompute // them in full if needed. TRYV(RecomputeCombinedConditionsIfNeeded(glyphs)); - // Note: we don't need to include the fallback segment/condition in - // conditions_and_glyphs since all downstream processing which - // utilizes that map ignores the fallback segment. It will be - // manually added to the final segmentation in ToGlyphSegmentation() - return absl::OkStatus(); } -void GlyphGroupings::CollectSegments(glyph_id_t gid, SegmentSet& segments) { - auto it = conditions_and_glyphs_.GlyphToCondition().find(gid); - if (it == conditions_and_glyphs_.GlyphToCondition().end()) { - return; - } - segments.union_set(it->second.TriggeringSegments()); -} - GlyphSet GlyphGroupings::ModifiedGlyphs(const SegmentSet& segments) const { GlyphSet glyphs; for (segment_index_t s : segments) { @@ -326,61 +199,6 @@ GlyphSet GlyphGroupings::ModifiedGlyphs(const SegmentSet& segments) const { return glyphs; } -Status GlyphGroupings::FindFallbackGlyphConditions( - const RequestedSegmentationInformation& segmentation_info, - const GlyphConditionSet& glyph_condition_set, - const SegmentSet& inscope_segments, GlyphClosureCache& closure_cache, - std::optional dependency_closure) { - if (unmapped_glyphs_.empty()) { - return absl::OkStatus(); - } - - SegmentSet inscope = SegmentSet::all(); - if (dependency_closure.has_value()) { - inscope = TRY( - dependency_closure.value()->SegmentsThatInteractWith(unmapped_glyphs_)); - VLOG(0) << "used dep graph to scope complex condition finding to " - << inscope.size() << " segments."; - } - - // Note: inscope_segments is not currently used, the approach needs more - // work. In testing in some cases it caused complex conditions to be larger - // than necessary when a segment which could shorten the condition isn't in - // scope. - // - // For example, with true condition (a or b) AND (b or c), if only {a, c} is - // inscope then we don't have the possibility of finding the superset of {b}. - btree_map complex_conditions = - TRY(FindSupersetDisjunctiveConditionsFor( - segmentation_info, glyph_condition_set, closure_cache, - unmapped_glyphs_, inscope)); - - found_condition_glyphs_.union_set(unmapped_glyphs_); - unmapped_glyphs_.clear(); - for (const auto& [s, g] : complex_conditions) { - if (s.empty()) { - return absl::InternalError("Complex conditions should never be empty."); - } - - ActivationCondition c = ActivationCondition::or_segments(s, 0); - if (s.size() == 1) { - segment_index_t segment = *s.begin(); - exclusive_glyph_groups_[segment].union_set(g); - c = ActivationCondition::exclusive_segment(segment, 0); - } else { - or_glyph_groups_[s].union_set(g); - } - - // There may be existing glyphs at this specific condition, so union into - // it. - TRYV(UnionConditionAndGlyphs(c, g)); - } - VLOG(0) - << "Unmapped glyphs patch removed and replaced with found conditions."; - - return absl::OkStatus(); -} - Status GlyphGroupings::RecomputeCombinedConditions() { // Find all conditions that are affected by combined_patches_. btree_set affected_conditions; diff --git a/ift/encoder/glyph_groupings.h b/ift/encoder/glyph_groupings.h index c83b576b..5a8e8205 100644 --- a/ift/encoder/glyph_groupings.h +++ b/ift/encoder/glyph_groupings.h @@ -8,8 +8,6 @@ #include "ift/common/int_set.h" #include "ift/encoder/activation_condition.h" #include "ift/encoder/condition_to_glyphs_index.h" -#include "ift/encoder/dependency_closure.h" -#include "ift/encoder/glyph_closure_cache.h" #include "ift/encoder/glyph_condition_set.h" #include "ift/encoder/glyph_partition.h" #include "ift/encoder/glyph_segmentation.h" @@ -33,8 +31,7 @@ class GlyphGroupings { simplify_combined_ == other.simplify_combined_ && conditions_and_glyphs_ == other.conditions_and_glyphs_ && conditions_and_glyphs_pre_combination_ == - other.conditions_and_glyphs_pre_combination_ && - unmapped_glyphs_ == other.unmapped_glyphs_; + other.conditions_and_glyphs_pre_combination_; } bool operator!=(const GlyphGroupings& other) { return !(*this == other); } @@ -88,16 +85,6 @@ class GlyphGroupings { return empty; } - // Returns the set of glyphs that are considered unmapped, - // which will be placed in the fallback (always loaded) patch. - ift::common::GlyphSet UnmappedGlyphs() const { return unmapped_glyphs_; } - - // Returns the set of glyphs that were unmapped but had conditions - // found for them. - const ift::common::GlyphSet& FoundConditionGlyphs() const { - return found_condition_glyphs_; - } - // Returns a list of conditions which include segment. const absl::flat_hash_set& TriggeringSegmentToConditions( segment_index_t segment) const { @@ -125,11 +112,6 @@ class GlyphGroupings { return empty; } - // Add a set of glyphs to an existing exclusive group (and_group of one - // segment). - absl::Status AddGlyphsToExclusiveGroup(segment_index_t exclusive_segment, - const ift::common::GlyphSet& glyphs); - // Specify that any patches containing glyphs from either a or b should be // merged into one patch. Only affects exclusive and disjunctive patches. // @@ -157,11 +139,8 @@ class GlyphGroupings { absl::Status GroupGlyphs( const RequestedSegmentationInformation& segmentation_info, const GlyphConditionSet& glyph_condition_set, - GlyphClosureCache& closure_cache, - std::optional dependency_closure, ift::common::GlyphSet glyphs, - const ift::common::SegmentSet& modified_segments, - bool additional_conditions_check = true); + const ift::common::SegmentSet& modified_segments); // Converts this grouping into a finalized GlyphSegmentation. absl::StatusOr ToGlyphSegmentation( @@ -189,21 +168,9 @@ class GlyphGroupings { } private: - void CollectSegments(glyph_id_t gid, ift::common::SegmentSet& segments); - ift::common::GlyphSet ModifiedGlyphs( const ift::common::SegmentSet& segments) const; - // Perform a more detailed analysis to try and find more granular conditions - // for fallback glyphs. Will replace the fallback glyphs with any found - // conditions. - absl::Status FindFallbackGlyphConditions( - const RequestedSegmentationInformation& segmentation_info, - const GlyphConditionSet& glyph_condition_set, - const ift::common::SegmentSet& inscope_segments, - GlyphClosureCache& closure_cache, - std::optional dependency_closure); - // Removes all stored grouping information related to glyph with the specified // condition. void InvalidateGlyphInformation(uint32_t gid); @@ -310,14 +277,6 @@ class GlyphGroupings { // An alternate representation of and/or_glyph_groups_, derived from them. ConditionToGlyphsIndex conditions_and_glyphs_; ConditionToGlyphsIndex conditions_and_glyphs_pre_combination_; - - // These glyphs aren't mapped by any conditions and as a result should be - // included in the fallback patch. - ift::common::GlyphSet unmapped_glyphs_; - - // These glyphs were previously considered unmapped, but have had conditions - // found for them. - ift::common::GlyphSet found_condition_glyphs_; }; } // namespace ift::encoder diff --git a/ift/encoder/glyph_groupings_test.cc b/ift/encoder/glyph_groupings_test.cc index c9698907..08ddaf6a 100644 --- a/ift/encoder/glyph_groupings_test.cc +++ b/ift/encoder/glyph_groupings_test.cc @@ -10,7 +10,6 @@ #include "ift/common/int_set.h" #include "ift/common/test_font_loader.h" #include "ift/encoder/activation_condition.h" -#include "ift/encoder/complex_condition_finder.h" #include "ift/encoder/dependency_closure.h" #include "ift/encoder/glyph_closure_cache.h" #include "ift/encoder/glyph_condition_set.h" @@ -21,9 +20,6 @@ #include "ift/encoder/types.h" #include "ift/freq/probability_bound.h" -using ift::config::FIND_CONDITIONS; -using ift::config::PATCH; - namespace ift::encoder { using absl::btree_map; @@ -65,15 +61,7 @@ class GlyphGroupingsTest : public ::testing::Test { glyph_groupings_(hb_face_get_glyph_count(roboto_.get())), glyph_groupings_simplification_(hb_face_get_glyph_count(roboto_.get()), - true), - segments_complex_({ - Segment({0x54}), // s0 - Segment({0x6C}), // s1 - Segment({0x13C}), // s2 - Segment({0x21A}), // s3 - Segment({0xF6C3}), // s4 - }), - glyph_groupings_complex_(hb_face_get_glyph_count(roboto_.get())) { + true) { uint32_t num_glyphs = hb_face_get_glyph_count(roboto_.get()); SubsetDefinition init_font_segment; @@ -83,7 +71,7 @@ class GlyphGroupingsTest : public ::testing::Test { closure_cache_ = std::move(*GlyphClosureCache::Create(roboto_.get(), *resolver)); requested_segmentation_info_ = *RequestedSegmentationInformation::Create( - segments_, init_font_segment, *closure_cache_, PATCH); + segments_, init_font_segment, *closure_cache_); glyph_conditions_ = std::make_unique(num_glyphs); @@ -125,34 +113,6 @@ class GlyphGroupingsTest : public ::testing::Test { // Disjunctive on segments 2 and 3 glyph_conditions_->AddOrCondition(cp_to_gid_['j'], 2); glyph_conditions_->AddOrCondition(cp_to_gid_['j'], 3); - - SetupComplexCase(); - } - - void SetupComplexCase() { - hb_font_t* font = hb_font_create(roboto_.get()); - std::vector codepoints = {0x54, 0x6C, 0x13C, 0x21A, 0xF6C3}; - - for (hb_codepoint_t cp : codepoints) { - glyph_id_t gid; - hb_font_get_nominal_glyph(font, cp, &gid); - cp_to_gid_[cp] = gid; - glyphs_to_group_complex_.insert(gid); - } - hb_font_destroy(font); - - glyph_conditions_complex_ = std::make_unique( - hb_face_get_glyph_count(roboto_.get())); - glyph_conditions_complex_->AddAndCondition(ToGlyph(0x54), 0); - glyph_conditions_complex_->AddAndCondition(ToGlyph(0x6C), 1); - - SubsetDefinition init_font_segment; - AddInitSubsetDefaults(init_font_segment); - - requested_segmentation_info_complex_ = - *RequestedSegmentationInformation::Create( - segments_complex_, init_font_segment, *closure_cache_, - FIND_CONDITIONS); } hb_face_unique_ptr from_file(const char* filename) { @@ -182,19 +142,11 @@ class GlyphGroupingsTest : public ::testing::Test { GlyphGroupings glyph_groupings_simplification_; GlyphSet glyphs_to_group_; flat_hash_map cp_to_gid_; - - std::vector segments_complex_; - std::unique_ptr glyph_conditions_complex_; - GlyphGroupings glyph_groupings_complex_; - GlyphSet glyphs_to_group_complex_; - std::unique_ptr - requested_segmentation_info_complex_; }; TEST_F(GlyphGroupingsTest, SimpleGrouping) { - auto sc = glyph_groupings_.GroupGlyphs(*requested_segmentation_info_, - *glyph_conditions_, *closure_cache_, - std::nullopt, glyphs_to_group_, {}); + auto sc = glyph_groupings_.GroupGlyphs( + *requested_segmentation_info_, *glyph_conditions_, glyphs_to_group_, {}); ASSERT_TRUE(sc.ok()) << sc; // Condition map: @@ -217,9 +169,8 @@ TEST_F(GlyphGroupingsTest, SimpleGrouping) { } TEST_F(GlyphGroupingsTest, SegmentChange) { - auto sc = glyph_groupings_.GroupGlyphs(*requested_segmentation_info_, - *glyph_conditions_, *closure_cache_, - std::nullopt, glyphs_to_group_, {}); + auto sc = glyph_groupings_.GroupGlyphs( + *requested_segmentation_info_, *glyph_conditions_, glyphs_to_group_, {}); ASSERT_TRUE(sc.ok()) << sc; // Now in the glyph condition set combine segments s1 into s0 @@ -253,8 +204,7 @@ TEST_F(GlyphGroupingsTest, SegmentChange) { // Recompute the grouping sc = glyph_groupings_.GroupGlyphs(*requested_segmentation_info_, - new_conditions, *closure_cache_, - std::nullopt, glyphs_to_group_, {}); + new_conditions, glyphs_to_group_, {}); ASSERT_TRUE(sc.ok()) << sc; // Condition map: @@ -280,8 +230,7 @@ TEST_F(GlyphGroupingsTest, CombinePatches) { ASSERT_TRUE(sc.ok()) << sc; sc = glyph_groupings_.GroupGlyphs(*requested_segmentation_info_, - *glyph_conditions_, *closure_cache_, - std::nullopt, glyphs_to_group_, {}); + *glyph_conditions_, glyphs_to_group_, {}); ASSERT_TRUE(sc.ok()) << sc; // Condition map: @@ -308,8 +257,7 @@ TEST_F(GlyphGroupingsTest, CombinePatches_WithSimpfliciation) { ASSERT_TRUE(sc.ok()) << sc; sc = glyph_groupings_simplification_.GroupGlyphs( - *requested_segmentation_info_, *glyph_conditions_, *closure_cache_, - std::nullopt, glyphs_to_group_, {}); + *requested_segmentation_info_, *glyph_conditions_, glyphs_to_group_, {}); ASSERT_TRUE(sc.ok()) << sc; // Starting Condition map: @@ -346,8 +294,7 @@ TEST_F(GlyphGroupingsTest, CombinePatches_SimplificationCollision) { ASSERT_TRUE(sc.ok()) << sc; sc = glyph_groupings_simplification_.GroupGlyphs( - *requested_segmentation_info_, *glyph_conditions_, *closure_cache_, - std::nullopt, glyphs_to_group_, {}); + *requested_segmentation_info_, *glyph_conditions_, glyphs_to_group_, {}); ASSERT_TRUE(sc.ok()) << sc; flat_hash_map expected = { @@ -375,74 +322,26 @@ TEST_F(GlyphGroupingsTest, CombinePatches_SimplificationCollisionInvalidation) { ASSERT_TRUE(sc.ok()) << sc; sc = glyph_groupings_simplification_.GroupGlyphs( - *requested_segmentation_info_, *glyph_conditions_, *closure_cache_, - std::nullopt, glyphs_to_group_, {}); + *requested_segmentation_info_, *glyph_conditions_, glyphs_to_group_, {}); ASSERT_TRUE(sc.ok()) << sc; // Invalidate 'l' by calling GroupGlyphs with 'l' in the glyphs set. sc = glyph_groupings_simplification_.GroupGlyphs( - *requested_segmentation_info_, *glyph_conditions_, *closure_cache_, - std::nullopt, ToGlyphs({'l'}), {}); - ASSERT_TRUE(sc.ok()) << sc; -} - -TEST_F(GlyphGroupingsTest, CombinePatches_WithInertSpecialCase) { - // Initial Condition map: - // s0 -> {a, b} - // s1 -> {c, d} - // s3 -> {k} - // s2 AND s3 -> {e, f} - // s2 OR s3 -> {j} - // s3 OR s4 -> {g, h} - auto sc = glyph_groupings_.CombinePatches(ToGlyphs({'a'}), ToGlyphs({'c'})); - ASSERT_TRUE(sc.ok()) << sc; - - sc = glyph_groupings_.GroupGlyphs(*requested_segmentation_info_, - *glyph_conditions_, *closure_cache_, - std::nullopt, glyphs_to_group_, {}); + *requested_segmentation_info_, *glyph_conditions_, ToGlyphs({'l'}), {}); ASSERT_TRUE(sc.ok()) << sc; - - // Combined Condition map: - // s0 OR s1 -> {a, b, c, d} - // s3 -> {k} - // s2 AND s3 -> {e, f} - // s2 OR s3 -> {j} - // s3 OR s4 -> {g, h} - - // Create a new exclusive patch without using GroupGlyphs(), merges s1 and s3 - // into s3 - sc = glyph_groupings_.AddGlyphsToExclusiveGroup(3, ToGlyphs({'c', 'd', 'k'})); - ASSERT_TRUE(sc.ok()) << sc; - - // s3 should get pulled into the combined patch of s0 and s1: - // s0 OR s1 OR s3 -> {a, b, c, d, k} - // s2 AND s3 -> {e, f} - // s2 OR s3 -> {j} - // s3 OR s4 -> {g, h} - flat_hash_map expected = { - {ActivationCondition::or_segments({0, 3}, 0), - ToGlyphs({'a', 'b', 'c', 'd', 'k'})}, - {ActivationCondition::and_segments({2, 3}, 0), ToGlyphs({'e', 'f'})}, - {ActivationCondition::or_segments({2, 3}, 0), ToGlyphs({'j'})}, - {ActivationCondition::or_segments({3, 4}, 0), ToGlyphs({'g', 'h'})}, - }; - - ASSERT_EQ(expected, glyph_groupings_.ConditionsAndGlyphs()); } TEST_F(GlyphGroupingsTest, CombinePatches_Invalidates) { // Form grouping without union's - auto sc = glyph_groupings_.GroupGlyphs(*requested_segmentation_info_, - *glyph_conditions_, *closure_cache_, - std::nullopt, glyphs_to_group_, {}); + auto sc = glyph_groupings_.GroupGlyphs( + *requested_segmentation_info_, *glyph_conditions_, glyphs_to_group_, {}); ASSERT_TRUE(sc.ok()) << sc; sc = glyph_groupings_.CombinePatches(ToGlyphs({'g'}), ToGlyphs({'b'})); ASSERT_TRUE(sc.ok()) << sc; sc = glyph_groupings_.GroupGlyphs(*requested_segmentation_info_, - *glyph_conditions_, *closure_cache_, - std::nullopt, {}, {}); + *glyph_conditions_, {}, {}); ASSERT_TRUE(sc.ok()) << sc; // UnionPatches + GroupGlyphs() will automatically invalidate and then fix @@ -466,9 +365,8 @@ TEST_F(GlyphGroupingsTest, CombinePatches_Invalidates) { TEST_F(GlyphGroupingsTest, CombinePatches_PartialUpdate) { // Form grouping without union's - auto sc = glyph_groupings_.GroupGlyphs(*requested_segmentation_info_, - *glyph_conditions_, *closure_cache_, - std::nullopt, glyphs_to_group_, {}); + auto sc = glyph_groupings_.GroupGlyphs( + *requested_segmentation_info_, *glyph_conditions_, glyphs_to_group_, {}); ASSERT_TRUE(sc.ok()) << sc; sc = glyph_groupings_.CombinePatches(ToGlyphs({'g'}), ToGlyphs({'b'})); @@ -478,8 +376,8 @@ TEST_F(GlyphGroupingsTest, CombinePatches_PartialUpdate) { // and see if grouping correctly reforms the full mapping. Invalidates: // s0 -> {a, b} sc = glyph_groupings_.GroupGlyphs(*requested_segmentation_info_, - *glyph_conditions_, *closure_cache_, - std::nullopt, ToGlyphs({'a', 'b'}), {}); + *glyph_conditions_, ToGlyphs({'a', 'b'}), + {}); ASSERT_TRUE(sc.ok()) << sc; // Expected condition map: @@ -501,8 +399,8 @@ TEST_F(GlyphGroupingsTest, CombinePatches_PartialUpdate) { // Do another partial invalidation this time on: s3 OR s4 -> {g, h} sc = glyph_groupings_.GroupGlyphs(*requested_segmentation_info_, - *glyph_conditions_, *closure_cache_, - std::nullopt, ToGlyphs({'g', 'h'}), {}); + *glyph_conditions_, ToGlyphs({'g', 'h'}), + {}); ASSERT_TRUE(sc.ok()) << sc; // Groupings should still be the same. @@ -513,8 +411,7 @@ TEST_F(GlyphGroupingsTest, CombinePatches_Noop) { auto sc = glyph_groupings_.CombinePatches(ToGlyphs({'a'}), ToGlyphs({'b'})); ASSERT_TRUE(sc.ok()) << sc; sc = glyph_groupings_.GroupGlyphs(*requested_segmentation_info_, - *glyph_conditions_, *closure_cache_, - std::nullopt, glyphs_to_group_, {}); + *glyph_conditions_, glyphs_to_group_, {}); ASSERT_TRUE(sc.ok()) << sc; // The combination is a noop since it only combines things already in the @@ -542,8 +439,7 @@ TEST_F(GlyphGroupingsTest, CombinePatches_AffectsConjunction) { ASSERT_TRUE(sc.ok()) << sc; sc = glyph_groupings_.GroupGlyphs(*requested_segmentation_info_, - *glyph_conditions_, *closure_cache_, - std::nullopt, glyphs_to_group_, {}); + *glyph_conditions_, glyphs_to_group_, {}); ASSERT_TRUE(sc.ok()) << sc; // Condition map after merging 'd' and 'e': @@ -571,8 +467,7 @@ TEST_F(GlyphGroupingsTest, CombinePatches_SegmentChanges) { ASSERT_TRUE(sc.ok()) << sc; sc = glyph_groupings_.GroupGlyphs(*requested_segmentation_info_, - *glyph_conditions_, *closure_cache_, - std::nullopt, glyphs_to_group_, {}); + *glyph_conditions_, glyphs_to_group_, {}); ASSERT_TRUE(sc.ok()) << sc; // Now in the glyph condition set combine segments s1 into s0 @@ -606,9 +501,9 @@ TEST_F(GlyphGroupingsTest, CombinePatches_SegmentChanges) { new_conditions.AddOrCondition(cp_to_gid_['j'], 3); // Recompute the grouping - sc = glyph_groupings_.GroupGlyphs( - *requested_segmentation_info_, new_conditions, *closure_cache_, - std::nullopt, ToGlyphs({'a', 'b', 'c', 'd'}), {}); + sc = glyph_groupings_.GroupGlyphs(*requested_segmentation_info_, + new_conditions, + ToGlyphs({'a', 'b', 'c', 'd'}), {}); ASSERT_TRUE(sc.ok()) << sc; // Condition map: @@ -632,13 +527,12 @@ TEST_F(GlyphGroupingsTest, EqualityRespectsPatchCombination) { ASSERT_TRUE(sc.ok()) << sc; sc = glyph_groupings_.GroupGlyphs(*requested_segmentation_info_, - *glyph_conditions_, *closure_cache_, - std::nullopt, glyphs_to_group_, {}); + *glyph_conditions_, glyphs_to_group_, {}); ASSERT_TRUE(sc.ok()) << sc; GlyphGroupings other(hb_face_get_glyph_count(roboto_.get())); sc = other.GroupGlyphs(*requested_segmentation_info_, *glyph_conditions_, - *closure_cache_, std::nullopt, glyphs_to_group_, {}); + glyphs_to_group_, {}); ASSERT_TRUE(sc.ok()) << sc; // other does not have the same patch combinations and so should not be equal @@ -649,7 +543,7 @@ TEST_F(GlyphGroupingsTest, EqualityRespectsPatchCombination) { ASSERT_TRUE(sc.ok()); sc = other.GroupGlyphs(*requested_segmentation_info_, *glyph_conditions_, - *closure_cache_, std::nullopt, glyphs_to_group_, {}); + glyphs_to_group_, {}); ASSERT_TRUE(sc.ok()) << sc; // Now that combined patches matches they should be equal. @@ -661,8 +555,7 @@ TEST_F(GlyphGroupingsTest, ExclusiveGlyphsRespectsPatchCombinations) { ASSERT_TRUE(sc.ok()) << sc; sc = glyph_groupings_.GroupGlyphs(*requested_segmentation_info_, - *glyph_conditions_, *closure_cache_, - std::nullopt, glyphs_to_group_, {}); + *glyph_conditions_, glyphs_to_group_, {}); ASSERT_TRUE(sc.ok()) << sc; // Condition map: @@ -679,228 +572,16 @@ TEST_F(GlyphGroupingsTest, ExclusiveGlyphsRespectsPatchCombinations) { ASSERT_EQ(glyph_groupings_.ExclusiveGlyphs(10), (GlyphSet{})); } -TEST_F(GlyphGroupingsTest, ComplexConditionFinding_LeaveUnmapped) { - SubsetDefinition init_font_segment; - AddInitSubsetDefaults(init_font_segment); - std::unique_ptr segmentation_info = - *RequestedSegmentationInformation::Create( - segments_complex_, init_font_segment, *closure_cache_, PATCH); - - auto sc = glyph_groupings_complex_.GroupGlyphs( - *segmentation_info, *glyph_conditions_complex_, *closure_cache_, - std::nullopt, glyphs_to_group_complex_, {}); - ASSERT_TRUE(sc.ok()) << sc; - - // Condition map: - // if (s0) then p0 => {56} [excl] - // if (s1) then p0 => {80} [excl] - flat_hash_map expected = { - {ActivationCondition::exclusive_segment(0, 0), ToGlyphs({0x54})}, - {ActivationCondition::exclusive_segment(1, 0), ToGlyphs({0x6C})}, - }; - - ASSERT_EQ(expected, glyph_groupings_complex_.ConditionsAndGlyphs()); - ASSERT_EQ(glyph_groupings_complex_.UnmappedGlyphs(), - (GlyphSet{442, 748, 782})); -} - -TEST_F(GlyphGroupingsTest, ComplexConditionFinding_Basic) { - auto sc = glyph_groupings_complex_.GroupGlyphs( - *requested_segmentation_info_complex_, *glyph_conditions_complex_, - *closure_cache_, std::nullopt, glyphs_to_group_complex_, {}); - ASSERT_TRUE(sc.ok()) << sc; - - // Condition map: - // if (s0) then p0 => {56} [excl] - // if (s1) then p0 => {80} [excl] - // if ((s0 OR s3 OR s4)) then p0 => {782} - // if ((s1 OR s2 OR s4)) then p0 => {748} - // if ((s2 OR s3 OR s4)) then p0 => {442} - flat_hash_map expected = { - {ActivationCondition::exclusive_segment(0, 0), ToGlyphs({0x54})}, - {ActivationCondition::exclusive_segment(1, 0), ToGlyphs({0x6C})}, - - {ActivationCondition::or_segments({0, 3, 4}, 0), {782}}, - {ActivationCondition::or_segments({1, 2, 4}, 0), {748}}, - {ActivationCondition::or_segments({2, 3, 4}, 0), {442}}, - }; - - ASSERT_EQ(expected, glyph_groupings_complex_.ConditionsAndGlyphs()); - ASSERT_TRUE(glyph_groupings_complex_.UnmappedGlyphs().empty()); -} - -TEST_F(GlyphGroupingsTest, ComplexConditionFinding_Basic_WithDependencyGraph) { - std::unique_ptr dep_closure = *DependencyClosure::Create( - requested_segmentation_info_complex_.get(), roboto_.get(), *resolver); - - auto sc = glyph_groupings_complex_.GroupGlyphs( - *requested_segmentation_info_complex_, *glyph_conditions_complex_, - *closure_cache_, dep_closure.get(), glyphs_to_group_complex_, {}); - ASSERT_TRUE(sc.ok()) << sc; - - // Condition map: - // if (s0) then p0 => {56} [excl] - // if (s1) then p0 => {80} [excl] - // if ((s0 OR s3 OR s4)) then p0 => {782} - // if ((s1 OR s2 OR s4)) then p0 => {748} - // if ((s2 OR s3 OR s4)) then p0 => {442} - flat_hash_map expected = { - {ActivationCondition::exclusive_segment(0, 0), ToGlyphs({0x54})}, - {ActivationCondition::exclusive_segment(1, 0), ToGlyphs({0x6C})}, - - {ActivationCondition::or_segments({0, 3, 4}, 0), {782}}, - {ActivationCondition::or_segments({1, 2, 4}, 0), {748}}, - {ActivationCondition::or_segments({2, 3, 4}, 0), {442}}, - }; - - ASSERT_EQ(expected, glyph_groupings_complex_.ConditionsAndGlyphs()); - ASSERT_TRUE(glyph_groupings_complex_.UnmappedGlyphs().empty()); - - // Also try incremental - sc = glyph_groupings_complex_.GroupGlyphs( - *requested_segmentation_info_complex_, *glyph_conditions_complex_, - *closure_cache_, dep_closure.get(), {748}, {}); - ASSERT_TRUE(sc.ok()) << sc; - ASSERT_EQ(expected, glyph_groupings_complex_.ConditionsAndGlyphs()); - ASSERT_TRUE(glyph_groupings_complex_.UnmappedGlyphs().empty()); -} - -TEST_F(GlyphGroupingsTest, ComplexConditionFinding_IncrementalUnchanged) { - auto sc = glyph_groupings_complex_.GroupGlyphs( - *requested_segmentation_info_complex_, *glyph_conditions_complex_, - *closure_cache_, std::nullopt, glyphs_to_group_complex_, {}); - ASSERT_TRUE(sc.ok()) << sc; - flat_hash_map expected = - glyph_groupings_complex_.ConditionsAndGlyphs(); - - // Incremental regrouping, should arrive back at the same mapping. - sc = glyph_groupings_complex_.GroupGlyphs( - *requested_segmentation_info_complex_, *glyph_conditions_complex_, - *closure_cache_, std::nullopt, {748}, {}); - ASSERT_TRUE(sc.ok()) << sc; - ASSERT_EQ(expected, glyph_groupings_complex_.ConditionsAndGlyphs()); - ASSERT_TRUE(glyph_groupings_complex_.UnmappedGlyphs().empty()); - - sc = glyph_groupings_complex_.GroupGlyphs( - *requested_segmentation_info_complex_, *glyph_conditions_complex_, - *closure_cache_, std::nullopt, ToGlyphs({0x54}), {2}); - ASSERT_TRUE(sc.ok()) << sc; - ASSERT_EQ(expected, glyph_groupings_complex_.ConditionsAndGlyphs()); - ASSERT_TRUE(glyph_groupings_complex_.UnmappedGlyphs().empty()); -} - -TEST_F(GlyphGroupingsTest, ComplexConditionFinding_IncrementalChanged) { - auto sc = glyph_groupings_complex_.GroupGlyphs( - *requested_segmentation_info_complex_, *glyph_conditions_complex_, - *closure_cache_, std::nullopt, glyphs_to_group_complex_, {}); - ASSERT_TRUE(sc.ok()) << sc; - - // Modify a segment definition and then incremental recompute the groupings. - Segment merged = segments_complex_[0]; - merged.Definition().Union(segments_complex_[3].Definition()); - requested_segmentation_info_complex_->AssignMergedSegment(0, {3}, merged); - - // Incremental regrouping, should arrive back at the same mapping. - sc = glyph_groupings_complex_.GroupGlyphs( - *requested_segmentation_info_complex_, *glyph_conditions_complex_, - *closure_cache_, std::nullopt, ToGlyphs({0x54}), {0, 3}); - ASSERT_TRUE(sc.ok()) << sc; - - flat_hash_map expected = { - {ActivationCondition::exclusive_segment(0, 0), ToGlyphs({0x54})}, - {ActivationCondition::exclusive_segment(1, 0), ToGlyphs({0x6C})}, - {ActivationCondition::or_segments({1, 2, 4}, 0), {748}}, - }; - - auto new_mappings = *FindSupersetDisjunctiveConditionsFor( - *requested_segmentation_info_complex_, *glyph_conditions_complex_, - *closure_cache_, {442, 782}, SegmentSet::all()); - - for (const auto& [s, g] : new_mappings) { - if (s.size() == 1) { - expected[ActivationCondition::exclusive_segment(*s.begin(), 0)].union_set( - g); - } else { - expected[ActivationCondition::or_segments(s, 0)].union_set(g); - } - } - - ASSERT_EQ(expected, glyph_groupings_complex_.ConditionsAndGlyphs()); - ASSERT_TRUE(glyph_groupings_complex_.UnmappedGlyphs().empty()); -} - -TEST_F(GlyphGroupingsTest, ComplexConditionFinding_CombinedPatches) { - auto sc = glyph_groupings_complex_.CombinePatches({ToGlyph(0x6C)}, {782}); - ASSERT_TRUE(sc.ok()) << sc; - - sc = glyph_groupings_complex_.GroupGlyphs( - *requested_segmentation_info_complex_, *glyph_conditions_complex_, - *closure_cache_, std::nullopt, glyphs_to_group_complex_, {}); - ASSERT_TRUE(sc.ok()) << sc; - - // Condition map: - // if (s0) then p0 => {56} [excl] - // if ((s0 OR s1 OR s3 OR s4)) then p0 => {80, 782} - // if ((s1 OR s2 OR s4)) then p0 => {748} - // if ((s2 OR s3 OR s4)) then p0 => {442} - flat_hash_map expected = { - {ActivationCondition::exclusive_segment(0, 0), ToGlyphs({0x54})}, - - {ActivationCondition::or_segments({0, 1, 3, 4}, 0), {ToGlyph(0x6C), 782}}, - {ActivationCondition::or_segments({1, 2, 4}, 0), {748}}, - {ActivationCondition::or_segments({2, 3, 4}, 0), {442}}, - }; - - ASSERT_EQ(expected, glyph_groupings_complex_.ConditionsAndGlyphs()); - ASSERT_TRUE(glyph_groupings_complex_.UnmappedGlyphs().empty()); -} - -TEST_F(GlyphGroupingsTest, - ComplexConditionFinding_IncrementalAndCombinedPatches) { - auto sc = glyph_groupings_complex_.CombinePatches({ToGlyph(0x6C)}, {782}); - ASSERT_TRUE(sc.ok()) << sc; - - sc = glyph_groupings_complex_.GroupGlyphs( - *requested_segmentation_info_complex_, *glyph_conditions_complex_, - *closure_cache_, std::nullopt, glyphs_to_group_complex_, {}); - ASSERT_TRUE(sc.ok()) << sc; - - // Modify a segment definition and then incremental recompute the groupings. - Segment merged = segments_complex_[0]; - merged.Definition().Union(segments_complex_[3].Definition()); - requested_segmentation_info_complex_->AssignMergedSegment(0, {3}, merged); - - sc = glyph_groupings_complex_.GroupGlyphs( - *requested_segmentation_info_complex_, *glyph_conditions_complex_, - *closure_cache_, std::nullopt, ToGlyphs({0x54}), {0, 3}); - ASSERT_TRUE(sc.ok()) << sc; - - // Condition map: - // if ((s0 OR s1)) then p0 => {56, 80, 782} - // if ((s1 OR s2 OR s4)) then p0 => {748} - // if ((s0 OR s2 OR s4)) then p0 => {442} - flat_hash_map expected = { - {ActivationCondition::or_segments({0, 1}, 0), - {ToGlyph(0x54), ToGlyph(0x6C), 782}}, - {ActivationCondition::or_segments({1, 2, 4}, 0), {748}}, - {ActivationCondition::or_segments({0, 2, 4}, 0), {442}}, - }; - - ASSERT_EQ(expected, glyph_groupings_complex_.ConditionsAndGlyphs()); - ASSERT_TRUE(glyph_groupings_complex_.UnmappedGlyphs().empty()); -} - TEST_F(GlyphGroupingsTest, IncrementalExclusiveConflict) { // Initial grouping for both - auto sc = glyph_groupings_.GroupGlyphs( - *requested_segmentation_info_, *glyph_conditions_, *closure_cache_, - std::nullopt, {ToGlyph('a'), ToGlyph('b')}, {}); + auto sc = glyph_groupings_.GroupGlyphs(*requested_segmentation_info_, + *glyph_conditions_, + {ToGlyph('a'), ToGlyph('b')}, {}); ASSERT_TRUE(sc.ok()) << sc; // Incremental grouping for ONLY one of them sc = glyph_groupings_.GroupGlyphs(*requested_segmentation_info_, - *glyph_conditions_, *closure_cache_, - std::nullopt, {ToGlyph('a')}, {}); + *glyph_conditions_, {ToGlyph('a')}, {}); EXPECT_TRUE(sc.ok()) << sc; ActivationCondition expected = ActivationCondition::exclusive_segment(0, 0); @@ -923,9 +604,8 @@ TEST_F(GlyphGroupingsTest, glyph_conditions_->AddOrCondition(gid_z, 1); // Initial grouping - auto sc = glyph_groupings_.GroupGlyphs(*requested_segmentation_info_, - *glyph_conditions_, *closure_cache_, - std::nullopt, glyphs_to_group_, {}); + auto sc = glyph_groupings_.GroupGlyphs( + *requested_segmentation_info_, *glyph_conditions_, glyphs_to_group_, {}); ASSERT_TRUE(sc.ok()) << sc; // Merge s0 and s1. This will create combined condition s0 OR s1. @@ -934,8 +614,7 @@ TEST_F(GlyphGroupingsTest, // Apply the merge. sc = glyph_groupings_.GroupGlyphs(*requested_segmentation_info_, - *glyph_conditions_, *closure_cache_, - std::nullopt, glyphs_to_group_, {}); + *glyph_conditions_, glyphs_to_group_, {}); ASSERT_TRUE(sc.ok()) << sc; // Trigger another merge that does NOT involve s0 or s1, which will cause @@ -944,9 +623,9 @@ TEST_F(GlyphGroupingsTest, ASSERT_TRUE(sc.ok()) << sc; // Apply the second merge. Only pass affected glyphs. - sc = glyph_groupings_.GroupGlyphs( - *requested_segmentation_info_, *glyph_conditions_, *closure_cache_, - std::nullopt, ToGlyphs({'g', 'h', 'j'}), {}); + sc = glyph_groupings_.GroupGlyphs(*requested_segmentation_info_, + *glyph_conditions_, + ToGlyphs({'g', 'h', 'j'}), {}); ASSERT_TRUE(sc.ok()) << sc; // Check if 'z' is still in s0 OR s1. diff --git a/ift/encoder/glyph_segmentation.cc b/ift/encoder/glyph_segmentation.cc index c7a94759..821fa5da 100644 --- a/ift/encoder/glyph_segmentation.cc +++ b/ift/encoder/glyph_segmentation.cc @@ -40,7 +40,7 @@ namespace ift::encoder { Status GlyphSegmentation::ConditionsToSegmentation( const btree_map& conditions, - const SegmentSet& fallback_group, GlyphSegmentation& segmentation) { + GlyphSegmentation& segmentation) { patch_id_t next_id = 0; segmentation.patches_.clear(); segmentation.conditions_.clear(); @@ -58,11 +58,8 @@ Status GlyphSegmentation::ConditionsToSegmentation( segmentation.conditions_.insert(ActivationCondition::exclusive_segment( *condition.TriggeringSegments().begin(), next_id++)); } else { - bool is_fallback = !fallback_group.empty() && - condition.IsPurelyDisjunctive() && - (condition.TriggeringSegments() == fallback_group); segmentation.conditions_.insert(ActivationCondition::composite_condition( - condition.conditions(), next_id++, is_fallback)); + condition.conditions(), next_id++)); } } diff --git a/ift/encoder/glyph_segmentation.h b/ift/encoder/glyph_segmentation.h index 82e29297..6023e297 100644 --- a/ift/encoder/glyph_segmentation.h +++ b/ift/encoder/glyph_segmentation.h @@ -30,16 +30,13 @@ namespace ift::encoder { class GlyphSegmentation { public: GlyphSegmentation(SubsetDefinition init_font_segment, - ift::common::GlyphSet init_font_glyph_closure, - ift::common::GlyphSet unmapped_glyphs) + ift::common::GlyphSet init_font_glyph_closure) : init_font_segment_(init_font_segment), - init_font_glyph_closure_(init_font_glyph_closure), - unmapped_glyphs_(unmapped_glyphs) {} + init_font_glyph_closure_(init_font_glyph_closure) {} bool operator==(const GlyphSegmentation& other) const { return init_font_segment_ == other.init_font_segment_ && init_font_glyph_closure_ == other.init_font_glyph_closure_ && - unmapped_glyphs_ == other.unmapped_glyphs_ && conditions_ == other.conditions_ && segments_ == other.segments_ && patches_ == other.patches_; } @@ -75,17 +72,6 @@ class GlyphSegmentation { return patches_; } - /* - * These glyphs where unable to be grouped into patches due to complex - * interactions. - * - * TODO(garretrieger): instead of treating them separately generate a catch - * all patch that contains the unmapped glyphs. - */ - const ift::common::GlyphSet& UnmappedGlyphs() const { - return unmapped_glyphs_; - }; - /* * These glyphs should be included in the initial font. */ @@ -107,7 +93,6 @@ class GlyphSegmentation { static absl::Status ConditionsToSegmentation( const absl::btree_map& conditions, - const common::SegmentSet& fallback_group, GlyphSegmentation& segmentation); void CopySegments(const std::vector& segments); @@ -115,7 +100,6 @@ class GlyphSegmentation { private: SubsetDefinition init_font_segment_; ift::common::GlyphSet init_font_glyph_closure_; - ift::common::GlyphSet unmapped_glyphs_; absl::btree_set conditions_; std::vector segments_; absl::btree_map patches_; diff --git a/ift/encoder/glyph_segmentation_test.cc b/ift/encoder/glyph_segmentation_test.cc index 9b41948a..ca5f84d2 100644 --- a/ift/encoder/glyph_segmentation_test.cc +++ b/ift/encoder/glyph_segmentation_test.cc @@ -14,8 +14,7 @@ #include "ift/proto/patch_encoding.h" #include "ift/proto/patch_map.h" -using ift::config::CLOSURE_ONLY; -using ift::config::PATCH; +using ift::config::DEP_GRAPH_ONLY; using google::protobuf::TextFormat; using ift::common::BazelDataFileResolver; @@ -54,7 +53,7 @@ class GlyphSegmentationTest : public ::testing::Test { }; TEST_F(GlyphSegmentationTest, SimpleSegmentation_ToConfigProto) { - ClosureGlyphSegmenter segmenter(8, 11, PATCH, CLOSURE_ONLY, resolver); + ClosureGlyphSegmenter segmenter(8, 11, DEP_GRAPH_ONLY, resolver); auto segmentation = segmenter.CodepointToGlyphSegments(roboto.get(), {'a'}, {{'b'}, {'c'}}); ASSERT_TRUE(segmentation.ok()) << segmentation.status(); @@ -125,7 +124,7 @@ initial_glyphs { } TEST_F(GlyphSegmentationTest, SimpleSegmentationWithFeatures_ToConfigProto) { - ClosureGlyphSegmenter segmenter(8, 11, PATCH, CLOSURE_ONLY, resolver); + ClosureGlyphSegmenter segmenter(8, 11, DEP_GRAPH_ONLY, resolver); SubsetDefinition smcp; smcp.feature_tags.insert(HB_TAG('s', 'm', 'c', 'p')); @@ -266,7 +265,7 @@ initial_glyphs { } TEST_F(GlyphSegmentationTest, MixedAndOr_ToConfigProto) { - ClosureGlyphSegmenter segmenter(8, 11, PATCH, CLOSURE_ONLY, resolver); + ClosureGlyphSegmenter segmenter(8, 11, DEP_GRAPH_ONLY, resolver); UnicodeFrequencies freq; auto segmentation = segmenter.CodepointToGlyphSegments( roboto.get(), {'a'}, {{'f', 0xc1}, {'i', 0x106}}); @@ -384,7 +383,7 @@ initial_glyphs { TEST_F(GlyphSegmentationTest, MergeBases_ToConfigProto) { // {e, f} is too smal, since no conditional patches exist it should merge with // the next available base which is {'j', 'k'} - ClosureGlyphSegmenter segmenter(8, 11, PATCH, CLOSURE_ONLY, resolver); + ClosureGlyphSegmenter segmenter(8, 11, DEP_GRAPH_ONLY, resolver); auto segmentation = segmenter.CodepointToGlyphSegments(roboto.get(), {}, { @@ -538,7 +537,4 @@ TEST_F(GlyphSegmentationTest, ActivationConditionOrdering) { ASSERT_FALSE(g < f); } -// TODO(garretrieger): add test where or_set glyphs are moved back to unmapped -// due to found "additional conditions". - } // namespace ift::encoder \ No newline at end of file diff --git a/ift/encoder/merger.cc b/ift/encoder/merger.cc index ba018f93..92ce7076 100644 --- a/ift/encoder/merger.cc +++ b/ift/encoder/merger.cc @@ -190,7 +190,7 @@ StatusOr> Merger::TryNextPatchMerge() { last_exclusive); it != Context().glyph_groupings.OrderedConditions().end(); ++it) { const auto& condition = *it; - if (condition.IsFallback() || condition.IsExclusive()) { + if (condition.IsExclusive()) { continue; } const GlyphSet& glyphs = @@ -758,10 +758,7 @@ Status Merger::CollectCompositeCandidateMerges( last_exclusive); it != context_->glyph_groupings.OrderedConditions().end(); it++) { const auto& next_condition = *it; - if (next_condition.IsFallback() || next_condition.IsExclusive()) { - // Merging the fallback will cause all segments to be merged into one, - // which is undesirable so don't consider the fallback. Also skip - // any non composite conditions. + if (next_condition.IsExclusive()) { continue; } @@ -823,7 +820,7 @@ GlyphSet Merger::ComputeCandidatePatchMergeGlyphs( it != context.glyph_groupings.OrderedConditions().end(); it++) { const auto& condition = *it; - if (condition.IsFallback() || condition.IsExclusive()) { + if (condition.IsExclusive()) { continue; } @@ -1047,10 +1044,9 @@ StatusOr> Merger::TryMergingACompositeCondition( for (const ActivationCondition& next_condition : context_->glyph_groupings.TriggeringSegmentToConditions( base_segment_index)) { - if (next_condition.IsFallback() || next_condition.IsUnitary()) { - // Merging the fallback will cause all segments to be merged into one, - // which is undesirable so don't consider the fallback. Unitary conditions - // only reference a single segment so there is nothing to merge. + if (next_condition.IsUnitary()) { + // Unitary conditions only reference a single segment so there is nothing + // to merge. continue; } diff --git a/ift/encoder/requested_segmentation_information.cc b/ift/encoder/requested_segmentation_information.cc index b33b3d74..e4a13d33 100644 --- a/ift/encoder/requested_segmentation_information.cc +++ b/ift/encoder/requested_segmentation_information.cc @@ -7,8 +7,6 @@ #include "ift/encoder/segment.h" #include "ift/encoder/subset_definition.h" -using ift::config::UnmappedGlyphHandling; - using absl::StatusOr; using ift::common::CodepointSet; using ift::common::SegmentSet; @@ -37,13 +35,11 @@ static bool CheckSegmentsAreDisjoint(const SubsetDefinition& init_segment, } StatusOr> -RequestedSegmentationInformation::Create( - std::vector segments, SubsetDefinition init_font_segment, - GlyphClosureCache& closure_cache, - UnmappedGlyphHandling unmapped_glyph_handling) { +RequestedSegmentationInformation::Create(std::vector segments, + SubsetDefinition init_font_segment, + GlyphClosureCache& closure_cache) { std::unique_ptr info( - new RequestedSegmentationInformation(segments, init_font_segment, - unmapped_glyph_handling)); + new RequestedSegmentationInformation(segments, init_font_segment)); TRYV(info->ReassignInitSubset(closure_cache, init_font_segment)); info->segments_disjoint_ = CheckSegmentsAreDisjoint(init_font_segment, info->segments_); @@ -51,11 +47,8 @@ RequestedSegmentationInformation::Create( } RequestedSegmentationInformation::RequestedSegmentationInformation( - std::vector segments, SubsetDefinition init_font_segment, - UnmappedGlyphHandling unmapped_glyph_handling) - : segments_(std::move(segments)), - init_font_segment_(), - unmapped_glyph_handling_(unmapped_glyph_handling) { + std::vector segments, SubsetDefinition init_font_segment) + : segments_(std::move(segments)), init_font_segment_() { // ReassignInitSubset expects full_definition_ is already populated. full_definition_ = init_font_segment; for (size_t i = 0; i < segments_.size(); ++i) { diff --git a/ift/encoder/requested_segmentation_information.h b/ift/encoder/requested_segmentation_information.h index 7bb5447c..13a9b649 100644 --- a/ift/encoder/requested_segmentation_information.h +++ b/ift/encoder/requested_segmentation_information.h @@ -6,7 +6,6 @@ #include "absl/container/flat_hash_map.h" #include "ift/common/int_set.h" #include "ift/common/try.h" -#include "ift/config/common.pb.h" #include "ift/encoder/glyph_closure_cache.h" #include "ift/encoder/init_subset_defaults.h" #include "ift/encoder/segment.h" @@ -22,13 +21,11 @@ class RequestedSegmentationInformation { public: static absl::StatusOr> Create(std::vector segments, SubsetDefinition init_font_segment, - GlyphClosureCache& closure_cache, - ift::config::UnmappedGlyphHandling unmapped_glyph_handling); + GlyphClosureCache& closure_cache); private: - RequestedSegmentationInformation( - std::vector segments, SubsetDefinition init_font_segment, - ift::config::UnmappedGlyphHandling unmapped_glyph_handling); + RequestedSegmentationInformation(std::vector segments, + SubsetDefinition init_font_segment); public: // Merge all of the segments in to_merge into base, assigned it @@ -83,10 +80,6 @@ class RequestedSegmentationInformation { return absl::OkStatus(); } - ift::config::UnmappedGlyphHandling GetUnmappedGlyphHandling() const { - return unmapped_glyph_handling_; - } - const SubsetDefinition& InitFontSegment() const { return init_font_segment_; } // Returns the init font segment with all default always included items @@ -173,7 +166,6 @@ class RequestedSegmentationInformation { ift::common::CodepointSet full_codepoint_closure_; bool segments_disjoint_; - enum ift::config::UnmappedGlyphHandling unmapped_glyph_handling_; absl::flat_hash_map codepoint_to_segments_; }; diff --git a/ift/encoder/requested_segmentation_information_test.cc b/ift/encoder/requested_segmentation_information_test.cc index 7fc1539d..8ea8035d 100644 --- a/ift/encoder/requested_segmentation_information_test.cc +++ b/ift/encoder/requested_segmentation_information_test.cc @@ -17,7 +17,6 @@ using ift::common::FontData; using ift::common::hb_face_unique_ptr; using ift::common::make_hb_face; using ift::common::SegmentSet; -using ift::config::PATCH; using ift::freq::ProbabilityBound; namespace ift::encoder { @@ -48,8 +47,7 @@ TEST_F(RequestedSegmentationInformationTest, SegmentsForCodepoints) { {{'d'}}, }; - auto info_or = - RequestedSegmentationInformation::Create(segments, {}, *cache, PATCH); + auto info_or = RequestedSegmentationInformation::Create(segments, {}, *cache); ASSERT_TRUE(info_or.ok()); auto& info = *info_or; @@ -71,8 +69,7 @@ TEST_F(RequestedSegmentationInformationTest, IndexUpdatesOnMerge) { {{'c'}}, }; - auto info_or = - RequestedSegmentationInformation::Create(segments, {}, *cache, PATCH); + auto info_or = RequestedSegmentationInformation::Create(segments, {}, *cache); ASSERT_TRUE(info_or.ok()); auto& info = *info_or; @@ -97,8 +94,7 @@ TEST_F(RequestedSegmentationInformationTest, IndexUpdatesOnReassignInit) { {{'b', 'c'}}, }; - auto info_or = - RequestedSegmentationInformation::Create(segments, {}, *cache, PATCH); + auto info_or = RequestedSegmentationInformation::Create(segments, {}, *cache); ASSERT_TRUE(info_or.ok()); auto& info = *info_or; diff --git a/ift/encoder/segmentation_context.cc b/ift/encoder/segmentation_context.cc index 1703161a..e5c97a5d 100644 --- a/ift/encoder/segmentation_context.cc +++ b/ift/encoder/segmentation_context.cc @@ -22,11 +22,8 @@ #include "ift/encoder/subset_definition.h" #include "ift/encoder/types.h" -using ift::config::CLOSURE_AND_VALIDATE_DEP_GRAPH; -using ift::config::CLOSURE_ONLY; -using ift::config::DEP_GRAPH_ONLY_WITH_SIMPLIFICATION; using ift::config::ConditionAnalysisMode; -using ift::config::UnmappedGlyphHandling; +using ift::config::DEP_GRAPH_ONLY_WITH_SIMPLIFICATION; using absl::Status; using absl::StatusOr; @@ -98,8 +95,7 @@ Status SegmentationContext::ValidateSegmentation( // In unicode range mode not all conditions get record into segmentation, so // check disjointness across the non-exclusive patches which were skipped. - TRYV(check_disjoint(glyph_groupings.UnmappedGlyphs())); - GlyphSet unreachable_glyphs = glyph_groupings.UnmappedGlyphs(); + GlyphSet unreachable_glyphs; for (const auto& cond : glyph_groupings.OrderedConditions()) { if (cond.IsExclusive()) { continue; @@ -214,84 +210,20 @@ void SegmentationContext::LogSegmentCodepointCountHistogram( } Status SegmentationContext::ReprocessChanged(InvalidationSet modified) { - segment_index_t last_merged_segment_index = modified.base_segment; - - if (!IsPureDepGraphAnalysisMode()) { - if (!InertSegments().contains(last_merged_segment_index)) { - VLOG(1) << "Re-analyzing segment " << last_merged_segment_index - << " due to merge."; - GlyphSet analysis_modified_gids = - TRY(ReprocessSegment(last_merged_segment_index)); - modified.glyphs.union_set(analysis_modified_gids); - } - } else { - modified.glyphs.union_set( - (*dependency_closure_) - ->SegmentsToAffectedGlyphs({modified.base_segment})); - TransferDependencyGraphGlyphConditions(modified.glyphs); - } + modified.glyphs.union_set( + dependency_closure_->SegmentsToAffectedGlyphs({modified.base_segment})); + TransferDependencyGraphGlyphConditions(modified.glyphs); return GroupGlyphs(modified.glyphs, modified.segments); } Status SegmentationContext::ReprocessAll() { - if (!IsPureDepGraphAnalysisMode()) { - for (segment_index_t segment_index = 0; - segment_index < SegmentationInfo().Segments().size(); - segment_index++) { - TRY(ReprocessSegment(segment_index)); - } - } else { - // Pull conditions directly out of the dep graph instead of running closure - // processing. - TransferDependencyGraphGlyphConditions( - segmentation_info_->NonInitFontGlyphs()); - } - - return GroupGlyphs(SegmentationInfo().NonInitFontGlyphs(), {}); -} - -StatusOr SegmentationContext::ReprocessSegment( - segment_index_t segment_index) { - if (segmentation_info_->Segments()[segment_index].Definition().Empty()) { - // Empty segment is a noop; - return GlyphSet{}; - } - - GlyphSet and_gids; - GlyphSet or_gids; - GlyphSet exclusive_gids; - TRYV(AnalyzeSegment({segment_index}, and_gids, or_gids, exclusive_gids)); - - GlyphSet changed_gids; - changed_gids.union_set(and_gids); - changed_gids.union_set(or_gids); - changed_gids.union_set(exclusive_gids); - - for (uint32_t gid : changed_gids) { - glyph_condition_set.InvalidateGlyphInformation(GlyphSet{gid}, - SegmentSet{segment_index}); - } - - if (and_gids.empty() && or_gids.empty()) { - inert_segments_.insert(segment_index); - } - - for (uint32_t and_gid : exclusive_gids) { - // TODO(garretrieger): if we are assigning an exclusive gid there should - // be no other and segments, check and error if this is violated. - glyph_condition_set.AddAndCondition(and_gid, segment_index); - } - - for (uint32_t and_gid : and_gids) { - glyph_condition_set.AddAndCondition(and_gid, segment_index); - } - - for (uint32_t or_gid : or_gids) { - glyph_condition_set.AddOrCondition(or_gid, segment_index); - } + // Pull conditions directly out of the dep graph instead of running closure + // processing. + TransferDependencyGraphGlyphConditions( + segmentation_info_->NonInitFontGlyphs()); - return changed_gids; + return GroupGlyphs(segmentation_info_->NonInitFontGlyphs(), {}); } /* @@ -319,19 +251,10 @@ Status SegmentationContext::ReassignInitSubset(SubsetDefinition new_def) { s_index++; } - SegmentSet segments_to_reprocess; - if (!segmentation_info_->Segments().empty()) { - segments_to_reprocess.insert_range( - 0, segmentation_info_->Segments().size() - 1); - } - if (dependency_closure_.has_value()) { - SubsetDefinition new_def_delta = new_def; - new_def_delta.Subtract(SegmentationInfo().InitFontSegment()); - // If dep graph is enabled we can use it to narrow the set of segments that - // need reprocessing. - segments_to_reprocess = - TRY((*dependency_closure_)->SegmentsThatInteractWith(new_def_delta)); - } + SubsetDefinition new_def_delta = new_def; + new_def_delta.Subtract(SegmentationInfo().InitFontSegment()); + SegmentSet segments_to_reprocess = + TRY(dependency_closure_->SegmentsThatInteractWith(new_def_delta)); TRYV(segmentation_info_->ReassignInitSubset(*glyph_closure_cache, new_def)); @@ -343,11 +266,9 @@ Status SegmentationContext::ReassignInitSubset(SubsetDefinition new_def) { original_face.get(), init_font_brotli_quality_); } - if (dependency_closure_.has_value()) { - // Dep closure doesn't need the expanded affected set, as it will traverse - // from the input segments to find the affected subgraph. - TRYV((*dependency_closure_)->InitFontChanged(segments_with_changed_defs)); - } + // Dep closure doesn't need the expanded affected set, as it will traverse + // from the input segments to find the affected subgraph. + TRYV(dependency_closure_->InitFontChanged(segments_with_changed_defs)); // All segments depend on the init subset def, so we must reprocess // everything. First reset condition set information: @@ -358,17 +279,11 @@ Status SegmentationContext::ReassignInitSubset(SubsetDefinition new_def) { inert_segments_.subtract(segments_to_reprocess); // Then reprocess segments: - if (!IsPureDepGraphAnalysisMode()) { - for (segment_index_t segment_index : segments_to_reprocess) { - TRY(ReprocessSegment(segment_index)); - } - } else { - GlyphSet gids; - for (segment_index_t s : segments_to_reprocess) { - gids.union_set((*dependency_closure_)->SegmentsToAffectedGlyphs({s})); - } - TransferDependencyGraphGlyphConditions(gids); + GlyphSet gids; + for (segment_index_t s : segments_to_reprocess) { + gids.union_set(dependency_closure_->SegmentsToAffectedGlyphs({s})); } + TransferDependencyGraphGlyphConditions(gids); // the groupings can be incrementally recomputed by looking at what conditions // have changed. @@ -388,9 +303,7 @@ Status SegmentationContext::ReassignInitSubset(SubsetDefinition new_def) { Status SegmentationContext::ResetSegments(std::vector segments) { segmentation_info_->ResetSegments(std::move(segments)); - if (dependency_closure_.has_value()) { - TRYV((*dependency_closure_)->SegmentsReset()); - } + TRYV(dependency_closure_->SegmentsReset()); uint32_t glyph_count = hb_face_get_glyph_count(original_face.get()); glyph_condition_set = GlyphConditionSet(glyph_count); glyph_groupings = GlyphGroupings( @@ -400,85 +313,10 @@ Status SegmentationContext::ResetSegments(std::vector segments) { return ReprocessAll(); } -static void PrintDiff(absl::string_view set_name, const GlyphSet& closure, - const GlyphSet& dep) { - std::string op = " == "; - if (closure != dep) { - op = " != "; - } - - LOG(ERROR) << "Set " << set_name << ": closure glyphs " << closure.ToString() - << op << " dependency glyphs " << dep.ToString(); -} - -Status SegmentationContext::AnalyzeSegment(const SegmentSet& segment_ids, - GlyphSet& and_gids, - GlyphSet& or_gids, - GlyphSet& exclusive_gids) { - ConditionAnalysisMode effective_mode = condition_analysis_mode_; - GlyphSet dep_and_gids = and_gids; - GlyphSet dep_or_gids = or_gids; - GlyphSet dep_exclusive_gids = exclusive_gids; - if (dependency_closure_.has_value()) { - auto accuracy = TRY((*dependency_closure_) - ->AnalyzeSegment(segment_ids, dep_and_gids, - dep_or_gids, dep_exclusive_gids, - effective_mode == DEP_GRAPH_ONLY_WITH_SIMPLIFICATION)); - if (!IsPureDepGraphAnalysisMode() && accuracy == DependencyClosure::INACCURATE) { - // Don't fallback to closure in pure dep graph mode. - effective_mode = CLOSURE_ONLY; - } - } - - if (effective_mode == CLOSURE_ONLY || - effective_mode == CLOSURE_AND_VALIDATE_DEP_GRAPH) { - TRYV(glyph_closure_cache->AnalyzeSegment( - *segmentation_info_, segment_ids, and_gids, or_gids, exclusive_gids)); - } else { - or_gids.union_set(dep_or_gids); - and_gids.union_set(dep_and_gids); - exclusive_gids.union_set(dep_exclusive_gids); - } - - if (effective_mode == CLOSURE_AND_VALIDATE_DEP_GRAPH) { - if (and_gids != dep_and_gids || or_gids != dep_or_gids || - exclusive_gids != dep_exclusive_gids) { - LOG(ERROR) << "Mismatch between closure and dependency analysis " - "conditions for segments " - << segment_ids.ToString(); - for (segment_index_t s : segment_ids) { - LOG(ERROR) << "segment[" << s << "].codepoints = " - << segmentation_info_->Segments() - .at(s) - .Definition() - .codepoints.ToString(); - LOG(ERROR) << "segment[" << s << "].features.size() = " - << segmentation_info_->Segments() - .at(s) - .Definition() - .feature_tags.size(); - } - PrintDiff("AND", and_gids, dep_and_gids); - PrintDiff("OR ", or_gids, dep_or_gids); - PrintDiff("EXC", exclusive_gids, dep_exclusive_gids); - LOG(ERROR) << "init codepoints = " - << segmentation_info_->InitFontSegment().codepoints.ToString(); - LOG(ERROR) << "init glyphs = " - << segmentation_info_->InitFontGlyphs().ToString(); - return absl::InternalError( - "Dependency graph conditions does not match the closure analysis " - "conditions"); - } - } - - return absl::OkStatus(); -} - StatusOr SegmentationContext::InitializeSegmentationContext( hb_face_t* face, SubsetDefinition initial_segment, std::vector segments, - UnmappedGlyphHandling unmapped_glyph_handling, ConditionAnalysisMode condition_analysis_mode, uint32_t brotli_quality, uint32_t init_font_brotli_quality, std::shared_ptr resolver, @@ -493,9 +331,8 @@ SegmentationContext::InitializeSegmentationContext( // No merging is done during init. SegmentationContext context = TRY(SegmentationContext::Create( - face, initial_segment, segments, unmapped_glyph_handling, - condition_analysis_mode, brotli_quality, init_font_brotli_quality, - std::move(resolver), target_mode)); + face, initial_segment, segments, condition_analysis_mode, brotli_quality, + init_font_brotli_quality, std::move(resolver), target_mode)); // ### Generate the initial conditions and groupings by processing all // segments and glyphs. ### @@ -507,7 +344,7 @@ SegmentationContext::InitializeSegmentationContext( void SegmentationContext::TransferDependencyGraphGlyphConditions( const GlyphSet& gids) { - const auto& conditions = (*dependency_closure_)->AllGlyphConditions(); + const auto& conditions = dependency_closure_->AllGlyphConditions(); for (glyph_id_t g : gids) { ActivationCondition condition = conditions.at(g); if (condition.IsUnitary()) { @@ -526,7 +363,7 @@ void SegmentationContext::TransferDependencyGraphGlyphConditions( } glyph_condition_set.SetCondition(g, condition); } - inert_segments_ = (*dependency_closure_)->InertSegments(); + inert_segments_ = dependency_closure_->InertSegments(); } } // namespace ift::encoder \ No newline at end of file diff --git a/ift/encoder/segmentation_context.h b/ift/encoder/segmentation_context.h index 564a1350..e1d705d8 100644 --- a/ift/encoder/segmentation_context.h +++ b/ift/encoder/segmentation_context.h @@ -4,6 +4,7 @@ #include #include #include +#include #include "absl/log/log.h" #include "absl/status/status.h" @@ -51,7 +52,6 @@ class SegmentationContext { static absl::StatusOr Create( hb_face_t* face, const SubsetDefinition& initial_segment, const std::vector& segments, - ift::config::UnmappedGlyphHandling unmapped_glyph_handling, ift::config::ConditionAnalysisMode condition_analysis_mode, uint32_t brotli_quality, uint32_t init_font_brotli_quality, std::shared_ptr resolver, @@ -63,13 +63,12 @@ class SegmentationContext { TRY(GlyphClosureCache::Create(face, *resolver)); std::unique_ptr segmentation_info = TRY(RequestedSegmentationInformation::Create(segments, initial_segment, - *closure_cache, - unmapped_glyph_handling)); + *closure_cache)); SegmentationContext context( - face, unmapped_glyph_handling, condition_analysis_mode, brotli_quality, - init_font_brotli_quality, std::move(closure_cache), - std::move(segmentation_info), std::move(resolver), target_mode); + face, condition_analysis_mode, brotli_quality, init_font_brotli_quality, + std::move(closure_cache), std::move(segmentation_info), + std::move(resolver), target_mode); TRYV(context.InitDependencyClosure()); return context; @@ -82,14 +81,13 @@ class SegmentationContext { std::unique_ptr segmentation_info = TRY(RequestedSegmentationInformation::Create( SegmentationInfo().Segments(), SegmentationInfo().InitFontSegment(), - *closure_cache, SegmentationInfo().GetUnmappedGlyphHandling())); + *closure_cache)); SegmentationContext context( - original_face.get(), SegmentationInfo().GetUnmappedGlyphHandling(), - condition_analysis_mode_, brotli_quality_, init_font_brotli_quality_, - std::move(closure_cache), std::move(segmentation_info), resolver_, - target_mode_, estimated_compression_ratio_, - estimated_woff2_base_overhead_); + original_face.get(), condition_analysis_mode_, brotli_quality_, + init_font_brotli_quality_, std::move(closure_cache), + std::move(segmentation_info), resolver_, target_mode_, + estimated_compression_ratio_, estimated_woff2_base_overhead_); context.initial_glyph_probabilities_ = initial_glyph_probabilities_; TRYV(context.InitDependencyClosure()); @@ -105,7 +103,6 @@ class SegmentationContext { static absl::StatusOr InitializeSegmentationContext( hb_face_t* face, SubsetDefinition initial_segment, std::vector segments, - ift::config::UnmappedGlyphHandling unmapped_glyph_handling, ift::config::ConditionAnalysisMode condition_analysis_mode, uint32_t brotli_quality, uint32_t init_font_brotli_quality, std::shared_ptr resolver, @@ -114,7 +111,6 @@ class SegmentationContext { private: SegmentationContext( hb_face_t* face, - ift::config::UnmappedGlyphHandling unmapped_glyph_handling, ift::config::ConditionAnalysisMode condition_analysis_mode, uint32_t brotli_quality, uint32_t init_font_brotli_quality, std::unique_ptr closure_cache, @@ -128,7 +124,7 @@ class SegmentationContext { glyph_closure_cache(std::move(closure_cache)), original_face(ift::common::make_hb_face(hb_face_reference(face))), segmentation_info_(std::move(segmentation_info)), - dependency_closure_(std::nullopt), + dependency_closure_(nullptr), glyph_condition_set(hb_face_get_glyph_count(face)), glyph_groupings(hb_face_get_glyph_count(face), condition_analysis_mode == @@ -144,11 +140,8 @@ class SegmentationContext { } absl::Status InitDependencyClosure() { - if (UsingDepGraph()) { - dependency_closure_ = TRY(DependencyClosure::Create( - segmentation_info_.get(), original_face.get(), *resolver_, - IsPureDepGraphAnalysisMode())); - } + dependency_closure_ = TRY(DependencyClosure::Create( + segmentation_info_.get(), original_face.get(), *resolver_)); return absl::OkStatus(); } @@ -178,39 +171,26 @@ class SegmentationContext { } void LogClosureStatistics() const { - uint64_t dep_graph_closures = 0; - uint64_t dep_graph_inaccurate = 0; - if (dependency_closure_.has_value()) { - dep_graph_closures = (*dependency_closure_)->AccurateResults() * 2; - dep_graph_inaccurate = (*dependency_closure_)->InaccurateResults(); - } - - uint64_t potential_closures = glyph_closure_cache->CacheHits() + - glyph_closure_cache->CacheMisses() + - dep_graph_closures; + uint64_t total_closures = + glyph_closure_cache->CacheHits() + glyph_closure_cache->CacheMisses(); double hb_subset_rate = - 100.0 * ((double)glyph_closure_cache->CacheMisses() / - (double)potential_closures); - double cache_hit_rate = 100.0 * ((double)glyph_closure_cache->CacheHits() / - (double)potential_closures); - double dep_graph_hit_rate = - 100.0 * ((double)dep_graph_closures / (double)potential_closures); - uint64_t other_closures = glyph_closure_cache->CacheHits() + - glyph_closure_cache->CacheMisses() - - (2 * dep_graph_inaccurate); - - VLOG(0) << ">> Of " << potential_closures - << " potential closure operations:" << std::endl + total_closures == 0 + ? 0.0 + : 100.0 * ((double)glyph_closure_cache->CacheMisses() / + (double)total_closures); + double cache_hit_rate = + total_closures == 0 + ? 0.0 + : 100.0 * ((double)glyph_closure_cache->CacheHits() / + (double)total_closures); + + VLOG(0) << ">> Of " << total_closures << " closure operations:" << std::endl << " " << glyph_closure_cache->CacheMisses() << " (" - << hb_subset_rate << "%) were handled by hb-subset-plan" + << hb_subset_rate << "%) were handled by hb-subset-plan" << std::endl - << " " << dep_graph_closures << " (" << dep_graph_hit_rate - << "%) were handled by dep graph" << std::endl << " " << glyph_closure_cache->CacheHits() << " (" - << cache_hit_rate << "%) were provided by the cache" << std::endl - << " " << other_closures - << " were from something other than AnalyzeSegment()" << std::endl; + << cache_hit_rate << "%) were provided by the cache" << std::endl; } const ift::common::SegmentSet& InertSegments() const { @@ -225,28 +205,6 @@ class SegmentationContext { return condition_analysis_mode_; } - bool IsPureDepGraphAnalysisMode() const { - switch (condition_analysis_mode_) { - case config::DEP_GRAPH_ONLY: - case config::DEP_GRAPH_ONLY_WITH_SIMPLIFICATION: - return true; - default: - return false; - } - } - - bool UsingDepGraph() const { - switch (condition_analysis_mode_) { - case config::CLOSURE_AND_DEP_GRAPH: - case config::CLOSURE_AND_VALIDATE_DEP_GRAPH: - case config::DEP_GRAPH_ONLY: - case config::DEP_GRAPH_ONLY_WITH_SIMPLIFICATION: - return true; - default: - return false; - } - } - // Assign a new merged segment to base and clear all of the segments that // were merged into it. uint32_t AssignMergedSegment(segment_index_t base, @@ -278,10 +236,7 @@ class SegmentationContext { // group glyphs is called. glyph_condition_set.InvalidateGlyphInformation(glyphs, segments); - if (dependency_closure_.has_value()) { - return (*dependency_closure_)->SegmentsMerged(base_segment, segments); - } - return absl::OkStatus(); + return dependency_closure_->SegmentsMerged(base_segment, segments); } /* @@ -305,13 +260,6 @@ class SegmentationContext { // Regenerates glyphs conditions and groupings for all segments. absl::Status ReprocessAll(); - // Performs a closure analysis on codepoints and returns the associated - // and, or, and exclusive glyph sets. - absl::Status AnalyzeSegment(const ift::common::SegmentSet& segment_ids, - ift::common::GlyphSet& and_gids, - ift::common::GlyphSet& or_gids, - ift::common::GlyphSet& exclusive_gids); - private: // Update the glyph groups for 'glyphs'. // @@ -319,23 +267,12 @@ class SegmentationContext { // calling this. absl::Status GroupGlyphs(const ift::common::GlyphSet& glyphs, const ift::common::SegmentSet& modified_segments) { - std::optional maybe_dep_closure = std::nullopt; - if (dependency_closure_.has_value()) { - maybe_dep_closure = (*dependency_closure_).get(); - } return glyph_groupings.GroupGlyphs(*segmentation_info_, glyph_condition_set, - *glyph_closure_cache, maybe_dep_closure, - glyphs, modified_segments, - !IsPureDepGraphAnalysisMode()); + glyphs, modified_segments); } void TransferDependencyGraphGlyphConditions(const common::GlyphSet& gids); - // Generates updated glyph conditions and glyph groupings for segment_index - // which has the provided set of codepoints. - absl::StatusOr ReprocessSegment( - segment_index_t segment_index); - /* * Ensures that the produce segmentation is: * - Disjoint (no duplicated glyphs) and doesn't overlap what's in the initial @@ -416,7 +353,7 @@ class SegmentationContext { private: std::unique_ptr segmentation_info_; - std::optional> dependency_closure_; + std::unique_ptr dependency_closure_; public: // == Phase 1 - derived from segments and init information diff --git a/ift/encoder/unicode_range_compiler_test.cc b/ift/encoder/unicode_range_compiler_test.cc index ea9677a7..36367fb9 100644 --- a/ift/encoder/unicode_range_compiler_test.cc +++ b/ift/encoder/unicode_range_compiler_test.cc @@ -21,9 +21,8 @@ using ift::common::FontData; using ift::common::FontHelper; using ift::common::hb_face_unique_ptr; using ift::common::Woff2; -using ift::config::CLOSURE_AND_VALIDATE_DEP_GRAPH; using ift::config::ConfigCompiler; -using ift::config::FIND_CONDITIONS; +using ift::config::DEP_GRAPH_ONLY; using ift::config::SegmentationPlan; namespace ift::encoder { @@ -64,8 +63,7 @@ TEST_F(UnicodeRangeCompilerTest, ExtractFontFamily) { } TEST_F(UnicodeRangeCompilerTest, CompileFromSegmentationPlan) { - ClosureGlyphSegmenter segmenter(1, 1, FIND_CONDITIONS, - CLOSURE_AND_VALIDATE_DEP_GRAPH, resolver, + ClosureGlyphSegmenter segmenter(1, 1, DEP_GRAPH_ONLY, resolver, ift::config::UNICODE_RANGE); auto segmentation = segmenter.CodepointToGlyphSegments( roboto.get(), {'a'}, {{'f'}, {'i'}, {'b', 'c', 'd'}}); diff --git a/util/fallback_analysis/count-fallback-glyphs-for-font.sh b/util/fallback_analysis/count-fallback-glyphs-for-font.sh deleted file mode 100755 index 2cb78260..00000000 --- a/util/fallback_analysis/count-fallback-glyphs-for-font.sh +++ /dev/null @@ -1,11 +0,0 @@ -#!/bin/bash - -FONT_NAME=$(basename $1) -./bazel-bin/util/gen_ift_segmentation_plan \ - --input_font="$1" \ - --config=util/fallback_analysis/fallback_count_config.txtpb \ - --noinclude_initial_codepoints_in_config \ - --nooutput_segmentation_analysis \ - --nooutput_segmentation_plan \ - --output_fallback_glyph_count 2> /dev/null | grep num_fallback_glyphs | \ - awk "BEGIN {FS=\",\"; OFS=\";\"} {print \"$FONT_NAME\", \$2, \$3, \$4, \$5}" diff --git a/util/fallback_analysis/count-fallback-glyphs.sh b/util/fallback_analysis/count-fallback-glyphs.sh deleted file mode 100755 index 0cae1b3f..00000000 --- a/util/fallback_analysis/count-fallback-glyphs.sh +++ /dev/null @@ -1,19 +0,0 @@ -#!/bin/bash -# -# Usage: ./util/fallback_analysis/count-fallback-glyphs.sh -# -# will be scanned recursively to locate all font files of the form *.ttf or *.otf. -# For each found font a closure analysis will be performed and the number of fallback glyphs -# and the compressed size in bytes of the fallback glyph patch will be output. -# -# sets the number of parallel executions of the closure analysis. -# -# Fallback glyphs are those that have conditions which could not be detected with the current -# closure analysis and as a result will always be loaded. -TARGET_DIR=$1 -CONCURRENCY=$2 - -bazel build -c opt util:gen_ift_segmentation_plan -echo "file; num_fallback_glyphs; total_glyphs; fallback_glyphs_compressed_bytes; all_glyphs_compressed_bytes" -find $TARGET_DIR \( -iname "*.ttf" -or -iname "*.otf" \) -print0 | xargs -0 -n 1 -P $CONCURRENCY \ - ./util/fallback_analysis/count-fallback-glyphs-for-font.sh diff --git a/util/fallback_analysis/fallback_count_config.txtpb b/util/fallback_analysis/fallback_count_config.txtpb deleted file mode 100644 index 07bb073a..00000000 --- a/util/fallback_analysis/fallback_count_config.txtpb +++ /dev/null @@ -1,19 +0,0 @@ -# This config is used to run the closure segmenter to test how many fallback glyphs result from -# segmenting a font with one codepoint per segment. - -# Don't move fallback glyphs to init font -move_fallback_glyphs_into_initial_font: false - -# Add segments for all non-default features -generate_feature_segments: true - -# Brotli shouldn't be used, but just in case set to lowest quality. -brotli_quality: 1 -brotli_quality_for_initial_font_merging: 1 - -# Ensure there is no merging. -preprocess_merging_group_size_for_ungrouped: 1 -ungrouped_config { - min_patch_size: 0 - max_patch_size: 4294967295 -} diff --git a/util/gen_ift_segmentation_plan.cc b/util/gen_ift_segmentation_plan.cc index 5cc9c022..5b628b17 100644 --- a/util/gen_ift_segmentation_plan.cc +++ b/util/gen_ift_segmentation_plan.cc @@ -39,8 +39,7 @@ using ift::common::BazelDataFileResolver; using ift::common::DataFileResolver; -using ift::config::CLOSURE_ONLY; -using ift::config::PATCH; +using ift::config::DEP_GRAPH_ONLY; using ift::config::SegmentationPlan; using ift::config::SegmenterConfig; using ift::freq::ProbabilityCalculator; @@ -76,10 +75,6 @@ ABSL_FLAG(bool, output_segmentation_analysis, false, ABSL_FLAG(bool, segmentation_analysis_split_cjk, false, "If set, then for analysis split \"Script_CJK\" into the individual scripts."); -ABSL_FLAG( - bool, output_fallback_glyph_count, false, - "If set the number of fallback glyphs in the segmentation will be output."); - ABSL_FLAG( int, verbosity, 0, "Log verbosity level from. 0 is least verbose, higher values are more."); @@ -188,7 +183,7 @@ static Status Analysis(hb_face_t* font, } } - ClosureGlyphSegmenter segmenter(11, 11, PATCH, CLOSURE_ONLY, resolver, + ClosureGlyphSegmenter segmenter(11, 11, DEP_GRAPH_ONLY, resolver, target_mode); auto costs = TRY(segmenter.TotalCosts(font, segmentation, strategy_probability_calculators)); @@ -233,28 +228,6 @@ static Status Analysis(hb_face_t* font, return absl::OkStatus(); } -static Status OutputFallbackGlyphCount(hb_face_t* original_face, - const ClosureGlyphSegmenter& segmenter, - const GlyphSegmentation& segmentation) { - uint32_t num_fallback_glyphs = segmentation.UnmappedGlyphs().size(); - uint32_t fallback_glyphs_size = 0; - uint32_t all_glyphs_size = 0; - TRYV(segmenter.FallbackCost(original_face, segmentation, fallback_glyphs_size, - all_glyphs_size)); - - GlyphSet all_glyphs; - for (const auto& [_, gids] : segmentation.GidSegments()) { - all_glyphs.union_set(gids); - } - - uint32_t num_glyphs = all_glyphs.size() + num_fallback_glyphs; - std::cout << "num_fallback_glyphs, " << num_fallback_glyphs << ", " - << num_glyphs << ", " << fallback_glyphs_size << ", " - << all_glyphs_size << std::endl; - - return absl::OkStatus(); -} - static Status Main(const std::vector args) { auto resolver = TRY(BazelDataFileResolver::Create(args[0])); @@ -296,16 +269,6 @@ static Status Main(const std::vector args) { std::cout << segmentation.ToString() << std::endl; } - if (absl::GetFlag(FLAGS_output_fallback_glyph_count)) { - ClosureGlyphSegmenter segmenter( - config.brotli_quality(), - config.brotli_quality_for_initial_font_merging(), - config.unmapped_glyph_handling(), config.condition_analysis_mode(), - resolver, config.target_mode(), - config.disjoint_conditions_probability_threshold()); - TRYV(OutputFallbackGlyphCount(font.get(), segmenter, segmentation)); - } - if (!absl::GetFlag(FLAGS_output_segmentation_analysis)) { return absl::OkStatus(); }