Repository navigation
Read the symbol table from lang-symbols.tsv instead of parsing lang.def - #270
Merged
Merged
Conversation
lang.def states nothing about its own shape: the first 60 lines pair a language name with its symbol, the next 10 are self-pairs, and the rest pair a symbol with its text the other way round. Three repositories each hand-parsed that, and the engine now generates lang.def from lang-symbols.tsv, whose kind column says which block a row belongs to. So read the table and drop this repository's parser. A row kind is what selects the rows now, not the symbol's prefix. That matters because 11 string symbols are LISTDEF_ rather than LANG_, so a prefix test silently drops them. Only the text column goes through lang_filter. Over a whole row it would also strip a text's leading space, where lang.def gave every text a line of its own. Proved equivalent by generating res/values-* from the engine's 30 catalogs with both versions: the output is byte-identical apart from the script path the generated header carries. The harness fixture becomes a TSV, keeps every case, and gains two: a row whose kind is not string must not resolve, and a row with a fourth column must stop the run. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com>
The review killed both guards I had added, and it was right. The engine's tools/gen-lang-def.py exits on a row without three columns, on a text that differs from its strip(), on a duplicate symbol, and on a meta row whose text does not repeat its symbol. We consume a committed blob that has already passed those checks. The column-count check was also stricter than the producer, which allows a blank line, so it would have refused a legal future table. The paste of two awks, which kept lang_filter off the symbol column, defended a padded text the producer forbids and took on an alignment risk instead: a later lang_filter rule that dropped a line would misalign every symbol from its text. One awk emitting both columns cannot. The case for "the kind column selects the rows, not the prefix" needed a meta row carrying a LANG_ symbol, which the producer forbids too. On any legal table a prefix test and a kind test agree, because language and meta symbols are all LANGUAGE_*, so that case was testing against a world that cannot exist. What the harness checks instead is the pinned table itself: it resolves that table's own first string symbol. Nothing else here reads it, so a pin losing lang-symbols.tsv would otherwise keep CI green. Verified by mutation: a crafted engine directory without the table reports it rather than skipping. Also moves the three not-found messages to stderr, which is where an error belongs and what lets the missing-table case assert the message rather than only the exit code. Still byte-identical over the engine's 30 catalogs. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
lang.defstates nothing about its own shape, and three repositories each hand-parsed it. The engine now generates it fromlang-symbols.tsv, whose kind column names each row's block, so read the table and drop the parser.Generating
res/values-*from the engine's 30 catalogs both ways gives byte-identical output, apart from the script path in the generated header. Nores/values-<lang>is checked in, so nothing ships either way today and the harness is the only observable. It now resolves one of the pinned table's own symbols, so a pin that loses the file turns the harness red.