Repository navigation
Conversation
The specification defines the X in string:X as the maximum length in
bytes, and strings are encoded as UTF-8. StringField.encode() however
compared the number of characters against max_length, so strings with
multi-byte characters could exceed the byte limit unnoticed.
For example, the Japanese material name
"シルクPLA トリプルカラー ゴールド＆シルバー＆カッパー"
("Silk PLA Tri-Color Gold & Silver & Copper") has 29 characters but
77 bytes, and was accepted for material_name (string:63).
Check the length of the UTF-8 encoded string instead and report the
actual and allowed size on failure.
Add a test that the above name is rejected, and an encode/decode test
with a multi-byte material name exactly at the 63-byte limit.
Signed-off-by: Christoph Müllner <christophm30@gmail.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
utils: Enforce string max_length in UTF-8 bytes
Problem
The specification defines the
Xinstring:Xas the maximum lengthin bytes (see "NFC data format", encoding rules), and strings are
encoded as UTF-8. The reference implementation however checked the number
of characters in
StringField.encode():For ASCII-only strings both are the same, but strings with multi-byte
characters could exceed the byte limit without any error. For example,
the Japanese material name
("Silk PLA Tri-Color Gold & Silver & Copper") has 29 characters but
77 UTF-8 bytes. It was accepted for
material_name(string:63) andwritten to the tag, and
rec_info --validatedid not report it either.Change
StringField.encode()now checks the length of the UTF-8 encoded stringand reports the actual and allowed size on failure:
Tests
tests/specific/string_too_long_utf8.yaml: the name above must berejected. This test fails without the fix.
tests/encode_decode/06_*: the Japanese material nameシルクPLA+ デュアルカラー ゴールド＆シルバー(25 characters, exactly63 bytes) must be encoded and decoded unchanged.
python3 tests/run_tests.pypasses,utils/gen_schema.pyproduces nochanges, and the sample data in
docs_src/sample_datastill encodes.