Skip to content

sbe: refuse sizes and ids past SBE's 16-bit header fields (#132) - #140

Merged
trendvidia merged 1 commit into
mainfrom
sbe-header-bounds-132
Nov 5, 2026
Merged

trendvidia merged 1 commit into
mainfrom
sbe-header-bounds-132

Conversation

@bfrancojr

Copy link
Copy Markdown
Collaborator

Fixes #132.

A schema or message that SBE's 16-bit header fields can't describe is now refused instead of being written with the wrong numbers. Before, (sbe.length) = 65537 bound as a 1-byte field. A block past 65535 bytes bound with wrapped offsets. A template id, schema id or version of 65537 was written as 1. A group of 65536 entries was written with a count of 0 and read back empty, with no error. Ported from protowire-go #254 (issue #250).

  • BuildTemplate, BuildGroupTemplate and BuildCompositeFields add up offsets in int and check each field's end before narrowing, as Go's endOfField does. Every offset in a template is then in range.
  • Codec::New refuses (sbe.template_id), (sbe.schema_id), (sbe.version) and (sbe.length) past 65535, with Go's text.
  • Marshal refuses a repeated field with more than 65535 entries: sbe: field M.e has 65536 entries, past 65535, the largest numInGroup SBE can state.

Also fixed: every bind error of a template now has Go's text and prefix, such as sbe: field m.v1.M.s: string field requires (sbe.length) annotation (it was string/bytes field requires (sbe.length)). Go's bound errors carry that prefix, so matching them meant matching the rest. No test pinned the old texts.

Stack and merge order: #123 → #124 → #137 (#134) → this PR → #133's PR. Base: sbe-template-lookup-134. The two are independent in code, but both edit Codec::New and the CHANGELOG, so stacking avoids a conflict.

Evidence, at db81e1c, on macOS:

Not in this PR: Go also refuses a oneof field at bind time (sbe: oneof field M.f not supported), and this port doesn't. That changes behaviour, so it's filed separately rather than folded in here.

@bfrancojr

Copy link
Copy Markdown
Collaborator Author

Rebased onto main e1c1b72 (#136 merged); head is now 95517ac. The PR's own diff is unchanged, checked by comparing its added and removed lines before and after the rebase. Rebased as part of the SBE stack; evidence is on #123.

@bfrancojr

Copy link
Copy Markdown
Collaborator Author

Rebased with the stack onto main df6b102; head is now 43eabbe. Evidence is on #124.

Base automatically changed from sbe-template-lookup-134 to main November 5, 2026 00:47
Every SBE header field is a uint16, and the codec kept the low 16 bits
of a larger value: (sbe.length) = 65537 bound as a 1-byte field, a block
past 65535 bytes bound with wrapped offsets, a template_id, schema_id or
version of 65537 was written as 1, and a group of 65536 entries was
written with numInGroup 0.

As protowire-go does (#250): offsets are accumulated in int and checked
before they become a uint16, Codec::New refuses each schema shape with
Go's text, and Marshal refuses a group of more than 65535 entries. The
template builders now word every bind error as Go does, prefixed with
the field, group or composite it is about.

Tests ported from Go's encoding/sbe/header_bounds_test.go, each refusal
paired with the same shape at 65535, which binds and round-trips.
@trendvidia
trendvidia force-pushed the sbe-header-bounds-132 branch from 43eabbe to 3486d29 Compare November 5, 2026 00:49
@bfrancojr

Copy link
Copy Markdown
Collaborator Author

Rebased onto main 197e05b, after #124 and #137 merged as squashes: 43eabbe → 3486d29. The old #124 and #137 commits were dropped (git rebase --onto origin/main bcedc19), so this PR's own commit is the only one replayed. Its diff is unchanged, apart from where its CHANGELOG line lands. At the stack top (39e1261): build OK, ctest 100% tests passed out of 705. The remote head was read back with git ls-remote: 3486d29.

@trendvidia
trendvidia merged commit 9525e9b into main Nov 5, 2026
11 checks passed
@trendvidia
trendvidia deleted the sbe-header-bounds-132 branch November 5, 2026 02:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

sbe: a size or id past 65535 is cut to 16 bits instead of refused (protowire-go#254)

2 participants