Skip to content

sbe: a size or id past 65535 is cut to 16 bits instead of refused (protowire-go#254) #132

Closed

Description

@bfrancojr

Every SBE header field is a uint16, and this port narrows a larger value to its low 16 bits with no error. protowire-go refuses each case since trendvidia/protowire-go#254 (issue #250).

What C++ does now (by reading main a632c6e, not measured):

  • BuildTemplate (src/sbe/template.cc) stores static_cast<uint16_t>(*tid), so (sbe.template_id) = 65537 is written as 1.
  • Codec::New (src/sbe/codec.cc) does the same for (sbe.schema_id) and (sbe.version).
  • FieldEncodingSize narrows (sbe.length) with static_cast<uint16_t>, so a 65537-byte field binds as 1 byte.
  • Offsets and block lengths are summed in uint16_t in BuildTemplate, BuildGroupTemplate and BuildCompositeFields, so a block past 65535 bytes wraps. In Go this bound and then panicked in Marshal. Here the wrapped length is written.
  • Codec::Marshal (src/sbe/marshal.cc) writes numInGroup as static_cast<uint16_t>(n), so a group of 65536 entries is written as 0 entries. It reads back as an empty group with no error.

Go's errors, to mirror:

  • sbe: message %s: (sbe.template_id) = %d is past 65535, the largest templateId SBE can state
  • sbe: file %s: (sbe.schema_id) = %d is past 65535, the largest schemaId SBE can state (and the same for (sbe.version) / version)
  • (sbe.length) = %d is past 65535, the largest blockLength SBE can state
  • ends at byte %d of its block, past 65535, the largest blockLength SBE can state, wrapped as sbe: field %s.%s: …, sbe: composite %s.%s: …, sbe: group field %s.%s: …, sbe: composite in group %s.%s: …
  • Marshal: sbe: field %s has %d entries, past 65535, the largest numInGroup SBE can state

Go accumulates offsets in int and checks before narrowing, so every offset in a template is in range by construction.

Done when each of these is refused with Go's text, and the same shape at 65535 still binds and round-trips (port encoding/sbe/header_bounds_test.go):

  • template_id, schema_id, version past 65535
  • (sbe.length) past 65535
  • a message block, group entry block and composite past 65535 bytes
  • Marshal of a group with more than 65535 entries

Note: #124 (stale-descriptor diagnosis) also edits Codec::New and RegisterMessage, so this work should stack on it or follow its merge.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions