From 3486d2968fe0ba6db300625c8bd8652d8f82fad4 Mon Sep 17 00:00:00 2001 From: "Benedicto Franco Jr." Date: Wed, 4 Nov 2026 04:56:29 -0800 Subject: [PATCH] sbe: refuse sizes and ids past SBE's 16-bit header fields (#132) Every SBE header field is a uint16, and the codec kept the low 16 bits of a larger value: (sbe.length) = 65537 bound as a 1-byte field, a block past 65535 bytes bound with wrapped offsets, a template_id, schema_id or version of 65537 was written as 1, and a group of 65536 entries was written with numInGroup 0. As protowire-go does (#250): offsets are accumulated in int and checked before they become a uint16, Codec::New refuses each schema shape with Go's text, and Marshal refuses a group of more than 65535 entries. The template builders now word every bind error as Go does, prefixed with the field, group or composite it is about. Tests ported from Go's encoding/sbe/header_bounds_test.go, each refusal paired with the same shape at 65535, which binds and round-trips. --- CHANGELOG.md | 1 + src/sbe/codec.cc | 11 ++ src/sbe/header_bounds.h | 14 ++ src/sbe/marshal.cc | 10 ++ src/sbe/template.cc | 124 ++++++++++++----- test/CMakeLists.txt | 2 +- test/sbe_header_bounds_test.cc | 242 +++++++++++++++++++++++++++++++++ 7 files changed, 371 insertions(+), 33 deletions(-) create mode 100644 src/sbe/header_bounds.h create mode 100644 test/sbe_header_bounds_test.cc diff --git a/CHANGELOG.md b/CHANGELOG.md index 263821a..0c6cc7b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -26,6 +26,7 @@ number says nothing about any other port. See the Releases section of - **A decoder rejects `null` on a field with `(pxf.required)` or `@required`** (#131, decided by the owner; trendvidia/protowire#569, protowire-go#268). **This narrows what decodes.** A required field holds a value, so `null` is now rejected as absence is, whatever the field's type and presence. Before, `null` counted as present, so `r = null` on `optional int32 r [(pxf.required) = true]` decoded and left a required field with no value. The error is `null is not allowed on required field "r": a required field holds a value`, in a body, a nested body, a `@dataset` cell and a dotted cell, from `Unmarshal` as well as `UnmarshalFull`. A plain scalar with `(pxf.required)`, which already rejected `null`, now names the field as required instead of suggesting `optional`. A zero value still satisfies the field. A `_null` mask that names an unset required field no longer makes `AppendDatasetRow` write `null` for it: the writer refuses it, as it does without the mask. - **`pxf::Marshal` writes an unset field that has `(pxf.default)` or `(pxf.required)` so that it reads back** (#129, protowire-go#259; draft -01 § Common Schema Layer, "`decode_PXF(encode_PXF(v)) == v`"), as protowire-go does. This changes what Marshal writes for annotated fields, and narrows what encodes. Marshal left out every unset field, so a reader that applies defaults filled it in: `int32 n = 1 [(pxf.default) = "5"]` holding 0 read back as 5, and a required scalar holding 0 made the full decode refuse the document. A proto3 scalar without presence is now written as its zero value, a required list or map as `[]` or `{}`, and a message-typed field with a default, such as a wrapper or a Timestamp, as `null`. An unset `optional` or oneof scalar with a default is still left out and reads back as its default, the owner's choice in protowire-go#267. An unset `optional` field that is `(pxf.required)` is now an error, `pxf: is unset, and the field is (pxf.required)`; with `emit_defaults` it is written as zero, as before. An Any whose body has such a field stays as `type_url` and `value`. Inside a `@dataset` value cell, the same error now names the field by its full name. Unannotated fields are unchanged. +- **`sbe::Codec::New` refuses a schema the SBE header cannot describe, and `Marshal` a group it cannot count** (#132, protowire-go#250). This narrows what binds and what encodes. Every SBE header field is a 16-bit number, and the codec kept the low 16 bits of a larger one. `bytes b = 1 [(sbe.length) = 65537]` bound as a 1-byte field. A block longer than 65535 bytes, in a message, a group entry or a composite, bound with a wrapped length and offsets. A `(sbe.template_id)`, `(sbe.schema_id)` or `(sbe.version)` of 65537 was written as 1. `Codec::New` now refuses each of these with protowire-go's text, naming the field, message or file, such as `sbe: field m.v1.M.b: (sbe.length) = 65537 is past 65535, the largest blockLength SBE can state`; 65535 still binds. `Marshal` refuses a repeated field with more than 65535 entries, with `sbe: field m.v1.M.e has 65536 entries, past 65535, the largest numInGroup SBE can state`: 65536 entries used to be written with a count of 0, and the message read back empty with no error. Every other bind error of a template now has protowire-go's text as well, prefixed with the field, group or composite it is about, such as `sbe: field m.v1.M.s: string field requires (sbe.length) annotation` where it used to be `string/bytes field requires (sbe.length)`. - **`sbe` tells two schemas' template ids apart** (#134, protowire-go#253 and #257; draft -01 § SBE Annotations, "sbe.template_id MUST be unique among messages within a schema", and § Header and Block Length). This narrows what binds and what decodes. `Codec::New` refuses two different messages with one `(sbe.template_id)` in one schema, with `sbe: messages A and B share (sbe.template_id) = 7 in schema 1; it must be unique within a schema`; the second used to replace the first, and `NewView` then read one message's bytes as the other's. A file passed twice still binds, and so do two schemas that each use a template id. `NewView` now finds a template by the header's schema id and template id together, and refuses an unknown pair with `sbe: unknown template ID 9 in schema 2`, even if another schema has that template id; it used to say `unknown template id 9`. `Unmarshal` and `UnmarshalDescriptor` refuse a header whose schema id is not the target message's, with `sbe: schema ID mismatch: got 1, want 2 for template 7`; the version is not checked. A caller that builds headers by hand must now set the schema id. - **`pxf::Marshal` refuses a message nested past `kMaxNestingDepth`** (#107, protowire#502), as protowire-go does (protowire-go#236), with `pxf: nesting depth exceeds MaxNestingDepth=100`. Depth is counted as a decoder counts it: the root is depth 0, and each `{` or `[` is one level, so a list of lists stops at 50. It used to write a document that a reader at the same limit refused. **This narrows what encodes.** `AppendDatasetRow` refuses too. An Any whose body would be too deep stays as `type_url` and `value`, so an Any's contents never cause a refusal; only an Any that itself sits past the limit does. An Any's value is also no longer cut off at libprotobuf's 100-level parse limit when the writer's limit is raised. `pb::Marshal` cannot refuse yet (#93). - **A schema that redefines a well-known type with the wrong shape is refused at bind time** (draft -01 §wkt-shapes, protowire#487, protowire-go#218). This narrows what binds. A message named, say, `pxf.Secret` or `google.protobuf.Timestamp` without the fields the codec reads now gives a violation of the new kind `ViolationKind::kWellKnownShape`. Copies of the canonical schemas are unaffected, and field numbers are not checked. The check also reaches a schema's own file at a `google/protobuf/` path, which the closure walk used to skip. diff --git a/src/sbe/codec.cc b/src/sbe/codec.cc index 3a44c17..9bf6cfd 100644 --- a/src/sbe/codec.cc +++ b/src/sbe/codec.cc @@ -5,6 +5,7 @@ #include #include +#include "header_bounds.h" #include "protowire/sbe/annotations.h" namespace protowire::sbe { @@ -31,6 +32,16 @@ StatusOr Codec::New(std::vector files, CodecOp " missing (sbe.schema_id) option"); } auto version = GetFileUint32Option(fd, kExtVersion).value_or(0); + if (*schema > static_cast(kMaxUint16)) { + return Status::Error("sbe: file " + std::string(fd->name()) + + ": (sbe.schema_id) = " + std::to_string(*schema) + " is past " + + std::to_string(kMaxUint16) + ", the largest schemaId SBE can state"); + } + if (version > static_cast(kMaxUint16)) { + return Status::Error("sbe: file " + std::string(fd->name()) + + ": (sbe.version) = " + std::to_string(version) + " is past " + + std::to_string(kMaxUint16) + ", the largest version SBE can state"); + } for (int i = 0; i < fd->message_type_count(); ++i) { Status st = c.RegisterMessage( fd->message_type(i), static_cast(*schema), static_cast(version)); diff --git a/src/sbe/header_bounds.h b/src/sbe/header_bounds.h new file mode 100644 index 0000000..3a4626f --- /dev/null +++ b/src/sbe/header_bounds.h @@ -0,0 +1,14 @@ +// SPDX-License-Identifier: MIT +// Copyright (c) 2026 TrendVidia, LLC. +#pragma once + +namespace protowire::sbe { + +// kMaxUint16 is the largest value of every SBE header field: blockLength, +// templateId, schemaId and version in the message header, blockLength and +// numInGroup in a group header. A schema or a message needing a larger value +// cannot be written: narrowing it would write a header that disagrees with +// the bytes after it (protowire-go#250). +inline constexpr int kMaxUint16 = 65535; + +} // namespace protowire::sbe diff --git a/src/sbe/marshal.cc b/src/sbe/marshal.cc index 7e22d33..76ea190 100644 --- a/src/sbe/marshal.cc +++ b/src/sbe/marshal.cc @@ -5,8 +5,11 @@ #include "protowire/sbe.h" #include +#include #include +#include "header_bounds.h" + namespace protowire::sbe { namespace pb = google::protobuf; @@ -156,6 +159,13 @@ StatusOr> Codec::Marshal(const pb::Message& msg) const { for (const GroupTemplate& gt : tmpl->groups) { const pb::Reflection* r = msg.GetReflection(); int n = r->FieldSize(msg, gt.fd); + // numInGroup is a uint16: a larger count would be written as its low 16 + // bits, and a reader would take the entries past them for trailing bytes. + if (n > kMaxUint16) { + return Status::Error("sbe: field " + std::string(gt.fd->full_name()) + " has " + + std::to_string(n) + " entries, past " + std::to_string(kMaxUint16) + + ", the largest numInGroup SBE can state"); + } size_t header_off = buf.size(); buf.resize(header_off + 4, 0); WriteU16(buf.data() + header_off + 0, gt.block_length); diff --git a/src/sbe/template.cc b/src/sbe/template.cc index 214a1b7..cd7f5e7 100644 --- a/src/sbe/template.cc +++ b/src/sbe/template.cc @@ -7,6 +7,8 @@ #include #include +#include "header_bounds.h" +#include "protowire/detail/go_quote.h" #include "protowire/sbe/annotations.h" namespace protowire::sbe { @@ -29,6 +31,28 @@ std::vector SortedFields(const pb::Descriptor* md) { return out; } +// "pkg.Msg.field", as Go's "%s.%s" of a message's full name and a field's +// name writes it. +std::string FieldPath(const pb::Descriptor* md, const pb::FieldDescriptor* fd) { + return std::string(md->full_name()) + "." + std::string(fd->name()); +} + +Status Wrap(const std::string& prefix, const Status& st) { + return Status::Error(prefix + ": " + st.message()); +} + +// EndOfField returns the offset after a field of size bytes placed at +// offset, refusing a field that ends past what a uint16 blockLength can +// state. Every offset in the block is then a uint16 as well. +StatusOr EndOfField(int offset, uint16_t size) { + const int end = offset + size; + if (end > kMaxUint16) { + return Status::Error("ends at byte " + std::to_string(end) + " of its block, past " + + std::to_string(kMaxUint16) + ", the largest blockLength SBE can state"); + } + return end; +} + StatusOr> FieldEncodingSize(const pb::FieldDescriptor* fd) { if (auto enc = GetFieldStringOption(fd, kExtEncoding); enc.has_value()) { const std::string& e = *enc; @@ -36,7 +60,7 @@ StatusOr> FieldEncodingSize(const pb::FieldDesc if (e == kEncInt16 || e == kEncUint16) return std::make_pair(e, uint16_t{2}); if (e == kEncInt32 || e == kEncUint32 || e == kEncFloat) return std::make_pair(e, uint16_t{4}); if (e == kEncInt64 || e == kEncUint64 || e == kEncDouble) return std::make_pair(e, uint16_t{8}); - return Status::Error("unknown encoding \"" + e + "\""); + return Status::Error("unknown encoding " + detail::GoQuote(e)); } switch (fd->cpp_type()) { case pb::FieldDescriptor::CPPTYPE_BOOL: @@ -58,12 +82,18 @@ StatusOr> FieldEncodingSize(const pb::FieldDesc case pb::FieldDescriptor::CPPTYPE_STRING: { auto len = GetFieldUint32Option(fd, kExtLength); if (!len.has_value()) { - return Status::Error("string/bytes field requires (sbe.length)"); + return Status::Error(std::string(fd->type_name()) + + " field requires (sbe.length) annotation"); + } + if (*len > static_cast(kMaxUint16)) { + return Status::Error("(sbe.length) = " + std::to_string(*len) + " is past " + + std::to_string(kMaxUint16) + + ", the largest blockLength SBE can state"); } return std::make_pair(std::string(kEncChar), static_cast(*len)); } default: - return Status::Error("unsupported proto type for SBE"); + return Status::Error("unsupported proto kind " + std::string(fd->type_name())); } } @@ -73,35 +103,41 @@ StatusOr> BuildCompositeFields(const pb::Descriptor* StatusOr> BuildCompositeFields(const pb::Descriptor* md, uint16_t* total_size) { std::vector out; - uint16_t offset = 0; + int offset = 0; for (const pb::FieldDescriptor* fd : SortedFields(md)) { + const std::string where = "composite field " + FieldPath(md, fd); if (fd->is_repeated() || fd->is_map()) { - return Status::Error("composite contains list/map field"); + return Status::Error("composite " + std::string(md->full_name()) + + " contains list/map field " + std::string(fd->name())); } if (fd->cpp_type() == pb::FieldDescriptor::CPPTYPE_MESSAGE) { uint16_t sub_size = 0; auto subs = BuildCompositeFields(fd->message_type(), &sub_size); if (!subs.ok()) return subs.status(); + auto end = EndOfField(offset, sub_size); + if (!end.ok()) return Wrap(where, end.status()); FieldTemplate ft; ft.fd = fd; - ft.offset = offset; + ft.offset = static_cast(offset); ft.size = sub_size; ft.composite = std::move(subs).consume(); out.push_back(std::move(ft)); - offset += sub_size; + offset = *end; continue; } auto es = FieldEncodingSize(fd); - if (!es.ok()) return es.status(); + if (!es.ok()) return Wrap(where, es.status()); + auto end = EndOfField(offset, es->second); + if (!end.ok()) return Wrap(where, end.status()); FieldTemplate ft; ft.fd = fd; - ft.offset = offset; + ft.offset = static_cast(offset); ft.encoding = es->first; ft.size = es->second; out.push_back(ft); - offset += ft.size; + offset = *end; } - *total_size = offset; + *total_size = static_cast(offset); return out; } @@ -109,35 +145,46 @@ StatusOr BuildGroupTemplate(const pb::FieldDescriptor* fd) { const pb::Descriptor* md = fd->message_type(); GroupTemplate gt; gt.fd = fd; - uint16_t offset = 0; + int offset = 0; for (const pb::FieldDescriptor* f : SortedFields(md)) { - if (f->is_map() || f->is_repeated()) { - return Status::Error("group field cannot be repeated/map"); + if (f->is_map()) { + return Status::Error("sbe: map field in group " + std::string(md->full_name()) + + " not supported"); + } + if (f->is_repeated()) { + return Status::Error("sbe: nested repeated field in group " + std::string(md->full_name()) + + " not supported"); } if (f->cpp_type() == pb::FieldDescriptor::CPPTYPE_MESSAGE) { + const std::string where = "sbe: composite in group " + FieldPath(md, f); uint16_t sub_size = 0; auto subs = BuildCompositeFields(f->message_type(), &sub_size); - if (!subs.ok()) return subs.status(); + if (!subs.ok()) return Wrap(where, subs.status()); + auto end = EndOfField(offset, sub_size); + if (!end.ok()) return Wrap(where, end.status()); FieldTemplate ft; ft.fd = f; - ft.offset = offset; + ft.offset = static_cast(offset); ft.size = sub_size; ft.composite = std::move(subs).consume(); gt.fields.push_back(std::move(ft)); - offset += sub_size; + offset = *end; continue; } + const std::string where = "sbe: group field " + FieldPath(md, f); auto es = FieldEncodingSize(f); - if (!es.ok()) return es.status(); + if (!es.ok()) return Wrap(where, es.status()); + auto end = EndOfField(offset, es->second); + if (!end.ok()) return Wrap(where, end.status()); FieldTemplate ft; ft.fd = f; - ft.offset = offset; + ft.offset = static_cast(offset); ft.encoding = es->first; ft.size = es->second; gt.fields.push_back(ft); - offset += ft.size; + offset = *end; } - gt.block_length = offset; + gt.block_length = static_cast(offset); return gt; } @@ -146,19 +193,25 @@ StatusOr BuildGroupTemplate(const pb::FieldDescriptor* fd) { StatusOr BuildTemplate(const pb::Descriptor* md, uint16_t schema_id, uint16_t version) { + const std::string name(md->full_name()); auto tid = GetMessageUint32Option(md, kExtTemplateID); if (!tid.has_value()) { - return Status::Error(std::string(md->full_name()) + " missing (sbe.template_id)"); + return Status::Error("sbe: message " + name + " missing (sbe.template_id)"); + } + if (*tid > static_cast(kMaxUint16)) { + return Status::Error("sbe: message " + name + ": (sbe.template_id) = " + std::to_string(*tid) + + " is past " + std::to_string(kMaxUint16) + + ", the largest templateId SBE can state"); } MessageTemplate tmpl; tmpl.template_id = static_cast(*tid); tmpl.schema_id = schema_id; tmpl.version = version; - uint16_t offset = 0; + int offset = 0; for (const pb::FieldDescriptor* fd : SortedFields(md)) { if (fd->is_map()) { - return Status::Error("map field not supported in SBE"); + return Status::Error("sbe: map field " + FieldPath(md, fd) + " not supported"); } if (fd->is_repeated() && fd->cpp_type() == pb::FieldDescriptor::CPPTYPE_MESSAGE) { auto gt = BuildGroupTemplate(fd); @@ -167,32 +220,39 @@ StatusOr BuildTemplate(const pb::Descriptor* md, continue; } if (fd->is_repeated()) { - return Status::Error("repeated scalar not supported in SBE"); + return Status::Error("sbe: repeated scalar field " + FieldPath(md, fd) + + " not supported; wrap in a message"); } if (fd->cpp_type() == pb::FieldDescriptor::CPPTYPE_MESSAGE) { + const std::string where = "sbe: composite " + FieldPath(md, fd); uint16_t sub_size = 0; auto subs = BuildCompositeFields(fd->message_type(), &sub_size); - if (!subs.ok()) return subs.status(); + if (!subs.ok()) return Wrap(where, subs.status()); + auto end = EndOfField(offset, sub_size); + if (!end.ok()) return Wrap(where, end.status()); FieldTemplate ft; ft.fd = fd; - ft.offset = offset; + ft.offset = static_cast(offset); ft.size = sub_size; ft.composite = std::move(subs).consume(); tmpl.fields.push_back(std::move(ft)); - offset += sub_size; + offset = *end; continue; } + const std::string where = "sbe: field " + FieldPath(md, fd); auto es = FieldEncodingSize(fd); - if (!es.ok()) return es.status(); + if (!es.ok()) return Wrap(where, es.status()); + auto end = EndOfField(offset, es->second); + if (!end.ok()) return Wrap(where, end.status()); FieldTemplate ft; ft.fd = fd; - ft.offset = offset; + ft.offset = static_cast(offset); ft.encoding = es->first; ft.size = es->second; tmpl.fields.push_back(ft); - offset += ft.size; + offset = *end; } - tmpl.block_length = offset; + tmpl.block_length = static_cast(offset); return tmpl; } diff --git a/test/CMakeLists.txt b/test/CMakeLists.txt index 1cc89dc..7ea43f1 100644 --- a/test/CMakeLists.txt +++ b/test/CMakeLists.txt @@ -64,7 +64,7 @@ endif() if(TARGET protowire_sbe) set(_sbe_test_srcs "") foreach(f sbe_codec_test sbe_convert_test sbe_navigation_test sbe_stream_test - sbe_template_id_test) + sbe_template_id_test sbe_header_bounds_test) if(EXISTS ${CMAKE_CURRENT_SOURCE_DIR}/${f}.cc) list(APPEND _sbe_test_srcs ${f}.cc) endif() diff --git a/test/sbe_header_bounds_test.cc b/test/sbe_header_bounds_test.cc new file mode 100644 index 0000000..d61c468 --- /dev/null +++ b/test/sbe_header_bounds_test.cc @@ -0,0 +1,242 @@ +// SPDX-License-Identifier: MIT +// Copyright (c) 2026 TrendVidia, LLC. +// +// Every SBE header field is a uint16, so each value a schema declares, and +// each block it lays out, must fit in 65535. Ported from protowire-go's +// encoding/sbe/header_bounds_test.go (protowire-go#250). + +#include + +#include +#include +#include +#include + +#include "protowire/sbe.h" +#include "sbe_files.h" + +namespace { + +namespace pb = google::protobuf; +using protowire::sbe::Codec; +using protowire_test::SbeFiles; + +bool BoundsFile(SbeFiles* files, + const std::string& schema_id, + const std::string& version, + const std::string& body) { + return files->Compile( + {{"bounds.proto", + "syntax = \"proto3\";\npackage bounds.v1;\nimport \"sbe/annotations.proto\";\noption " + "(sbe.schema_id) = " + + schema_id + ";\noption (sbe.version) = " + version + ";\n" + body + "\n"}}); +} + +uint16_t Le16(const std::vector& b, size_t at) { + return static_cast(b[at] | (b[at + 1] << 8U)); +} + +// TestNewCodecHeaderBounds pairs each schema the SBE header cannot describe +// with the same shape at the bound, which must bind and round-trip. +TEST(SbeHeaderBounds, NewCodec) { + struct Case { + const char* name; + std::string schema_id; + std::string version; + std::string body; + std::string reject; // a substring of the error; empty means it must bind + uint16_t block_size = 0; // the root blockLength when it binds + }; + const std::vector cases = { + {"length past 65535", + "", + "", + "message M { option (sbe.template_id) = 1; bytes b = 1 [(sbe.length) = 65537]; uint32 t = " + "2; }", + "field bounds.v1.M.b: (sbe.length) = 65537 is past 65535"}, + {"length at 65535", + "", + "", + "message M { option (sbe.template_id) = 1; bytes b = 1 [(sbe.length) = 65535]; }", + "", + 65535}, + {"string length past 65535", + "", + "", + "message M { option (sbe.template_id) = 1; string s = 1 [(sbe.length) = 70000]; }", + "field bounds.v1.M.s: (sbe.length) = 70000 is past 65535"}, + {"root block past 65535", + "", + "", + "message M { option (sbe.template_id) = 1; bytes a = 1 [(sbe.length) = 30000]; bytes b = " + "2 [(sbe.length) = 30000]; bytes c = 3 [(sbe.length) = 30000]; uint32 t = 4; }", + "field bounds.v1.M.c: ends at byte 90000 of its block, past 65535"}, + {"root block one byte past 65535", + "", + "", + "message M { option (sbe.template_id) = 1; bytes a = 1 [(sbe.length) = 65532]; uint32 t = " + "2; }", + "field bounds.v1.M.t: ends at byte 65536 of its block"}, + {"root block at 65535", + "", + "", + "message M { option (sbe.template_id) = 1; bytes a = 1 [(sbe.length) = 65531]; uint32 t = " + "2; }", + "", + 65535}, + {"group block past 65535", + "", + "", + "message E { bytes a = 1 [(sbe.length) = 40000]; bytes b = 2 [(sbe.length) = 40000]; } " + "message M { option (sbe.template_id) = 1; repeated E e = 1; }", + "group field bounds.v1.E.b: ends at byte 80000 of its block"}, + {"group block at 65535", + "", + "", + "message E { bytes a = 1 [(sbe.length) = 40000]; bytes b = 2 [(sbe.length) = 25535]; } " + "message M { option (sbe.template_id) = 1; repeated E e = 1; }", + ""}, + {"composite past 65535", + "", + "", + "message C { bytes a = 1 [(sbe.length) = 40000]; bytes b = 2 [(sbe.length) = 40000]; } " + "message M { option (sbe.template_id) = 1; C c = 1; }", + "composite bounds.v1.M.c: composite field bounds.v1.C.b: ends at byte 80000"}, + {"composite fits, block past 65535", + "", + "", + "message C { bytes a = 1 [(sbe.length) = 40000]; } message M { option (sbe.template_id) = " + "1; C c = 1; C d = 2; }", + "composite bounds.v1.M.d: ends at byte 80000"}, + {"composite in group past 65535", + "", + "", + "message C { bytes a = 1 [(sbe.length) = 40000]; } message E { C c = 1; C d = 2; } message " + "M { option (sbe.template_id) = 1; repeated E e = 1; }", + "composite in group bounds.v1.E.d: ends at byte 80000"}, + {"nested composite past 65535", + "", + "", + "message D { bytes a = 1 [(sbe.length) = 40000]; } message C { D x = 1; D y = 2; } message " + "M { option (sbe.template_id) = 1; C c = 1; }", + "composite field bounds.v1.C.y: ends at byte 80000"}, + {"template_id past 65535", + "", + "", + "message M { option (sbe.template_id) = 65537; uint32 t = 1; }", + "message bounds.v1.M: (sbe.template_id) = 65537 is past 65535"}, + {"template_id at 65535", + "", + "", + "message M { option (sbe.template_id) = 65535; uint32 t = 1; }", + "", + 4}, + {"schema_id past 65535", + "65536", + "", + "message M { option (sbe.template_id) = 1; uint32 t = 1; }", + "file bounds.proto: (sbe.schema_id) = 65536 is past 65535"}, + {"schema_id at 65535", + "65535", + "", + "message M { option (sbe.template_id) = 1; uint32 t = 1; }", + "", + 4}, + {"version past 65535", + "", + "65536", + "message M { option (sbe.template_id) = 1; uint32 t = 1; }", + "file bounds.proto: (sbe.version) = 65536 is past 65535"}, + {"version at 65535", + "", + "65535", + "message M { option (sbe.template_id) = 1; uint32 t = 1; }", + "", + 4}, + }; + for (const Case& tc : cases) { + SCOPED_TRACE(tc.name); + const std::string schema_id = tc.schema_id.empty() ? "1" : tc.schema_id; + const std::string version = tc.version.empty() ? "0" : tc.version; + SbeFiles files; + if (!BoundsFile(&files, schema_id, version, tc.body)) { + ADD_FAILURE() << files.error(); + continue; + } + auto codec = Codec::New(files.files()); + if (!tc.reject.empty()) { + if (codec.ok()) { + ADD_FAILURE() << "bound; want " << tc.reject; + continue; + } + EXPECT_NE(codec.status().message().find(tc.reject), std::string::npos) + << codec.status().message(); + continue; + } + if (!codec.ok()) { + ADD_FAILURE() << codec.status().ToString(); + continue; + } + + const pb::Descriptor* md = files.Message("bounds.v1.M"); + auto msg = files.New(md); + if (const pb::FieldDescriptor* e = md->FindFieldByName("e"); e != nullptr) { + msg->GetReflection()->AddMessage(msg.get(), e); + } + auto data = codec->Marshal(*msg); + if (!data.ok()) { + ADD_FAILURE() << data.status().ToString(); + continue; + } + + // The header states exactly what the schema declared. + if (data->size() < 8) { + ADD_FAILURE() << "short header"; + continue; + } + if (tc.block_size != 0) EXPECT_EQ(Le16(*data, 0), tc.block_size) << "blockLength"; + EXPECT_EQ(std::to_string(Le16(*data, 4)), schema_id) << "schemaId"; + EXPECT_EQ(std::to_string(Le16(*data, 6)), version) << "version"; + + auto back = codec->UnmarshalDescriptor(*data, md); + EXPECT_TRUE(back.ok()) << back.status().ToString(); + } +} + +// TestMarshalGroupCountBound: numInGroup is a uint16. Before the fix 65536 +// entries were written with a count of 0, and the message read back with no +// entries and no error. +TEST(SbeHeaderBounds, MarshalGroupCount) { + SbeFiles files; + ASSERT_TRUE(BoundsFile( + &files, + "1", + "0", + "message E { uint32 a = 1; } message M { option (sbe.template_id) = 1; repeated E e = 1; }")) + << files.error(); + auto codec = Codec::New(files.files()); + ASSERT_TRUE(codec.ok()) << codec.status().ToString(); + const pb::Descriptor* md = files.Message("bounds.v1.M"); + const pb::FieldDescriptor* efd = md->FindFieldByName("e"); + + const auto with_entries = [&](int n) { + auto msg = files.New(md); + for (int i = 0; i < n; ++i) msg->GetReflection()->AddMessage(msg.get(), efd); + return msg; + }; + + auto data = codec->Marshal(*with_entries(65535)); + ASSERT_TRUE(data.ok()) << data.status().ToString(); + EXPECT_EQ(Le16(*data, 8 + 2), 65535) << "numInGroup"; + auto back = codec->UnmarshalDescriptor(*data, md); + ASSERT_TRUE(back.ok()) << back.status().ToString(); + EXPECT_EQ((*back)->GetReflection()->FieldSize(**back, efd), 65535); + + auto past = codec->Marshal(*with_entries(65536)); + ASSERT_FALSE(past.ok()); + EXPECT_NE(past.status().message().find("field bounds.v1.M.e has 65536 entries, past 65535"), + std::string::npos) + << past.status().message(); +} + +} // namespace