sbe: refuse a value its (sbe.encoding) primitive cannot hold (#133)

CHANGELOG.md

@@ -26,6 +26,7 @@ number says nothing about any other port. See the Releases section of
 
 - **A decoder rejects `null` on a field with `(pxf.required)` or `@required`** (#131, decided by the owner; trendvidia/protowire#569, protowire-go#268). **This narrows what decodes.** A required field holds a value, so `null` is now rejected as absence is, whatever the field's type and presence. Before, `null` counted as present, so `r = null` on `optional int32 r [(pxf.required) = true]` decoded and left a required field with no value. The error is `null is not allowed on required field "r": a required field holds a value`, in a body, a nested body, a `@dataset` cell and a dotted cell, from `Unmarshal` as well as `UnmarshalFull`. A plain scalar with `(pxf.required)`, which already rejected `null`, now names the field as required instead of suggesting `optional`. A zero value still satisfies the field. A `_null` mask that names an unset required field no longer makes `AppendDatasetRow` write `null` for it: the writer refuses it, as it does without the mask.
 - **`pxf::Marshal` writes an unset field that has `(pxf.default)` or `(pxf.required)` so that it reads back** (#129, protowire-go#259; draft -01 § Common Schema Layer, "`decode_PXF(encode_PXF(v)) == v`"), as protowire-go does. This changes what Marshal writes for annotated fields, and narrows what encodes. Marshal left out every unset field, so a reader that applies defaults filled it in: `int32 n = 1 [(pxf.default) = "5"]` holding 0 read back as 5, and a required scalar holding 0 made the full decode refuse the document. A proto3 scalar without presence is now written as its zero value, a required list or map as `[]` or `{}`, and a message-typed field with a default, such as a wrapper or a Timestamp, as `null`. An unset `optional` or oneof scalar with a default is still left out and reads back as its default, the owner's choice in protowire-go#267. An unset `optional` field that is `(pxf.required)` is now an error, `pxf: <full name> is unset, and the field is (pxf.required)`; with `emit_defaults` it is written as zero, as before. An Any whose body has such a field stays as `type_url` and `value`. Inside a `@dataset` value cell, the same error now names the field by its full name. Unannotated fields are unchanged.
+- **`sbe` refuses a value its `(sbe.encoding)` primitive cannot hold, in both directions** (#133, protowire-go#252; draft -01 § SBE Annotations, "Emitters MUST reject values that fall outside the narrowed type's range"). This narrows what encodes, decodes and binds. `Marshal` used to keep the low bits: `int32 v [(sbe.encoding) = "int8"]` set to 300 was written as 44, and read back as 44 with no error. It now refuses the value with `sbe: field m.v1.M.v: value 300 is outside the range of int8, [-128, 127]`, for every integer primitive, and a `double` narrowed to `float` past ±3.4028234663852886e+38 (NaN and the infinities still pass). An enum is laid out as `uint8`, so an enum value past 255, or a negative one, is refused too. `Unmarshal` refuses a wire value the field cannot hold, such as `uint32` 2^31 into an `int32` field, with `sbe: field m.v1.M.v: wire value 2147483648 (uint32) does not fit a int32 field, [-2147483648, 2147483647]`; it used to cut it to the field's low bits. A signed primitive is now sign-extended whatever the field, so `int8` 0xFF in a `uint32` field is -1 and refused, where it used to read as 255. `Codec::New` refuses `(sbe.encoding)` on a `string` or `bytes` field, and a primitive that crosses between integers and floating point (`int32` with `"float"`, `double` with `"int32"`). A `double` field under `"float"`, and a `float` field under `"double"`, used to write and read the wrong bits; they now convert the value.
 - **`sbe::Codec::New` refuses a schema the SBE header cannot describe, and `Marshal` a group it cannot count** (#132, protowire-go#250). This narrows what binds and what encodes. Every SBE header field is a 16-bit number, and the codec kept the low 16 bits of a larger one. `bytes b = 1 [(sbe.length) = 65537]` bound as a 1-byte field. A block longer than 65535 bytes, in a message, a group entry or a composite, bound with a wrapped length and offsets. A `(sbe.template_id)`, `(sbe.schema_id)` or `(sbe.version)` of 65537 was written as 1. `Codec::New` now refuses each of these with protowire-go's text, naming the field, message or file, such as `sbe: field m.v1.M.b: (sbe.length) = 65537 is past 65535, the largest blockLength SBE can state`; 65535 still binds. `Marshal` refuses a repeated field with more than 65535 entries, with `sbe: field m.v1.M.e has 65536 entries, past 65535, the largest numInGroup SBE can state`: 65536 entries used to be written with a count of 0, and the message read back empty with no error. Every other bind error of a template now has protowire-go's text as well, prefixed with the field, group or composite it is about, such as `sbe: field m.v1.M.s: string field requires (sbe.length) annotation` where it used to be `string/bytes field requires (sbe.length)`.
 - **`sbe` tells two schemas' template ids apart** (#134, protowire-go#253 and #257; draft -01 § SBE Annotations, "sbe.template_id MUST be unique among messages within a schema", and § Header and Block Length). This narrows what binds and what decodes. `Codec::New` refuses two different messages with one `(sbe.template_id)` in one schema, with `sbe: messages A and B share (sbe.template_id) = 7 in schema 1; it must be unique within a schema`; the second used to replace the first, and `NewView` then read one message's bytes as the other's. A file passed twice still binds, and so do two schemas that each use a template id. `NewView` now finds a template by the header's schema id and template id together, and refuses an unknown pair with `sbe: unknown template ID 9 in schema 2`, even if another schema has that template id; it used to say `unknown template id 9`. `Unmarshal` and `UnmarshalDescriptor` refuse a header whose schema id is not the target message's, with `sbe: schema ID mismatch: got 1, want 2 for template 7`; the version is not checked. A caller that builds headers by hand must now set the schema id.
 - **`pxf::Marshal` refuses a message nested past `kMaxNestingDepth`** (#107, protowire#502), as protowire-go does (protowire-go#236), with `pxf: nesting depth exceeds MaxNestingDepth=100`. Depth is counted as a decoder counts it: the root is depth 0, and each `{` or `[` is one level, so a list of lists stops at 50. It used to write a document that a reader at the same limit refused. **This narrows what encodes.** `AppendDatasetRow` refuses too. An Any whose body would be too deep stays as `type_url` and `value`, so an Any's contents never cause a refusal; only an Any that itself sits past the limit does. An Any's value is also no longer cut off at libprotobuf's 100-level parse limit when the writer's limit is raised. `pb::Marshal` cannot refuse yet (#93).

src/sbe/encoding_range.h

@@ -0,0 +1,137 @@
+// SPDX-License-Identifier: MIT
+// Copyright (c) 2026 TrendVidia, LLC.
+//
+// The range of an (sbe.encoding) primitive and of the field it narrows, for
+// the checks of draft -01 § SBE Annotations: "Emitters MUST reject values
+// that fall outside the narrowed type's range". Marshal refuses a value the
+// primitive cannot hold, and Unmarshal one the field cannot hold, rather
+// than keep its low bits (protowire-go#252).
+#pragma once
+
+#include <charconv>
+#include <cmath>
+#include <cstdint>
+#include <limits>
+#include <string>
+#include <string_view>
+
+#include <google/protobuf/descriptor.h>
+
+#include "protowire/sbe/template.h"
+
+namespace protowire::sbe {
+
+inline bool IsIntEncoding(std::string_view e) {
+  return e == kEncInt8 || e == kEncInt16 || e == kEncInt32 || e == kEncInt64 || e == kEncUint8 ||
+         e == kEncUint16 || e == kEncUint32 || e == kEncUint64;
+}
+
+// EncodingRange holds, for an integer primitive, its range and the field's,
+// and whether a value can fall outside the other. For a float primitive, the
+// checks are a double that float cannot hold.
+struct EncodingRange {
+  int64_t enc_lo = 0;
+  uint64_t enc_hi = 0;
+  int64_t field_lo = 0;
+  uint64_t field_hi = 0;
+  bool enc_signed = false;
+  bool write_check = false;
+  bool read_check = false;
+};
+
+// FieldRange is the range of values an integer, bool or enum field holds. An
+// enum is open in proto3, so any int32.
+inline void FieldRange(const google::protobuf::FieldDescriptor* fd, int64_t* lo, uint64_t* hi) {
+  using FD = google::protobuf::FieldDescriptor;
+  switch (fd->type()) {
+    case FD::TYPE_INT32:
+    case FD::TYPE_SINT32:
+    case FD::TYPE_SFIXED32:
+    case FD::TYPE_ENUM:
+      *lo = std::numeric_limits<int32_t>::min();
+      *hi = std::numeric_limits<int32_t>::max();
+      return;
+    case FD::TYPE_INT64:
+    case FD::TYPE_SINT64:
+    case FD::TYPE_SFIXED64:
+      *lo = std::numeric_limits<int64_t>::min();
+      *hi = std::numeric_limits<int64_t>::max();
+      return;
+    case FD::TYPE_UINT32:
+    case FD::TYPE_FIXED32:
+      *lo = 0;
+      *hi = std::numeric_limits<uint32_t>::max();
+      return;
+    case FD::TYPE_BOOL:
+      *lo = 0;
+      *hi = 1;
+      return;
+    default:  // TYPE_UINT64, TYPE_FIXED64
+      *lo = 0;
+      *hi = std::numeric_limits<uint64_t>::max();
+      return;
+  }
+}
+
+inline EncodingRange RangeOf(const FieldTemplate& ft) {
+  using FD = google::protobuf::FieldDescriptor;
+  EncodingRange r;
+  if (ft.encoding == kEncFloat) {
+    r.write_check = ft.fd->type() == FD::TYPE_DOUBLE;
+  } else if (ft.encoding == kEncDouble) {
+    r.read_check = ft.fd->type() == FD::TYPE_FLOAT;
+  } else if (IsIntEncoding(ft.encoding)) {
+    r.enc_signed = ft.encoding[0] == 'i';
+    const unsigned bits = 8U * ft.size;
+    if (r.enc_signed) {
+      r.enc_hi = (uint64_t{1} << (bits - 1)) - 1;
+      r.enc_lo = -static_cast<int64_t>(r.enc_hi) - 1;
+    } else {
+      r.enc_lo = 0;
+      r.enc_hi = std::numeric_limits<uint64_t>::max() >> (64 - bits);
+    }
+    FieldRange(ft.fd, &r.field_lo, &r.field_hi);
+    r.write_check = r.field_lo < r.enc_lo || r.field_hi > r.enc_hi;
+    // A bool reads any nonzero primitive as true.
+    r.read_check =
+        ft.fd->type() != FD::TYPE_BOOL && (r.enc_lo < r.field_lo || r.enc_hi > r.field_hi);
+  }
+  return r;
+}
+
+// InRange reports whether the integer v is within [lo, hi]. v holds a
+// uint64 when is_unsigned is true, and an int64's bits otherwise.
+inline bool InRange(uint64_t v, bool is_unsigned, int64_t lo, uint64_t hi) {
+  if (!is_unsigned && static_cast<int64_t>(v) < 0) return static_cast<int64_t>(v) >= lo;
+  return v <= hi;
+}
+
+// IntString formats an integer held as in InRange.
+inline std::string IntString(uint64_t v, bool is_unsigned) {
+  return is_unsigned ? std::to_string(v) : std::to_string(static_cast<int64_t>(v));
+}
+
+// FloatFits reports whether a double is within float's range. NaN and the
+// infinities are, as float has them; a finite value past ±max float is not,
+// even one that would round down to it.
+inline bool FloatFits(double f) {
+  constexpr double kMax = std::numeric_limits<float>::max();
+  return std::isinf(f) || !(f > kMax || f < -kMax);
+}
+
+// GoG formats f as Go's %g does: the shortest digits that read back as f,
+// in %e form when the exponent is below -4 or at least 6.
+inline std::string GoG(double f) {
+  if (std::isnan(f)) return "NaN";
+  if (std::isinf(f)) return f > 0 ? "+Inf" : "-Inf";
+  char buf[64];
+  auto sci = std::to_chars(buf, buf + sizeof(buf), f, std::chars_format::scientific);
+  std::string s(buf, sci.ptr);
+  const size_t e = s.find('e');
+  const int exp = std::stoi(s.substr(e + 1));
+  if (exp < -4 || exp >= 6) return s;
+  auto fix = std::to_chars(buf, buf + sizeof(buf), f, std::chars_format::fixed);
+  return std::string(buf, fix.ptr);
+}
+
+}  // namespace protowire::sbe

src/sbe/marshal.cc

@@ -5,9 +5,11 @@
 #include "protowire/sbe.h"
 
 #include <cstring>
+#include <limits>
 #include <string>
 #include <vector>
 
+#include "encoding_range.h"
 #include "header_bounds.h"
 
 namespace protowire::sbe {
@@ -27,59 +29,89 @@ void WriteU64(uint8_t* p, uint64_t v) {
   for (int i = 0; i < 8; ++i) p[i] = static_cast<uint8_t>(v >> (i * 8));
 }
 
-void EnsureAndWriteScalar(std::vector<uint8_t>& buf,
-                          size_t offset,
-                          const FieldTemplate& ft,
-                          uint64_t v_bits) {
+// WriteScalar writes a numeric field as its primitive. The value is read by
+// the field's kind and, for an integer primitive, written as the low bytes of
+// its two's complement once it is known to fit; a value the primitive cannot
+// hold is refused rather than cut to its low bits (draft -01 § SBE
+// Annotations, protowire-go#252).
+Status WriteScalar(std::vector<uint8_t>& buf,
+                   size_t offset,
+                   const FieldTemplate& ft,
+                   const pb::Message& msg) {
   while (buf.size() < offset + ft.size) buf.push_back(0);
   uint8_t* p = buf.data() + offset;
+  const pb::Reflection* r = msg.GetReflection();
+  const pb::FieldDescriptor* fd = ft.fd;
+  const EncodingRange rg = RangeOf(ft);
+
+  if (ft.encoding == kEncFloat || ft.encoding == kEncDouble) {
+    const double f = fd->cpp_type() == pb::FieldDescriptor::CPPTYPE_FLOAT
+                         ? static_cast<double>(r->GetFloat(msg, fd))
+                         : r->GetDouble(msg, fd);
+    if (ft.encoding == kEncDouble) {
+      uint64_t b = 0;
+      std::memcpy(&b, &f, 8);
+      WriteU64(p, b);
+      return Status::OK();
+    }
+    if (rg.write_check && !FloatFits(f)) {
+      return Status::Error("sbe: field " + std::string(fd->full_name()) + ": value " + GoG(f) +
+                           " is outside the range of float, ±" +
+                           GoG(std::numeric_limits<float>::max()));
+    }
+    const auto narrowed = static_cast<float>(f);
+    uint32_t b = 0;
+    std::memcpy(&b, &narrowed, 4);
+    WriteU32(p, b);
+    return Status::OK();
+  }
+
+  uint64_t v = 0;
+  bool is_unsigned = false;
+  switch (fd->cpp_type()) {
+    case pb::FieldDescriptor::CPPTYPE_BOOL:
+      v = r->GetBool(msg, fd) ? 1 : 0;
+      is_unsigned = true;
+      break;
+    case pb::FieldDescriptor::CPPTYPE_ENUM:
+      v = static_cast<uint64_t>(int64_t{r->GetEnumValue(msg, fd)});
+      break;
+    case pb::FieldDescriptor::CPPTYPE_UINT32:
+      v = r->GetUInt32(msg, fd);
+      is_unsigned = true;
+      break;
+    case pb::FieldDescriptor::CPPTYPE_UINT64:
+      v = r->GetUInt64(msg, fd);
+      is_unsigned = true;
+      break;
+    case pb::FieldDescriptor::CPPTYPE_INT32:
+      v = static_cast<uint64_t>(int64_t{r->GetInt32(msg, fd)});
+      break;
+    default:  // CPPTYPE_INT64
+      v = static_cast<uint64_t>(r->GetInt64(msg, fd));
+      break;
+  }
+  if (rg.write_check && !InRange(v, is_unsigned, rg.enc_lo, rg.enc_hi)) {
+    return Status::Error("sbe: field " + std::string(fd->full_name()) + ": value " +
+                         IntString(v, is_unsigned) + " is outside the range of " + ft.encoding +
+                         ", [" + std::to_string(rg.enc_lo) + ", " + std::to_string(rg.enc_hi) +
+                         "]");
+  }
   switch (ft.size) {
     case 1:
-      p[0] = static_cast<uint8_t>(v_bits);
+      p[0] = static_cast<uint8_t>(v);
       break;
     case 2:
-      WriteU16(p, static_cast<uint16_t>(v_bits));
+      WriteU16(p, static_cast<uint16_t>(v));
       break;
     case 4:
-      WriteU32(p, static_cast<uint32_t>(v_bits));
-      break;
-    case 8:
-      WriteU64(p, v_bits);
+      WriteU32(p, static_cast<uint32_t>(v));
       break;
-  }
-}
-
-uint64_t EncodeScalar(const pb::Message& msg, const FieldTemplate& ft) {
-  const pb::Reflection* r = msg.GetReflection();
-  const pb::FieldDescriptor* fd = ft.fd;
-  switch (fd->cpp_type()) {
-    case pb::FieldDescriptor::CPPTYPE_BOOL:
-      return r->GetBool(msg, fd) ? 1 : 0;
-    case pb::FieldDescriptor::CPPTYPE_INT32:
-      return static_cast<uint64_t>(r->GetInt32(msg, fd));
-    case pb::FieldDescriptor::CPPTYPE_INT64:
-      return static_cast<uint64_t>(r->GetInt64(msg, fd));
-    case pb::FieldDescriptor::CPPTYPE_UINT32:
-      return r->GetUInt32(msg, fd);
-    case pb::FieldDescriptor::CPPTYPE_UINT64:
-      return r->GetUInt64(msg, fd);
-    case pb::FieldDescriptor::CPPTYPE_FLOAT: {
-      float f = r->GetFloat(msg, fd);
-      uint32_t b;
-      std::memcpy(&b, &f, 4);
-      return b;
-    }
-    case pb::FieldDescriptor::CPPTYPE_DOUBLE: {
-      double d = r->GetDouble(msg, fd);
-      uint64_t b;
-      std::memcpy(&b, &d, 8);
-      return b;
-    }
-    case pb::FieldDescriptor::CPPTYPE_ENUM:
-      return static_cast<uint64_t>(r->GetEnumValue(msg, fd));
     default:
-      return 0;
+      WriteU64(p, v);
+      break;
   }
+  return Status::OK();
 }
 
 // CutLength is how many bytes of s fit in a field of size bytes. A string is
@@ -109,34 +141,35 @@ void WriteFixedString(std::vector<uint8_t>& buf,
   for (size_t i = n; i < ft.size; ++i) buf[offset + i] = 0;
 }
 
-void WriteFields(std::vector<uint8_t>& buf,
-                 size_t base,
-                 const std::vector<FieldTemplate>& fields,
-                 const pb::Message& msg);
+Status WriteFields(std::vector<uint8_t>& buf,
+                   size_t base,
+                   const std::vector<FieldTemplate>& fields,
+                   const pb::Message& msg);
 
-void WriteCompositeField(std::vector<uint8_t>& buf,
-                         size_t base,
-                         const FieldTemplate& ft,
-                         const pb::Message& msg) {
+Status WriteCompositeField(std::vector<uint8_t>& buf,
+                           size_t base,
+                           const FieldTemplate& ft,
+                           const pb::Message& msg) {
   const pb::Message& sub = msg.GetReflection()->GetMessage(msg, ft.fd);
-  WriteFields(buf, base + ft.offset, ft.composite, sub);
+  return WriteFields(buf, base + ft.offset, ft.composite, sub);
 }
 
-void WriteFields(std::vector<uint8_t>& buf,
-                 size_t base,
-                 const std::vector<FieldTemplate>& fields,
-                 const pb::Message& msg) {
+Status WriteFields(std::vector<uint8_t>& buf,
+                   size_t base,
+                   const std::vector<FieldTemplate>& fields,
+                   const pb::Message& msg) {
   for (const FieldTemplate& ft : fields) {
     if (!ft.composite.empty()) {
-      WriteCompositeField(buf, base, ft, msg);
+      if (Status st = WriteCompositeField(buf, base, ft, msg); !st.ok()) return st;
       continue;
     }
     if (ft.encoding == kEncChar) {
       WriteFixedString(buf, base + ft.offset, ft, msg);
       continue;
     }
-    EnsureAndWriteScalar(buf, base + ft.offset, ft, EncodeScalar(msg, ft));
+    if (Status st = WriteScalar(buf, base + ft.offset, ft, msg); !st.ok()) return st;
   }
+  return Status::OK();
 }
 
 }  // namespace
@@ -153,7 +186,7 @@ StatusOr<std::vector<uint8_t>> Codec::Marshal(const pb::Message& msg) const {
   WriteU16(buf.data() + 2, tmpl->template_id);
   WriteU16(buf.data() + 4, tmpl->schema_id);
   WriteU16(buf.data() + 6, tmpl->version);
-  WriteFields(buf, 8, tmpl->fields, msg);
+  if (Status st = WriteFields(buf, 8, tmpl->fields, msg); !st.ok()) return st;
 
   // Repeating groups.
   for (const GroupTemplate& gt : tmpl->groups) {
@@ -174,7 +207,7 @@ StatusOr<std::vector<uint8_t>> Codec::Marshal(const pb::Message& msg) const {
       size_t entry_off = buf.size();
       buf.resize(entry_off + gt.block_length, 0);
       const pb::Message& entry = r->GetRepeatedMessage(msg, gt.fd, i);
-      WriteFields(buf, entry_off, gt.fields, entry);
+      if (Status st = WriteFields(buf, entry_off, gt.fields, entry); !st.ok()) return st;
     }
   }
   return buf;