Read FlatBuffers buffers when you don't have the schema.
No dependencies. Every accessor is bounds-checked, so malformed input returns a zero value instead of panicking.
The normal FlatBuffers workflow compiles a .fbs schema into generated accessors, and the field names live in that generated code. If you're handed a binary payload without its schema there's nothing to generate from, and the official library can't help, because it needs the types you're trying to work out.
That comes up more often than the docs suggest:
- reverse engineering a protocol from a capture
- inspecting a payload from a service that doesn't publish its schema
- triaging a file that decodes on one machine but not another
- checking what an SDK actually sends, rather than what its docs claim
flatread reads a buffer positionally. Fields are addressed by vtable slot number instead of by name. That's enough to walk an unknown buffer, and enough to write a real decoder once you've worked out which slot means what.
go get github.com/Mootjelh/flatreadgo install github.com/Mootjelh/flatread/cmd/flatdump@latestPoint flatdump at a payload and it describes every populated field, guessing each one's type from its bytes:
$ flatdump payload.bin
root @0x40 (168 bytes)
slot 4 scalar u32=4242 u16=4242 u8=146
slot 6 string "hello"
slot 8 bytes 3
slot 10 table
slot 4 scalar u32=7 u16=7 u8=7
slot 12 vector 2 tables
[0]
slot 4 scalar u32=11 u16=11 u8=11
[1]
slot 4 scalar u32=22 u16=22 u8=22
It reads stdin when given no file, so it drops into a pipeline:
curl -s https://example.com/payload | flatdump -depth 8With -json the same walk comes out machine-readable, which is what you want once you are grepping across a directory of captures:
flatdump -json payload.bin | jq '.fields[] | select(.kind == "string")'Capture the same request twice, or the same object before and after something changed, and ask which slots moved:
$ flatdump -diff before.bin after.bin
4 4242 -> 9999
6 "hello" -> "world"
10.4 7 -> 8
12[0].4 11 -> 12
The path is the slot route from the root: a dot for a nested table, brackets for an element of a table vector. That works without a schema, because a slot number is a stable handle whether or not you know what the slot means.
Exit codes follow diff(1): 0 when the buffers agree, 1 when they differ, 2 on
an error. In code it is flatread.Diff(a, b), returning a []Change.
A slot present on only one side is reported rather than skipped, which is the case a walk over one buffer's slots quietly misses.
Two things are worth checking before you conclude a payload is empty.
Size-prefixed buffers. gRPC and any format that stores several messages back to back put a uint32 byte count in front of the buffer. flatdump detects that and says so; in code it is RootSizePrefixed, with IsSizePrefixed for the guess.
This matters because feeding such a buffer to plain Root does not fail loudly. It reads the size as the root offset, lands somewhere arbitrary, and usually reports a table with no fields at all:
$ flatdump -prefix no sized.bin
root @0xa8 (172 bytes)
No error, no fields, and identical to a buffer that genuinely has nothing in it.
File identifiers. A schema can attach four bytes of identifier directly after the root offset, and when it is there it is usually the fastest way to find out what you are holding, because it is picked to be human-readable:
$ flatdump payload.bin
file identifier "FLTR"
root @0x40 (168 bytes)
...
Nothing in the buffer records whether that field is present, so FileIdentifier returns the bytes plus whether they look like an identifier at all. A false there means "these are not plausible", not "this schema has none".
A struct is not a table. It is a fixed-size record stored inline, with no vtable and no offsets, so a vector of them is a count followed by raw bytes. A vector of bytes is also a count followed by raw bytes. Nothing in the buffer tells them apart, and Guess reports KindBytes for both.
That reads badly by default. Three 8-byte structs is 24 bytes of data, and the count in front of them is 3, so the dump says:
$ flatdump payload.bin
slot 4 bytes 3 01 00 00
which looks like a tiny byte vector rather than three records. Tell it the element size and it becomes readable:
$ flatdump -struct 4:8 payload.bin
slot 4 structs 3 x 8 bytes
[0] 01 00 00 00 02 00 00 00 u32 1 2
[1] 03 00 00 00 04 00 00 00 u32 3 4
[2] 05 00 00 00 06 00 00 00 u32 5 6
The uint32 reading is shown beside the bytes rather than instead of them, on the same reasoning as the scalar line showing three widths at once: the field layout also comes from a schema, so which reading is right is exactly what you do not know yet.
The argument is path:size, and the path is slot numbers from the root, dotted. -struct 6:8 is slot 6 of the root; -struct 4.6:12 is slot 6 of the table in root slot 4. It has to be a path rather than a bare slot number, because the dump walks every nested table and a bare number would claim that slot in all of them, turning ordinary byte vectors into nonsense records. The flag is repeatable.
A size that cannot fit is reported rather than quietly falling back:
slot 4 structs no vector of 4096-byte elements fits here
You asked a question about a schema and that is the answer to it. Falling back to the byte reading would show you the answer to a different one.
In code it is DumpOptions.StructSize func(path []uint16) int, returning 0 for anything that is not a struct vector, with Table.StructAt and Table.StructVectorLen underneath for reading the elements yourself.
On a payload of any size the dump is all or nothing. -depth cuts everywhere at once, so seeing one nested table in full means raising the depth and then reading past everything that came with it. -slot takes the same dotted path as -struct and starts there instead:
$ flatdump -slot 12.4 payload.bin
root @0x40 (168 bytes)
slot 12 vector 2 tables
[0]
slot 4 scalar u32=11 u16=11 u8=11
[1]
slot 4 scalar u32=22 u16=22 u8=22
The slots on the way are kept, so the output says where in the buffer you are looking. Below the target the walk carries on as usual, -depth and -elems included: the flag picks where to start, not how much to print. Depth is still counted from the root, so reaching a long way in wants both.
A path that is not there says so:
$ flatdump -slot 10.99 payload.bin
root @0x40 (168 bytes)
slot 10 table
no slot 10.99 in this buffer
Note that slot 10 does exist, so the dump is not empty and judging by output length would call that a hit. An empty dump is indistinguishable from a buffer with nothing in it, which is the confusion this tool exists to remove, so it is answered directly. Under -json the same answer is "only_found": false.
For selecting rather than focusing, -json and jq were always enough and still are:
flatdump -json payload.bin | jq '.fields[] | select(.kind == "string")'In code it is DumpOptions.Only []uint16, with flatread.OnPath exported for anyone writing their own walk who wants the same rule rather than a second copy of it.
root, err := flatread.Root(buf)
if err != nil {
return err
}
for _, slot := range root.Slots() {
fmt.Println(slot, root.Guess(slot))
}
name := root.String(6)
ids := root.Int32Vector(8)
if pricing, ok := root.Table(10); ok {
total := pricing.Uint32(4)
}
for i := 0; i < root.VectorLen(12); i++ {
if offer, ok := root.TableAt(12, i); ok {
_ = offer.String(4)
}
}Scalars (Byte, Bool, Int8 through Int64, Uint16 through Uint64, Float32, Float64), strings, byte vectors, numeric and string vectors, nested tables and vectors of tables. Has distinguishes an absent field from one that's present and zero.
Entry points: Root for an ordinary buffer, RootSizePrefixed for one carrying a length, TableAtOffset when you already know where a table starts.
Structs are not tables. They're fixed-size records stored inline in the vector, with no vtable and no offsets, so TableAt cannot read them: it treats each element as an offset, adds a number that was never one, and returns a table pointing at whatever sits there. No error, no panic, just plausible nonsense.
StructAt reads them, given the element size from the schema:
n := root.StructVectorLen(6, 8) // 8-byte records
for i := 0; i < n; i++ {
rec := root.StructAt(6, i, 8)
x := binary.LittleEndian.Uint32(rec)
y := binary.LittleEndian.Uint32(rec[4:])
}StructVectorLen also catches a wrong size, because the declared count multiplied by that size stops fitting the buffer. Plain VectorLen can't: it doesn't know how wide an element is.
A union becomes two fields in consecutive slots: the type tag at n, the value at n+2.
switch kind, value, ok := root.Union(6); {
case !ok:
// absent, or the tag is UnionNone
case kind == 1:
_ = value.String(4)
}
for i := 0; i < root.UnionVectorLen(10); i++ {
kind, value, ok := root.UnionAt(10, i)
...
}A tag of UnionNone in a union vector is a hole, not the end. Stopping there skips everything after it.
Worth having to hand while reading a dump. Everything is little-endian.
buf[0:4] uoffset to the root table
table[-soffset] the table's vtable. soffset is a SIGNED int32
stored at the table's own position, so the
vtable normally sits before the table
vtable[0:2] vtable size in bytes
vtable[2:4] inline table size in bytes
vtable[4+2i : 6+2i] byte offset of field i inside the table,
0 meaning absent
Field i lives at slot number 4+2i, so the first field is slot 4, the second slot 6, the third slot 8. Slot numbers are what you pass to every accessor. Without a schema they're the only stable handle you have.
Offsets to strings, vectors and nested tables are uoffsets: unsigned, and relative to the position of the offset itself.
flatread recovers structure, not types. Guess works from a length prefix that fits, a run of printable bytes, a known magic number, or an offset that resolves to a readable vtable. That's genuinely all the information the buffer holds.
The clearest consequence, which has a test pinning it so it doesn't get "fixed" later: a vector of uint32{1, 2, 3} and the twelve bytes that encode it are the same twelve bytes, so Guess reports bytes. Nothing in a FlatBuffers buffer records a vector's element width.
The same applies to vectors of structs, which is why they need StructAt and an element size you supply.
So use Guess to find the slots you care about, confirm each one against real data, then read them with the typed accessors. Don't build a decoder on it.
flatread also doesn't write buffers. Use the official library for that. Once you know the schema you should be generating code from it anyway.
Every accessor is bounds-checked and returns a zero value rather than panicking. This package is by definition pointed at bytes nobody has validated, so malformed input is the ordinary case here, not an exceptional one.
That property is enforced rather than claimed:
go test -fuzz=FuzzReaderThe fuzz target walks every accessor over arbitrary input, including offset cycles: a buffer whose field points back at its own table, which an encoder would never emit but a corrupt or hostile file can. CI runs a 60-second pass on every push.
The trade-off is that a zero return is ambiguous. It means either "the field holds zero" or "there is no such field". Use Has when the difference matters.
Byte vectors carrying a RoaringBitmap serial cookie are recognised and labelled, since Roaring turns up often in search and analytics payloads. Decoding one needs a dependency this package doesn't take, so there's a hook for it:
flatread.DumpWith(buf, flatread.DumpOptions{
Annotate: func(b []byte) string {
if !flatread.LooksRoaring(b) {
return ""
}
bm := roaring.New()
if _, err := bm.ReadFrom(bytes.NewReader(b)); err != nil {
return "roaring: " + err.Error()
}
return fmt.Sprintf("%d bits set", bm.GetCardinality())
},
})MIT