Skip to content

GH-3561 Harden variants against malformed metadata. #3562

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
steveloughran wants to merge 4 commits into
base: master
Choose a base branch
from
Open

GH-3561 Harden variants against malformed metadata. #3562

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
ce40d68
GH-3561 Harden variant decoding
steveloughran May 13, 2026
ff3da30
Review feedback
steveloughran Jul 10, 2026
4be59ad
use Math.toIntExact when casting to int to guarantee overflows raise …
steveloughran Jul 13, 2026
04f79ab
Hardening ImmutableMetadata loading, with test
steveloughran Jul 14, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions parquet-variant/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -69,6 +69,17 @@
<version>${slf4j.version}</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.apache.parquet</groupId>
<artifactId>parquet-hadoop</artifactId>

@laskoviymishka laskoviymishka Jul 10, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The Iceberg tests for this stayed at direct ByteBuffer construction, so I was a little surprised to see the round-trip harness pull parquet-hadoop and hadoop-client into parquet-variant's test scope. Since Parquet's binary column encoding is just a length-prefixed byte copy, I don't think most of these cases exercise a different construction path than wrapping the bytes directly. Might be worth keeping the round-trip for the one or two cases where encode/decode genuinely matters — but you'll know the parquet-java test conventions better than I do.

@steveloughran steveloughran Jul 10, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wanted to create the test files for apache/parquet-testing#113 ; so while I agree it's overkill, it's a test only dependency and an easy way to generate test data for interop testing.

<version>${project.version}</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
<scope>test</scope>
</dependency>
</dependencies>

<build>
Expand Down
76 changes: 60 additions & 16 deletions parquet-variant/src/main/java/org/apache/parquet/variant/Variant.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -61,6 +61,8 @@ public final class Variant {
* Lazy cache for the parsed array header.
*/
private VariantUtil.ArrayInfo cachedArrayInfo;
/** Nesting depth of this Variant relative to the top-level value (0 = top-level). */
private final int depth;

@laskoviymishka laskoviymishka Jul 10, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor: depth here counts navigation depth through Variant objects, so it also ticks up for leaf primitives and short strings, not just container nesting. Harmless — a one-line note on the field might just save the next reader a double-take.


/**
* The threshold to switch from linear search to binary search when looking up a field by key in
Expand All @@ -69,10 +71,31 @@ public final class Variant {
*/
static final int BINARY_SEARCH_THRESHOLD = 32;

/**
* Create a Variant from a tuple of (value, metadata) byte arrays.
* Includes validation of the version and the top-level structure.
* @param value value buffer
* @param metadata metadata buffer
* @throws UnsupportedOperationException if there is a version mismatch
* @throws IllegalArgumentException for any validation failure.
*/
public Variant(byte[] value, byte[] metadata) {
this(value, 0, value.length, metadata, 0, metadata.length);
}

/**
* Create a Variant from a subset of the (value, metadata) buffers
* supplied.
* Includes validation of the version and the toplevel structure.
* @param value value buffer
* @param valuePos offset where the value data begins
* @param valueLength length of value data
* @param metadata metadata buffer
* @param metadataPos offset where the metadata begins.
* @param metadataLength length of the metadata.
* @throws UnsupportedOperationException if there is a version mismatch
* @throws IllegalArgumentException for any validation failure.
*/
public Variant(byte[] value, int valuePos, int valueLength, byte[] metadata, int metadataPos, int metadataLength) {
this(ByteBuffer.wrap(value, valuePos, valueLength), ByteBuffer.wrap(metadata, metadataPos, metadataLength));
}
Expand All @@ -81,10 +104,20 @@ public Variant(byte[] value, int valuePos, int valueLength, byte[] metadata, int
this(value, metadata.getEncodedBuffer());
}

/**
* Create a Variant from a tuple of (value, metadata) buffers.
* Includes validation of the version and the toplevel structure.
* @param value value buffer
* @param metadata metadata buffer
* @throws UnsupportedOperationException if there is a version mismatch
* @throws IllegalArgumentException for any validation failure.
*/
public Variant(ByteBuffer value, ByteBuffer metadata) {
this.value = value.asReadOnlyBuffer().order(ByteOrder.LITTLE_ENDIAN);
this.metadata = metadata.asReadOnlyBuffer().order(ByteOrder.LITTLE_ENDIAN);

this.depth = 0;
// A metadata buffer must contain at least the version byte
Preconditions.checkArgument(this.metadata.remaining() >= 1, "variant metadata is empty");
// There is currently only one allowed version.
if ((metadata.get(metadata.position()) & VariantUtil.VERSION_MASK) != VariantUtil.VERSION) {
throw new UnsupportedOperationException(String.format(
Expand All @@ -95,29 +128,39 @@ public Variant(ByteBuffer value, ByteBuffer metadata) {
// Pre-compute dictionary size for lazy metadata cache allocation.
int pos = this.metadata.position();
int metaOffsetSize = ((this.metadata.get(pos) >> 6) & 0x3) + 1;
if (this.metadata.remaining() > 1) {
Preconditions.checkArgument(
this.metadata.remaining() >= 1 + metaOffsetSize,
"variant metadata truncated: offsetSize=" + metaOffsetSize);
this.dictSize = VariantUtil.readUnsignedLittleEndian(this.metadata, pos + 1, metaOffsetSize);
long dictTableEnd = 1L + metaOffsetSize + ((long) this.dictSize + 1) * metaOffsetSize;
Preconditions.checkArgument(
dictTableEnd <= this.metadata.remaining(),
"variant metadata dictionary extends past buffer: dictSize=" + this.dictSize);
} else {
this.dictSize = 0;
}
Preconditions.checkArgument(
this.metadata.remaining() >= 1 + metaOffsetSize,

@laskoviymishka laskoviymishka Jul 10, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks like a genuine fix, but it's also a public-API behavior change that might be worth a release note. A 1-byte metadata buffer (just the version byte) used to construct successfully with dictSize=0 via the old else branch, and now throws — so any downstream caller relying on that leniency would start seeing IllegalArgumentException.

"variant metadata truncated: offsetSize=" + metaOffsetSize);
this.dictSize = VariantUtil.readUnsignedLittleEndian(this.metadata, pos + 1, metaOffsetSize);
long dictTableEnd = 1L + metaOffsetSize + ((long) this.dictSize + 1) * metaOffsetSize;
Preconditions.checkArgument(
dictTableEnd <= this.metadata.remaining(),
"variant metadata dictionary extends past buffer: dictSize=" + this.dictSize);
this.metadataCache = null;
VariantUtil.validateValueShallow(this.value, dictSize);

@Fokko Fokko May 27, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since we now validate in the constructor, should we add some Javadoc to this public function? Including the @throws IllegalArgumentException

@steveloughran steveloughran Jun 25, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

}

/**
* Package-private constructor that shares pre-parsed metadata state from a parent Variant.
* Package-private constructor that shares pre-parsed metadata state from a parent Variant, performing shallow validation of the structure.
* @param value value buffer
* @param metadata metadata buffer
* @param metadataCache shared metadata cache.
* @param dictSize shared dictionary size.
* @param depth depth of this variant in a recursive structure.
* @throws IllegalArgumentException if the depth of variants is too high or the structure
* invalid in some other form.
*/
Variant(ByteBuffer value, ByteBuffer metadata, String[] metadataCache, int dictSize) {
Variant(ByteBuffer value, ByteBuffer metadata, String[] metadataCache, int dictSize, int depth) {
Preconditions.checkArgument(
depth <= VariantUtil.MAX_VARIANT_DEPTH,
"variant nesting depth exceeds maximum %s",
VariantUtil.MAX_VARIANT_DEPTH);
this.value = value.asReadOnlyBuffer().order(ByteOrder.LITTLE_ENDIAN);
this.metadata = metadata.asReadOnlyBuffer().order(ByteOrder.LITTLE_ENDIAN);
this.metadataCache = metadataCache;
this.dictSize = dictSize;
this.depth = depth;
VariantUtil.validateValueShallow(this.value, dictSize);
}

public ByteBuffer getValueBuffer() {
Expand Down Expand Up @@ -359,9 +402,10 @@ public Variant getElementAtIndex(int index) {

/**
* Creates a child Variant that shares this instance's metadata cache.
* @throws IllegalArgumentException validation error during construction.
*/
private Variant childVariant(ByteBuffer childValue) {
return new Variant(childValue, metadata, metadataCache, dictSize);
return new Variant(childValue, metadata, metadataCache, dictSize, depth + 1);
}

/**
Expand Down
4 changes: 3 additions & 1 deletion parquet-variant/src/main/java/org/apache/parquet/variant/VariantJsonParser.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,8 @@
*/
package org.apache.parquet.variant;

import static org.apache.parquet.variant.VariantUtil.MAX_VARIANT_DEPTH;

import com.fasterxml.jackson.core.JsonFactory;
import com.fasterxml.jackson.core.JsonParseException;
import com.fasterxml.jackson.core.JsonParser;
Expand All @@ -37,7 +39,7 @@ public final class VariantJsonParser {

private static final JsonFactory JSON_FACTORY = JsonFactory.builder()
.streamReadConstraints(StreamReadConstraints.builder()
.maxNestingDepth(500)
.maxNestingDepth(MAX_VARIANT_DEPTH)

@laskoviymishka laskoviymishka Jul 10, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This bumps the JSON parse-depth boundary from 500 to 1000 as a side effect of sharing the constant, so documents that used to overflow at 501 now parse. Since buildJson/buildJsonArray/buildJsonObject recurse on the real JVM stack, I wasn't sure whether 1000 had been validated as safe on this path specifically (as opposed to the descent path). Might be worth a release note, and maybe a test that parses ~1000-level JSON on a realistic stack. wdyt?

@steveloughran steveloughran Jul 10, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That original 500 was completely chosen without any data as a general resilience number, the goal being "a malicious file can damage the stack of the thread processing it, but not the rest of the process". I went with the same number for the variant for consistency, and after a discussion in the mailing list went to 1k.

we haven't shipped the json parser yet, so no need for a release note. I will look at #3415 notes to see if we should update it though

.maxStringLength(10_000_000)
.maxDocumentLength(50_000_000L)
.build())
Expand Down
Loading
Loading