Serialized engine size failure of TensorRT 11.2 when building an unfused pointwise op on a large activation on GPU H100

[himmelroman]

Description

TensorRT 11.2.1.2 appends a large zero-filled region to the serialized plan for any
pointwise op that cannot be fused into an adjacent Conv. The appended region is
exactly 7x the fp16 output activation tensor, so the plan size scales with activation
geometry rather than with weights or graph size.

A single-node LeakyRelu graph on a [1,64,2944,1664] fp16 tensor serializes to
4,389,349,804 bytes (99.998% zeros) on TensorRT 11.2.1.2, versus 10,508 bytes on
TensorRT 10.16.1.11 — a 417,714x increase for the same ONNX on the same GPU, driver,
container and build script.

The engine is functionally correct: it deserializes and produces bit-identical output. The
problem is purely the serialized artifact size.

The same LeakyRelu fused into a Conv produces a 210,812-byte plan that is
byte-identical on 10.16.1.11 and 11.2.1.2. Only fusability differs between the working
and failing case — the activation shape is the same.

We first hit this on a real model (RealBasicVSR x4 upsample head), where a DepthToSpace
sat between the Conv and its LeakyReLU and blocked the fusion twice. That engine built
at 5,487,774,180 bytes under 11.2.1.2 versus 1,041,804 bytes under 10.16.1.11
(5268x). Byte layout of the 5.2 GB file: real content occupies the leading 1,100,136 bytes,
followed by a single contiguous zero run of 5,486,674,044 bytes to EOF. That equals
exactly 7x the two unfused activations combined:

7 * 2 * (1*64*1472*832)  = 1,097,334,784
7 * 2 * (1*64*2944*1664) = 4,389,339,136
                           -------------
                           5,486,673,920   (vs 5,486,674,044 observed)

Environment

TensorRT Version: 11.2.1.2 (regression vs 10.16.1.11, which is unaffected)

NVIDIA GPU: NVIDIA H100 80GB HBM3

NVIDIA Driver Version: 570.172.08

CUDA Version: 12.8.1

CUDNN Version: 91002

Operating System: Ubuntu (Linux 6.8.0-1037-gcp x86_64, glibc 2.39)

Python Version (if applicable): 3.12.3

Tensorflow Version (if applicable): n/a

PyTorch Version (if applicable): 2.9.1+cu128 (used only to cross-check numerics; not needed for the repro)

Baremetal or Container (if so, version): Container, based on nvidia/cuda:12.8.1 with TensorRT 11.2.1.2 installed

Relevant Files

Model link: none needed — the repro below builds the graph programmatically in ~20 lines
and requires no model file, no weights, and no external data.

Steps To Reproduce

Commands or scripts:

Save as trt_repro.py and run python trt_repro.py. Requires only tensorrt, onnx, numpy.

"""Minimal repro: TensorRT 11.2 emits a huge zero-filled serialized plan for an
unfused pointwise op.

TensorRT 10.16.1.11 ->        10,508 bytes
TensorRT 11.2.1.2   -> 4,389,349,804 bytes  (99.998% zeros)
"""

import numpy as np
import onnx
import tensorrt as trt
from onnx import TensorProto as T
from onnx import helper, numpy_helper

SHAPE = [1, 64, 2944, 1664]  # fp16 activation; 313,524,224 elements


def build(nodes, initializers, path):
    x = helper.make_tensor_value_info("x", T.FLOAT16, SHAPE)
    y = helper.make_tensor_value_info("y", T.FLOAT16, SHAPE)
    graph = helper.make_graph(nodes, "repro", [x], [y], initializers)
    model = helper.make_model(graph, opset_imports=[helper.make_opsetid("", 20)])
    model.ir_version = 10
    onnx.save(model, path)

    logger = trt.Logger(trt.Logger.ERROR)
    builder = trt.Builder(logger)
    network = builder.create_network(
        1 << int(trt.NetworkDefinitionCreationFlag.STRONGLY_TYPED)
    )
    parser = trt.OnnxParser(network, logger)
    with open(path, "rb") as f:
        assert parser.parse(f.read()), [
            parser.get_error(i) for i in range(parser.num_errors)
        ]

    config = builder.create_builder_config()
    config.set_memory_pool_limit(trt.MemoryPoolType.WORKSPACE, 4 << 30)
    profile = builder.create_optimization_profile()
    profile.set_shape("x", SHAPE, SHAPE, SHAPE)
    config.add_optimization_profile(profile)
    return bytes(builder.build_serialized_network(network, config))


def report(label, plan):
    elements = 1
    for d in SHAPE:
        elements *= d
    print(
        f"{label:34s} {len(plan):>14,} bytes   "
        f"zeros={100 * plan.count(0) / len(plan):6.3f}%   "
        f"plan/(fp16 tensor)={len(plan) / (elements * 2):.3f}"
    )


print(f"TensorRT {trt.__version__}\n")

# 1. A single unfused LeakyRelu -> plan is ~7x the output activation, all zeros.
report(
    "LeakyRelu alone (BUG)",
    build([helper.make_node("LeakyRelu", ["x"], ["y"], alpha=0.1)], [], "/tmp/a.onnx"),
)

# 2. The same LeakyRelu fused into a Conv -> normal plan size.
#    Only the fusability differs; the activation shape is identical.
w = numpy_helper.from_array(np.zeros((64, 64, 3, 3), dtype=np.float16), "w")
b = numpy_helper.from_array(np.zeros((64,), dtype=np.float16), "b")
report(
    "Conv+LeakyRelu (fuses, OK)",
    build(
        [
            helper.make_node(
                "Conv", ["x", "w", "b"], ["t"],
                pads=[1, 1, 1, 1], strides=[1, 1], dilations=[1, 1], group=1,
            ),
            helper.make_node("LeakyRelu", ["t"], ["y"], alpha=0.1),
        ],
        [w, b],
        "/tmp/b.onnx",
    ),
)

Actual output (same machine, same GPU, same script; only the container's TensorRT differs):

########## TensorRT 11.2.1.2 ##########
TensorRT 11.2.1.2

LeakyRelu alone (BUG)               4,389,349,804 bytes   zeros=100.000%   plan/(fp16 tensor)=7.000
Conv+LeakyRelu (fuses, OK)                210,812 bytes   zeros=62.273%   plan/(fp16 tensor)=0.000

########## TensorRT 10.16.1.11 ##########
TensorRT 10.16.1.11

LeakyRelu alone (BUG)                      10,508 bytes   zeros=48.677%   plan/(fp16 tensor)=0.000
Conv+LeakyRelu (fuses, OK)                210,812 bytes   zeros=62.273%   plan/(fp16 tensor)=0.000

Note the Conv+LeakyRelu plan is byte-identical (210,812) across both versions, while
the standalone LeakyRelu differs by 417,714x.

The engine inspector shows the unfused op falling to Myelin

On the real model, the two affected activations appear as Myelin nodes:

__myl_CastMulLtSeleCast_myl6_0
__myl_CastMulLtSeleCast_myl13_0

TensorRT 10.16.1.11 produces the same 17 layers with the same two __myl_ nodes at
1,041,804 bytes, so the layer structure is not what changed between versions — only the
amount of zero padding written into the plan.

Not specific to LeakyRelu

An exact Mul+Max rewrite of the same LeakyRelu bloats identically
(4,389,350,020 bytes, 100% zeros), so the trigger is an unfused pointwise op in general,
not this operator.

No builder configuration avoids it

All of the following produce a byte-identical 4,389,349,804-byte plan on 11.2.1.2:

setting	plan size
default	4,389,349,804
tiling_optimization_level = NONE	4,389,349,804
l2_limit_for_tiling = 0	4,389,349,804
max_num_tactics = 1	4,389,349,804
BuilderFlag.STRIP_PLAN	4,389,349,804
BuilderFlag.DISABLE_COMPILATION_CACHE	4,389,349,804
builder_optimization_level = 1..5	4,389,349,804
builder_optimization_level = 0	10,532 (clean)

STRIP_PLAN having no effect confirms the region is not weight data.
builder_optimization_level = 0 is the only clean setting, but it disables tactic
selection and is not usable for a production engine.

EXCLUDE_LEAN_RUNTIME is rejected with
kEXCLUDE_LEAN_RUNTIME can only be set if kVERSION_COMPATIBLE is set, which confirms
VERSION_COMPATIBLE is off and no lean runtime is being embedded.

Workaround

Reordering the graph so the pointwise op can fuse into its Conv avoids the issue entirely.
In our case DepthToSpace is a pure position permutation and LeakyReLU is elementwise,
so they commute exactly; emitting Conv -> LeakyReLU -> DepthToSpace instead of
Conv -> DepthToSpace -> LeakyReLU gave:

• 5,487,774,180 bytes -> 1,056,260 bytes
• zero __myl_ nodes; all activations fused as conv + PWN(leaky_relu)
• bit-identical output (torch.equal True) against both the bloated 11.2 engine and a
  10.16.1.11 engine built from the same ONNX

This workaround is only available when the graph happens to admit a commuting reorder, so
it is not a general fix.

Have you tried the latest release?:
Yes — 11.2.1.2 is the version exhibiting the issue. 10.16.1.11 is unaffected.

Can this model run on other frameworks?:
Yes. The graph is a single standard ONNX LeakyRelu node (opset 20) and runs correctly
under ONNXRuntime. The TensorRT engine also executes correctly — the defect is the size of
the serialized plan, not its numerics.

Impact

Engines are distributed to inference workers by download. A 1 MB artifact becoming 5.2 GB
inflates storage, transfer, and cold-start time for every worker, for a graph whose real
content is ~1 MB. Because the padding scales with activation size, it is worst exactly
where it hurts most — high-resolution upsampling heads.