Description
I am attempting to build a TensorRT engine for a proprietary PointCNN-derived point cloud classification model on Jetson Orin using TensorRT 10.3.0.
The model executes correctly in TensorFlow and ONNX Runtime, but TensorRT consistently crashes during engine generation. The failure occurs during Myelin compilation, before inference execution.
The crash is reproducible with both NVIDIA trtexec and a custom application using the TensorRT API, suggesting the issue is not application-specific.
The model contains PointCNN XConv blocks involving neighborhood gathering, MatMul, reshapes, transposes, and depthwise convolutions.
TensorRT successfully:
-Parses the ONNX model
-Performs tactic selection
-Profiles kernels
-Starts Myelin compilation
The segmentation fault always occurs during compilation of the same XConv region.
Environment
TensorRT Version: 10.3.0
NVIDIA GPU: Jetson Orin NX 16GB
NVIDIA Driver Version: 540.4.0
CUDA Version: CUDA 12.6 (JetPack 6.2)
Operating System: Ubuntu: 22.04
Python Version (if applicable): 3.10.12
Tensorflow Version (if applicable): TensorFlow 1.15.2
PyTorch Version (if applicable): N/A
Baremetal or Container (if so, version): Baremetal Jetson device
Relevant Files
Model link: Unfortunately I cannot share the model. However, the model is: TensorFlow 1.15 PointCNN-derived point cloud classifier and the problematic section corresponds to the first XConv block. TensorFlow tensor shapes in the failing region are:
xconv_1_nn_fts_input:0 (1, ?, 7, 18)
xconv_1_X_2_KK:0 (?, 1024, 7, 7)
xconv_1_fts_X:0 (?, 1024, 7, 18)
The failure occurs around:
xconv_1_X_2_KK
↓
MatMul (xconv_1_fts_X)
↓
Transpose
↓
Depthwise Conv
Steps To Reproduce
Commands or scripts: Build using:
trtexec
--onnx=model.onnx
--minShapes=features:0:1x1024x1,points:0:1x1024x3
--optShapes=features:0:1x1024x1,points:0:1x1024x3
--maxShapes=features:0:1x1024x1,points:0:1x1024x3
--verbose
I also tried --fp16, --fp32, --noTF32, builderOptimizationLevel=0, 1, 2, 3.
All configurations fail in the same location.
I also tried exporting the ONNX with --target tensorrt
Have you tried the latest release?: TensorRT version: 10.3.0
Can this model run on other frameworks? For example run ONNX model with ONNXRuntime (polygraphy run <model.onnx> --onnxrt): yes, the model runs successfully in TensorFlow and ONNX Runtime
Question
Is there any recommended way to disable the problematic fusion or obtain additional diagnostics around the Myelin compilation stage?
Description
I am attempting to build a TensorRT engine for a proprietary PointCNN-derived point cloud classification model on Jetson Orin using TensorRT 10.3.0.
The model executes correctly in TensorFlow and ONNX Runtime, but TensorRT consistently crashes during engine generation. The failure occurs during Myelin compilation, before inference execution.
The crash is reproducible with both NVIDIA trtexec and a custom application using the TensorRT API, suggesting the issue is not application-specific.
The model contains PointCNN XConv blocks involving neighborhood gathering, MatMul, reshapes, transposes, and depthwise convolutions.
TensorRT successfully:
-Parses the ONNX model
-Performs tactic selection
-Profiles kernels
-Starts Myelin compilation
The segmentation fault always occurs during compilation of the same XConv region.
Environment
TensorRT Version: 10.3.0
NVIDIA GPU: Jetson Orin NX 16GB
NVIDIA Driver Version: 540.4.0
CUDA Version: CUDA 12.6 (JetPack 6.2)
Operating System: Ubuntu: 22.04
Python Version (if applicable): 3.10.12
Tensorflow Version (if applicable): TensorFlow 1.15.2
PyTorch Version (if applicable): N/A
Baremetal or Container (if so, version): Baremetal Jetson device
Relevant Files
Model link: Unfortunately I cannot share the model. However, the model is: TensorFlow 1.15 PointCNN-derived point cloud classifier and the problematic section corresponds to the first XConv block. TensorFlow tensor shapes in the failing region are:
xconv_1_nn_fts_input:0 (1, ?, 7, 18)
xconv_1_X_2_KK:0 (?, 1024, 7, 7)
xconv_1_fts_X:0 (?, 1024, 7, 18)
The failure occurs around:
xconv_1_X_2_KK
↓
MatMul (xconv_1_fts_X)
↓
Transpose
↓
Depthwise Conv
Steps To Reproduce
Commands or scripts: Build using:
trtexec
--onnx=model.onnx
--minShapes=features:0:1x1024x1,points:0:1x1024x3
--optShapes=features:0:1x1024x1,points:0:1x1024x3
--maxShapes=features:0:1x1024x1,points:0:1x1024x3
--verbose
I also tried --fp16, --fp32, --noTF32, builderOptimizationLevel=0, 1, 2, 3.
All configurations fail in the same location.
I also tried exporting the ONNX with --target tensorrt
Have you tried the latest release?: TensorRT version: 10.3.0
Can this model run on other frameworks? For example run ONNX model with ONNXRuntime (
polygraphy run <model.onnx> --onnxrt): yes, the model runs successfully in TensorFlow and ONNX RuntimeQuestion
Is there any recommended way to disable the problematic fusion or obtain additional diagnostics around the Myelin compilation stage?