# [ONNX][Autotune] Integrated quantization does not preserve AutoTune Q/DQ placement on ViT

[HannisLee]

[ONNX][Autotune] Integrated quantization does not preserve AutoTune Q/DQ placement on ViT

Description

When using ONNX Integrated AutoTune with:

python -m modelopt.onnx.quantization ... --autotune

I found that the Q/DQ placement selected during the AutoTune search is not preserved in the final calibrated INT8 ONNX model.

This also causes a noticeable TensorRT performance regression.

For autotune_schemes_per_region=50:

Model	QPS	Speedup	P50
FP16 baseline	744.41	1.00x	1.34 ms
Standard INT8 PTQ	1019.40	1.37x	0.98 ms
Integrated AutoTune optimized_final.onnx	1045.54	1.41x	0.95 ms
Integrated final calibrated model	915.37	1.23x	1.09 ms

At the consumer-input edge level:

SAME          = 86
AUTOTUNE ONLY = 48
FINAL ONLY    = 0

So all activation boundaries in the final model are already present in the AutoTune model, but 48 AutoTune-selected boundaries disappear in the final calibrated model.

The missing boundaries follow a regular pattern:

12 x Cast -> Sqrt
12 x Div  -> Sqrt_1
12 x Div  -> Sqrt_2
11 x fc1/Add -> Gelu
 1 x Add -> final LayerNormalization

Because each Div Q/DQ is shared by two Sqrt consumers, these 48 missing consumer edges correspond exactly to the 36 missing activation Q/DQ pairs:

122 - 86 = 36

For example:

/blocks/blocks.0/attn/Cast_output_0_cast_to_fp16
    -> /blocks/blocks.0/attn/Sqrt.input[0]
/blocks/blocks.0/attn/Div_output_0

-> /blocks/blocks.0/attn/Sqrt_1.input[0]
/blocks/blocks.0/attn/Div_output_0

-> /blocks/blocks.0/attn/Sqrt_2.input[0]
/blocks/blocks.0/mlp/fc1/Add_output_0

-> /blocks/blocks.0/mlp/act/Gelu.input[0]

Expected behavior

I expected Integrated AutoTune to preserve the Q/DQ placement selected by AutoTune while using calibration data to compute the final quantization scales.

This also appears consistent with the existing integration test:

tests/gpu/onnx/quantization/test_autotune_quantization_integration.py

which compares DQ-fed (node_name, input_index) sets between standalone AutoTune and Integrated AutoTune and asserts that they are equal.

Could you please check whether the final Integrated AutoTune model is expected to preserve the exact AutoTune Q/DQ placement, and if so, fix the topology preservation for this ViT case?

I can provide the ONNX models, calibration data, topology comparison script, and full AutoTune/TensorRT logs if needed.