Add multimodal evaluation and component-aware quantization support - #2531
Add multimodal evaluation and component-aware quantization support#2531Delwin Kim (DelwinKim) wants to merge 44 commits into
Conversation
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Outdated benchmark snapshot (historical only)Warning This July 28 snapshot is outdated and is not part of the current PR validation or merge claims. It is retained only for historical provenance. Current results are intentionally omitted from this draft. Show outdated July 28 benchmark snapshotOutdated July 28 benchmark snapshot
These runs exercise Protocols
Exact variant definitions
Standalone W8 RTN and symmetric Google-policy RTN results are intentionally excluded. W8 source packages remain only in the matched INT8-embedding overlay experiment. Gemma-4 E2B - package size
Gemma-4 E2B - full image and fusion benchmarksHigher is better. Values are percentages. The top four scores in each row
Gemma-4 E2B - full audio benchmarksWER/CER is lower-is-better; BLEU is higher-is-better. The top four scores in
All 12 newly added PR558 runs completed with status 0 and zero empty PR558 recipe reproduction and harness compatibilityYou built PR558's original CUDA package by running its five component
The FLEURS numbers use different prompts and normalizers and therefore are
On matched 64-example compatibility runs, the recipe-native score was 10.33 Gemma-4 E2B Q4_0 GGUF importBoth Q4_0 variants now have complete full-image results and are included in The deployed package imports Google's official QAT Q4_0 decoder as 275
Verified image/fusion results on the first 500 examples:
Corrected audio results on the first 500 examples:
These are the corrected v13 audio packages. The original GGUF audio import Qwen2.5-VL-3B - full image benchmarksImage rows use complete benchmark splits. MathVision uses the same ordered
The post-export RTN-dec package produced empty generations on several tasks. K-Quant retained most FP16 quality without empty outputs. A separate matched INT8-token-embedding matrix completed 14/14 runs; each first-500 metric changed by at most 0.8 percentage points with no consistent direction. Qwen2.5-VL-3B - INT8 token-embedding overlaysEach row is a matched first-500 source → overlay pair. The overlay changes
All 14 source/overlay runs completed successfully. Across these bundles, each Qwen3-VL-2B - full image benchmarksImage rows use complete benchmark splits. MathVision uses the same ordered
Decoder-only K-Quant preserves more quality than all-MatMul K-Quant on Qwen3-VL, particularly OCRBench and TextVQA. Main takeaways
These tables are evidence that the evaluator handles official vision, document/OCR, audio, translation, and cross-modal scorers on deployed quantized packages. They should not be read as a claim that every difference is statistically significant. |
Adds LMMSEvaluator (olive/evaluator/olive_evaluator.py) and an ORT-GenAI multimodal adapter (olive/evaluator/lmms_ort.py) for evaluating multimodal ONNX models via lmms-eval.
Build on top of the LMMSEvaluator + ORT-GenAI multimodal adapter foundation:
- LMMSEvaluator now dispatches HfModelHandler inputs to lmms-eval's native
per-architecture wrappers (phi4_multimodal, qwen2_5_vl, whisper, ...),
with auto-detection from HF model_type and a forwarded-kwargs filter
that only passes args the target wrapper actually declares (handles
wrappers like qwen2_5_vl which assert kwargs == {}). Enables
FP-vs-quantized comparison in a single recipe via evaluate_input_model.
- lmms_ort.py adapter: tolerant audio/image disambiguation (audio dicts
with "path" no longer get mis-routed to PIL.Image.open), Whisper-specific
prompt + EOS-collision handling so ASR works end-to-end through
ortgenai_mm without the Phi-4-MM chat-template scaffolding interfering.
- New CompositeToOnnxPackage pass: flattens nested CompositeModel ORT-GenAI
packages (subdir-per-component or root-level) into the flat layout
LMMSEvaluator expects. Tolerates extensionless component filenames
produced by some upstream quant passes.
- Tests: 32 in test_lmms_ort.py (entry-point/registry, HF dispatch,
kwargs filter, prompt builder, score_continuation, partition_visuals,
run_generation), 9 in test_composite_to_onnx_package.py (flatten +
external-data rewrites + fallback entry-point).
Validated end-to-end:
- whisper-large-v3 via HfModel -> ModelBuilder fp16 -> KQuant int8 ->
CompositeToOnnxPackage -> ortgenai_mm eval on LibriSpeech.
FP HF WER 1.52/2.26 (clean/other), INT8 ONNX WER 1.68/2.36.
…ocessor args - MobiusBuilder: add `mobius_ep_override` config knob. Lets a workflow force the mobius execution_provider (e.g. "default") independent of the Olive accelerator EP. Needed because mobius's cuda-EP attention fusions (PackedMultiHeadAttention for Qwen2.5-VL vision, GQA for Gemma-4 decoder) produce graphs the ORT-GenAI fused-attention kernels reject. "default" EP skips those fusions; the resulting INT4 graph is numerically equivalent. - lmms_ort: support torchcodec.AudioDecoder visuals (HF datasets 5.x audio feature) in _normalize_audio via duck-typed get_all_samples(). - lmms_ort: branch processor-arg shape on model type - Phi-4-MM needs a bare string, Whisper needs [prompt]. Passing a list to Phi-4-MM raised "Number of image tokens does not match the number of images". Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Remove the setup.py lmms_eval.models entry point, the _model_manifest factory, and its registration tests. The Olive LMMSEvaluator path imports LMMSORTGenAIEvaluator directly, so the entry point only affected the standalone lmms-eval CLI; dropping it keeps setup.py out of this change.
LMMSEvaluator/ortgenai_mm gains two knobs: - ignore_stop_strings: drops spurious stop strings (e.g. lmms-eval's default until=["\n\n"] fewshot_delimiter) from a task's until list so step-by-step reasoning runs to EOS instead of truncating at the first blank line. - system_prompt now defaults to None and is resolved per model family (_default_system_prompt_for_model_type) to match lmms-eval's per-model wrappers, keeping HF-input and ORT-GenAI prompts identical without per-run config. Adds unit tests for both. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
… preserve nested layout Adopts upstream microsoft#2529's _find_genai_config/_get_genai_model_dir discovery in LMMSEvaluator (replacing the naive 2-level _resolve_model_dir), so nested multi-component ORT-GenAI packages are located by searching upward from the entry ONNX file. CompositeToOnnxPackage no longer flattens the package: since ORT-GenAI loads nested layouts directly and the evaluator now discovers genai_config.json, the pass just hardlink-copies the nested tree and returns an ONNXModelHandler pointing at the entry component (default 'decoder'). This drops the expensive onnx re-serialization / external-data rewriting while keeping the essential Composite->ONNX handler conversion (LocalSystem.evaluate_model rejects composite models). Rewrites the pass tests for the nested-preserving behavior. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Drop local 'import json as _json' in LMMSEvaluator.evaluate (json already imported at module level) to clear PYLINT reimport warning. - Apply ruff-format (single-line create_pass_from_dict call) in the CompositeToOnnxPackage test. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…lanning Introduce a planner pass that annotates a model with mixed_precision_info (components to keep in full precision) consumed by prepare_model in quant_utils.py, which skips excluded modules by name. Registers the pass in olive_config.json. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Align HF and ORT-GenAI request construction, generation settings, prompts, metrics, package sizing, and result persistence. Correct multimodal continuation scoring and expand regression coverage for media, Whisper, and composite packages. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: a07152ad-b18a-4e15-9af2-2a9a0149483e
08dfd47 to
1c2b96a
Compare
Implement MBQ decoder reparameterization with modality-balanced calibration masks and downstream quantizer consistency checks. Add MQuant-inspired component activation range diagnostics and shared multimodal calibration primitives, with tests and documentation. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 9e5f326b-249c-4121-a665-8eadd74782d1
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: a07152ad-b18a-4e15-9af2-2a9a0149483e
Add Gemma 4-aware native GPTQ calibration and module discovery, preserve uncalibrated modules in float, support GPTQModel API variants, and retain nested ORT-GenAI packages. Add RTN node globs and robust audio decoding fallback with focused tests. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: a07152ad-b18a-4e15-9af2-2a9a0149483e
Combine the multimodal evaluator and deployment-package work with the multimodal quantization pass stack while preserving the original pass commits and merge history. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: a07152ad-b18a-4e15-9af2-2a9a0149483e
Add processor-ready AI2D and TextVQA calibration recipes, support the current Qwen2.5-VL decoder layout, and preserve BF16 during MBQ reconstruction search. Include full deployed AI2D evaluation recipes for component-aware RTN and MBQ+RTN. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: a07152ad-b18a-4e15-9af2-2a9a0149483e
Classify every architecture-defined embedding module, serialize mixed-precision exclusions for downstream exporters, and add matched Gemma-4 PyTorch KQuant AI2D recipes. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: a07152ad-b18a-4e15-9af2-2a9a0149483e
Track the complete Delwin_testing recipe and validation workspace for Gemma-4, Qwen2.5-VL, Qwen3-VL, Phi-4-MM, and related benchmark experiments. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: a07152ad-b18a-4e15-9af2-2a9a0149483e
Remove Delwin_testing from the repository index and ignore the local workspace while preserving its files on disk. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: eeac47b5-2bf9-470e-a3c4-b3205d220039
Add Qwen3-VL module mappings and DeepStack-aware calibration replay. Support glob-based KQuant node overrides with exact-name precedence, add coverage, and ignore generated submissions.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 7271f78d-9d14-4ce5-a154-5a7ffa9a70c2
Clear the remaining cache CodeQL finding without changing package copy behavior. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Retain generated token IDs so ORT-GenAI evaluations can exclude Gemma's optional thought channel before task scoring. Fail loudly when Gemma response delimiters do not resolve to single tokenizer IDs and preserve incremental decoding for other model families. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Delwin Kim <139003345+DelwinKim@users.noreply.github.com>
Preserve lmms-eval's default model-aware TaskManager when no external path is configured, and pass the active model when resolving custom tasks. Preserve tuple-shaped loglikelihood generations in sample artifacts without misclassifying them as empty. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Mirror AudioDecoderEx selection for speech processors that advertise multiple target rates. Preserve compatible 8 kHz input, select 16 kHz for higher-rate audio, and persist the supported-rate provenance. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Resolve standard ORT-GenAI Whisper packages through audio_processor_config.json and use the unparameterized AudioDecoder runtime default of 16 kHz while retaining strict failures for unsupported package shapes. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Treat AudioDecoder target_sample_rate as a validated scalar package declaration alongside sample_rate and sampling_rate. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Let package-declared AudioDecoder pipelines perform their native resampling, reserving host resampling for explicit overrides. Include media construction in fail_on_error handling and record the effective preprocessing mode. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Apply explicit audio_target_sample_rate overrides directly, including valid upsampling, while retaining downsample-only selection for package-declared AudioDecoder rates. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Adapt ModelBuilder to the new validated option-preparation API without logging tokens or truncating values, while preserving legacy typed options. Resolve new kv_cache_dim symbols from genai_config for discrepancy checks. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Represent FP16 promotions as exclusions and add validated module scoping for multimodal sensitivity runs. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Delwin Kim <139003345+DelwinKim@users.noreply.github.com>
The HQQ pass replaced each MatMul with a MatMulNBits node consuming newly registered quantized initializers, but never removed the original weight tensor it superseded. Because no node referenced it any more, the weight stayed in the graph as an orphaned initializer and was still serialized, inflating the saved model by the full size of the original weights while changing nothing numerically. On a Gemma-4 E2B decoder this left all 275 replaced FP16 weights in place, adding 3.73 GB to a decoder-only package (4.53 GB for all-MatMul) on top of the 0.94 GB of packed INT4 payload, so HQQ packages could not be compared against K-Quant or RTN ones by size. Prune unused initializers after quantization, matching the existing step in kquant_quantization.py and rtn_quantization.py. Initializers still consumed by a node, such as the weight of an excluded MatMul, are preserved. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
nodes_to_exclude and nodes_to_include were matched with exact string equality, so a caller had to enumerate every MatMul name to target a subgraph. Other Olive quantization passes accept wildcard patterns, and listing hundreds of generated node names is impractical for transformer decoders where the interesting selections are structural, such as every node under a vision encoder prefix. Match both lists with fnmatch when the entry contains a wildcard, keeping exact matching for plain names so existing configurations behave the same. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
MatMul stores its B operand as (K, N), but MatMulNBits expects each output channel's K-axis blocks laid out contiguously as (N, k_blocks, blob_size). The pass quantized the (K, N) tensor directly and then reshaped the packed result to (cols, k_blocks, blob_size), so quantized weights, scales, and zero points were all written in input-channel-major order while the runtime read them as output-channel-major. Blocks therefore spanned unrelated output channels and were paired with the wrong scales and zero points. Packages built this way loaded and generated non-empty text, but the output was incoherent, which made the corruption easy to mistake for a quality regression rather than a layout bug. Transpose to (N, K) and quantize along K so all three tensors share the runtime's ordering, and derive the reshape dimensions from the original weight rather than the transposed tensor. Non-zero axis values are now rejected explicitly, since only axis 0 was ever handled correctly. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
When HQQ replaces a weight that is also exposed as a graph input, remove both the unused initializer and its obsolete input declaration so the quantized model remains runnable. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Recognize bracket-expression globs and remove replaced weight initializers recursively in every graph scope while preserving captured parent initializers. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Serialize mapping-valued ORT-GenAI options with their documented selector syntax and treat both FP16 and FP32 SelectiveMixedPrecision promotions as quantization exclusions. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Use the standard AudioDecoder 16 kHz fallback when only unrelated decoder attributes are configured and no explicit sample rate is declared. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 37 out of 38 changed files in this pull request and generated 1 comment.
Suppressed comments (1)
olive/passes/pytorch/multimodal_mixed_precision.py:333
- _load_meta_model unconditionally imports accelerate.init_empty_weights. If accelerate isn't installed (it’s treated as optional elsewhere, e.g. SelectiveMixedPrecision), this pass will raise ImportError and fail even though it can fall back to a real model load for classification.
import transformers
from accelerate import init_empty_weights
from transformers import AutoModel
Fall back to real model loading when Accelerate is unavailable, document the intentional calibration sentinel, and consolidate the evaluator test import used for dispatch monkeypatching. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Delwin Kim <139003345+DelwinKim@users.noreply.github.com>
| import PIL.Image | ||
| import pytest | ||
|
|
||
| import olive.evaluator.olive_evaluator as olive_evaluator_module |
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 37 out of 38 changed files in this pull request and generated no new comments.
Suppressed comments (1)
olive/passes/pytorch/train_utils.py:302
get_calibration_datasetassumes every dataloader batch is a 2-tuple(input_data, labels)and unpacks it unconditionally. Olive datasets can also be unsupervised and yield a singledict(seeBaseDataset), and in that case this will crash with an unpacking error. Consider accepting both shapes:(dict, labels)anddict(labels=None), and raise a clear error for anything else.
for input_data, labels in dataloader:
Thanks for the suggestion, implemented these solutions to these 2 Olive "gaps" include_path now accepts one task directory or a list, validates each path, and passes a model-aware TaskManager(include_path=..., model_name=lm) for both ONNX and HF dispatch. When include_path is absent, Olive does not construct a manager, preserving lmms-eval's normal default-manager behavior. End-to-end resolution was tested with an external task directory and a real ORT-GenAI package. Metric direction now comes from lmms-eval's per-task higher_is_better map rather than defaulting every metric to True; the regression test verifies that WER is lower-is-better. Numeric metrics returned by custom task process_results continue to flow into Olive without a new Olive metric registration. One limitation remains important: include_path can add collision-free task names, but it cannot override a built-in task with the same name. Corrections to existing lmms-eval tasks still need to go upstream rather than being hidden in a local task directory. For a benchmark that is absent from lmms-eval, I would follow the three paths you outlined: For a public benchmark with a published protocol, add a versioned lmms-eval YAML/utils.py task, use include_path while it is under development, and contribute it upstream where possible. So the intended boundary is: use lmms-eval for public, protocol-defined benchmarks; use Olive's custom evaluation path for internal/proprietary workloads; and change the adapter only when the benchmark exists but deployed ORT-GenAI inference lacks a required capability. |
Avoid invoking the native ORT-GenAI tokenizer with an empty generated sequence or an empty response-channel suffix. Return the existing semantic empty response instead and cover both cases with a regression test. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Delwin Kim <139003345+DelwinKim@users.noreply.github.com>
|
Delwin Kim (@DelwinKim) please read the following Contributor License Agreement(CLA). If you agree with the CLA, please reply with the following information.
Contributor License AgreementContribution License AgreementThis Contribution License Agreement (“Agreement”) is agreed to by the party signing below (“You”),
|
Describe your changes
Important
The evaluator and ORT-GenAI package-handling changes (i.e. just the
LMMSEvaluatorintegration) have been isolated in draft PR #2615, fromDelwinKim/Olive:t-delwinkim/lmms-evaluator-onlyat08a6a2b2. The remaining quantization and calibration features should be reviewed as separate changes.Results are tracked in microsoft/Olive#2613.
This draft adds an end-to-end multimodal evaluation and quantization workflow to Olive. It integrates the lmms-eval benchmark harness, makes Mobius multi-component ORT-GenAI packages directly evaluable, and adds component-aware quantization/calibration support for vision-language and omni models.
The PR branch intentionally serves as a single reproduction branch (for reproducing results) for the relevant Olive changes, including the multimodal evaluator, sensitivity-based mixed precision, and HQQ fixes, etc.
Validated environment
3.12.9olive-ai==0.11.0.dev0;t-delwinkim/lmms-ort-evaluatorated853908mobius-ai==0.1.0; published integration brancht-delwinkim/multimodal-quant-integration-20260803atd369f992lmms-eval==0.7.2; evaluation-fix revision8a252a73onnxruntime-gpu==1.29.0; source revision761185d2onnxruntime-genai-cuda==0.15.0.dev0; source revision9f7a24fb; ONNX Runtime Extensions319f8d2ctorch==2.12.0;transformers==5.11.0;datasets==5.0.0;onnxscript==0.7.1;protobuf==6.33.6All benchmark results in the updated sample-results comment come from deployed ONNX/ORT-GenAI packages evaluated through
LMMSEvaluator, not from PyTorch proxy evaluations.This is the current public reproduction environment. Historical pre-integration result rows retain their recorded package/build provenance; updating this table does not relabel those artifacts as having been built by the current Mobius revision.
Post-review validation covered 463 affected Olive tests and 50 focused lmms-eval tests, changed-file lint/format checks, Olive import/CLI checks, and fresh deployed-package probes for external task resolution, package-native audio preprocessing, PNG/JPEG85 image profiles, MMLU loglikelihood, AI2D direct extraction, and limit-aware OCRBench. The exact fast-test workflow also passed against
onnxruntime-genai==0.15.1(4 tests and 13 subtests), while preserving compatibility with the pinned 0.15.0 development runtime. The pinned lmms-eval revision also includes the exact Google-promptfleurs_en_google_asrtask as a built-in task. Machine-local artifacts remain outside this PR.Reported quantization variants
MobiusBuilder(precision="fp16"); no weight quantizationMatMulNBitslayout; replaced weights are removed from root and nested graph scopesdesc_act=false, 1% Hessian dampinggemma-4-E2B-it-qat-w4a16-ctcompressed-tensors QAT checkpoint: symmetric W4A16 group-32 Linear weights with BF16 activations; vision/audio/projectors/embeddings/LM head remain BF16; no additional PTQ passMatMulNBits; encoders/embeddings/LM head remain floating pointStandalone W8 RTN and symmetric Google-policy RTN results are intentionally excluded. W8 packages appear only as source controls in the matched INT8-embedding experiment.
What is added
1.
LMMSEvaluatorA new evaluator (
"type": "LMMSEvaluator") runs official lmms-eval tasks inside an Olive workflow.Supported input handlers:
ONNXModelHandlerpointing into a complete ORT-GenAI package (genai_config.jsonplus decoder/vision/audio/embedding graphs). This dispatches to the newortgenai_mmadapter.HfModelHandlerfor native Hugging Face multimodal evaluation. The lmms-eval model wrapper is inferred frommodel_typeor selected explicitly withmodel_class.The evaluator:
generate_until;loglikelihoodfor compatible tasks;og.Imagesandog.Audios;genai_config.jsonfrom nested package layouts;2.
CompositeToOnnxPackageMobiusBuilderemits aCompositeModelHandlercontaining decoder, vision, audio, and embedding components. Generic evaluation cannot open that handler as one inference session.CompositeToOnnxPackage:decoder);ONNXModelHandler;3. Component-aware mixed precision
MultiModalMixedPrecisionclassifies quantizable modules into:visionaudiotextprojectorlm_headembedsIt writes
mixed_precision_infoconsumed by downstream PyTorch RTN, GPTQ, and K-Quant passes:The pass plans component precision but does not quantize by itself.
SelectiveMixedPrecisionprovides the complementary layer-sensitivity path. FP16/FP32 promotions are emitted as quantization exclusions, and optional module globs are validated against fused QKV groups and tied embedding/LM-head pairs.4. Multimodal calibration
MultimodalActivationRangeCalibrationrecords per-component activation ranges and derived qparams for vision/audio/text/projector analysis. It is a diagnostic/planning pass; it does not claim to implement MQuant's MSQ, AIFS, or runtime activation kernels.MultiModalMixedPrecisionexclusions and avoid quantizing modules for which calibration statistics were not collected.5. Quantizer and workflow integration
nodes_to_includeandnodes_to_exclude; ONNX K-Quant acceptsnodes_to_excludeplus per-nodecustomized_weight_configoverrides. Both filters match exact node names and*/?glob patterns.MatMulNBitspacking, and recursive orphan-initializer cleanup.GptqModelaccepts the current GPTQModel API/configuration surface.olive/olive_config.jsonand documented in the pass reference and quantization guide.Current validated recipes
The following examples use the exact current pass/evaluator field names and the tested quantization values. Machine-local snapshot, cache, model, and result paths from the validation workspace are replaced with portable paths.
Recipe A: Gemma-4 E2B decoder-only ONNX K-Quant plus AI2D evaluation
{ "input_model": { "type": "HfModel", "model_path": "google/gemma-4-E2B-it" }, "systems": { "local_system": { "type": "LocalSystem", "accelerators": [ { "device": "gpu", "execution_providers": ["CUDAExecutionProvider"] } ] } }, "passes": { "mobius_build": { "type": "MobiusBuilder", "precision": "fp16" }, "int4_quantize": { "type": "OnnxKQuantQuantization", "bits": 4, "block_size": 32, "save_as_external_data": true, "nodes_to_exclude": [ "vision_encoder/*", "audio_encoder/*", "embedding/*", "decoder/lm_head/*" ] }, "package": { "type": "CompositeToOnnxPackage" } }, "evaluators": { "evaluator": { "type": "LMMSEvaluator", "tasks": ["ai2d_direct"], "batch_size": 1, "max_length": 32768, "ignore_stop_strings": ["\n\n"], "log_samples": true, "output_path": "results/gemma4_kquant_decoder_ai2d.json", "fail_on_error": true } }, "evaluator": "evaluator", "evaluate_input_model": false, "target": "local_system", "output_dir": "models/gemma4_kquant_decoder", "cache_dir": "cache/gemma4_kquant_decoder", "no_artifacts": true, "clean_evaluation_cache": true }Run one benchmark per Olive process. To evaluate English audio instead, replace
ai2d_directand the output filename with one oflibrispeech_test_clean,librispeech_test_other, orfleurs_en. Keep custom tasks such asai2d_directin a versioned lmms-eval task bundle when they are not built into the pinned dependency.Recipe B: Qwen2.5-VL component-aware PyTorch RTN
{ "input_model": { "type": "HfModel", "model_path": "Qwen/Qwen2.5-VL-3B-Instruct", "task": "image-text-to-text", "load_kwargs": { "torch_dtype": "bfloat16", "attn_implementation": "sdpa" } }, "systems": { "local_system": { "type": "LocalSystem", "accelerators": [ { "device": "gpu", "execution_providers": ["CUDAExecutionProvider"] } ] } }, "passes": { "component_plan": { "type": "MultiModalMixedPrecision", "component_precision": { "vision": 16, "projector": 16, "text": 4, "lm_head": 16, "embeds": 16 }, "bits": 4, "group_size": 32, "sym": false }, "rtn": { "type": "Rtn", "bits": 4, "group_size": 32, "sym": false, "lm_head": false, "embeds": false }, "mobius_build": { "type": "MobiusBuilder", "precision": "fp16" }, "package": { "type": "CompositeToOnnxPackage" } }, "evaluators": { "evaluator": { "type": "LMMSEvaluator", "tasks": ["ai2d_direct"], "batch_size": 1, "max_length": 32768, "ignore_stop_strings": ["\n\n"], "log_samples": true, "output_path": "results/qwen25vl_component_rtn_ai2d.json", "fail_on_error": true } }, "evaluator": "evaluator", "evaluate_input_model": false, "target": "local_system", "output_dir": "models/qwen25vl_component_rtn", "cache_dir": "cache/qwen25vl_component_rtn", "no_artifacts": true, "clean_evaluation_cache": true }Standalone Hugging Face evaluation
{ "input_model": { "type": "HfModel", "model_path": "Qwen/Qwen2.5-VL-3B-Instruct" }, "evaluators": { "evaluator": { "type": "LMMSEvaluator", "model_class": "qwen2_5_vl", "trust_remote_code": true, "tasks": ["ai2d_direct"], "batch_size": 1, "log_samples": true, "output_path": "results/qwen25vl_hf_ai2d.json", "fail_on_error": true } }, "evaluator": "evaluator", "evaluate_input_model": true, "output_dir": "models/qwen25vl_hf", "cache_dir": "cache/qwen25vl_hf", "no_artifacts": true, "clean_evaluation_cache": true }Run any recipe with:
LMMSEvaluatorconfigurationtasksmodel_classlimitbatch_size1max_new_tokens256max_length32768system_promptinclude_pathaudio_target_sample_rateimage_serialization_profilelosslessjpeg85parity modeignore_stop_strings"\n\n"trust_remote_codefalselog_samplesfalseoutput_pathfail_on_errortrueTests
test/evaluator/test_lmms_ort.pytest/evaluator/test_olive_evaluator.pytest/test_cache.pytest/passes/onnx/test_composite_to_onnx_package.pytest/passes/onnx/test_hqq_quantization.pytest/passes/onnx/test_kquant_quantization.pytest/passes/onnx/test_rtn_quantization.pytest/passes/pytorch/test_multimodal_activation_range.pytest/passes/pytorch/test_multimodal_mixed_precision.pytest/passes/pytorch/test_multimodal_quantization.pytest/passes/pytorch/test_quant_utils.pytest/passes/pytorch/test_selective_mixed_precision.pyCurrent limitations
batch_size=1).MultimodalActivationRangeCalibrationis diagnostic metadata only; it does not add MQuant runtime kernels.Checklist
Release note
Add multimodal vision/audio evaluation through lmms-eval, plus component-aware mixed precision, multimodal calibration and deployable ORT-GenAI package evaluation for Olive workflows.