Skip to content

Add fixed diffusion tensor execution for SDXL and PixArt - #314

Draft
schopde-nvidia wants to merge 1 commit into
NVIDIA:mainfrom
schopde-nvidia:codex/prepared-diffusion-execution
Draft

schopde-nvidia wants to merge 1 commit into
NVIDIA:mainfrom
schopde-nvidia:codex/prepared-diffusion-execution

Conversation

@schopde-nvidia

Copy link
Copy Markdown

Flare's token-serving contract does not describe complete diffusion tensor requests. A meaningful comparison with PyTorch needs a separate native process that loads the exported text encoder, denoiser, solver updates, and image decoder. This change keeps those computations in Ignition's prepared graphs and adds shared native tensor transport and graph replay.

Before this change, signed cores required token-serving metadata and the CLI had no offline tensor-request execution path. This PR adds:

  • A generic Helios TensorRunner and flare run --tensor-request --tensor-output. They validate tensor extents, input overlap, and graph order, retain resource owners until queued work drains, and write intermediate tensors, repeated outputs, and timing samples.
  • Signed physical graph-only core publication and loading. Token-serving cores retain their paired frontend/sampling metadata and serving-contract checks; physical-only cores cannot enter token serving. Existing core signatures and wire identifiers are preserved.
  • Fixed SDXL and PixArt preparation recipes that capture text encoding, denoising, guidance, solver arithmetic, and decoding. Model-specific computation remains in Python preparation and the resulting graphs. PixArt uses the same native executor implementation as SDXL; the 4- and 30-step SDXL profiles differ in prepared graph sequences.
  • A comparator that separates canonical eager output, captured compiled output, and independently loaded native output. Native/captured acceptance requires finite, identical bytes, including signed zero. Failed numerical comparisons return nonzero and suppress performance ratios.
  • Setup and reproduction instructions, focused tests, and a separately marked proposal for concurrent iterative serving. The proposal is not implemented.

The demonstrated result is complete native execution of two diffusion architectures and byte-identical export/replay for the tested profiles. Canonical eager PyTorch numerical equivalence remains unresolved. Compiled outputs differ from the canonical references, so this is a draft PR with no speedup claim. The command handles fixed offline tensor requests; it does not provide an HTTP image endpoint, concurrent diffusion scheduling, native prompt processing, or general diffusion-model support.

Validation used one NVIDIA RTX PRO 6000 Blackwell Server Edition (SM120), PyTorch 2.14.1+cu132, Triton 3.8.0, and Diffusers 0.43.0. Every profile used two prompts with seeds 42 and 43, three warmups, and five measured native repetitions.

Prepared profile Graphs / exposed outputs Composed eager vs. canonical Diffusers Native and repeat vs. captured Compiled/native vs. canonical
SDXL base 1.0, 1024x1024, 4 Euler updates 7 / 8 Exact for both cases Byte-identical for both cases Exact comparison fails
SDXL base 1.0, 1024x1024, 30 Euler updates 33 / 34 Exact for both cases Byte-identical for both cases Exact comparison fails
PixArt-XL-2-512x512, 20 DPM-Solver++ updates 23 / 43 Exact for both cases Byte-identical for both cases Exact comparison fails

SDXL uses FP16 text encoders/UNet, the canonical FP32 VAE upcast, guidance 5, and negative prompt blurry, low quality. PixArt uses canonical mixed FP16/FP32 T5 parameters, an FP16 transformer/VAE, guidance 4.5, 120 text positions, and a second-order midpoint solver with an empty negative prompt. Checkpoint revisions, prompts, and executable reproduction commands are in the diffusion design document linked below.

Independent native processes loaded neither Python nor Torch. Isolated compiled update and decoder comparisons also differ when supplied canonical intermediates; PixArt's second update differs with canonical solver history. Native replay fidelity therefore does not establish canonical numerical correctness. Raw decoded-image RMSE for the two SDXL 30-step cases is 0.009159 and 0.018393; for PixArt it is 0.008709 and 0.085828. No performance ratio is claimed.

Focused validation passed 114 Rust tests covering schema, protection, export, conversion, tensor execution, and CLI behavior; 28 SDXL Python checks; 41 PixArt Python checks; and 38 geometry/schedule checks. The PixArt count includes the original 13 comparator tests. The byte-sensitive comparator now passes 15 tests, and rechecking all six retained GPU comparisons preserves the native-pass/canonical-fail result. The lightweight local Python run also reported four dependency-related skips. These are focused checks, not a full repository suite.

GPU validation used the implementation based on a3e8778. The combined tree with subsequent upstream changes has not been built or GPU-tested. Broader checkpoints, shapes, solvers, tensor parallelism, concurrent serving, and numerical acceptance bounds beyond exact comparison remain unqualified. Open qualification work is recorded in docs/TODO.txt.

Reproduction: pinned checkpoints, environment setup, preparation, and native comparison commands. Qualification requirements: diffusion TODOs.

Flare's token-serving contract does not describe complete diffusion tensor
requests. Comparing diffusion against native Flare requires loading exported
computation independently of Python. Keep model composition and solver math
in Ignition's prepared graphs, and let Helios validate tensor storage and
replay the declared graph sequence.

Before this change, signed cores and the CLI required token-serving metadata
and offered no offline tensor-request command. Add physical graph-only core
publication/loading, a generic TensorRunner, and flare run tensor input,
output, repetition, and timing options. Validate tensor bounds and input
overlap, retain GPU resource owners through queued work, and preserve the
paired metadata requirements for token-serving cores. Add fixed SDXL and
PixArt preparation recipes, geometry and solver helpers, intermediate-output
comparison, and reproducible setup instructions. A separate iterative-serving
contract is documented as proposed and is not implemented.

The resulting executor runs both diffusion architectures without model-specific
native computation. The measured result is byte-identical native replay of
captured compiled outputs for the tested profiles. Canonical eager PyTorch
numerical equivalence remains unresolved: exact comparisons fail, and the
comparator withholds performance ratios. This does not establish concurrent
diffusion serving, broader model support, or a speedup.

On an RTX PRO 6000 Blackwell Server Edition (SM120), tested SDXL at 1024x1024
with 4 and 30 updates and PixArt at 512x512 with 20 updates, each with two
prompts and seeds 42/43. Composed eager recipes match canonical Diffusers;
independently loaded native outputs and repetitions match captured outputs.
The native process loads neither Python nor Torch. Focused validation passed
114 Rust tests, 28 SDXL Python checks, 41 PixArt Python checks, and 38 geometry
and schedule checks. After tightening native acceptance to compare bytes
including signed zero, 15 comparator tests pass and all six retained GPU
comparisons retain their native-pass/canonical-fail result. These are focused
checks, not the full repository suite. GPU execution used source based on
a3e8778; integration with subsequent upstream changes has not been built or
GPU-tested. Reproduction details and open qualification are in
docs/diffusion-design.md and docs/TODO.txt.

Design-Signed-Off-By: Codex (GPT-6); design-review=ed22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant