Skip to content

[VPTO] Represent MAD CTRL requirements as ctrl_state_guard and materialize minimal hardware accesses - #1504

Draft
jimmychou0 wants to merge 1 commit into
hw-native-sys:masterfrom
jimmychou0:zjm/ctrl_opt_issue1279
Draft

[VPTO] Represent MAD CTRL requirements as ctrl_state_guard and materialize minimal hardware accesses#1504
jimmychou0 wants to merge 1 commit into
hw-native-sys:masterfrom
jimmychou0:zjm/ctrl_opt_issue1279

Conversation

@jimmychou0

Copy link
Copy Markdown
Contributor

Summary

Implements the ctrl_state_guard design for issue #1279: MAD semantic-to-raw lowering no longer materializes the fixed 7-instruction CTRL round-trip per MAD (get_ctrl -> 4x sbitset -> set_ctrl -> mad_raw -> set_ctrl). Instead the temporary CTRL requirement is represented structurally and a new VPTOOptimizeCtrlStatePass materializes the minimal set of hardware CTRL accesses.

Changes

  • New state interfaces (StateAccessOpInterface with Query/Write/Consume/Clobber, StateGuardOpInterface, StateResource/StateAccessKind enums): get_ctrl=Query, set_ctrl=Write, the four raw MAD ops=Consume.
  • New internal op pto.ctrl_state_guard{controlled_bits, required_bits} wrapping the raw MAD consumer; verifier enforces R ⊆ C, C ≠ 0, exactly one CTRL-Consume (must implement MadRawOpInterface), no nesting, func-local. Eliminated before emission validation.
  • VPTOExpandWrapperOps computes (C,R) statically (HiF8 bit45, TF32 bit46/47, explicit sat bit48, n_dir bit51; unspecified sat inherits the entry state) and emits the guard instead of the hardware sequence.
  • New VPTOOptimizeCtrlStatePass (after the scheduler, before emission validation):
    • bottom-up scf.for summaries: CTRL cleanliness, unique requirement, statically proven ≥1 trip count;
    • hoists one shared configuration (get_ctrl + bit updates + set_ctrl) to the outermost legal loop entry, unwraps covered guards, restores once after the loop — loops with dynamic/possibly-zero trip count, mixed requirements, or explicit CTRL accesses are never hoisted (zero-trip paths never execute a configuration);
    • remaining guards: per-block forward scan sharing one logical entry read, reusing the installed active state for an equal requirement, and deferring restores to the first real observation point (explicit get_ctrl/set_ctrl, a Consume outside a guard, an unknown op, or the block end);
    • guards inside scf.if regions materialize branch-locally.
  • Emission-stage VPTO validation rejects leftover ctrl_state_guard.

Validation

  • 144 (LLVM 19.1.7, assert build): full lit suite 1876 tests / 1875 passed / 0 failed / 1 unsupported (pre-existing environment item). 7 new tests under test/lit/vpto/ctrl/: shared configuration, explicit observation point forcing a restore, requirement switching without intermediate restores, loop hoisting, nested M/K loops (single config at the outermost entry), dynamic zero-trip no-hoist, in-loop boundary no-hoist.
  • 238 (Ascend950PR A5, TileLang 8192³ bf16 manual cube+vector GEMM): numerics identical to the pre-change baseline (max_diff 6.409e-04 PASS on both); emitted VPTO IR for the real kernel drops from 4× get_ctrl + 8× set_ctrl + 16× sbitset inside the innermost loop (28 scalar CTRL ops per K iteration) to 1× get_ctrl + 2× set_ctrl + 4× sbitset hoisted outside three nesting levels with zero CTRL accesses in any loop body.

Closes #1279 (IR-level scalar elimination; issue-case performance regression numbers to be quantified on the 12 reported GEMM shapes as follow-up — the validated manual GEMM is deeply software-pipelined and shows no end-to-end delta, as expected for that kernel class).

Design doc

https://github.com/KurrinQu/PTOAS/blob/docs/vpto-ctrl-state-optimization-design/docs/designs/vpto-ctrl-state-optimization-design.md (implementation phase 1 + phase 2 loop hoisting; scf.if meet-based sharing and cross-block propagation left as future refinement per §6.8 conservative-exit rules)

@jimmychou0
jimmychou0 force-pushed the zjm/ctrl_opt_issue1279 branch 4 times, most recently from ea4e3f8 to 70a2459 Compare September 14, 2026 06:27
…mal accesses

Issue hw-native-sys#1279: every semantic MAD expansion emitted a fixed
get_ctrl -> 4x sbitset -> set_ctrl -> mad_raw -> set_ctrl restore chain,
adding scalar-issue pressure that cost 1-8% on GEMM kernels. CSE cannot
remove the sequence because the ops carry ordered implicit-state effects.

Represent the temporary requirement structurally instead:

- Add StateAccessOpInterface (Query/Write/Consume/Clobber) and
  StateGuardOpInterface with StateResource/StateAccessKind enums;
  get_ctrl/set_ctrl declare Query/Write and the four raw MAD ops declare
  Consume.
- Add internal ctrl_state_guard{controlled_bits, required_bits} wrapping
  the raw MAD (verifier: R subset of C, C nonzero, exactly one CTRL
  Consume implementing MadRawOpInterface, no nesting, func-local).
  VPTOExpandWrapperOps now computes (C,R) statically (HiF8 bit45, TF32
  bit46/47, explicit sat bit48, n_dir bit51) and emits the guard instead
  of the hardware access sequence.
- Add VPTOOptimizeCtrlStatePass after the scheduler: bottom-up scf.for
  summaries (CTRL cleanliness, unique requirement, statically proven
  at-least-once trip count) hoist one shared configuration to the
  outermost legal loop entry, unwrap covered guards, and restore once
  after the loop. Remaining guards use a per-block forward scan sharing
  one logical entry read, reusing an installed active state for an equal
  requirement, and deferring restores to the first real observation
  point (explicit get/set_ctrl, outside Consume, unknown op, block end).
- Emission-stage VPTO validation now rejects leftover ctrl_state_guard.

Validation (144, LLVM 19.1.7 assert build): full lit suite 1876 tests,
1875 passed, 0 failed; new tests cover shared configuration, explicit
observation points, requirement switching, loop hoisting, nested M/K
loops, dynamic zero-trip no-hoist, and in-loop boundary no-hoist.

Validation (238, Ascend950PR, TileLang 8192 bf16 manual GEMM): numerics
identical to baseline (max_diff 6.409e-04); emitted VPTO IR drops from
4x get_ctrl + 8x set_ctrl + 16x sbitset inside the loop nest to a single
configuration hoisted outside three nesting levels with zero CTRL
accesses in any loop body.
@jimmychou0
jimmychou0 force-pushed the zjm/ctrl_opt_issue1279 branch from 70a2459 to 6dcd2d5 Compare September 14, 2026 06:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Performance] PTOAS 每次展开 mad 都生成固定 7 条标量指令,增加了scalar开销。

1 participant