[Example] Optimize Tilus Hopper matmuls to beat cuBLAS - #159
Merged
Conversation
WilliamZhang20
changed the title
WilliamZhang20
changed the title
yaoyaoding
reviewed
Aug 11, 2026
yaoyaoding
left a comment
yaoyaoding
left a comment
Member
There was a problem hiding this comment.
Thanks @WilliamZhang20 ! It's cool.
Could you double check whether the performance change after reverting the change to mbarrier_alloc_ctx.py?
Member
|
/ok to test cd59da3 |
Member
|
added #160 to fix a issue due to tvm-ffi upgrade, which breaks the ci. need rebase to main after merging the PR. |
Member
|
Hi @WilliamZhang20 , can you rebase to the main branch to include the #160 fix? thanks! |
Signed-off-by: William Zhang <wzhang20@yahoo.com>
Signed-off-by: William Zhang <wzhang20@yahoo.com>
Signed-off-by: William Zhang <wzhang20@yahoo.com>
This reverts commit 6e8bbbe. Tutorial docs are deferred to a separate branch/PR to keep this one scoped to the Hopper matmul kernels. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: William Zhang <wzhang20@yahoo.com>
v4 was pinned to num_stages=2 on the theory that a deliberately shallow pipeline would place it between v3 and v5. Measured in a paired benchmark it was 0.1% *slower* than v3, so the two consumer warp groups bought nothing: the consumer is synchronous (wait_group(0) after every commit), and with two stages the producer can be at most one tile ahead, so every consumer drain becomes a producer stall shortly after. Use three stages, and enable the 4-wide tile raster the kernel already implements. On an H100 at 8192^3 fp16, median of three fresh processes: v3 1.93 ms -> v4 1.73 ms -> v5 1.62 ms -> v6 1.36 ms monotonic in 3/3 runs and under NCU. v4 moves 572 -> 637 TFLOP/s, tensor pipe utilization 68 -> 81%, DRAM throughput 35 -> 27%. swizzle_size=4 was picked from a 7-config sweep: it had the tightest run-to-run spread (1.2% vs 7.4% for swizzle 2), and matching v5 keeps v5 attributable purely to its overlapped WGMMA groups. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: William Zhang <wzhang20@yahoo.com>
Restore upstream mbarrier_alloc_ctx.py (NVIDIA#158) and fix the kernels instead. v0-v2 called free_shared() on their staging buffers before the epilogue. Barriers are placed after the whole function is emitted, from the allocator static free list, so that returned a slot the TMA engine writes throughout the loop above and a barrier could land inside a live TMA destination. The kernel still ran and still produced mostly-correct output, but dropped NaNs into ~0.5-1% of the result on some launches and not others. The frees reclaimed nothing (the epilogue allocates no shared memory), so drop them. Verified: 20 runs each of v0/v1/v2 are bit-exact with zero non-finite values, and in the generated CUDA the barrier now sits at byte offset 16384, directly past both 8192-byte staging buffers. No measurable performance change -- v3-v6 never called free_shared, so their codegen is identical either way. Also collapse `with self.single_warp(): with self.single_thread():` to a single `with self.single_thread():` in v4, v5 and v6 (8 sites). Both narrow to thread 0 of the enclosing group, so the outer scope was a no-op. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: William Zhang <wzhang20@yahoo.com>
WilliamZhang20
force-pushed
the
hopper-faster
branch
from
August 11, 2026 21:33
cd59da3 to
a69d42c
Compare
Contributor
Author
|
rebased |
Member
|
/ok to test a69d42c |
Member
|
thank you! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Workload: FP16
8192 x 8192 x 8192GEMM on H100 80 GB HBM3 SXM.