CI: retry arm64 GH200 nightly-standard row + add GB300#2296
Open
leofang wants to merge 9 commits into
Open
Conversation
Reverts the disable in 4c70cfa now that the runner team has fixed the pool-side hang on stream-ordered memory allocator calls.
Remove before merging.
Contributor
Member
Author
|
/ok to test 8d51cf7 |
|
The cufile BAR-size query returns CUDA_ERROR_NOT_SUPPORTED on Grace+Hopper (unified memory, no discrete PCIe BAR). Tracked in NVIDIA#2299; remove this deselect once the test skipif is fixed upstream. Uses PYTEST_ADDOPTS so no changes to run-tests are needed.
Member
Author
|
/ok to test f148214 |
leofang
commented
Jul 13, 2026
leofang
changed the title
Member
Author
|
/ok to test 5bbfa09 |
leofang
added
enhancement
…3 / GB300) -arch=all-major only emits X.0 cubins (sm_100, sm_90, ...), so nvJitLink returns ERROR_INVALID_INPUT when the RDC saxpy fixture is linked on GB300 (sm_103): Blackwell no longer serves an sm_10x cubin from the sm_100 image. -arch=all additionally emits sm_103 (and other minor archs) so the linker finds a usable input. The fixture is built in-job by build_test_binaries.py, so there is no committed artifact to regenerate.
github-actions
Bot
added
the
cuda.core
…deselect gate to 0.4.1 test_fortran_contiguous calls cp.random, which dlopens libcurand at runtime; the mlir nightly install didn't provide it, unlike the numba-cuda branch (which installs cuda-toolkit[curand,cublas] + cupy). Mirror that here. Bump the version-gated mlir deselect from <=0.4.0 to <=0.4.1: 0.4.1 still fails the cuobjdump-invoking nvjitlink test (wheel envs lack cuobjdump) and the NVIDIA#135 pytest-contamination tests, so keep them deselected. cu12 rows remain red due to the separate NVIDIA#2320 LTO-IR regression (not in the deselect list).
Member
Author
|
/ok to test 7f44b44 |
leofang
commented
Jul 13, 2026
| "-dc", | ||
| *nvcc_extra_flags, | ||
| "-arch=all-major", | ||
| "-arch=all", |
Member
Author
There was a problem hiding this comment.
This should fix a bug on GB300 (cc 10.3). Unfortunately, we can't verify this in the PR because of the binary fetch logic.
This reverts commit 8d51cf7.
leofang
added
bug
Member
Author
|
/ok to test 4b61b3c |
leofang
commented
Jul 13, 2026
| # allocator (cudaMallocAsync); runner pool needs fixing first. | ||
| # - { ARCH: 'arm64', PY_VER: '3.14', CUDA_VER: '13.3.0', LOCAL_CTK: '1', GPU: 'gh200', GPU_COUNT: '1', DRIVER: 'latest', ENV: { MODE: 'nightly-standard' } } | ||
| - { ARCH: 'arm64', PY_VER: '3.13', CUDA_VER: '13.3.0', LOCAL_CTK: '1', GPU: 'gh200', GPU_COUNT: '1', DRIVER: 'latest', ENV: { MODE: 'nightly-standard' } } | ||
| - { ARCH: 'arm64', PY_VER: '3.14', CUDA_VER: '13.3.0', LOCAL_CTK: '1', GPU: 'gb300', GPU_COUNT: '1', DRIVER: 'latest', ENV: { MODE: 'nightly-standard' } } |
Member
Author
There was a problem hiding this comment.
These two lines are real, non-cosmetic changes (gh200/gb300).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Re-enable the
gh200nightly-standardrow inci/test-matrix.yml(previously disabled in 4c70cfa because the runner hung oncudaMallocAsync). Runner team indicated the pool-side issue has been fixed, so let's give it another CI-visible run.Also added access to gb300, which exposes a minor issue in the way we generate object/library files for testing.
Also fixed a few nightly CI issues due to the recent numba-cuda-mlir v0.4.1 release