Dockerfile - Add ROCm6.4 dockerfile - #837
Open
polarG wants to merge 5 commits into
Open
Conversation
Add dockerfile/rocm6.4.x.dockerfile based on rocm/pytorch:rocm6.4.4_ubuntu24.04_py3.12_pytorch_release_2.7.1 (ROCm 6.4.4, Python 3.12, torch 2.7.1). Builds RCCL, rocBLAS and hipBLASLt from release(-staging)/rocm-rel-6.4, OpenMPI 4.1.x (--with-rocm), Intel MLC v3.12, OFED user-space via NVIDIA DOCA-Host 3.2.3, Docker CLI 29.6.2, and TransformerEngine pinned to 386bd316. .github/workflows/build-image.yml: add rocm6.4 build matrix entry (superbench/main:rocm6.4) and drop the commented-out rocm6.2 entry. third_party: bump rccl-tests submodule 46375b1 -> 0039629 and update the corresponding version comment in third_party/Makefile.
The hipBLASLt build (step 17/24) failed with 'RuntimeError: Set changed size during iteration' from joblib's Parallel(return_as="generator_unordered") path (joblib issue #1788, fixed upstream in PR #1789 but not yet in a released joblib). Tensile's install.sh creates a fresh virtualenv with 'python -m venv --system-site-packages --clear' and pip-installs tensilelite (pulling an unpatched joblib) into it, then runs the parallel library build that crashes. The previous joblib source sed-patches could not fix this: the base-env 'find /' patch runs before the venv exists, and the post-install 'find /opt' patch was unreachable because it followed './install.sh' in the same '&&' chain (install.sh crashes first) and would be wiped by the venv's --clear on any retry. Fix: patch Tensile's own source (hipBLASLt/tensilelite/**/*.py) to use return_as="generator" instead of "generator_unordered" BEFORE running ./install.sh, which avoids the buggy joblib code path entirely and survives the venv --clear. The sed handles both single- and double-quote styles via a matched backreference. Removed the now-dead post-install joblib find/sed; kept the base-env joblib patch as defense-in-depth.
Without AMDGPU_TARGETS set, hipcc defaults to gfx906 at docker-build time (no GPU present during build), so gpu_copy and other micro-benchmarks were compiled for the wrong GPU arch and ran incorrectly on MI300X: gpu-copy-bw:correctness failed its CheckBuf data check (20/20). Building for gfx942 fixes it (0/20, verified in-container via roc-obj-ls + rebuild). Mirrors the AMDGPU_TARGETS setting in rocm6.3.x.dockerfile.
…LAGS override
When AMDGPU_TARGETS is set (the ROCm 6.4 Dockerfile sets ENV AMDGPU_TARGETS=gfx942), the third_party 'rocm_rccl_tests' target failed to compile rccl-tests with:
/opt/rocm/include/hip/hip_runtime.h: error: ("Must define exactly one of __HIP_PLATFORM_AMD__ or __HIP_PLATFORM_NVIDIA__")
which cascaded into 'unknown type name hipStream_t' / '__host__' errors from rccl.h and 'make ... rocm_rccl_tests Error 2'.
Root cause: the target passes HIPCUFLAGS=... on the rccl-tests make command line. A command-line variable assignment overrides (does not append to) every 'HIPCUFLAGS +=' inside rccl-tests/src/Makefile, so the platform flags it normally adds ('-x hip -D__HIP_PLATFORM_AMD__ -D__HIPCC__') were dropped. The compiler is amdclang++ (not the hipcc wrapper), which does not auto-define __HIP_PLATFORM_AMD__, so hip_runtime.h aborted before any HIP type was declared.
Fix: re-add '-x hip -D__HIP_PLATFORM_AMD__ -D__HIPCC__' to the HIPCUFLAGS override so it matches what rccl-tests/src/Makefile supplies by default, and document why in the comment. The empty-AMDGPU_TARGETS branch is unchanged. Verified with 'make -n rocm_rccl_tests AMDGPU_TARGETS=gfx942'.
Contributor
There was a problem hiding this comment.
Pull request overview
Adds a new ROCm 6.4.4 SuperBench container image (targeting MI300X/gfx942) and wires it into the CI image build matrix, plus adjusts third-party build flags to keep rccl-tests compiling correctly when AMDGPU_TARGETS is set.
Changes:
- Add
dockerfile/rocm6.4.x.dockerfileto build a ROCm 6.4.4 Ubuntu 24.04 / Python 3.12 image with gfx942 targeting and updated dependency/tooling installs. - Update
third_party/Makefilerccl-tests build to preserve required HIP platform defines when overridingHIPCUFLAGS. - Register the
rocm6.4image in.github/workflows/build-image.yml.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
third_party/Makefile |
Updates rccl-tests build flags to stay compatible with AMDGPU_TARGETS overrides under ROCm 6.4. |
dockerfile/rocm6.4.x.dockerfile |
New ROCm 6.4.4 image definition targeting MI300X (gfx942) and updated build/install steps. |
.github/workflows/build-image.yml |
Adds rocm6.4 to the image build matrix for CI. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+72
to
+76
| # Check if CMake is installed and its version | ||
| RUN cmake_version=$(cmake --version 2>/dev/null | grep -oP "(?<=cmake version )(\d+\.\d+)" || echo "0.0") && \ | ||
| required_version="3.24.1" && \ | ||
| if [ "$(printf "%s\n" "$required_version" "$cmake_version" | sort -V | head -n 1)" != "$required_version" ]; then \ | ||
| echo "existing cmake version is ${cmake_version}" && \ |
Comment on lines
+140
to
+142
| ldconfig && \ | ||
| cd / && \ | ||
| rm -rf /tmp/openmpi-${OPENMPI_VERSION}* |
Comment on lines
+199
to
+201
| RUN pip install "joblib>=1.4.2" && \ | ||
| find / -path '*/joblib/parallel.py' -not -path '*/.git/*' -exec sed -i \ | ||
| 's/timeout_control_job = next(iter(self\._jobs_set), None)/timeout_control_job = next(iter(set(self._jobs_set)), None)/' {} + |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #837 +/- ##
=======================================
Coverage 86.02% 86.02%
=======================================
Files 103 103
Lines 7950 7950
=======================================
Hits 6839 6839
Misses 1111 1111
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
torch pulls in an unpinned setuptools during 'pip install .[test,cpuworker]', which upgrades the 65.7 pinned by the pipeline to 83.0.0 and makes the setup.py guard raise VersionConflict on the python-3.10 lint job. setuptools 66+ only drops Python 3.7, so keep the cap there only.
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 5 out of 5 changed files in this pull request and generated no new comments.
Suppressed comments (3)
dockerfile/rocm6.4.x.dockerfile:74
- The CMake version parsing only captures major.minor (e.g., "3.24"), so patch versions like 3.24.2 are treated as 3.24 and will incorrectly trigger a full CMake source build. This can significantly slow builds and adds avoidable variability.
# Check if CMake is installed and its version
RUN cmake_version=$(cmake --version 2>/dev/null | grep -oP "(?<=cmake version )(\d+\.\d+)" || echo "0.0") && \
required_version="3.24.1" && \
dockerfile/rocm6.4.x.dockerfile:142
- This cleanup command removes "/tmp/openmpi-${OPENMPI_VERSION}*", but the OpenMPI source tree is cloned into "/tmp/ompi". As written, the build directory is not deleted and will bloat the final image.
make -j $(nproc) install && \
ldconfig && \
cd / && \
rm -rf /tmp/openmpi-${OPENMPI_VERSION}*
dockerfile/rocm6.4.x.dockerfile:201
- Running
find /during docker build will traverse the entire filesystem and can be very slow (and occasionally problematic on special mount points). Since joblib is a Python package, you can locate its installedparallel.pyvia Python and patch just that file.
RUN pip install "joblib>=1.4.2" && \
find / -path '*/joblib/parallel.py' -not -path '*/.git/*' -exec sed -i \
's/timeout_control_job = next(iter(self\._jobs_set), None)/timeout_control_job = next(iter(set(self._jobs_set)), None)/' {} +
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Adds a ROCm 6.4.4 SuperBench Docker image targeting AMD MI300X (gfx942), based on #812, plus fixes discovered while building and verifying the image end-to-end on MI300X hardware.
Major Revision
dockerfile/rocm6.4.x.dockerfile— new ROCm 6.4.4 image (Ubuntu 24.04, Python 3.12, PyTorch 2.7.1, gfx942), baserocm/pytorch:rocm6.4.4_ubuntu24.04_py3.12_pytorch_release_2.7.1.doca-ofed-userspace=3.2.3-019000), matching the target host's OFED version; MLNX_OFED standalone tarballs are gone upstream.29.6.2(extracts only thedockerclient binary, dropping dockerd/containerd/runc/shims to shrink the CVE surface — SuperBench only shells out to the Docker client, never runs its own daemon).flexto the apt package list (fixes an OpenMPI "developer build"/autogen.plfailure that requires it).AMDGPU_TARGETS=gfx942explicitly (previously unset, a regression vsrocm6.3.x.dockerfile) — without a GPU present atdocker buildtime, hipcc silently defaulted togfx906, which caused incorrect zero-copy reads /gpu-copy-bw:correctnessfailures on MI300X.third_party/rccl-testssubmodule to0039629(adds a missing<cstring>include required for ROCm 6.4's HIP headers, Add cstring header explictly as it is removed from HIP ROCm/rccl-tests#132).third_party/Makefilerccl-tests build: repeat the HIP platform defines (-x hip -D__HIP_PLATFORM_AMD__ -D__HIPCC__) inside theHIPCUFLAGSoverride, since settingAMDGPU_TARGETScauses the override to replace (not append to) rccl-tests' ownHIPCUFLAGS, which otherwise drops those defines and breaks the HIP headers.rocm6.4in.github/workflows/build-image.yml.Minor Revision
if/elseshell fix (missing;beforeelse).rm -rf .gitcleanup in the finalmake postinstallstep (image size).apt-get cleanin the apt/OFED install blocks.Verification
A full
sb runwas executed on an 8x MI300X host: 460/462 executor runs passed. The 3 failures (resnetrendezvous port collision,gpu-copy-bw:perfOOM,gpu-copy-bw:correctness) were root-caused; the correctness failure was theAMDGPU_TARGETSissue above, confirmed in-container with the full correctness suite passing 321/321 after rebuilding withgfx942. The other two failures are fixed outside this Dockerfile (runner/config changes) and are not blockers for this PR.