[CI] Fix old NVCC dependency in GH CI

[fheinecke]

Description

CI is failing because the NVCC version in the CUDA container image used is too old. Newer container images don't include a sufficiently new version of NVCC.

This installs the dependency via apt. This is less than ideal, but it unblocks CI while I work on a better solution.

Note: this fixes the fatal error: nccl_device.h: No such file or directory compilation issue, but pipelines still won't pass due to issues farther down in the build process (compilation choking out the runner agent process, runners failing to compile everything within 6 hour limit, etc).

Type of change

• Documentation change (change only to the documentation, either a fix or a new content)
• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• Infra/Build change
• Code refactoring

Changes

Please list the changes introduced in this PR:

• Install nvcc via apt, bypassing old nvcc version in base image.

• Fixed setup installing python3-llfuse instead of just python

  $ docker run --rm -it --entrypoint sh nvcr.io/nvidia/cuda:12.9.2-devel-ubuntu22.04 -c "apt update > /dev/null && apt-cache search python3.9"
  
  WARNING: apt does not have a stable CLI interface. Use with caution in scripts.
  
  python3-llfuse - Python 3 bindings for the low-level FUSE API

Checklist:

• I have read and followed the contributing guidelines
• The functionality is complete
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• My changes generate no new warnings
• I have added tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes

[greptile-apps]

Greptile Summary

This PR unblocks CI by upgrading the CUDA container image to 12.9.2-devel-ubuntu22.04 in the core and pytorch jobs, fixing the nccl_device.h: No such file or directory compile error by explicitly installing pinned NCCL packages via apt, and correcting the Python package names (python3 / python3-pip instead of python3.9 / pip).

• Container images bumped from 12.1.0 (core) and 12.8.0 (pytorch) to 12.9.2; both jobs now add apt-mark unhold libnccl2 libnccl-dev before installing libnccl2=2.30.7-1+cuda12.9 and libnccl-dev=2.30.7-1+cuda12.9.
• Python apt packages corrected from python3.9 pip to python3 python3-pip, matching what is available in the newer Ubuntu 22.04 CUDA image.
• The author notes this is an interim fix; pipelines still won't fully pass due to compilation time limits on the runners.

Confidence Score: 4/5

Safe to merge as an unblocking CI fix; the hardcoded NCCL package version is the only maintenance risk.

The changes are straightforward container and package updates. The only concern is the hardcoded libnccl2=2.30.7-1+cuda12.9 pin duplicated across two jobs — if that exact version disappears from the apt repo, both jobs break at the install step. No logic changes to the build itself; the Python package name corrections are unambiguously right.

.github/workflows/build.yml — the duplicated NCCL version pins in the core and pytorch jobs are the only spot that warrants a follow-up once a longer-term solution is in place.

Important Files Changed

Filename	Overview
.github/workflows/build.yml	Updates CUDA container images to 12.9.2, fixes Python package names (python3/python3-pip), and adds apt-pinned NCCL packages with unhold to resolve nccl_device.h compilation failures; hardcoded NCCL version is a future maintenance concern

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[PR Trigger] --> B[core job\ncuda:12.9.2-devel-ubuntu22.04]
    A --> C[pytorch job\ncuda:12.9.2-devel-ubuntu22.04]
    A --> D[jax job\nghcr.io/nvidia/jax:jax]
    A --> E[all job\nghcr.io/nvidia/jax:jax]

    B --> B1[apt-mark unhold libnccl2 libnccl-dev]
    B1 --> B2[apt-get install git python3 python3-pip\ncudnn9-cuda-12 libnccl2=2.30.7 libnccl-dev=2.30.7]
    B2 --> B3[pip install cmake pybind11 ninja cudnn-frontend]
    B3 --> B4[Checkout + sccache]
    B4 --> B5[pip install TE - NVTE_FRAMEWORK=none]
    B5 --> B6[Sanity check: import transformer_engine]

    C --> C1[docker run cuda:12.9.2 container]
    C1 --> C2[apt-mark unhold libnccl2 libnccl-dev]
    C2 --> C3[apt-get install git python3 python3-pip\ncudnn9-cuda-12 libnccl2=2.30.7 libnccl-dev=2.30.7]
    C3 --> C4[pip install cmake torch ninja...]
    C4 --> C5[pip install TE - NVTE_FRAMEWORK=pytorch]
    C5 --> C6[Sanity check: test_sanity_import.py]

    D --> D1[pip install cmake pybind11 cudnn-frontend]
    D1 --> D2[pip install TE - NVTE_FRAMEWORK=jax]
    D2 --> D3[Sanity check: jax test_sanity_import.py]

    E --> E1[docker run jax container]
    E1 --> E2[pip install cmake torch cudnn-frontend...]
    E2 --> E3[pip install TE - NVTE_FRAMEWORK=all]
    E3 --> E4[Sanity check: pytorch + jax]

Loading

_{Reviews (1): Last reviewed commit: "[CI] Fix old NVCC dependency in GH CI" | Re-trigger Greptile}