Skip to content

[Fix] Add CUDA-enabled Spark image - #312

Merged
assaf758 merged 1 commit into
mlrun:developmentfrom
gtopper:ORIS-4194-cuda
Aug 24, 2026
Merged

[Fix] Add CUDA-enabled Spark image#312
assaf758 merged 1 commit into
mlrun:developmentfrom
gtopper:ORIS-4194-cuda

Conversation

@gtopper

@gtopper gtopper commented Aug 23, 2026

Copy link
Copy Markdown
Collaborator

📝 Description

Adds the CUDA-enabled Spark image required when MLRun requests GPU resources for Spark drivers or executors.

The image matches the existing CE Spark 3.5.6 stack while adding CUDA 12.8.1 and cuDNN 9.8.


🛠️ Changes Made

  • Added docker/spark/Dockerfile.cuda based on nvidia/cuda:12.8.1-cudnn-devel-ubuntu22.04.
  • Shared Python 3.11, connector JAR, and filesystem setup through docker/spark/scripts/ce-customize.sh.
  • Added CUDA build and validation targets to docker/spark/Makefile.
  • Added local checks for Spark, Scala, Java, Python, connector JARs, CUDA, cuDNN, and NVIDIA container metadata.
  • Documented image build, validation, and manual GCR publication in docker/spark/README.md.
  • Bumped the chart version from 0.12.0-rc.10 to 0.12.0-rc.11.

✅ Checklist

  • I have tested the changes in this PR
  • I confirmed whether my changes require a change in documentation and if so, I created another PR in MLRun for the relevant documentation.
  • I confirmed whether my changes require a changes in QA tests, for example: credentials changes, resources naming change and if so, I updated the relevant Jira ticket for QA.
  • I increased the Chart version in charts/mlrun-ce/Chart.yaml.
  • I confirmed that the installation works both on a local Docker Desktop environment and on a real cluster when using the required prerequisites.
    • If installation issues were found, I updated the relevant Jira ticket with the issue and steps to reproduce, or updated the prerequisites documentation if the issue is related to missing or outdated prerequisites.
  • If needed, update https://github.com/mlrun/ce/blob/development/charts/mlrun-ce/README.md with the relevant installation instructions and version Matrix.
  • If needed, update the following values files for multi namespace support:

🧪 Testing

  • Built the linux/amd64 CUDA image with make -C docker/spark build-cuda.
  • Ran make -C docker/spark validate-cuda.
  • Verified Spark 3.5.6, Scala 2.12, Java 17, Python 3.11, connector JARs, CUDA 12.8.1, cuDNN 9.8, the Spark entrypoint, and NVIDIA container metadata.
  • GPU device visibility and Spark driver/executor startup still require validation in an ORIS GPU environment.

🔗 References


🚨 Breaking Changes?

  • Yes (explain below)
  • No

🔍️ Additional Notes

  • The CUDA image is published manually because this repository has no provisioned GCR publishing credentials.
  • Adding the image to the Mlefi release manifest and validating the GPU Spark scenario are follow-up tasks.

### 📝 Description

Adds the CUDA-enabled Spark image required when MLRun requests GPU resources for Spark drivers or executors.

The image matches the existing CE Spark 3.5.6 stack while adding CUDA 12.8.1 and cuDNN 9.8.

---

### 🛠️ Changes Made

- Added `docker/spark/Dockerfile.cuda` based on `nvidia/cuda:12.8.1-cudnn-devel-ubuntu22.04`.
- Shared Python 3.11, connector JAR, and filesystem setup through `docker/spark/scripts/ce-customize.sh`.
- Added CUDA build and validation targets to `docker/spark/Makefile`.
- Added local checks for Spark, Scala, Java, Python, connector JARs, CUDA, cuDNN, and NVIDIA container metadata.
- Documented image build, validation, and manual GCR publication in `docker/spark/README.md`.
- Bumped the chart version from `0.12.0-rc.10` to `0.12.0-rc.11`.

---

### ✅ Checklist

- [x] I have tested the changes in this PR
- [x] I confirmed whether my changes require a change in documentation and if so, I created another PR in MLRun for the relevant documentation.
- [ ] I confirmed whether my changes require a changes in QA tests, for example: credentials changes, resources naming change and if so, I updated the relevant Jira ticket for QA.
- [x] I increased the Chart version in `charts/mlrun-ce/Chart.yaml`.
- [ ] I confirmed that the installation works both on a local Docker Desktop environment and on a real cluster when using the required [prerequisites](https://docs.mlrun.org/en/stable/install-mlrun-ce/kubernetes-install.html#prerequisites).
  - [ ] If installation issues were found, I updated the relevant Jira ticket with the issue and steps to reproduce, or updated the prerequisites documentation if the issue is related to missing or outdated prerequisites.
- [x] If needed, update https://github.com/mlrun/ce/blob/development/charts/mlrun-ce/README.md with the relevant installation instructions and version Matrix.
- [x] If needed, update the following values files for multi namespace support:
  - [x] [Admin values](https://github.com/mlrun/ce/blob/development/charts/mlrun-ce/admin_installation_values.yaml)
  - [x] [User values Node Port](https://github.com/mlrun/ce/blob/development/charts/mlrun-ce/non_admin_installation_values.yaml)
  - [x] [User values ClusterIP](https://github.com/mlrun/ce/blob/development/charts/mlrun-ce/non_admin_cluster_ip_installation_values.yaml)

---

### 🧪 Testing

- Built the `linux/amd64` CUDA image with `make -C docker/spark build-cuda`.
- Ran `make -C docker/spark validate-cuda`.
- Verified Spark 3.5.6, Scala 2.12, Java 17, Python 3.11, connector JARs, CUDA 12.8.1, cuDNN 9.8, the Spark entrypoint, and NVIDIA container metadata.
- GPU device visibility and Spark driver/executor startup still require validation in an ORIS GPU environment.

---

### 🔗 References

- Ticket link: ORIS-4194
- External links: [Apache Spark Docker images](https://github.com/apache/spark-docker), [NVIDIA CUDA container images](https://hub.docker.com/r/nvidia/cuda)
- Design docs links (Optional): N/A

---

### 🚨 Breaking Changes?

- [ ] Yes (explain below)
- [x] No

---

### 🔍️ Additional Notes

- The CUDA image is published manually because this repository has no provisioned GCR publishing credentials.
- Adding the image to the Mlefi release manifest and validating the GPU Spark scenario are follow-up tasks.
@github-actions github-actions Bot added the bug Something isn't working label Aug 23, 2026
@assaf758
assaf758 merged commit 1c414dd into mlrun:development Aug 24, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants