Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
35 changes: 35 additions & 0 deletions .github/workflows/nfd-device-id-check.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
name: nfd-device-id-check

# Verifies that every copy of the AMD GPU PCI device-ID list agrees with the
# source of truth, hack/k8s-patch/template-patch/gpu-nfd-default-rule.yaml.
#
# The list is duplicated across the NodeFeatureRule template, its generated
# helm copy, and two copy-paste YAML blocks in the OpenShift install docs.
# Nothing in the build enforces agreement and they have already diverged once:
# commit 9c5ef17e added the Radeon AI PRO R9700 (0x7551) to the rule but not to
# the docs, so OLM users on Radeon hardware received no GPU labels at all and
# every test failed with "No nodes with AMD/GPU found" (GPUOP-1062).
#
# This is a thin caller of the Makefile target so that contributors get the
# identical result locally via 'make check-nfd-device-ids'.
#
# Deliberately not filtered on paths: a path-filtered workflow never reports a
# status on unrelated PRs, which stalls merges if it is configured as a
# required status check. The check takes about a second, so it runs on every PR.

on:
pull_request:

permissions:
contents: read

jobs:
device-id-drift:
name: AMD GPU device-ID lists in sync
runs-on: ubuntu-latest
steps:
- name: Check out PR head
uses: actions/checkout@v4

- name: Check device-ID lists
run: make check-nfd-device-ids
4 changes: 4 additions & 0 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -345,6 +345,10 @@ docs-lint: ## Run docs Markdown lint + spelling (full ROCm-style docs lint).
${MAKE} docs-lint-markdown
${MAKE} docs-lint-spelling

.PHONY: check-nfd-device-ids
check-nfd-device-ids: ## Verify the AMD GPU PCI device-ID lists agree across the NFD rule templates and the OpenShift docs.
python3 $(shell pwd)/hack/check-nfd-device-ids.py --root $(shell pwd)

##@ Build

manager: $(shell find -name "*.go") go.mod go.sum ## Build manager binary.
Expand Down
50 changes: 46 additions & 4 deletions docs/installation/openshift-olm.md
Original file line number Diff line number Diff line change
Expand Up @@ -295,7 +295,25 @@ spec:
"7408", # MI250X
"740c", # MI250/MI250X
"738c", # MI100
"738e" # MI100
"738e", # MI100
# AMD Radeon Pro
"7460", # V710
"7448", # W7900
"744b", # W7900D
"744a", # W7900 Dual Slot
"7449", # W7800 48GB
"745e", # W7800
"73a2", # W6900X
"73a3", # W6800 GL-XL
"73ab", # W6800X / W6800X Duo
"73a1", # V620
"7551", # AI PRO R9700 / R9700S / R9600D
# AMD Radeon
"7550", # RX 9070 / 9070 XT
"744c", # RX 7900 XT / 7900 XTX / 7900 GRE / 7900M
"73af", # RX 6900 XT
"73bf", # RX 6800 / 6800 XT / 6900 XT
"7590" # RX 9060 XT
]}
- name: amd-vgpu
labels:
Expand All @@ -314,7 +332,10 @@ spec:
"74bc", # MI308X-HF VF
"74b5", # MI300X VF
"74bd", # MI300X-HF VF
"7410" # MI210 VF
"7410", # MI210 VF
# AMD Radeon Pro
"7461", # V710 MxGPU
"73ae" # V620 MxGPU
]}
```

Expand Down Expand Up @@ -352,7 +373,25 @@ spec:
"7408", # MI250X
"740c", # MI250/MI250X
"738c", # MI100
"738e" # MI100
"738e", # MI100
# AMD Radeon Pro
"7460", # V710
"7448", # W7900
"744b", # W7900D
"744a", # W7900 Dual Slot
"7449", # W7800 48GB
"745e", # W7800
"73a2", # W6900X
"73a3", # W6800 GL-XL
"73ab", # W6800X / W6800X Duo
"73a1", # V620
"7551", # AI PRO R9700 / R9700S / R9600D
# AMD Radeon
"7550", # RX 9070 / 9070 XT
"744c", # RX 7900 XT / 7900 XTX / 7900 GRE / 7900M
"73af", # RX 6900 XT
"73bf", # RX 6800 / 6800 XT / 6900 XT
"7590" # RX 9060 XT
]}
- name: amd-vgpu
labels:
Expand All @@ -371,7 +410,10 @@ spec:
"74bc", # MI308X-HF VF
"74b5", # MI300X VF
"74bd", # MI300X-HF VF
"7410" # MI210 VF
"7410", # MI210 VF
# AMD Radeon Pro
"7461", # V710 MxGPU
"73ae" # V620 MxGPU
]}
```

Expand Down
25 changes: 25 additions & 0 deletions docs/troubleshooting.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,31 @@ To collect logs from the AMD GPU Operator:
kubectl logs -n kube-amd-gpu <pod-name>
```

## No AMD GPU Nodes Detected

If the operator deploys no pods, or tooling reports that no AMD GPU nodes were found in the cluster, the nodes are most likely missing the NFD label that the operator selects on.

Check whether the label is present on the node:

```bash
kubectl get node <node-name> -o jsonpath='{.metadata.labels}' | tr ',' '\n' | grep amd-gpu
```

That label is produced only by an NFD rule that you supply. Two common reasons for it to be missing:

1. **No rule was applied.** On OpenShift the OLM bundle does not include a `NodeFeatureDiscovery` or `NodeFeatureRule` resource, so nothing labels the node until you create one. Follow the [OpenShift installation guide](./installation/openshift-olm) and apply the NFD resource from that page in full.

2. **The rule does not list your GPU's device ID.** Find the device ID of the installed GPU and compare it against the rule you applied:

```bash
lspci -nn -d 1002:
kubectl get nodefeaturerule -A -o yaml
```

If your device ID is absent, add it and re-apply the resource.

On Kubernetes the Helm chart installs the full device list for you when `installdefaultNFDRule` is enabled.

## Potential Issues with ``DeviceConfig``

* Please refer to {ref}`typical-deployment-scenarios` for more information and get corresponding ```helm install``` commands and configs that fits your specific use case.
Expand Down
Loading
Loading