From 1fbae7b25c34a40d48fefdab2247ce856f7c4fb7 Mon Sep 17 00:00:00 2001 From: Kevin Wang Date: Thu, 16 Jul 2026 03:16:17 -0700 Subject: [PATCH 01/32] fix(guest): bind GPU attestation to measured launch state --- docs/security/security-model.md | 28 +- dstack/dstack-util/src/system_setup.rs | 367 ++++++++++++++++-- .../files/10-nvidia-gpu-ordering.conf | 2 +- .../files/dstack-gpu-attestation.rego | 29 ++ .../nvattest/nvattest_2026.06.09.bb | 12 +- .../nvidia/files/nvidia-persistenced.service | 2 +- 6 files changed, 403 insertions(+), 37 deletions(-) create mode 100644 os/yocto/layers/meta-nvidia/recipes-graphics/nvattest/files/dstack-gpu-attestation.rego diff --git a/docs/security/security-model.md b/docs/security/security-model.md index b85a83bf3..fa291c10e 100644 --- a/docs/security/security-model.md +++ b/docs/security/security-model.md @@ -8,7 +8,7 @@ This document helps you evaluate whether dstack's security model fits your needs dstack removes the need to trust most infrastructure operators. On TEE platforms such as Intel TDX and AMD SEV-SNP, the cloud or host operator cannot read your protected memory, modify your measured code, or access your secrets without detection. On AWS EC2 NitroTPM attested instances, AWS Nitro is part of the trusted platform, and the untrusted party is the workload AWS account administrator/operator. Network attackers cannot intercept your traffic because TLS terminates inside the attested environment with keys controlled by that environment (Zero Trust HTTPS). Docker registries cannot serve malicious images because the guest verifies SHA256 digests before pulling. -The primary trust input is **attested platform hardware**. Intel TDX is the production TEE path. AMD SEV-SNP is available where the selected dstack OS image and host support it, but it is new and experimental. AWS EC2 NitroTPM attested instances use a different trust root: the AWS Nitro system and AWS NitroTPM attestation PKI. In that mode, the threat model protects against the workload AWS account administrator and EC2 operator actions, but AWS remains trusted. For GPU workloads, you also trust **NVIDIA GPU hardware** and NVIDIA's Remote Attestation Service (NRAS). These are hardware-level trust assumptions. +The primary trust input is **attested platform hardware**. Intel TDX is the production TEE path. AMD SEV-SNP is available where the selected dstack OS image and host support it, but it is new and experimental. AWS EC2 NitroTPM attested instances use a different trust root: the AWS Nitro system and AWS NitroTPM attestation PKI. In that mode, the threat model protects against the workload AWS account administrator and EC2 operator actions, but AWS remains trusted. For GPU workloads, you also trust **NVIDIA GPU hardware**, NVIDIA's attestation PKI/RIMs, and its revocation service (or NRAS when a deployment selects remote verification). These are hardware-level trust assumptions. Everything else is verifiable. @@ -55,11 +55,27 @@ dstack supports NVIDIA H100, H200, and B200 GPUs in confidential compute mode fo ### How It Works -GPUs are passed through via VFIO directly to the TEE-protected CVM. The GPU operates in confidential compute mode, encrypting data during computation. Both the CPU TEE and NVIDIA GPU provide hardware isolation together. If either component fails verification, the security model breaks. +GPUs are passed through via VFIO to the TEE-protected CVM. Before key provisioning, `dstack-util setup` inventories every VGA/3D-controller PCI function, compares that count with `vm_config.num_gpus`, and runs NVIDIA's local `nvattest` verifier over every NVIDIA-driver-visible GPU with a fresh nonce. A mandatory relying-party policy requires secure boot and disabled debug status. Separate fail-closed `nvidia-smi` queries require the current CC feature to be ON and DevTools mode to be OFF. Only then does dstack set the GPU ready state. ### Dual Attestation -GPU workloads require verification of both hardware components. The CPU TEE provides the quote that verifies CPU and memory isolation. NVIDIA's Remote Attestation Service (NRAS) independently verifies the GPU is genuine and running in confidential mode. Both attestations must pass for complete verification. +GPU workloads require verification of both hardware components. The CPU TEE quote verifies the CVM and its measured guest code. NVIDIA-signed evidence, checked against NVIDIA RIMs and certificate status by `nvattest`, verifies the GPU appraisal. After the ready-state operation succeeds, dstack emits a `gpu-attestation` launch event before `system-ready`. Its versioned payload records the number of appraised devices, policy version, asserted CC/DevTools state, and SHA-256 of the complete nvattest JSON output (claims and detached EAT). On TDX this event is append-only in RTMR3; it is never derived from application-controlled `report_data`. + +A verifier must replay the measured event log, require exactly one pre-`system-ready` `gpu-attestation` event when GPU protection is required, require `devices == vm_config.num_gpus > 0`, and accept the named policy version. The raw file under `/run/nvidia-gpu-attestation/attestation.out` is not trusted by itself; if it is supplied for inspection, its digest must match the event. + +### GPU Threat Model and Lifetime + +The GPU gate assumes a malicious host/VMM and untrusted host-provided PCI topology, while trusting the CPU TEE, the measured dstack guest/kernel, NVIDIA hardware/firmware roots, and the cryptography used by both attestation chains. Availability is out of scope. + +The event makes the following **boot-time** statement: immediately before key provisioning, all attached VGA/3D PCI functions were NVIDIA devices, their count matched the quote-bound VM configuration, nvattest returned one fresh production-policy claim for each device, CC was ON, DevTools was OFF, and setting the GPU ready state succeeded. This closes these cases: + +- A GPU-less launch cannot be presented as a GPU-verified launch: it has `num_gpus == 0` and no `gpu-attestation` event. +- A mixed launch cannot attest only its TEE-capable subset. Non-NVIDIA display GPUs are rejected, and the sysfs, `vm_config`, and nvattest claim counts must all agree. A non-CC NVIDIA GPU either prevents evidence collection/appraisal or causes that count/policy check to fail. +- Copying another CVM's result into a file or `report_data` does not work. Only measured pre-application code can place the event before `system-ready`, and event-log replay binds it to the quoted RTMR/PCR value. + +This is **not a lifetime or physical co-location guarantee**. After `system-ready`, an application with sufficient guest privileges can unload the NVIDIA driver, and a malicious host may attempt PCI hot-remove/replacement or proxy GPU traffic. The boot event remains a true historical statement but does not prove that the same device is still attached. dstack also cannot rule out a live relay/cuckoo attack to a genuine remote GPU: current Hopper/Blackwell deployments do not provide a CPU-TEE-verifiable TEE-I/O/TDISP device binding. Applications that mutate the driver or PCI topology are outside this guarantee; higher-assurance deployments must prevent that behavior and re-attest before using a newly initialized GPU. + +AMD SEV-SNP has no runtime measurement register in the current dstack stack. The local boot gate can still fail closed, but a `gpu-attestation` event carried beside an SNP report is not remotely bound to that report and must not be accepted as dual-attestation evidence. SNP needs a measured vTPM/PCR channel before it can provide the same remote binding. ### AI Workload Protection @@ -156,6 +172,12 @@ Use this checklist to verify a workload running in a dstack CVM. - [ ] Config commitment matches the expected app/config target (on AWS: PCR14 replay; PCR8 is an optional shortcut — see the [AWS verifier runbook](../aws-ec2-production-verifier-runbook.md)) - [ ] reportData contains your challenge (replay protection) +**GPU verification (when required):** +- [ ] `vm_config.num_gpus` is greater than zero and matches the expected deployment +- [ ] Exactly one `gpu-attestation` event appears before `system-ready` +- [ ] The event device count matches `vm_config.num_gpus`, and its policy/CC/DevTools fields match the production policy +- [ ] The platform binds the event log to a quoted RTMR/PCR (do not accept it from current SEV-SNP evidence) + **Key management verification:** - [ ] key-provider matches expected KMS identity - [ ] KMS attestation is valid diff --git a/dstack/dstack-util/src/system_setup.rs b/dstack/dstack-util/src/system_setup.rs index 73823d254..be6652fd5 100644 --- a/dstack/dstack-util/src/system_setup.rs +++ b/dstack/dstack-util/src/system_setup.rs @@ -1080,28 +1080,90 @@ mod gpu { const NVATTEST: &str = "/usr/bin/nvattest"; const NVIDIA_SMI: &str = "/usr/bin/nvidia-smi"; + const NVIDIA_POLICY: &str = "/usr/share/nvattest/dstack-gpu-attestation.rego"; const ATTESTATION_OUTPUT: &str = "/run/nvidia-gpu-attestation/attestation.out"; const ATTESTATION_TIMEOUT: Duration = Duration::from_secs(300); const NVIDIA_SMI_TIMEOUT: Duration = Duration::from_secs(60); + const EVENT_VERSION: u32 = 1; + const POLICY_VERSION: &str = "nvidia-cc-production-v1"; - /// True if a passed-through NVIDIA GPU is present, detected via sysfs PCI - /// (vendor 0x10de, class VGA 0x0300xx or 3D controller 0x0302xx) so it - /// works even before the nvidia driver is loaded. Fails (rather than - /// reporting "no GPU") when the PCI bus cannot be enumerated, so a broken - /// /sys cannot bypass the attestation gate. - pub(super) fn nvidia_gpu_present() -> Result { - let entries = - fs::read_dir("/sys/bus/pci/devices").context("failed to enumerate PCI devices")?; - Ok(entries.filter_map(|e| e.ok()).any(|dev| { - let read = |name: &str| { - fs::read_to_string(dev.path().join(name)) - .unwrap_or_default() - .trim() - .to_string() - }; - read("vendor") == "0x10de" - && matches!(read("class").get(..6), Some("0x0300") | Some("0x0302")) - })) + #[derive(Debug, Clone, Copy, PartialEq, Eq)] + pub(super) struct GpuInventory { + pub(super) total: u32, + pub(super) nvidia: u32, + } + + #[derive(Debug, Serialize)] + struct GpuAttestationEvent { + version: u32, + provider: &'static str, + devices: u32, + policy: &'static str, + cc_mode: &'static str, + devtools: bool, + evidence_sha256: String, + } + + #[derive(Deserialize)] + struct NvattestOutput { + result_code: i64, + claims: Vec, + } + + /// Count passed-through display-class GPUs through sysfs so devices which + /// the NVIDIA driver did not bind cannot be hidden from the gate. Reading + /// the inventory is fail-closed: a mixed NVIDIA/non-NVIDIA set must not be + /// represented by an attestation result for only the NVIDIA subset. + pub(super) fn gpu_inventory() -> Result { + gpu_inventory_at(Path::new("/sys/bus/pci/devices")) + } + + fn gpu_inventory_at(devices_path: &Path) -> Result { + let entries = fs::read_dir(devices_path).context("failed to enumerate PCI devices")?; + let mut inventory = GpuInventory { + total: 0, + nvidia: 0, + }; + for entry in entries { + let device = entry.context("failed to read PCI device entry")?; + let class_path = device.path().join("class"); + let class = fs::read_to_string(&class_path) + .with_context(|| format!("failed to read {}", class_path.display()))?; + if !matches!(class.trim().get(..6), Some("0x0300") | Some("0x0302")) { + continue; + } + inventory.total += 1; + let vendor_path = device.path().join("vendor"); + let vendor = fs::read_to_string(&vendor_path) + .with_context(|| format!("failed to read {}", vendor_path.display()))?; + if vendor.trim() == "0x10de" { + inventory.nvidia += 1; + } + } + Ok(inventory) + } + + /// Bind the runtime inventory to `vm_config.num_gpus`. On TDX this config + /// affects the measured launch layout, so a remote verifier can compare the + /// event's device count with the same quote-bound value. + pub(super) fn expected_gpu_count(vm_config: &str, inventory: GpuInventory) -> Result { + if inventory.total != inventory.nvidia { + bail!( + "unsupported non-NVIDIA GPU attached: found {} display GPUs, {} NVIDIA", + inventory.total, + inventory.nvidia + ); + } + let vm_config: dstack_types::VmConfig = serde_json::from_str(vm_config) + .context("failed to parse vm_config for GPU attestation")?; + let expected = vm_config.num_gpus; + if inventory.total != expected { + bail!( + "gpu count mismatch: vm_config requires {expected}, found {}", + inventory.total + ); + } + Ok(expected) } /// Run a GPU tool with a bounded timeout so a wedged driver/GPU cannot @@ -1121,6 +1183,50 @@ mod gpu { .with_context(|| format!("failed to run {program}")) } + fn parse_nvidia_smi_state(stdout: &[u8], name: &str) -> Result { + let output = std::str::from_utf8(stdout) + .with_context(|| format!("nvidia-smi returned non-UTF-8 {name}"))?; + let states = output + .split(|ch: char| !ch.is_ascii_alphanumeric()) + .filter_map(|word| match word.to_ascii_lowercase().as_str() { + "on" | "enabled" => Some(true), + "off" | "disabled" => Some(false), + _ => None, + }) + .collect::>(); + if states.len() != 1 { + bail!("unable to parse nvidia-smi {name}"); + } + states.first().copied().context("missing nvidia-smi state") + } + + async fn query_nvidia_smi_state(argument: &str, name: &str) -> Result { + let output = + run_command(NVIDIA_SMI, &["conf-compute", argument], NVIDIA_SMI_TIMEOUT).await?; + if !output.status.success() { + bail!( + "nvidia-smi conf-compute {argument} failed ({}): {}", + output.status, + truncated_lossy(&output.stderr, 512), + ); + } + parse_nvidia_smi_state(&output.stdout, name) + } + + /// Require the current system-wide GPU state to be production CC. NVAT's + /// NVML collector accepts protected-PCIe as well as CC and its default + /// appraisal does not reject DevTools, so a successful appraisal alone is + /// not a sufficient confidentiality check. + async fn verify_production_cc_mode() -> Result<()> { + if !query_nvidia_smi_state("-f", "CC feature state").await? { + bail!("nvidia confidential compute mode is not enabled"); + } + if query_nvidia_smi_state("-d", "DevTools mode").await? { + bail!("nvidia DevTools mode is enabled"); + } + Ok(()) + } + /// Mark the GPU as ready to accept work. Only meaningful (and only /// succeeds) when the GPU runs in CC mode. pub(super) async fn set_gpu_ready_state() -> Result<()> { @@ -1141,14 +1247,90 @@ mod gpu { Ok(()) } - /// Run local GPU attestation via nvattest with a fresh nonce, keeping the - /// verifier output in /run for debugging. Fails on any non-zero exit — - /// including a GPU that cannot produce an attestation report at all (a - /// non-CC GPU, or CC mode left off by the host). - pub(super) async fn attest_gpu() -> Result<()> { + fn validate_attestation_output( + stdout: &[u8], + nonce: &str, + expected_devices: u32, + ) -> Result<()> { + let output: NvattestOutput = + serde_json::from_slice(stdout).context("failed to parse nvattest JSON output")?; + if output.result_code != 0 { + bail!("nvattest JSON result is not successful"); + } + if output.claims.len() != expected_devices as usize { + bail!( + "gpu attestation count mismatch: expected {expected_devices}, got {}", + output.claims.len() + ); + } + for (index, claim) in output.claims.iter().enumerate() { + let claim = claim + .as_object() + .with_context(|| format!("invalid GPU claim at index {index}"))?; + let string_claim = |name: &str| claim.get(name).and_then(Value::as_str); + let valid_cert_chain = |name: &str| { + let cert = claim.get(name).and_then(Value::as_object)?; + Some( + cert.get("x-nvidia-cert-status").and_then(Value::as_str) == Some("valid") + && cert + .get("x-nvidia-cert-ocsp-status") + .and_then(Value::as_str) + == Some("good") + && cert + .get("x-nvidia-cert-ocsp-nonce-matches") + .and_then(Value::as_bool) + == Some(true) + && cert + .get("x-nvidia-cert-ocsp-response-valid") + .and_then(Value::as_bool) + == Some(true), + ) + }; + if string_claim("x-nvidia-device-type") != Some("gpu") + || claim.get("secboot").and_then(Value::as_bool) != Some(true) + || string_claim("dbgstat") != Some("disabled") + || string_claim("measres") != Some("success") + || valid_cert_chain("x-nvidia-gpu-attestation-report-cert-chain") != Some(true) + || valid_cert_chain("x-nvidia-gpu-driver-rim-cert-chain") != Some(true) + || valid_cert_chain("x-nvidia-gpu-vbios-rim-cert-chain") != Some(true) + { + bail!("gpu claim at index {index} violates the production policy"); + } + if string_claim("eat_nonce") != Some(nonce) + || claim + .get("x-nvidia-gpu-attestation-report-nonce-match") + .and_then(Value::as_bool) + != Some(true) + { + bail!("gpu claim at index {index} has an invalid nonce"); + } + } + Ok(()) + } + + fn attestation_event(stdout: &[u8], devices: u32) -> Result> { + let event = GpuAttestationEvent { + version: EVENT_VERSION, + provider: "nvidia", + devices, + policy: POLICY_VERSION, + cc_mode: "on", + devtools: false, + evidence_sha256: hex::encode(sha256(stdout)), + }; + serde_json::to_vec(&event).context("failed to serialize GPU attestation event") + } + + /// Run local GPU attestation via nvattest with a fresh nonce and a + /// mandatory relying-party policy. The JSON output is preserved under + /// /run; its hash is returned in a versioned summary for RTMR3. + pub(super) async fn attest_gpu(expected_devices: u32) -> Result> { if !Path::new(NVATTEST).exists() { bail!("nvattest is not available in this image"); } + if !Path::new(NVIDIA_POLICY).exists() { + bail!("gpu attestation policy is not available in this image"); + } // Certificate/OCSP validation needs a sane clock even when // secure_time is off; best-effort step chrony before attesting. if let Err(err) = cmd!(chronyc makestep) { @@ -1163,8 +1345,12 @@ mod gpu { "gpu", "--verifier", "local", + "--relying-party-policy", + NVIDIA_POLICY, "--nonce", &nonce, + "--format", + "json", ], ATTESTATION_TIMEOUT, ) @@ -1179,10 +1365,10 @@ mod gpu { truncated_lossy(&output.stderr, 512), ); } - if let Err(err) = save_attestation_output(&output.stdout) { - warn!("failed to save GPU attestation output: {err:?}"); - } - Ok(()) + validate_attestation_output(&output.stdout, &nonce, expected_devices)?; + verify_production_cc_mode().await?; + save_attestation_output(&output.stdout).context("failed to save GPU attestation output")?; + attestation_event(&output.stdout, expected_devices) } fn save_attestation_output(stdout: &[u8]) -> Result<()> { @@ -1206,6 +1392,117 @@ mod gpu { None => text.to_string(), } } + + #[cfg(test)] + mod tests { + use super::*; + + fn add_pci_device(root: &Path, name: &str, vendor: &str, class: &str) { + let device = root.join(name); + fs::create_dir_all(&device).unwrap(); + fs::write(device.join("vendor"), vendor).unwrap(); + fs::write(device.join("class"), class).unwrap(); + } + + fn nvattest_output(nonce: &str, claims: usize) -> Vec { + let cert_chain = serde_json::json!({ + "x-nvidia-cert-status": "valid", + "x-nvidia-cert-ocsp-status": "good", + "x-nvidia-cert-ocsp-nonce-matches": true, + "x-nvidia-cert-ocsp-response-valid": true + }); + let claims = (0..claims) + .map(|_| { + serde_json::json!({ + "x-nvidia-device-type": "gpu", + "secboot": true, + "dbgstat": "disabled", + "measres": "success", + "eat_nonce": nonce, + "x-nvidia-gpu-attestation-report-nonce-match": true, + "x-nvidia-gpu-attestation-report-cert-chain": cert_chain, + "x-nvidia-gpu-driver-rim-cert-chain": cert_chain, + "x-nvidia-gpu-vbios-rim-cert-chain": cert_chain + }) + }) + .collect::>(); + serde_json::to_vec(&serde_json::json!({ + "result_code": 0, + "claims": claims, + "detached_eat": {} + })) + .unwrap() + } + + #[test] + fn inventory_counts_nvidia_and_non_nvidia_gpus() { + let root = tempfile::tempdir().unwrap(); + add_pci_device(root.path(), "0000:01:00.0", "0x10de\n", "0x030200\n"); + add_pci_device(root.path(), "0000:02:00.0", "0x1234\n", "0x030000\n"); + add_pci_device(root.path(), "0000:03:00.0", "0x1af4\n", "0x020000\n"); + assert_eq!( + gpu_inventory_at(root.path()).unwrap(), + GpuInventory { + total: 2, + nvidia: 1 + } + ); + } + + #[test] + fn expected_count_rejects_mixed_or_unmeasured_gpus() { + let mixed = GpuInventory { + total: 2, + nvidia: 1, + }; + assert!(expected_gpu_count(r#"{"num_gpus":2}"#, mixed).is_err()); + + let nvidia = GpuInventory { + total: 2, + nvidia: 2, + }; + let err = expected_gpu_count(r#"{"num_gpus":1}"#, nvidia).unwrap_err(); + assert!(err.to_string().contains("gpu count mismatch")); + assert_eq!(expected_gpu_count(r#"{"num_gpus":2}"#, nvidia).unwrap(), 2); + } + + #[test] + fn nvidia_smi_state_parser_is_fail_closed() { + assert!(parse_nvidia_smi_state(b"CC status: ON\n", "CC").unwrap()); + assert!(!parse_nvidia_smi_state(b"DevTools Mode: OFF\n", "DevTools").unwrap()); + assert!(parse_nvidia_smi_state(b"unknown\n", "CC").is_err()); + assert!(parse_nvidia_smi_state(b"current: ON, pending: OFF\n", "CC").is_err()); + } + + #[test] + fn nvattest_output_requires_every_expected_production_gpu() { + let nonce = "11".repeat(32); + let valid = nvattest_output(&nonce, 2); + validate_attestation_output(&valid, &nonce, 2).unwrap(); + assert!(validate_attestation_output(&valid, &nonce, 1).is_err()); + + let mut invalid: Value = serde_json::from_slice(&valid).unwrap(); + invalid["claims"][1]["dbgstat"] = Value::String("enabled".to_string()); + assert!( + validate_attestation_output(&serde_json::to_vec(&invalid).unwrap(), &nonce, 2) + .is_err() + ); + } + + #[test] + fn event_commits_to_complete_nvattest_output() { + let nonce = "22".repeat(32); + let output = nvattest_output(&nonce, 1); + let event: Value = + serde_json::from_slice(&attestation_event(&output, 1).unwrap()).unwrap(); + assert_eq!(event["version"], EVENT_VERSION); + assert_eq!(event["devices"], 1); + assert_eq!(event["policy"], POLICY_VERSION); + assert_eq!(event["cc_mode"], "on"); + assert_eq!(event["devtools"], false); + assert_eq!(event["evidence_sha256"], hex::encode(sha256(&output))); + } + } } impl Stage0<'_> { @@ -1213,10 +1510,11 @@ impl Stage0<'_> { /// NVIDIA GPU before continuing to key provisioning, or — when explicitly /// disabled — set the GPU ready state without verification. async fn setup_gpu(&self) -> Result<()> { - if !gpu::nvidia_gpu_present()? { - return Ok(()); - } + let inventory = gpu::gpu_inventory()?; if !self.shared.app_compose.attest_gpu() { + if inventory.nvidia == 0 { + return Ok(()); + } warn!("requirements.attest_gpu is false; setting GPU ready state without attestation"); // Best-effort: a GPU with CC mode off has no ready state to set. if let Err(err) = gpu::set_gpu_ready_state().await { @@ -1224,10 +1522,17 @@ impl Stage0<'_> { } return Ok(()); } + let expected_devices = + gpu::expected_gpu_count(&self.shared.sys_config.vm_config, inventory)?; + if expected_devices == 0 { + return Ok(()); + } self.vmm.notify_q("boot.progress", "attesting GPU").await; info!("verifying GPU TEE attestation"); - gpu::attest_gpu().await?; + let event = gpu::attest_gpu(expected_devices).await?; gpu::set_gpu_ready_state().await?; + emit_runtime_event("gpu-attestation", &event) + .context("failed to emit GPU attestation event")?; info!("GPU TEE attestation succeeded"); Ok(()) } diff --git a/os/yocto/layers/meta-nvidia/recipes-graphics/nvattest/files/10-nvidia-gpu-ordering.conf b/os/yocto/layers/meta-nvidia/recipes-graphics/nvattest/files/10-nvidia-gpu-ordering.conf index 50e0b7d1f..64d356563 100644 --- a/os/yocto/layers/meta-nvidia/recipes-graphics/nvattest/files/10-nvidia-gpu-ordering.conf +++ b/os/yocto/layers/meta-nvidia/recipes-graphics/nvattest/files/10-nvidia-gpu-ordering.conf @@ -1,6 +1,6 @@ # Ordering only, not a hard dependency: on GPU instances, `dstack-util setup` # (run by dstack-prepare.service) performs local GPU TEE attestation per the -# app-compose `requirements.verify_gpu` policy, which needs the nvidia driver +# app-compose `requirements.attest_gpu` policy, which needs the nvidia driver # loaded (nvidia-persistenced) and the NVLink fabric up (nvidia-fabricmanager). # Both units skip cleanly on GPU-less instances via nvidia-gpu-detect. [Unit] diff --git a/os/yocto/layers/meta-nvidia/recipes-graphics/nvattest/files/dstack-gpu-attestation.rego b/os/yocto/layers/meta-nvidia/recipes-graphics/nvattest/files/dstack-gpu-attestation.rego new file mode 100644 index 000000000..8cf8ee479 --- /dev/null +++ b/os/yocto/layers/meta-nvidia/recipes-graphics/nvattest/files/dstack-gpu-attestation.rego @@ -0,0 +1,29 @@ +# SPDX-FileCopyrightText: © 2026 Phala Network +# +# SPDX-License-Identifier: Apache-2.0 + +package policy + +import future.keywords.every + +default nv_match := false + +nv_match { + count(input) > 0 + every claim in input { + claim["x-nvidia-device-type"] == "gpu" + claim.measres == "success" + claim.secboot == true + claim.dbgstat == "disabled" + valid_cert_chain(claim["x-nvidia-gpu-attestation-report-cert-chain"]) + valid_cert_chain(claim["x-nvidia-gpu-driver-rim-cert-chain"]) + valid_cert_chain(claim["x-nvidia-gpu-vbios-rim-cert-chain"]) + } +} + +valid_cert_chain(cert_chain) { + cert_chain["x-nvidia-cert-status"] == "valid" + cert_chain["x-nvidia-cert-ocsp-status"] == "good" + cert_chain["x-nvidia-cert-ocsp-nonce-matches"] == true + cert_chain["x-nvidia-cert-ocsp-response-valid"] == true +} diff --git a/os/yocto/layers/meta-nvidia/recipes-graphics/nvattest/nvattest_2026.06.09.bb b/os/yocto/layers/meta-nvidia/recipes-graphics/nvattest/nvattest_2026.06.09.bb index 7d907d69d..5eee6f712 100644 --- a/os/yocto/layers/meta-nvidia/recipes-graphics/nvattest/nvattest_2026.06.09.bb +++ b/os/yocto/layers/meta-nvidia/recipes-graphics/nvattest/nvattest_2026.06.09.bb @@ -1,5 +1,5 @@ SUMMARY = "NVIDIA GPU attestation CLI" -DESCRIPTION = "Builds NVIDIA's nvattest CLI. dstack-util setup runs it at boot to gate readiness on local GPU TEE attestation (app-compose requirements.verify_gpu)." +DESCRIPTION = "Builds NVIDIA's nvattest CLI. dstack-util setup runs it at boot to gate readiness on local GPU TEE attestation (app-compose requirements.attest_gpu)." HOMEPAGE = "https://github.com/NVIDIA/attestation-sdk" LICENSE = "Apache-2.0" LIC_FILES_CHKSUM = "file://LICENSE;md5=e620fc90e76c4aa0c3efdd1673ca0b3b" @@ -7,6 +7,7 @@ LIC_FILES_CHKSUM = "file://LICENSE;md5=e620fc90e76c4aa0c3efdd1673ca0b3b" SRC_URI = " \ git://github.com/NVIDIA/attestation-sdk.git;protocol=https;branch=main \ file://10-nvidia-gpu-ordering.conf \ + file://dstack-gpu-attestation.rego \ " SRCREV = "9d12801cea8a198ea0f29640dfaf8a4017c841c5" @@ -100,10 +101,19 @@ do_install() { install -d ${D}${systemd_system_unitdir}/dstack-prepare.service.d install -m 0644 ${UNPACKDIR}/10-nvidia-gpu-ordering.conf \ ${D}${systemd_system_unitdir}/dstack-prepare.service.d/10-nvidia-gpu-ordering.conf + + # dstack-util passes this policy explicitly. It retains every check from + # the SDK's default appraisal and additionally requires production debug + # status. (The pinned SDK does not expose GPU mode as a Rego claim, so + # dstack-util separately requires CC=ON and DevTools=OFF.) + install -d ${D}${datadir}/nvattest + install -m 0644 ${UNPACKDIR}/dstack-gpu-attestation.rego \ + ${D}${datadir}/nvattest/dstack-gpu-attestation.rego } FILES:${PN} += " \ ${systemd_system_unitdir}/dstack-prepare.service.d/10-nvidia-gpu-ordering.conf \ + ${datadir}/nvattest/dstack-gpu-attestation.rego \ ${libdir}/lib*.so \ ${libdir}/lib*.so.* \ " diff --git a/os/yocto/layers/meta-nvidia/recipes-graphics/nvidia/files/nvidia-persistenced.service b/os/yocto/layers/meta-nvidia/recipes-graphics/nvidia/files/nvidia-persistenced.service index f40aef461..166e6295d 100644 --- a/os/yocto/layers/meta-nvidia/recipes-graphics/nvidia/files/nvidia-persistenced.service +++ b/os/yocto/layers/meta-nvidia/recipes-graphics/nvidia/files/nvidia-persistenced.service @@ -19,7 +19,7 @@ ExecStart=/usr/bin/env nvidia-persistenced --uvm-persistence-mode # NOTE: the GPU ready state (conf-compute -srs 1) is intentionally NOT set # here. It is set by `dstack-util setup` (dstack-prepare.service), only after # local GPU TEE attestation via nvattest succeeds -- unless the deployment -# opts out with app-compose requirements.verify_gpu=false. +# opts out with app-compose requirements.attest_gpu=false. RemainAfterExit=yes [Install] From 6bb7d9d26413e6103d99501d4a137ca7622a8567 Mon Sep 17 00:00:00 2001 From: Kevin Wang Date: Thu, 16 Jul 2026 03:24:13 -0700 Subject: [PATCH 02/32] docs(security): clarify NVIDIA GPU secboot claim --- docs/security/security-model.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/security/security-model.md b/docs/security/security-model.md index fa291c10e..38e20ccc1 100644 --- a/docs/security/security-model.md +++ b/docs/security/security-model.md @@ -55,7 +55,7 @@ dstack supports NVIDIA H100, H200, and B200 GPUs in confidential compute mode fo ### How It Works -GPUs are passed through via VFIO to the TEE-protected CVM. Before key provisioning, `dstack-util setup` inventories every VGA/3D-controller PCI function, compares that count with `vm_config.num_gpus`, and runs NVIDIA's local `nvattest` verifier over every NVIDIA-driver-visible GPU with a fresh nonce. A mandatory relying-party policy requires secure boot and disabled debug status. Separate fail-closed `nvidia-smi` queries require the current CC feature to be ON and DevTools mode to be OFF. Only then does dstack set the GPU ready state. +GPUs are passed through via VFIO to the TEE-protected CVM. Before key provisioning, `dstack-util setup` inventories every VGA/3D-controller PCI function, compares that count with `vm_config.num_gpus`, and runs NVIDIA's local `nvattest` verifier over every NVIDIA-driver-visible GPU with a fresh nonce. A mandatory relying-party policy requires successful GPU measurement appraisal, the NVIDIA GPU claim `secboot == true`, and `dbgstat == "disabled"`. This `secboot` field describes the GPU attestation result; it does not require UEFI Secure Boot in the CVM. The dstack CPU/guest boot chain is verified independently through measured boot. Separate fail-closed `nvidia-smi` queries require the current CC feature to be ON and DevTools mode to be OFF. Only then does dstack set the GPU ready state. ### Dual Attestation From e9b31581ff188584759f8f54c55ef3c4ca4f9ce0 Mon Sep 17 00:00:00 2001 From: Kevin Wang Date: Thu, 16 Jul 2026 03:54:23 -0700 Subject: [PATCH 03/32] feat(guest): enforce measured app GPU policies --- docs/security/security-model.md | 17 +- dstack/Cargo.lock | 285 ++++++++++++++++++ dstack/Cargo.toml | 1 + dstack/dstack-types/src/lib.rs | 30 +- dstack/dstack-util/Cargo.toml | 1 + dstack/dstack-util/src/system_setup.rs | 242 ++++++++++----- .../files/10-nvidia-gpu-ordering.conf | 2 +- .../files/dstack-gpu-attestation.rego | 29 -- .../nvattest/nvattest_2026.06.09.bb | 11 +- .../nvidia/files/nvidia-persistenced.service | 4 +- 10 files changed, 495 insertions(+), 127 deletions(-) delete mode 100644 os/yocto/layers/meta-nvidia/recipes-graphics/nvattest/files/dstack-gpu-attestation.rego diff --git a/docs/security/security-model.md b/docs/security/security-model.md index 38e20ccc1..47a98a0dc 100644 --- a/docs/security/security-model.md +++ b/docs/security/security-model.md @@ -55,22 +55,26 @@ dstack supports NVIDIA H100, H200, and B200 GPUs in confidential compute mode fo ### How It Works -GPUs are passed through via VFIO to the TEE-protected CVM. Before key provisioning, `dstack-util setup` inventories every VGA/3D-controller PCI function, compares that count with `vm_config.num_gpus`, and runs NVIDIA's local `nvattest` verifier over every NVIDIA-driver-visible GPU with a fresh nonce. A mandatory relying-party policy requires successful GPU measurement appraisal, the NVIDIA GPU claim `secboot == true`, and `dbgstat == "disabled"`. This `secboot` field describes the GPU attestation result; it does not require UEFI Secure Boot in the CVM. The dstack CPU/guest boot chain is verified independently through measured boot. Separate fail-closed `nvidia-smi` queries require the current CC feature to be ON and DevTools mode to be OFF. Only then does dstack set the GPU ready state. +GPUs are passed through via VFIO to the TEE-protected CVM. Before key provisioning, `dstack-util setup` inventories every VGA/3D-controller PCI function, compares that count with `vm_config.num_gpus`, and runs NVIDIA's local `nvattest` verifier over every NVIDIA-driver-visible GPU with a fresh nonce. dstack does not ship or pass a custom relying-party policy to this command; nvattest's built-in appraisal still applies. The complete JSON result (`result_code`, `result_message`, `claims`, and `detached_eat`) is saved at `/run/nvidia-gpu-attestation/attestation.out`. + +An application may additionally set `requirements.gpu_policy` to a Rego v0 script. Its input is the nvattest output's `claims` array, and it must define the boolean entrypoint `data.policy.nv_match` in `package policy`. Immediately after measuring `compose-hash`, dstack measures `sha256(gpu_policy UTF-8 bytes)` in a `gpu-policy` event and then evaluates the policy. A false, undefined, malformed, or non-boolean result stops boot before key provisioning. If no GPU is configured, the policy receives an empty array. `gpu_policy` cannot be combined with `attest_gpu: false`. + +Separate fail-closed `nvidia-smi` queries require the current CC feature to be ON and DevTools mode to be OFF. Only after the default appraisal, optional application policy, and these checks succeed does dstack set the GPU ready state. The dstack CPU/guest boot chain is verified independently through measured boot; a GPU claim named `secboot` refers to the GPU appraisal, not UEFI Secure Boot in the CVM. ### Dual Attestation -GPU workloads require verification of both hardware components. The CPU TEE quote verifies the CVM and its measured guest code. NVIDIA-signed evidence, checked against NVIDIA RIMs and certificate status by `nvattest`, verifies the GPU appraisal. After the ready-state operation succeeds, dstack emits a `gpu-attestation` launch event before `system-ready`. Its versioned payload records the number of appraised devices, policy version, asserted CC/DevTools state, and SHA-256 of the complete nvattest JSON output (claims and detached EAT). On TDX this event is append-only in RTMR3; it is never derived from application-controlled `report_data`. +GPU workloads require verification of both hardware components. The CPU TEE quote verifies the CVM and its measured guest code. NVIDIA-signed evidence, checked against NVIDIA RIMs and certificate status by `nvattest`, verifies the GPU appraisal. After the optional policy and ready-state operations succeed, dstack emits a `gpu-attestation` launch event before `system-ready`. Its versioned payload records the number of appraised devices, asserted CC/DevTools state, and SHA-256 of the complete nvattest JSON output (claims and detached EAT). On TDX, both `gpu-policy` and `gpu-attestation` are append-only RTMR3 events; they are never derived from application-controlled `report_data`. -A verifier must replay the measured event log, require exactly one pre-`system-ready` `gpu-attestation` event when GPU protection is required, require `devices == vm_config.num_gpus > 0`, and accept the named policy version. The raw file under `/run/nvidia-gpu-attestation/attestation.out` is not trusted by itself; if it is supplied for inspection, its digest must match the event. +A verifier must replay the measured event log, require exactly one pre-`system-ready` `gpu-attestation` event when GPU protection is required, and require `devices == vm_config.num_gpus > 0`. If an application policy is required, the verifier must also require exactly one `gpu-policy` event immediately after `compose-hash` and compare its 32-byte payload with the expected policy digest. The raw `attestation.out` file is not trusted by itself; if it is supplied for inspection, its digest must match the `gpu-attestation` event. ### GPU Threat Model and Lifetime The GPU gate assumes a malicious host/VMM and untrusted host-provided PCI topology, while trusting the CPU TEE, the measured dstack guest/kernel, NVIDIA hardware/firmware roots, and the cryptography used by both attestation chains. Availability is out of scope. -The event makes the following **boot-time** statement: immediately before key provisioning, all attached VGA/3D PCI functions were NVIDIA devices, their count matched the quote-bound VM configuration, nvattest returned one fresh production-policy claim for each device, CC was ON, DevTools was OFF, and setting the GPU ready state succeeded. This closes these cases: +The events make the following **boot-time** statement: immediately before key provisioning, all attached VGA/3D PCI functions were NVIDIA devices, their count matched the quote-bound VM configuration, nvattest returned one fresh successfully appraised claim for each device, the measured application policy accepted those claims when present, CC was ON, DevTools was OFF, and setting the GPU ready state succeeded. This closes these cases: - A GPU-less launch cannot be presented as a GPU-verified launch: it has `num_gpus == 0` and no `gpu-attestation` event. -- A mixed launch cannot attest only its TEE-capable subset. Non-NVIDIA display GPUs are rejected, and the sysfs, `vm_config`, and nvattest claim counts must all agree. A non-CC NVIDIA GPU either prevents evidence collection/appraisal or causes that count/policy check to fail. +- A mixed launch cannot attest only its TEE-capable subset. Non-NVIDIA display GPUs are rejected, and the sysfs, `vm_config`, and nvattest claim counts must all agree. A non-CC NVIDIA GPU either prevents evidence collection/appraisal or causes the default appraisal, application policy, or CC-state check to fail. - Copying another CVM's result into a file or `report_data` does not work. Only measured pre-application code can place the event before `system-ready`, and event-log replay binds it to the quoted RTMR/PCR value. This is **not a lifetime or physical co-location guarantee**. After `system-ready`, an application with sufficient guest privileges can unload the NVIDIA driver, and a malicious host may attempt PCI hot-remove/replacement or proxy GPU traffic. The boot event remains a true historical statement but does not prove that the same device is still attached. dstack also cannot rule out a live relay/cuckoo attack to a genuine remote GPU: current Hopper/Blackwell deployments do not provide a CPU-TEE-verifiable TEE-I/O/TDISP device binding. Applications that mutate the driver or PCI topology are outside this guarantee; higher-assurance deployments must prevent that behavior and re-attest before using a newly initialized GPU. @@ -174,8 +178,9 @@ Use this checklist to verify a workload running in a dstack CVM. **GPU verification (when required):** - [ ] `vm_config.num_gpus` is greater than zero and matches the expected deployment +- [ ] If `requirements.gpu_policy` is required, exactly one `gpu-policy` event follows `compose-hash`, and its payload matches the expected SHA-256 digest - [ ] Exactly one `gpu-attestation` event appears before `system-ready` -- [ ] The event device count matches `vm_config.num_gpus`, and its policy/CC/DevTools fields match the production policy +- [ ] The event device count matches `vm_config.num_gpus`, and its CC/DevTools fields indicate production mode - [ ] The platform binds the event log to a quoted RTMR/PCR (do not accept it from current SEV-SNP evidence) **Key management verification:** diff --git a/dstack/Cargo.lock b/dstack/Cargo.lock index f12355b98..22f63d33b 100644 --- a/dstack/Cargo.lock +++ b/dstack/Cargo.lock @@ -55,6 +55,20 @@ dependencies = [ "subtle", ] +[[package]] +name = "ahash" +version = "0.8.12" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5a15f179cd60c4584b8a8c596927aadc462e27f2ca70c04e0071964a73ba7a75" +dependencies = [ + "cfg-if", + "getrandom 0.3.4", + "once_cell", + "serde", + "version_check", + "zerocopy", +] + [[package]] name = "aho-corasick" version = "1.1.4" @@ -401,6 +415,21 @@ dependencies = [ "syn 2.0.117", ] +[[package]] +name = "bit-set" +version = "0.8.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "08807e080ed7f9d5433fa9b275196cfc35414f66a0c79d864dc51a0d825231a3" +dependencies = [ + "bit-vec", +] + +[[package]] +name = "bit-vec" +version = "0.8.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5e764a1d40d510daf35e07be9eb06e75770908c27d411ee6c92109c9840eaaf7" + [[package]] name = "bitfield" version = "0.15.0" @@ -552,6 +581,12 @@ dependencies = [ "syn 2.0.117", ] +[[package]] +name = "borrow-or-share" +version = "0.2.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "dc0b364ead1874514c8c2855ab558056ebfeb775653e7ae45ff72f28f8f3166c" + [[package]] name = "borsh" version = "1.6.1" @@ -608,6 +643,12 @@ version = "1.2.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "7575182f7272186991736b70173b0ea045398f984bf5ebbb3804736ce1330c9d" +[[package]] +name = "bytecount" +version = "0.6.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "175812e0be2bccb6abe50bb8d566126198344f707e304f45c648fd8f2cc0365e" + [[package]] name = "bytemuck" version = "1.25.0" @@ -755,6 +796,16 @@ dependencies = [ "windows-link 0.2.1", ] +[[package]] +name = "chrono-tz" +version = "0.10.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a6139a8597ed92cf816dfb33f5dd6cf0bb93a6adc938f11039f371bc5bcd26c3" +dependencies = [ + "chrono", + "phf", +] + [[package]] name = "ciborium" version = "0.2.2" @@ -1976,6 +2027,7 @@ dependencies = [ "ra-tls", "rand 0.8.6", "regex", + "regorus", "safe-write", "schnorrkel", "scopeguard", @@ -2196,6 +2248,15 @@ dependencies = [ "zeroize", ] +[[package]] +name = "email_address" +version = "0.2.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e079f19b08ca6239f47f8ba8509c11cf3ea30095831f7fed61441475edd8c449" +dependencies = [ + "serde", +] + [[package]] name = "encode_unicode" version = "1.0.0" @@ -2306,6 +2367,17 @@ dependencies = [ "winapi", ] +[[package]] +name = "fancy-regex" +version = "0.14.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6e24cb5a94bcae1e5408b0effca5cd7172ea3c5755049c5f3af4cd283a165298" +dependencies = [ + "bit-set", + "regex-automata", + "regex-syntax", +] + [[package]] name = "fastrand" version = "2.4.1" @@ -2399,6 +2471,17 @@ dependencies = [ "miniz_oxide", ] +[[package]] +name = "fluent-uri" +version = "0.3.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1918b65d96df47d3591bed19c5cca17e3fa5d0707318e4b5ef2eae01764df7e5" +dependencies = [ + "borrow-or-share", + "ref-cast", + "serde", +] + [[package]] name = "fnv" version = "1.0.7" @@ -2426,6 +2509,16 @@ dependencies = [ "percent-encoding", ] +[[package]] +name = "fraction" +version = "0.15.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e076045bb43dac435333ed5f04caf35c7463631d0dae2deb2638d94dd0a5b872" +dependencies = [ + "lazy_static", + "num", +] + [[package]] name = "fs-err" version = "3.3.0" @@ -3467,6 +3560,15 @@ dependencies = [ "either", ] +[[package]] +name = "itertools" +version = "0.11.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b1c173a5686ce8bfa551b3563d0c2170bf24ca44da99c7ca4bfdab5418c3fe57" +dependencies = [ + "either", +] + [[package]] name = "itertools" version = "0.13.0" @@ -3582,6 +3684,30 @@ dependencies = [ "wasm-bindgen", ] +[[package]] +name = "jsonschema" +version = "0.29.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "161c33c3ec738cfea3288c5c53dfcdb32fd4fc2954de86ea06f71b5a1a40bfcd" +dependencies = [ + "ahash", + "base64 0.22.1", + "bytecount", + "email_address", + "fancy-regex", + "fraction", + "idna", + "itoa", + "num-cmp", + "once_cell", + "percent-encoding", + "referencing", + "regex-syntax", + "serde", + "serde_json", + "uuid-simd", +] + [[package]] name = "k256" version = "0.13.4" @@ -4160,6 +4286,20 @@ dependencies = [ "windows-sys 0.61.2", ] +[[package]] +name = "num" +version = "0.4.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "35bd024e8b2ff75562e5f34e7f4905839deb4b22955ef5e73d2fea1b9813cb23" +dependencies = [ + "num-bigint", + "num-complex", + "num-integer", + "num-iter", + "num-rational", + "num-traits", +] + [[package]] name = "num-bigint" version = "0.4.6" @@ -4186,6 +4326,21 @@ dependencies = [ "zeroize", ] +[[package]] +name = "num-cmp" +version = "0.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "63335b2e2c34fae2fb0aa2cecfd9f0832a1e24b3b32ecec612c3426d46dc8aaa" + +[[package]] +name = "num-complex" +version = "0.4.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "73f88a1307638156682bada9d7604135552957b7818057dcef22705b4d509495" +dependencies = [ + "num-traits", +] + [[package]] name = "num-conv" version = "0.2.2" @@ -4218,6 +4373,7 @@ version = "0.4.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "f83d14da390562dca69fc84082e73e548e1ad308d24accdedd2720017cb37824" dependencies = [ + "num-bigint", "num-integer", "num-traits", ] @@ -4342,6 +4498,12 @@ dependencies = [ "windows-sys 0.61.2", ] +[[package]] +name = "outref" +version = "0.5.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1a80800c0488c3a21695ea981a54918fbb37abf04f4d0720c453632255e2ff0e" + [[package]] name = "p256" version = "0.13.2" @@ -4693,6 +4855,15 @@ dependencies = [ "universal-hash", ] +[[package]] +name = "pori" +version = "0.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a4a63d338dec139f56dacc692ca63ad35a6be6a797442479b55acd611d79e906" +dependencies = [ + "nom", +] + [[package]] name = "portable-atomic" version = "1.13.1" @@ -5335,6 +5506,20 @@ version = "0.1.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "09c30c54dffee5b40af088d5d50aa3455c91a0127164b51f0215efc4cb28fb3c" +[[package]] +name = "referencing" +version = "0.29.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "40a64b3a635fad9000648b4d8a59c8710c523ab61a23d392a7d91d47683f5adc" +dependencies = [ + "ahash", + "fluent-uri", + "once_cell", + "parking_lot 0.12.5", + "percent-encoding", + "serde_json", +] + [[package]] name = "regex" version = "1.12.3" @@ -5364,6 +5549,35 @@ version = "0.8.10" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "dc897dd8d9e8bd1ed8cdad82b5966c3e0ecae09fb1907d58efaa013543185d0a" +[[package]] +name = "regorus" +version = "0.4.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "bf70615014ce5c89fe427197fd9b52e6506916838b449e80ee141adc43adec6b" +dependencies = [ + "anyhow", + "chrono", + "chrono-tz", + "constant_time_eq 0.4.2", + "data-encoding", + "hex", + "hmac 0.12.1", + "jsonschema", + "lazy_static", + "md-5", + "rand 0.9.4", + "regex", + "scientific", + "semver", + "serde", + "serde_json", + "serde_yaml", + "sha2 0.10.9", + "url", + "uuid", + "wax", +] + [[package]] name = "reqwest" version = "0.13.4" @@ -6042,6 +6256,26 @@ dependencies = [ "zeroize", ] +[[package]] +name = "scientific" +version = "0.5.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "38a4b339a8de779ecb098a772ecbba2ace74e23ed959a5b4f30631d8bf1799a8" +dependencies = [ + "scientific-macro", +] + +[[package]] +name = "scientific-macro" +version = "0.5.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d2ee4885492bb655bfa05d039cd9163eb8fe9f79ddebf00ca23a1637510c2fd2" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.117", +] + [[package]] name = "scoped-tls" version = "1.0.1" @@ -6299,6 +6533,19 @@ dependencies = [ "time", ] +[[package]] +name = "serde_yaml" +version = "0.9.34+deprecated" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6a8b1a1a2ebf674015cc02edccce75287f1a0130d394307b36743c2f5d504b47" +dependencies = [ + "indexmap 2.14.0", + "itoa", + "ryu", + "serde", + "unsafe-libyaml", +] + [[package]] name = "sev" version = "6.0.0" @@ -7422,6 +7669,12 @@ dependencies = [ "subtle", ] +[[package]] +name = "unsafe-libyaml" +version = "0.2.11" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "673aac59facbab8a9007c7f6108d11f63b603f7cabff99fabf650fea5c32b861" + [[package]] name = "untrusted" version = "0.7.1" @@ -7478,10 +7731,22 @@ checksum = "d258b83ceec21034727ecee8c382cfa6c3e133699b0742c64571814fb420c9f7" dependencies = [ "getrandom 0.4.2", "js-sys", + "rand 0.10.1", "serde_core", "wasm-bindgen", ] +[[package]] +name = "uuid-simd" +version = "0.8.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "23b082222b4f6619906941c17eb2297fff4c2fb96cb60164170522942a200bd8" +dependencies = [ + "outref", + "uuid", + "vsimd", +] + [[package]] name = "valuable" version = "0.1.1" @@ -7506,6 +7771,12 @@ version = "1.0.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "6a02e4885ed3bc0f2de90ea6dd45ebcbb66dacffe03547fadbb0eeae2770887d" +[[package]] +name = "vsimd" +version = "0.8.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5c3082ca00d5a5ef149bb8b555a72ae84c9c59f7250f013ac822ac2e49b19c64" + [[package]] name = "vsock" version = "0.5.4" @@ -7686,6 +7957,20 @@ dependencies = [ "tracing", ] +[[package]] +name = "wax" +version = "0.6.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8d12a78aa0bab22d2f26ed1a96df7ab58e8a93506a3e20adb47c51a93b4e1357" +dependencies = [ + "const_format", + "itertools 0.11.0", + "nom", + "pori", + "regex", + "thiserror 1.0.69", +] + [[package]] name = "web-sys" version = "0.3.99" diff --git a/dstack/Cargo.toml b/dstack/Cargo.toml index eb6f8f6d6..c3c109e24 100644 --- a/dstack/Cargo.toml +++ b/dstack/Cargo.toml @@ -128,6 +128,7 @@ log = "0.4.26" moka = { version = "0.12.15", default-features = false, features = ["sync"] } notify = "8.0.0" rand = "0.8.5" +regorus = { version = "0.4.0", default-features = false, features = ["full-opa"] } tracing = "0.1.40" tracing-subscriber = { version = "0.3.20", features = ["env-filter"] } safe-write = "0.1.2" diff --git a/dstack/dstack-types/src/lib.rs b/dstack/dstack-types/src/lib.rs index a6fe1901f..24896d1ed 100644 --- a/dstack/dstack-types/src/lib.rs +++ b/dstack/dstack-types/src/lib.rs @@ -159,6 +159,15 @@ pub struct Requirements { /// are unaffected either way. #[serde(skip_serializing_if = "Option::is_none")] pub attest_gpu: Option, + /// Optional application-supplied Rego policy evaluated against the + /// `claims` array in NVIDIA nvattest's JSON output before key + /// provisioning. The policy must define the boolean rule + /// `data.policy.nv_match`. + /// + /// The SHA-256 digest of the exact UTF-8 policy bytes is emitted as the + /// `gpu-policy` launch event immediately after `compose-hash`. + #[serde(skip_serializing_if = "Option::is_none")] + pub gpu_policy: Option, } impl Requirements { @@ -168,6 +177,7 @@ impl Requirements { && self.tdx_measure_acpi_tables.is_none() && self.launch_token_hash.is_none() && self.attest_gpu.is_none() + && self.gpu_policy.is_none() } } @@ -455,7 +465,8 @@ mod app_compose_tests { "os_version": ">=0.6.1", "platforms": ["dstack-gcp-tdx", "dstack-tdx"], "tdx_measure_acpi_tables": true, - "launch_token_hash": "9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08" + "launch_token_hash": "9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08", + "gpu_policy": "package policy\n\ndefault nv_match = false\n" } })) .unwrap(); @@ -470,6 +481,10 @@ mod app_compose_tests { requirements.launch_token_hash.as_deref(), Some("9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08") ); + assert_eq!( + requirements.gpu_policy.as_deref(), + Some("package policy\n\ndefault nv_match = false\n") + ); let err = serde_json::from_value::(serde_json::json!({ "manifest_version": "3", @@ -534,6 +549,19 @@ mod app_compose_tests { let requirements = launch_token.requirements.as_ref().unwrap(); assert!(requirements.launch_token_hash.is_some()); assert!(!requirements.is_empty()); + + let gpu_policy: AppCompose = serde_json::from_value(serde_json::json!({ + "manifest_version": "3", + "name": "test", + "runner": "docker-compose", + "requirements": { + "gpu_policy": "package policy\n\ndefault nv_match = false\n" + } + })) + .unwrap(); + let requirements = gpu_policy.requirements.as_ref().unwrap(); + assert!(requirements.gpu_policy.is_some()); + assert!(!requirements.is_empty()); } #[test] diff --git a/dstack/dstack-util/Cargo.toml b/dstack/dstack-util/Cargo.toml index e70b58fd0..f43a243d7 100644 --- a/dstack/dstack-util/Cargo.toml +++ b/dstack/dstack-util/Cargo.toml @@ -46,6 +46,7 @@ dcap-qvl.workspace = true k256 = { workspace = true, features = ["ecdsa"] } dstack-types.workspace = true rand.workspace = true +regorus.workspace = true sha3.workspace = true dstack-attest.workspace = true cert-client.workspace = true diff --git a/dstack/dstack-util/src/system_setup.rs b/dstack/dstack-util/src/system_setup.rs index be6652fd5..699e303ed 100644 --- a/dstack/dstack-util/src/system_setup.rs +++ b/dstack/dstack-util/src/system_setup.rs @@ -838,6 +838,15 @@ fn verify_manifest_feature_requirements(app_compose: &AppCompose) -> Result<()> "requirements requires manifest_version >= {MANIFEST_VERSION_3}; use string manifest_version \"{MANIFEST_VERSION_3}\" so older guests fail closed" ); } + if app_compose + .requirements + .as_ref() + .is_some_and(|requirements| { + requirements.gpu_policy.is_some() && requirements.attest_gpu == Some(false) + }) + { + bail!("requirements.gpu_policy requires GPU attestation"); + } Ok(()) } @@ -1060,18 +1069,19 @@ async fn do_sys_setup(stage0: Stage0<'_>) -> Result<()> { } else { info!("System time will be synchronized by chronyd in background"); } - stage0 + let gpu_attestation = stage0 .setup_gpu() .await .context("Failed to verify GPU TEE attestation")?; - let stage1 = stage0.setup_fs().await?; + let stage1 = stage0.setup_fs(gpu_attestation).await?; stage1.setup().await } /// GPU TEE attestation gate (`requirements.attest_gpu`, defaults to true). /// /// Runs before key provisioning so a CVM whose GPU cannot prove it is a -/// genuine, CC-enabled NVIDIA TEE never gets its app keys. The GPU +/// genuine, CC-enabled NVIDIA TEE never gets its app keys. An optional +/// application policy is measured and evaluated after `compose-hash`. The GPU /// "ready" state (`nvidia-smi conf-compute -srs 1`) is only set from here — /// nvidia-persistenced deliberately does not set it — so CUDA work cannot be /// submitted to an unverified GPU either. @@ -1080,12 +1090,11 @@ mod gpu { const NVATTEST: &str = "/usr/bin/nvattest"; const NVIDIA_SMI: &str = "/usr/bin/nvidia-smi"; - const NVIDIA_POLICY: &str = "/usr/share/nvattest/dstack-gpu-attestation.rego"; const ATTESTATION_OUTPUT: &str = "/run/nvidia-gpu-attestation/attestation.out"; const ATTESTATION_TIMEOUT: Duration = Duration::from_secs(300); const NVIDIA_SMI_TIMEOUT: Duration = Duration::from_secs(60); - const EVENT_VERSION: u32 = 1; - const POLICY_VERSION: &str = "nvidia-cc-production-v1"; + const EVENT_VERSION: u32 = 2; + const POLICY_ENTRYPOINT: &str = "data.policy.nv_match"; #[derive(Debug, Clone, Copy, PartialEq, Eq)] pub(super) struct GpuInventory { @@ -1098,12 +1107,26 @@ mod gpu { version: u32, provider: &'static str, devices: u32, - policy: &'static str, cc_mode: &'static str, devtools: bool, evidence_sha256: String, } + pub(super) struct GpuAttestationResult { + claims: Vec, + event: Vec, + } + + impl GpuAttestationResult { + pub(super) fn claims(&self) -> &[Value] { + &self.claims + } + + pub(super) fn event(&self) -> &[u8] { + &self.event + } + } + #[derive(Deserialize)] struct NvattestOutput { result_code: i64, @@ -1251,7 +1274,7 @@ mod gpu { stdout: &[u8], nonce: &str, expected_devices: u32, - ) -> Result<()> { + ) -> Result> { let output: NvattestOutput = serde_json::from_slice(stdout).context("failed to parse nvattest JSON output")?; if output.result_code != 0 { @@ -1268,33 +1291,8 @@ mod gpu { .as_object() .with_context(|| format!("invalid GPU claim at index {index}"))?; let string_claim = |name: &str| claim.get(name).and_then(Value::as_str); - let valid_cert_chain = |name: &str| { - let cert = claim.get(name).and_then(Value::as_object)?; - Some( - cert.get("x-nvidia-cert-status").and_then(Value::as_str) == Some("valid") - && cert - .get("x-nvidia-cert-ocsp-status") - .and_then(Value::as_str) - == Some("good") - && cert - .get("x-nvidia-cert-ocsp-nonce-matches") - .and_then(Value::as_bool) - == Some(true) - && cert - .get("x-nvidia-cert-ocsp-response-valid") - .and_then(Value::as_bool) - == Some(true), - ) - }; - if string_claim("x-nvidia-device-type") != Some("gpu") - || claim.get("secboot").and_then(Value::as_bool) != Some(true) - || string_claim("dbgstat") != Some("disabled") - || string_claim("measres") != Some("success") - || valid_cert_chain("x-nvidia-gpu-attestation-report-cert-chain") != Some(true) - || valid_cert_chain("x-nvidia-gpu-driver-rim-cert-chain") != Some(true) - || valid_cert_chain("x-nvidia-gpu-vbios-rim-cert-chain") != Some(true) - { - bail!("gpu claim at index {index} violates the production policy"); + if string_claim("x-nvidia-device-type") != Some("gpu") { + bail!("gpu claim at index {index} has an invalid device type"); } if string_claim("eat_nonce") != Some(nonce) || claim @@ -1305,7 +1303,7 @@ mod gpu { bail!("gpu claim at index {index} has an invalid nonce"); } } - Ok(()) + Ok(output.claims) } fn attestation_event(stdout: &[u8], devices: u32) -> Result> { @@ -1313,7 +1311,6 @@ mod gpu { version: EVENT_VERSION, provider: "nvidia", devices, - policy: POLICY_VERSION, cc_mode: "on", devtools: false, evidence_sha256: hex::encode(sha256(stdout)), @@ -1321,16 +1318,14 @@ mod gpu { serde_json::to_vec(&event).context("failed to serialize GPU attestation event") } - /// Run local GPU attestation via nvattest with a fresh nonce and a - /// mandatory relying-party policy. The JSON output is preserved under - /// /run; its hash is returned in a versioned summary for RTMR3. - pub(super) async fn attest_gpu(expected_devices: u32) -> Result> { + /// Run local GPU attestation via nvattest with a fresh nonce and no custom + /// relying-party policy. nvattest's built-in appraisal still applies. The + /// complete JSON output is preserved under /run, while its claims and a + /// versioned summary are returned for the application policy and RTMR3. + pub(super) async fn attest_gpu(expected_devices: u32) -> Result { if !Path::new(NVATTEST).exists() { bail!("nvattest is not available in this image"); } - if !Path::new(NVIDIA_POLICY).exists() { - bail!("gpu attestation policy is not available in this image"); - } // Certificate/OCSP validation needs a sane clock even when // secure_time is off; best-effort step chrony before attesting. if let Err(err) = cmd!(chronyc makestep) { @@ -1345,8 +1340,6 @@ mod gpu { "gpu", "--verifier", "local", - "--relying-party-policy", - NVIDIA_POLICY, "--nonce", &nonce, "--format", @@ -1358,6 +1351,7 @@ mod gpu { if !output.stderr.is_empty() { info!("nvattest: {}", truncated_lossy(&output.stderr, 2048)); } + save_attestation_output(&output.stdout).context("failed to save GPU attestation output")?; if !output.status.success() { bail!( "nvattest exited with {}: {}", @@ -1365,10 +1359,37 @@ mod gpu { truncated_lossy(&output.stderr, 512), ); } - validate_attestation_output(&output.stdout, &nonce, expected_devices)?; + let claims = validate_attestation_output(&output.stdout, &nonce, expected_devices)?; verify_production_cc_mode().await?; - save_attestation_output(&output.stdout).context("failed to save GPU attestation output")?; - attestation_event(&output.stdout, expected_devices) + Ok(GpuAttestationResult { + claims, + event: attestation_event(&output.stdout, expected_devices)?, + }) + } + + pub(super) fn policy_digest(policy: &str) -> [u8; 32] { + sha256(policy.as_bytes()) + } + + /// Evaluate the app-provided Rego v0 policy using the same input shape as + /// NVIDIA relying-party policies: the nvattest `claims` JSON array. + pub(super) fn evaluate_policy(policy: &str, claims: &[Value]) -> Result<()> { + let mut engine = regorus::Engine::new(); + engine.set_rego_v0(true); + engine + .add_policy("gpu-policy.rego".to_string(), policy.to_string()) + .context("failed to load GPU policy")?; + let input = serde_json::to_string(claims).context("failed to serialize GPU claims")?; + engine + .set_input_json(&input) + .context("failed to set GPU policy input")?; + if !engine + .eval_bool_query(POLICY_ENTRYPOINT.to_string(), false) + .context("failed to evaluate GPU policy")? + { + bail!("gpu policy rejected the attestation claims"); + } + Ok(()) } fn save_attestation_output(stdout: &[u8]) -> Result<()> { @@ -1405,24 +1426,12 @@ mod gpu { } fn nvattest_output(nonce: &str, claims: usize) -> Vec { - let cert_chain = serde_json::json!({ - "x-nvidia-cert-status": "valid", - "x-nvidia-cert-ocsp-status": "good", - "x-nvidia-cert-ocsp-nonce-matches": true, - "x-nvidia-cert-ocsp-response-valid": true - }); let claims = (0..claims) .map(|_| { serde_json::json!({ "x-nvidia-device-type": "gpu", - "secboot": true, - "dbgstat": "disabled", - "measres": "success", "eat_nonce": nonce, - "x-nvidia-gpu-attestation-report-nonce-match": true, - "x-nvidia-gpu-attestation-report-cert-chain": cert_chain, - "x-nvidia-gpu-driver-rim-cert-chain": cert_chain, - "x-nvidia-gpu-vbios-rim-cert-chain": cert_chain + "x-nvidia-gpu-attestation-report-nonce-match": true }) }) .collect::>(); @@ -1475,18 +1484,25 @@ mod gpu { } #[test] - fn nvattest_output_requires_every_expected_production_gpu() { + fn nvattest_output_requires_every_expected_gpu_and_fresh_nonce() { let nonce = "11".repeat(32); let valid = nvattest_output(&nonce, 2); validate_attestation_output(&valid, &nonce, 2).unwrap(); assert!(validate_attestation_output(&valid, &nonce, 1).is_err()); let mut invalid: Value = serde_json::from_slice(&valid).unwrap(); - invalid["claims"][1]["dbgstat"] = Value::String("enabled".to_string()); + invalid["claims"][1]["eat_nonce"] = Value::String("stale".to_string()); assert!( validate_attestation_output(&serde_json::to_vec(&invalid).unwrap(), &nonce, 2) .is_err() ); + + // Claim appraisal belongs to nvattest's default policy and the + // optional app policy, not a second policy hard-coded here. + let mut extra_claims: Value = serde_json::from_slice(&valid).unwrap(); + extra_claims["claims"][0]["dbgstat"] = Value::String("enabled".to_string()); + validate_attestation_output(&serde_json::to_vec(&extra_claims).unwrap(), &nonce, 2) + .unwrap(); } #[test] @@ -1497,11 +1513,37 @@ mod gpu { serde_json::from_slice(&attestation_event(&output, 1).unwrap()).unwrap(); assert_eq!(event["version"], EVENT_VERSION); assert_eq!(event["devices"], 1); - assert_eq!(event["policy"], POLICY_VERSION); + assert!(event.get("policy").is_none()); assert_eq!(event["cc_mode"], "on"); assert_eq!(event["devtools"], false); assert_eq!(event["evidence_sha256"], hex::encode(sha256(&output))); } + + #[test] + fn app_policy_receives_claims_array_and_must_return_true() { + let claims = vec![serde_json::json!({"status": "accepted"})]; + let policy = r#" + package policy + default nv_match = false + nv_match { + count(input) == 1 + input[0].status == "accepted" + } + "#; + evaluate_policy(policy, &claims).unwrap(); + + let rejected = vec![serde_json::json!({"status": "rejected"})]; + assert!(evaluate_policy(policy, &rejected).is_err()); + assert!(evaluate_policy("package policy", &claims).is_err()); + assert!(evaluate_policy("not valid rego", &claims).is_err()); + } + + #[test] + fn policy_digest_uses_exact_utf8_bytes() { + let policy = "package policy\n\ndefault nv_match = true\n"; + assert_eq!(policy_digest(policy), sha256(policy.as_bytes())); + assert_ne!(policy_digest(policy), policy_digest(policy.trim())); + } } } @@ -1509,32 +1551,28 @@ impl Stage0<'_> { /// Enforce `requirements.attest_gpu` (default true): attest an attached /// NVIDIA GPU before continuing to key provisioning, or — when explicitly /// disabled — set the GPU ready state without verification. - async fn setup_gpu(&self) -> Result<()> { + async fn setup_gpu(&self) -> Result> { let inventory = gpu::gpu_inventory()?; if !self.shared.app_compose.attest_gpu() { if inventory.nvidia == 0 { - return Ok(()); + return Ok(None); } warn!("requirements.attest_gpu is false; setting GPU ready state without attestation"); // Best-effort: a GPU with CC mode off has no ready state to set. if let Err(err) = gpu::set_gpu_ready_state().await { warn!("failed to set GPU ready state: {err:?}"); } - return Ok(()); + return Ok(None); } let expected_devices = gpu::expected_gpu_count(&self.shared.sys_config.vm_config, inventory)?; if expected_devices == 0 { - return Ok(()); + return Ok(None); } self.vmm.notify_q("boot.progress", "attesting GPU").await; info!("verifying GPU TEE attestation"); - let event = gpu::attest_gpu(expected_devices).await?; - gpu::set_gpu_ready_state().await?; - emit_runtime_event("gpu-attestation", &event) - .context("failed to emit GPU attestation event")?; - info!("GPU TEE attestation succeeded"); - Ok(()) + let attestation = gpu::attest_gpu(expected_devices).await?; + Ok(Some(attestation)) } } @@ -2146,7 +2184,10 @@ impl<'a> Stage0<'a> { Ok(()) } - fn measure_app_info(&self) -> Result { + async fn measure_app_info( + &self, + gpu_attestation: Option<&gpu::GpuAttestationResult>, + ) -> Result { let compose_hash = sha256_file(self.shared.dir.app_compose_file())?; let truncated_compose_hash = truncate(&compose_hash, 20); let key_provider = self.shared.app_compose.key_provider(); @@ -2198,6 +2239,29 @@ impl<'a> Stage0<'a> { emit_runtime_event("system-preparing", &[])?; emit_runtime_event("app-id", &instance_info.app_id)?; emit_runtime_event("compose-hash", &compose_hash)?; + + if let Some(policy) = self + .shared + .app_compose + .requirements + .as_ref() + .and_then(|requirements| requirements.gpu_policy.as_deref()) + { + emit_runtime_event("gpu-policy", &gpu::policy_digest(policy))?; + let claims = gpu_attestation + .map(gpu::GpuAttestationResult::claims) + .unwrap_or(&[]); + gpu::evaluate_policy(policy, claims).context("failed to apply GPU policy")?; + info!("application GPU policy accepted the attestation claims"); + } + + if let Some(attestation) = gpu_attestation { + gpu::set_gpu_ready_state().await?; + emit_runtime_event("gpu-attestation", attestation.event()) + .context("failed to emit GPU attestation event")?; + info!("GPU TEE attestation succeeded"); + } + emit_runtime_event("instance-id", &instance_id)?; emit_runtime_event("boot-mr-done", &[])?; @@ -2259,9 +2323,13 @@ impl<'a> Stage0<'a> { Ok(()) } - async fn setup_fs(self) -> Result> { + async fn setup_fs( + self, + gpu_attestation: Option, + ) -> Result> { let app_info = self - .measure_app_info() + .measure_app_info(gpu_attestation.as_ref()) + .await .context("Failed to measure app info")?; if self.shared.app_compose.key_provider().is_kms() { cmd_show_mrs()?; @@ -2789,6 +2857,24 @@ fn test_empty_requirements_require_v3_manifest() { assert!(err.to_string().contains("requires manifest_version")); } +#[test] +fn test_gpu_policy_requires_gpu_attestation() { + let app_compose: AppCompose = serde_json::from_value(serde_json::json!({ + "manifest_version": "3", + "name": "test", + "runner": "docker-compose", + "requirements": { + "attest_gpu": false, + "gpu_policy": "package policy\n\ndefault nv_match = true\n" + } + })) + .unwrap(); + let err = verify_manifest_feature_requirements(&app_compose).unwrap_err(); + assert!(err + .to_string() + .contains("requirements.gpu_policy requires GPU attestation")); +} + #[test] fn test_platform_requirements_omitted_accepts_any_platform() { let app_compose = test_app_compose(serde_json::json!("3"), None, None); diff --git a/os/yocto/layers/meta-nvidia/recipes-graphics/nvattest/files/10-nvidia-gpu-ordering.conf b/os/yocto/layers/meta-nvidia/recipes-graphics/nvattest/files/10-nvidia-gpu-ordering.conf index 64d356563..53923fd4e 100644 --- a/os/yocto/layers/meta-nvidia/recipes-graphics/nvattest/files/10-nvidia-gpu-ordering.conf +++ b/os/yocto/layers/meta-nvidia/recipes-graphics/nvattest/files/10-nvidia-gpu-ordering.conf @@ -1,6 +1,6 @@ # Ordering only, not a hard dependency: on GPU instances, `dstack-util setup` # (run by dstack-prepare.service) performs local GPU TEE attestation per the -# app-compose `requirements.attest_gpu` policy, which needs the nvidia driver +# app-compose `requirements.attest_gpu` setting, which needs the nvidia driver # loaded (nvidia-persistenced) and the NVLink fabric up (nvidia-fabricmanager). # Both units skip cleanly on GPU-less instances via nvidia-gpu-detect. [Unit] diff --git a/os/yocto/layers/meta-nvidia/recipes-graphics/nvattest/files/dstack-gpu-attestation.rego b/os/yocto/layers/meta-nvidia/recipes-graphics/nvattest/files/dstack-gpu-attestation.rego deleted file mode 100644 index 8cf8ee479..000000000 --- a/os/yocto/layers/meta-nvidia/recipes-graphics/nvattest/files/dstack-gpu-attestation.rego +++ /dev/null @@ -1,29 +0,0 @@ -# SPDX-FileCopyrightText: © 2026 Phala Network -# -# SPDX-License-Identifier: Apache-2.0 - -package policy - -import future.keywords.every - -default nv_match := false - -nv_match { - count(input) > 0 - every claim in input { - claim["x-nvidia-device-type"] == "gpu" - claim.measres == "success" - claim.secboot == true - claim.dbgstat == "disabled" - valid_cert_chain(claim["x-nvidia-gpu-attestation-report-cert-chain"]) - valid_cert_chain(claim["x-nvidia-gpu-driver-rim-cert-chain"]) - valid_cert_chain(claim["x-nvidia-gpu-vbios-rim-cert-chain"]) - } -} - -valid_cert_chain(cert_chain) { - cert_chain["x-nvidia-cert-status"] == "valid" - cert_chain["x-nvidia-cert-ocsp-status"] == "good" - cert_chain["x-nvidia-cert-ocsp-nonce-matches"] == true - cert_chain["x-nvidia-cert-ocsp-response-valid"] == true -} diff --git a/os/yocto/layers/meta-nvidia/recipes-graphics/nvattest/nvattest_2026.06.09.bb b/os/yocto/layers/meta-nvidia/recipes-graphics/nvattest/nvattest_2026.06.09.bb index 5eee6f712..121794421 100644 --- a/os/yocto/layers/meta-nvidia/recipes-graphics/nvattest/nvattest_2026.06.09.bb +++ b/os/yocto/layers/meta-nvidia/recipes-graphics/nvattest/nvattest_2026.06.09.bb @@ -1,5 +1,5 @@ SUMMARY = "NVIDIA GPU attestation CLI" -DESCRIPTION = "Builds NVIDIA's nvattest CLI. dstack-util setup runs it at boot to gate readiness on local GPU TEE attestation (app-compose requirements.attest_gpu)." +DESCRIPTION = "Builds NVIDIA's nvattest CLI. dstack-util setup runs it at boot to gate readiness on local GPU TEE attestation and an optional application policy." HOMEPAGE = "https://github.com/NVIDIA/attestation-sdk" LICENSE = "Apache-2.0" LIC_FILES_CHKSUM = "file://LICENSE;md5=e620fc90e76c4aa0c3efdd1673ca0b3b" @@ -7,7 +7,6 @@ LIC_FILES_CHKSUM = "file://LICENSE;md5=e620fc90e76c4aa0c3efdd1673ca0b3b" SRC_URI = " \ git://github.com/NVIDIA/attestation-sdk.git;protocol=https;branch=main \ file://10-nvidia-gpu-ordering.conf \ - file://dstack-gpu-attestation.rego \ " SRCREV = "9d12801cea8a198ea0f29640dfaf8a4017c841c5" @@ -102,18 +101,10 @@ do_install() { install -m 0644 ${UNPACKDIR}/10-nvidia-gpu-ordering.conf \ ${D}${systemd_system_unitdir}/dstack-prepare.service.d/10-nvidia-gpu-ordering.conf - # dstack-util passes this policy explicitly. It retains every check from - # the SDK's default appraisal and additionally requires production debug - # status. (The pinned SDK does not expose GPU mode as a Rego claim, so - # dstack-util separately requires CC=ON and DevTools=OFF.) - install -d ${D}${datadir}/nvattest - install -m 0644 ${UNPACKDIR}/dstack-gpu-attestation.rego \ - ${D}${datadir}/nvattest/dstack-gpu-attestation.rego } FILES:${PN} += " \ ${systemd_system_unitdir}/dstack-prepare.service.d/10-nvidia-gpu-ordering.conf \ - ${datadir}/nvattest/dstack-gpu-attestation.rego \ ${libdir}/lib*.so \ ${libdir}/lib*.so.* \ " diff --git a/os/yocto/layers/meta-nvidia/recipes-graphics/nvidia/files/nvidia-persistenced.service b/os/yocto/layers/meta-nvidia/recipes-graphics/nvidia/files/nvidia-persistenced.service index 166e6295d..6c703bef0 100644 --- a/os/yocto/layers/meta-nvidia/recipes-graphics/nvidia/files/nvidia-persistenced.service +++ b/os/yocto/layers/meta-nvidia/recipes-graphics/nvidia/files/nvidia-persistenced.service @@ -18,8 +18,8 @@ ExecStartPre=/usr/bin/env modprobe nvidia-uvm ExecStart=/usr/bin/env nvidia-persistenced --uvm-persistence-mode # NOTE: the GPU ready state (conf-compute -srs 1) is intentionally NOT set # here. It is set by `dstack-util setup` (dstack-prepare.service), only after -# local GPU TEE attestation via nvattest succeeds -- unless the deployment -# opts out with app-compose requirements.attest_gpu=false. +# local GPU TEE attestation and any application GPU policy succeed -- unless +# the deployment opts out with app-compose requirements.attest_gpu=false. RemainAfterExit=yes [Install] From 73b26e952220c241cfa6549f38b6d4e2ceac67c1 Mon Sep 17 00:00:00 2001 From: Kevin Wang Date: Thu, 16 Jul 2026 04:59:26 -0700 Subject: [PATCH 04/32] feat(guest): structure GPU policy settings --- docs/security/security-model.md | 13 ++- dstack/Cargo.lock | 1 + dstack/dstack-types/src/lib.rs | 55 ++++++++++--- dstack/dstack-util/Cargo.toml | 1 + dstack/dstack-util/src/system_setup.rs | 105 +++++++++++++++++-------- 5 files changed, 127 insertions(+), 48 deletions(-) diff --git a/docs/security/security-model.md b/docs/security/security-model.md index 47a98a0dc..c289ac37f 100644 --- a/docs/security/security-model.md +++ b/docs/security/security-model.md @@ -57,9 +57,14 @@ dstack supports NVIDIA H100, H200, and B200 GPUs in confidential compute mode fo GPUs are passed through via VFIO to the TEE-protected CVM. Before key provisioning, `dstack-util setup` inventories every VGA/3D-controller PCI function, compares that count with `vm_config.num_gpus`, and runs NVIDIA's local `nvattest` verifier over every NVIDIA-driver-visible GPU with a fresh nonce. dstack does not ship or pass a custom relying-party policy to this command; nvattest's built-in appraisal still applies. The complete JSON result (`result_code`, `result_message`, `claims`, and `detached_eat`) is saved at `/run/nvidia-gpu-attestation/attestation.out`. -An application may additionally set `requirements.gpu_policy` to a Rego v0 script. Its input is the nvattest output's `claims` array, and it must define the boolean entrypoint `data.policy.nv_match` in `package policy`. Immediately after measuring `compose-hash`, dstack measures `sha256(gpu_policy UTF-8 bytes)` in a `gpu-policy` event and then evaluates the policy. A false, undefined, malformed, or non-boolean result stops boot before key provisioning. If no GPU is configured, the policy receives an empty array. `gpu_policy` cannot be combined with `attest_gpu: false`. +An application may additionally set `requirements.gpu_policy` to an object with the following deny-unknown-fields schema: -Separate fail-closed `nvidia-smi` queries require the current CC feature to be ON and DevTools mode to be OFF. Only after the default appraisal, optional application policy, and these checks succeed does dstack set the GPU ready state. The dstack CPU/guest boot chain is verified independently through measured boot; a GPU claim named `secboot` refers to the GPU appraisal, not UEFI Secure Boot in the CVM. +- `rego` (optional string): a Rego v0 script evaluated with the nvattest output's `claims` array as `input`. It must define the boolean entrypoint `data.policy.nv_match` in `package policy`. +- `allow_devtools` (boolean, default `false`): permit NVIDIA DevTools mode. Production applications should leave this disabled because DevTools removes the expected GPU memory-confidentiality guarantee. + +Immediately after measuring `compose-hash`, dstack applies field defaults, JCS-canonicalizes the complete `gpu_policy` structure, and measures its SHA-256 digest in a `gpu-policy` event. It then applies the basic settings and optional Rego policy. A false, undefined, malformed, or non-boolean Rego result stops boot before key provisioning. If no GPU is configured, Rego receives an empty array. `gpu_policy` cannot be combined with `attest_gpu: false`. + +Separate fail-closed `nvidia-smi` queries always require the current CC feature to be ON and require DevTools mode to be OFF unless the measured policy explicitly permits it. Only after the default appraisal, optional application policy, and these checks succeed does dstack set the GPU ready state. The dstack CPU/guest boot chain is verified independently through measured boot; a GPU claim named `secboot` refers to the GPU appraisal, not UEFI Secure Boot in the CVM. ### Dual Attestation @@ -71,7 +76,7 @@ A verifier must replay the measured event log, require exactly one pre-`system-r The GPU gate assumes a malicious host/VMM and untrusted host-provided PCI topology, while trusting the CPU TEE, the measured dstack guest/kernel, NVIDIA hardware/firmware roots, and the cryptography used by both attestation chains. Availability is out of scope. -The events make the following **boot-time** statement: immediately before key provisioning, all attached VGA/3D PCI functions were NVIDIA devices, their count matched the quote-bound VM configuration, nvattest returned one fresh successfully appraised claim for each device, the measured application policy accepted those claims when present, CC was ON, DevTools was OFF, and setting the GPU ready state succeeded. This closes these cases: +The events make the following **boot-time** statement: immediately before key provisioning, all attached VGA/3D PCI functions were NVIDIA devices, their count matched the quote-bound VM configuration, nvattest returned one fresh successfully appraised claim for each device, the measured application policy accepted those claims and GPU state when present, CC was ON, DevTools complied with that policy (OFF by default), and setting the GPU ready state succeeded. This closes these cases: - A GPU-less launch cannot be presented as a GPU-verified launch: it has `num_gpus == 0` and no `gpu-attestation` event. - A mixed launch cannot attest only its TEE-capable subset. Non-NVIDIA display GPUs are rejected, and the sysfs, `vm_config`, and nvattest claim counts must all agree. A non-CC NVIDIA GPU either prevents evidence collection/appraisal or causes the default appraisal, application policy, or CC-state check to fail. @@ -180,7 +185,7 @@ Use this checklist to verify a workload running in a dstack CVM. - [ ] `vm_config.num_gpus` is greater than zero and matches the expected deployment - [ ] If `requirements.gpu_policy` is required, exactly one `gpu-policy` event follows `compose-hash`, and its payload matches the expected SHA-256 digest - [ ] Exactly one `gpu-attestation` event appears before `system-ready` -- [ ] The event device count matches `vm_config.num_gpus`, and its CC/DevTools fields indicate production mode +- [ ] The event device count matches `vm_config.num_gpus`, CC is ON, and its DevTools field complies with the measured policy - [ ] The platform binds the event log to a quoted RTMR/PCR (do not accept it from current SEV-SNP evidence) **Key management verification:** diff --git a/dstack/Cargo.lock b/dstack/Cargo.lock index 22f63d33b..873323d91 100644 --- a/dstack/Cargo.lock +++ b/dstack/Cargo.lock @@ -2034,6 +2034,7 @@ dependencies = [ "semver", "serde", "serde-human-bytes", + "serde_jcs", "serde_json", "sha2 0.10.9", "sha3", diff --git a/dstack/dstack-types/src/lib.rs b/dstack/dstack-types/src/lib.rs index 24896d1ed..dcd728f2e 100644 --- a/dstack/dstack-types/src/lib.rs +++ b/dstack/dstack-types/src/lib.rs @@ -121,6 +121,19 @@ pub struct AppCompose { pub requirements: Option, } +#[derive(Deserialize, Serialize, Debug, Clone, Default, PartialEq, Eq)] +#[serde(default, deny_unknown_fields)] +pub struct GpuPolicy { + /// Optional Rego v0 policy evaluated against NVIDIA nvattest's `claims` + /// array. It must define the boolean rule `data.policy.nv_match`. + #[serde(skip_serializing_if = "Option::is_none")] + pub rego: Option, + /// Permit NVIDIA DevTools mode. This defaults to false because DevTools + /// disables the GPU memory-confidentiality guarantees expected in + /// production. + pub allow_devtools: bool, +} + #[derive(Deserialize, Serialize, Debug, Clone, Default, PartialEq, Eq)] #[serde(default, deny_unknown_fields)] pub struct Requirements { @@ -159,15 +172,12 @@ pub struct Requirements { /// are unaffected either way. #[serde(skip_serializing_if = "Option::is_none")] pub attest_gpu: Option, - /// Optional application-supplied Rego policy evaluated against the - /// `claims` array in NVIDIA nvattest's JSON output before key - /// provisioning. The policy must define the boolean rule - /// `data.policy.nv_match`. + /// Optional application GPU policy applied before key provisioning. /// - /// The SHA-256 digest of the exact UTF-8 policy bytes is emitted as the - /// `gpu-policy` launch event immediately after `compose-hash`. + /// Its default-expanded, JCS-canonicalized JSON SHA-256 digest is emitted + /// as the `gpu-policy` launch event immediately after `compose-hash`. #[serde(skip_serializing_if = "Option::is_none")] - pub gpu_policy: Option, + pub gpu_policy: Option, } impl Requirements { @@ -466,7 +476,10 @@ mod app_compose_tests { "platforms": ["dstack-gcp-tdx", "dstack-tdx"], "tdx_measure_acpi_tables": true, "launch_token_hash": "9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08", - "gpu_policy": "package policy\n\ndefault nv_match = false\n" + "gpu_policy": { + "rego": "package policy\n\ndefault nv_match = false\n", + "allow_devtools": true + } } })) .unwrap(); @@ -481,10 +494,12 @@ mod app_compose_tests { requirements.launch_token_hash.as_deref(), Some("9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08") ); + let gpu_policy = requirements.gpu_policy.as_ref().unwrap(); assert_eq!( - requirements.gpu_policy.as_deref(), + gpu_policy.rego.as_deref(), Some("package policy\n\ndefault nv_match = false\n") ); + assert!(gpu_policy.allow_devtools); let err = serde_json::from_value::(serde_json::json!({ "manifest_version": "3", @@ -555,13 +570,31 @@ mod app_compose_tests { "name": "test", "runner": "docker-compose", "requirements": { - "gpu_policy": "package policy\n\ndefault nv_match = false\n" + "gpu_policy": { + "rego": "package policy\n\ndefault nv_match = false\n" + } } })) .unwrap(); let requirements = gpu_policy.requirements.as_ref().unwrap(); - assert!(requirements.gpu_policy.is_some()); + let gpu_policy = requirements.gpu_policy.as_ref().unwrap(); + assert!(gpu_policy.rego.is_some()); + assert!(!gpu_policy.allow_devtools); assert!(!requirements.is_empty()); + + let err = serde_json::from_value::(serde_json::json!({ + "manifest_version": "3", + "name": "test", + "runner": "docker-compose", + "requirements": { + "gpu_policy": { + "rego": "package policy", + "allow_debug": true + } + } + })) + .unwrap_err(); + assert!(err.to_string().contains("unknown field")); } #[test] diff --git a/dstack/dstack-util/Cargo.toml b/dstack/dstack-util/Cargo.toml index f43a243d7..9a567f6a6 100644 --- a/dstack/dstack-util/Cargo.toml +++ b/dstack/dstack-util/Cargo.toml @@ -25,6 +25,7 @@ serde.workspace = true serde-human-bytes.workspace = true semver.workspace = true serde_json.workspace = true +serde_jcs.workspace = true sha2.workspace = true tokio = { workspace = true, features = ["full"] } tracing.workspace = true diff --git a/dstack/dstack-util/src/system_setup.rs b/dstack/dstack-util/src/system_setup.rs index 699e303ed..2c38b01e6 100644 --- a/dstack/dstack-util/src/system_setup.rs +++ b/dstack/dstack-util/src/system_setup.rs @@ -20,7 +20,7 @@ use dstack_types::{ APP_COMPOSE, APP_KEYS, DECRYPTED_ENV, DECRYPTED_ENV_JSON, ENCRYPTED_ENV, HOST_SHARED_DIR_NAME, HOST_SHARED_DISK_LABEL, INSTANCE_INFO, SYS_CONFIG, USER_CONFIG, }, - KeyProvider, KeyProviderInfo, + GpuPolicy, KeyProvider, KeyProviderInfo, }; use fs_err as fs; use luks2::{ @@ -1114,7 +1114,8 @@ mod gpu { pub(super) struct GpuAttestationResult { claims: Vec, - event: Vec, + output: Vec, + devices: u32, } impl GpuAttestationResult { @@ -1122,8 +1123,8 @@ mod gpu { &self.claims } - pub(super) fn event(&self) -> &[u8] { - &self.event + pub(super) fn event(&self, devtools: bool) -> Result> { + attestation_event(&self.output, self.devices, devtools) } } @@ -1236,20 +1237,28 @@ mod gpu { parse_nvidia_smi_state(&output.stdout, name) } - /// Require the current system-wide GPU state to be production CC. NVAT's - /// NVML collector accepts protected-PCIe as well as CC and its default - /// appraisal does not reject DevTools, so a successful appraisal alone is - /// not a sufficient confidentiality check. - async fn verify_production_cc_mode() -> Result<()> { - if !query_nvidia_smi_state("-f", "CC feature state").await? { + fn validate_cc_state(cc_enabled: bool, devtools: bool, allow_devtools: bool) -> Result<()> { + if !cc_enabled { bail!("nvidia confidential compute mode is not enabled"); } - if query_nvidia_smi_state("-d", "DevTools mode").await? { + if devtools && !allow_devtools { bail!("nvidia DevTools mode is enabled"); } Ok(()) } + /// Require the current system-wide GPU state to use CC. NVAT's NVML + /// collector accepts protected-PCIe as well as CC, so a successful + /// appraisal alone is not a sufficient confidentiality check. DevTools is + /// rejected by default but can be explicitly allowed by the measured app + /// policy. + pub(super) async fn verify_cc_mode(allow_devtools: bool) -> Result { + let cc_enabled = query_nvidia_smi_state("-f", "CC feature state").await?; + let devtools = query_nvidia_smi_state("-d", "DevTools mode").await?; + validate_cc_state(cc_enabled, devtools, allow_devtools)?; + Ok(devtools) + } + /// Mark the GPU as ready to accept work. Only meaningful (and only /// succeeds) when the GPU runs in CC mode. pub(super) async fn set_gpu_ready_state() -> Result<()> { @@ -1306,13 +1315,13 @@ mod gpu { Ok(output.claims) } - fn attestation_event(stdout: &[u8], devices: u32) -> Result> { + fn attestation_event(stdout: &[u8], devices: u32, devtools: bool) -> Result> { let event = GpuAttestationEvent { version: EVENT_VERSION, provider: "nvidia", devices, cc_mode: "on", - devtools: false, + devtools, evidence_sha256: hex::encode(sha256(stdout)), }; serde_json::to_vec(&event).context("failed to serialize GPU attestation event") @@ -1320,8 +1329,8 @@ mod gpu { /// Run local GPU attestation via nvattest with a fresh nonce and no custom /// relying-party policy. nvattest's built-in appraisal still applies. The - /// complete JSON output is preserved under /run, while its claims and a - /// versioned summary are returned for the application policy and RTMR3. + /// complete JSON output is preserved under /run and retained with its + /// claims for the application policy and versioned RTMR3 summary. pub(super) async fn attest_gpu(expected_devices: u32) -> Result { if !Path::new(NVATTEST).exists() { bail!("nvattest is not available in this image"); @@ -1360,15 +1369,16 @@ mod gpu { ); } let claims = validate_attestation_output(&output.stdout, &nonce, expected_devices)?; - verify_production_cc_mode().await?; Ok(GpuAttestationResult { claims, - event: attestation_event(&output.stdout, expected_devices)?, + output: output.stdout, + devices: expected_devices, }) } - pub(super) fn policy_digest(policy: &str) -> [u8; 32] { - sha256(policy.as_bytes()) + pub(super) fn policy_digest(policy: &GpuPolicy) -> Result<[u8; 32]> { + let canonical = serde_jcs::to_vec(policy).context("failed to canonicalize GPU policy")?; + Ok(sha256(&canonical)) } /// Evaluate the app-provided Rego v0 policy using the same input shape as @@ -1483,6 +1493,14 @@ mod gpu { assert!(parse_nvidia_smi_state(b"current: ON, pending: OFF\n", "CC").is_err()); } + #[test] + fn basic_policy_requires_cc_and_rejects_devtools_by_default() { + validate_cc_state(true, false, false).unwrap(); + assert!(validate_cc_state(false, false, false).is_err()); + assert!(validate_cc_state(true, true, false).is_err()); + validate_cc_state(true, true, true).unwrap(); + } + #[test] fn nvattest_output_requires_every_expected_gpu_and_fresh_nonce() { let nonce = "11".repeat(32); @@ -1510,12 +1528,12 @@ mod gpu { let nonce = "22".repeat(32); let output = nvattest_output(&nonce, 1); let event: Value = - serde_json::from_slice(&attestation_event(&output, 1).unwrap()).unwrap(); + serde_json::from_slice(&attestation_event(&output, 1, true).unwrap()).unwrap(); assert_eq!(event["version"], EVENT_VERSION); assert_eq!(event["devices"], 1); assert!(event.get("policy").is_none()); assert_eq!(event["cc_mode"], "on"); - assert_eq!(event["devtools"], false); + assert_eq!(event["devtools"], true); assert_eq!(event["evidence_sha256"], hex::encode(sha256(&output))); } @@ -1539,10 +1557,22 @@ mod gpu { } #[test] - fn policy_digest_uses_exact_utf8_bytes() { - let policy = "package policy\n\ndefault nv_match = true\n"; - assert_eq!(policy_digest(policy), sha256(policy.as_bytes())); - assert_ne!(policy_digest(policy), policy_digest(policy.trim())); + fn policy_digest_commits_to_the_canonical_policy_structure() { + let policy = GpuPolicy { + rego: Some("package policy\n\ndefault nv_match = true\n".to_string()), + allow_devtools: true, + }; + let canonical = serde_jcs::to_vec(&policy).unwrap(); + assert_eq!(policy_digest(&policy).unwrap(), sha256(&canonical)); + + let production_policy = GpuPolicy { + allow_devtools: false, + ..policy.clone() + }; + assert_ne!( + policy_digest(&policy).unwrap(), + policy_digest(&production_policy).unwrap() + ); } } } @@ -2240,27 +2270,34 @@ impl<'a> Stage0<'a> { emit_runtime_event("app-id", &instance_info.app_id)?; emit_runtime_event("compose-hash", &compose_hash)?; - if let Some(policy) = self + let gpu_policy = self .shared .app_compose .requirements .as_ref() - .and_then(|requirements| requirements.gpu_policy.as_deref()) - { - emit_runtime_event("gpu-policy", &gpu::policy_digest(policy))?; + .and_then(|requirements| requirements.gpu_policy.as_ref()); + if let Some(policy) = gpu_policy { + emit_runtime_event("gpu-policy", &gpu::policy_digest(policy)?)?; let claims = gpu_attestation .map(gpu::GpuAttestationResult::claims) .unwrap_or(&[]); - gpu::evaluate_policy(policy, claims).context("failed to apply GPU policy")?; - info!("application GPU policy accepted the attestation claims"); + if let Some(rego) = policy.rego.as_deref() { + gpu::evaluate_policy(rego, claims).context("failed to apply GPU Rego policy")?; + } } if let Some(attestation) = gpu_attestation { + let allow_devtools = gpu_policy.is_some_and(|policy| policy.allow_devtools); + let devtools = gpu::verify_cc_mode(allow_devtools).await?; gpu::set_gpu_ready_state().await?; - emit_runtime_event("gpu-attestation", attestation.event()) + let event = attestation.event(devtools)?; + emit_runtime_event("gpu-attestation", &event) .context("failed to emit GPU attestation event")?; info!("GPU TEE attestation succeeded"); } + if gpu_policy.is_some() { + info!("application GPU policy accepted the attestation claims and state"); + } emit_runtime_event("instance-id", &instance_id)?; emit_runtime_event("boot-mr-done", &[])?; @@ -2865,7 +2902,9 @@ fn test_gpu_policy_requires_gpu_attestation() { "runner": "docker-compose", "requirements": { "attest_gpu": false, - "gpu_policy": "package policy\n\ndefault nv_match = true\n" + "gpu_policy": { + "rego": "package policy\n\ndefault nv_match = true\n" + } } })) .unwrap(); From 34f194ae3d681a8c80fea8fe0b4bf74bde3172d4 Mon Sep 17 00:00:00 2001 From: Kevin Wang Date: Thu, 16 Jul 2026 05:52:58 -0700 Subject: [PATCH 05/32] refactor(guest): consolidate post-measurement GPU setup --- docs/security/security-model.md | 2 +- dstack/dstack-util/src/system_setup.rs | 85 +++++++++++--------------- 2 files changed, 38 insertions(+), 49 deletions(-) diff --git a/docs/security/security-model.md b/docs/security/security-model.md index c289ac37f..04ee9165c 100644 --- a/docs/security/security-model.md +++ b/docs/security/security-model.md @@ -62,7 +62,7 @@ An application may additionally set `requirements.gpu_policy` to an object with - `rego` (optional string): a Rego v0 script evaluated with the nvattest output's `claims` array as `input`. It must define the boolean entrypoint `data.policy.nv_match` in `package policy`. - `allow_devtools` (boolean, default `false`): permit NVIDIA DevTools mode. Production applications should leave this disabled because DevTools removes the expected GPU memory-confidentiality guarantee. -Immediately after measuring `compose-hash`, dstack applies field defaults, JCS-canonicalizes the complete `gpu_policy` structure, and measures its SHA-256 digest in a `gpu-policy` event. It then applies the basic settings and optional Rego policy. A false, undefined, malformed, or non-boolean Rego result stops boot before key provisioning. If no GPU is configured, Rego receives an empty array. `gpu_policy` cannot be combined with `attest_gpu: false`. +After measuring `compose-hash`, dstack enters the GPU setup gate. If GPUs are configured, it attests them, applies field defaults, JCS-canonicalizes the complete `gpu_policy` structure, and measures its SHA-256 digest in a `gpu-policy` event. It then applies the basic settings and optional Rego policy before setting the GPU ready state. A false, undefined, malformed, or non-boolean Rego result stops boot before key provisioning. GPU-less launches return from the gate without measuring or evaluating a GPU policy. `gpu_policy` cannot be combined with `attest_gpu: false`. Separate fail-closed `nvidia-smi` queries always require the current CC feature to be ON and require DevTools mode to be OFF unless the measured policy explicitly permits it. Only after the default appraisal, optional application policy, and these checks succeed does dstack set the GPU ready state. The dstack CPU/guest boot chain is verified independently through measured boot; a GPU claim named `secboot` refers to the GPU appraisal, not UEFI Secure Boot in the CVM. diff --git a/dstack/dstack-util/src/system_setup.rs b/dstack/dstack-util/src/system_setup.rs index 2c38b01e6..4e87fdaa1 100644 --- a/dstack/dstack-util/src/system_setup.rs +++ b/dstack/dstack-util/src/system_setup.rs @@ -1069,11 +1069,7 @@ async fn do_sys_setup(stage0: Stage0<'_>) -> Result<()> { } else { info!("System time will be synchronized by chronyd in background"); } - let gpu_attestation = stage0 - .setup_gpu() - .await - .context("Failed to verify GPU TEE attestation")?; - let stage1 = stage0.setup_fs(gpu_attestation).await?; + let stage1 = stage0.setup_fs().await?; stage1.setup().await } @@ -1581,28 +1577,53 @@ impl Stage0<'_> { /// Enforce `requirements.attest_gpu` (default true): attest an attached /// NVIDIA GPU before continuing to key provisioning, or — when explicitly /// disabled — set the GPU ready state without verification. - async fn setup_gpu(&self) -> Result> { + async fn setup_gpu(&self) -> Result<()> { let inventory = gpu::gpu_inventory()?; if !self.shared.app_compose.attest_gpu() { if inventory.nvidia == 0 { - return Ok(None); + return Ok(()); } warn!("requirements.attest_gpu is false; setting GPU ready state without attestation"); // Best-effort: a GPU with CC mode off has no ready state to set. if let Err(err) = gpu::set_gpu_ready_state().await { warn!("failed to set GPU ready state: {err:?}"); } - return Ok(None); + return Ok(()); } let expected_devices = gpu::expected_gpu_count(&self.shared.sys_config.vm_config, inventory)?; if expected_devices == 0 { - return Ok(None); + return Ok(()); } self.vmm.notify_q("boot.progress", "attesting GPU").await; info!("verifying GPU TEE attestation"); let attestation = gpu::attest_gpu(expected_devices).await?; - Ok(Some(attestation)) + + let gpu_policy = self + .shared + .app_compose + .requirements + .as_ref() + .and_then(|requirements| requirements.gpu_policy.as_ref()); + if let Some(policy) = gpu_policy { + emit_runtime_event("gpu-policy", &gpu::policy_digest(policy)?)?; + if let Some(rego) = policy.rego.as_deref() { + gpu::evaluate_policy(rego, attestation.claims()) + .context("failed to apply GPU Rego policy")?; + } + } + + let allow_devtools = gpu_policy.is_some_and(|policy| policy.allow_devtools); + let devtools = gpu::verify_cc_mode(allow_devtools).await?; + gpu::set_gpu_ready_state().await?; + let event = attestation.event(devtools)?; + emit_runtime_event("gpu-attestation", &event) + .context("failed to emit GPU attestation event")?; + if gpu_policy.is_some() { + info!("application GPU policy accepted the attestation claims and state"); + } + info!("GPU TEE attestation succeeded"); + Ok(()) } } @@ -2214,10 +2235,7 @@ impl<'a> Stage0<'a> { Ok(()) } - async fn measure_app_info( - &self, - gpu_attestation: Option<&gpu::GpuAttestationResult>, - ) -> Result { + async fn measure_app_info(&self) -> Result { let compose_hash = sha256_file(self.shared.dir.app_compose_file())?; let truncated_compose_hash = truncate(&compose_hash, 20); let key_provider = self.shared.app_compose.key_provider(); @@ -2269,35 +2287,9 @@ impl<'a> Stage0<'a> { emit_runtime_event("system-preparing", &[])?; emit_runtime_event("app-id", &instance_info.app_id)?; emit_runtime_event("compose-hash", &compose_hash)?; - - let gpu_policy = self - .shared - .app_compose - .requirements - .as_ref() - .and_then(|requirements| requirements.gpu_policy.as_ref()); - if let Some(policy) = gpu_policy { - emit_runtime_event("gpu-policy", &gpu::policy_digest(policy)?)?; - let claims = gpu_attestation - .map(gpu::GpuAttestationResult::claims) - .unwrap_or(&[]); - if let Some(rego) = policy.rego.as_deref() { - gpu::evaluate_policy(rego, claims).context("failed to apply GPU Rego policy")?; - } - } - - if let Some(attestation) = gpu_attestation { - let allow_devtools = gpu_policy.is_some_and(|policy| policy.allow_devtools); - let devtools = gpu::verify_cc_mode(allow_devtools).await?; - gpu::set_gpu_ready_state().await?; - let event = attestation.event(devtools)?; - emit_runtime_event("gpu-attestation", &event) - .context("failed to emit GPU attestation event")?; - info!("GPU TEE attestation succeeded"); - } - if gpu_policy.is_some() { - info!("application GPU policy accepted the attestation claims and state"); - } + self.setup_gpu() + .await + .context("failed to verify GPU TEE attestation")?; emit_runtime_event("instance-id", &instance_id)?; emit_runtime_event("boot-mr-done", &[])?; @@ -2360,12 +2352,9 @@ impl<'a> Stage0<'a> { Ok(()) } - async fn setup_fs( - self, - gpu_attestation: Option, - ) -> Result> { + async fn setup_fs(self) -> Result> { let app_info = self - .measure_app_info(gpu_attestation.as_ref()) + .measure_app_info() .await .context("Failed to measure app info")?; if self.shared.app_compose.key_provider().is_kms() { From fb47c52d2ef5fb491cbb4ed9eef833f2cc316469 Mon Sep 17 00:00:00 2001 From: Kevin Wang Date: Thu, 16 Jul 2026 06:21:06 -0700 Subject: [PATCH 06/32] test(guest): add real H100 attestation fixture --- dstack/dstack-util/src/system_setup.rs | 25 +++++++ .../tests/fixtures/gpu_attestation_h100.json | 67 +++++++++++++++++++ 2 files changed, 92 insertions(+) create mode 100644 dstack/dstack-util/tests/fixtures/gpu_attestation_h100.json diff --git a/dstack/dstack-util/src/system_setup.rs b/dstack/dstack-util/src/system_setup.rs index 4e87fdaa1..8e04de2ee 100644 --- a/dstack/dstack-util/src/system_setup.rs +++ b/dstack/dstack-util/src/system_setup.rs @@ -1449,6 +1449,11 @@ mod gpu { .unwrap() } + // Captured with the pinned nvattest SDK on an Ubuntu 22.04 GCP A3 TDX + // VM with an H100 and the NVIDIA 580 open kernel driver. + const H100_ATTESTATION_OUTPUT: &[u8] = + include_bytes!("../tests/fixtures/gpu_attestation_h100.json"); + #[test] fn inventory_counts_nvidia_and_non_nvidia_gpus() { let root = tempfile::tempdir().unwrap(); @@ -1519,6 +1524,26 @@ mod gpu { .unwrap(); } + #[test] + fn real_h100_attestation_fixture_validates_and_drives_rego() { + let nonce = "11".repeat(32); + let claims = validate_attestation_output(H100_ATTESTATION_OUTPUT, &nonce, 1).unwrap(); + assert_eq!(claims[0]["hwmodel"], "GH100 A01 GSP BROM"); + assert_eq!(claims[0]["x-nvidia-gpu-claims-version"], "3.0"); + + let policy = r#" + package policy + default nv_match = false + nv_match { + count(input) == 1 + input[0].secboot == true + input[0].dbgstat == "disabled" + input[0].measres == "success" + } + "#; + evaluate_policy(policy, &claims).unwrap(); + } + #[test] fn event_commits_to_complete_nvattest_output() { let nonce = "22".repeat(32); diff --git a/dstack/dstack-util/tests/fixtures/gpu_attestation_h100.json b/dstack/dstack-util/tests/fixtures/gpu_attestation_h100.json new file mode 100644 index 000000000..c8e52e018 --- /dev/null +++ b/dstack/dstack-util/tests/fixtures/gpu_attestation_h100.json @@ -0,0 +1,67 @@ +{ + "claims": [ + { + "dbgstat": "disabled", + "eat_nonce": "1111111111111111111111111111111111111111111111111111111111111111", + "hwmodel": "GH100 A01 GSP BROM", + "measres": "success", + "oemid": "5703", + "secboot": true, + "ueid": "457723480349051719100670602112458195243836813799", + "x-nvidia-device-type": "gpu", + "x-nvidia-gpu-arch-check": true, + "x-nvidia-gpu-attestation-report-cert-chain": { + "x-nvidia-cert-expiration-date": "9999-12-31T23:59:59Z", + "x-nvidia-cert-ocsp-nonce-matches": true, + "x-nvidia-cert-ocsp-response-valid": true, + "x-nvidia-cert-ocsp-status": "good", + "x-nvidia-cert-revocation-reason": null, + "x-nvidia-cert-status": "valid" + }, + "x-nvidia-gpu-attestation-report-cert-chain-fwid-match": true, + "x-nvidia-gpu-attestation-report-nonce-match": true, + "x-nvidia-gpu-attestation-report-parsed": true, + "x-nvidia-gpu-attestation-report-signature-verified": true, + "x-nvidia-gpu-claims-version": "3.0", + "x-nvidia-gpu-driver-rim-cert-chain": { + "x-nvidia-cert-expiration-date": "2028-04-23T05:31:21Z", + "x-nvidia-cert-ocsp-nonce-matches": true, + "x-nvidia-cert-ocsp-response-valid": true, + "x-nvidia-cert-ocsp-status": "good", + "x-nvidia-cert-revocation-reason": null, + "x-nvidia-cert-status": "valid" + }, + "x-nvidia-gpu-driver-rim-fetched": true, + "x-nvidia-gpu-driver-rim-measurements-available": true, + "x-nvidia-gpu-driver-rim-signature-verified": true, + "x-nvidia-gpu-driver-rim-version-match": true, + "x-nvidia-gpu-driver-version": "580.159.03", + "x-nvidia-gpu-vbios-index-no-conflict": true, + "x-nvidia-gpu-vbios-rim-cert-chain": { + "x-nvidia-cert-expiration-date": "2027-04-30T23:59:59Z", + "x-nvidia-cert-ocsp-nonce-matches": true, + "x-nvidia-cert-ocsp-response-valid": true, + "x-nvidia-cert-ocsp-status": "good", + "x-nvidia-cert-revocation-reason": null, + "x-nvidia-cert-status": "valid" + }, + "x-nvidia-gpu-vbios-rim-fetched": true, + "x-nvidia-gpu-vbios-rim-measurements-available": true, + "x-nvidia-gpu-vbios-rim-signature-verified": true, + "x-nvidia-gpu-vbios-rim-version-match": true, + "x-nvidia-gpu-vbios-version": "96.00.D9.00.01", + "x-nvidia-mismatch-measurement-records": null + } + ], + "detached_eat": [ + [ + "JWT", + "eyJhbGciOiJub25lIn0.eyJlYXRfbm9uY2UiOiIxMTExMTExMTExMTExMTExMTExMTExMTExMTExMTExMTExMTExMTExMTExMTExMTExMTExMTExMTExMTExMTExIiwiZXhwIjoxNzg0MjExNDYyLCJpYXQiOjE3ODQyMDc4NjIsImlzcyI6Ik5WQVQtTE9DQUwtVkVSSUZJRVIiLCJqdGkiOiI2MjcwZTgxMTVhNWMxOTExODNjNjg0ZWY4YzFkYWRmYzYxYzYzMjQyZGUwOGY0MjQzOGU3M2NmMjNjODBkMTg1Iiwic3ViIjoiTlZJRElBLVBMQVRGT1JNLUFUVEVTVEFUSU9OIiwic3VibW9kcyI6eyJHUFUtMCI6WyJESUdFU1QiLFsiU0hBMjU2IiwiMDNkZTczYjliMTU5OTM2OTNkZmMyMzY3Y2QyZjNmMDE3ZWVkYjQ2ZWZhNzY3Yzc5ZGJkODY3ODZlMGQ3OWZlYSJdXX0sIngtbnZpZGlhLW92ZXJhbGwtYXR0LXJlc3VsdCI6dHJ1ZSwieC1udmlkaWEtdmVyIjoiMy4wIn0." + ], + { + "GPU-0": "eyJhbGciOiJub25lIn0.eyJkYmdzdGF0IjoiZGlzYWJsZWQiLCJlYXRfbm9uY2UiOiIxMTExMTExMTExMTExMTExMTExMTExMTExMTExMTExMTExMTExMTExMTExMTExMTExMTExMTExMTExMTExMTExIiwiZXhwIjoxNzg0MjExNDYyLCJod21vZGVsIjoiR0gxMDAgQTAxIEdTUCBCUk9NIiwiaWF0IjoxNzg0MjA3ODYyLCJpc3MiOiJOVkFULUxPQ0FMLVZFUklGSUVSIiwianRpIjoiYTU5ZWQ5ODAwYThhNjFhZDc5NjVlNmI1MGRmMjEyODVlNjMxZTg5ZWNlZTM5NmMwZjc1ZDkwNjIwNWI2MDUzYiIsIm1lYXNyZXMiOiJzdWNjZXNzIiwib2VtaWQiOiI1NzAzIiwic2VjYm9vdCI6dHJ1ZSwidWVpZCI6IjQ1NzcyMzQ4MDM0OTA1MTcxOTEwMDY3MDYwMjExMjQ1ODE5NTI0MzgzNjgxMzc5OSIsIngtbnZpZGlhLWRldmljZS10eXBlIjoiZ3B1IiwieC1udmlkaWEtZ3B1LWFyY2gtY2hlY2siOnRydWUsIngtbnZpZGlhLWdwdS1hdHRlc3RhdGlvbi1yZXBvcnQtY2VydC1jaGFpbiI6eyJ4LW52aWRpYS1jZXJ0LWV4cGlyYXRpb24tZGF0ZSI6Ijk5OTktMTItMzFUMjM6NTk6NTlaIiwieC1udmlkaWEtY2VydC1vY3NwLW5vbmNlLW1hdGNoZXMiOnRydWUsIngtbnZpZGlhLWNlcnQtb2NzcC1yZXNwb25zZS12YWxpZCI6dHJ1ZSwieC1udmlkaWEtY2VydC1vY3NwLXN0YXR1cyI6Imdvb2QiLCJ4LW52aWRpYS1jZXJ0LXJldm9jYXRpb24tcmVhc29uIjpudWxsLCJ4LW52aWRpYS1jZXJ0LXN0YXR1cyI6InZhbGlkIn0sIngtbnZpZGlhLWdwdS1hdHRlc3RhdGlvbi1yZXBvcnQtY2VydC1jaGFpbi1md2lkLW1hdGNoIjp0cnVlLCJ4LW52aWRpYS1ncHUtYXR0ZXN0YXRpb24tcmVwb3J0LW5vbmNlLW1hdGNoIjp0cnVlLCJ4LW52aWRpYS1ncHUtYXR0ZXN0YXRpb24tcmVwb3J0LXBhcnNlZCI6dHJ1ZSwieC1udmlkaWEtZ3B1LWF0dGVzdGF0aW9uLXJlcG9ydC1zaWduYXR1cmUtdmVyaWZpZWQiOnRydWUsIngtbnZpZGlhLWdwdS1jbGFpbXMtdmVyc2lvbiI6IjMuMCIsIngtbnZpZGlhLWdwdS1kcml2ZXItcmltLWNlcnQtY2hhaW4iOnsieC1udmlkaWEtY2VydC1leHBpcmF0aW9uLWRhdGUiOiIyMDI4LTA0LTIzVDA1OjMxOjIxWiIsIngtbnZpZGlhLWNlcnQtb2NzcC1ub25jZS1tYXRjaGVzIjp0cnVlLCJ4LW52aWRpYS1jZXJ0LW9jc3AtcmVzcG9uc2UtdmFsaWQiOnRydWUsIngtbnZpZGlhLWNlcnQtb2NzcC1zdGF0dXMiOiJnb29kIiwieC1udmlkaWEtY2VydC1yZXZvY2F0aW9uLXJlYXNvbiI6bnVsbCwieC1udmlkaWEtY2VydC1zdGF0dXMiOiJ2YWxpZCJ9LCJ4LW52aWRpYS1ncHUtZHJpdmVyLXJpbS1mZXRjaGVkIjp0cnVlLCJ4LW52aWRpYS1ncHUtZHJpdmVyLXJpbS1tZWFzdXJlbWVudHMtYXZhaWxhYmxlIjp0cnVlLCJ4LW52aWRpYS1ncHUtZHJpdmVyLXJpbS1zaWduYXR1cmUtdmVyaWZpZWQiOnRydWUsIngtbnZpZGlhLWdwdS1kcml2ZXItcmltLXZlcnNpb24tbWF0Y2giOnRydWUsIngtbnZpZGlhLWdwdS1kcml2ZXItdmVyc2lvbiI6IjU4MC4xNTkuMDMiLCJ4LW52aWRpYS1ncHUtdmJpb3MtaW5kZXgtbm8tY29uZmxpY3QiOnRydWUsIngtbnZpZGlhLWdwdS12Ymlvcy1yaW0tY2VydC1jaGFpbiI6eyJ4LW52aWRpYS1jZXJ0LWV4cGlyYXRpb24tZGF0ZSI6IjIwMjctMDQtMzBUMjM6NTk6NTlaIiwieC1udmlkaWEtY2VydC1vY3NwLW5vbmNlLW1hdGNoZXMiOnRydWUsIngtbnZpZGlhLWNlcnQtb2NzcC1yZXNwb25zZS12YWxpZCI6dHJ1ZSwieC1udmlkaWEtY2VydC1vY3NwLXN0YXR1cyI6Imdvb2QiLCJ4LW52aWRpYS1jZXJ0LXJldm9jYXRpb24tcmVhc29uIjpudWxsLCJ4LW52aWRpYS1jZXJ0LXN0YXR1cyI6InZhbGlkIn0sIngtbnZpZGlhLWdwdS12Ymlvcy1yaW0tZmV0Y2hlZCI6dHJ1ZSwieC1udmlkaWEtZ3B1LXZiaW9zLXJpbS1tZWFzdXJlbWVudHMtYXZhaWxhYmxlIjp0cnVlLCJ4LW52aWRpYS1ncHUtdmJpb3MtcmltLXNpZ25hdHVyZS12ZXJpZmllZCI6dHJ1ZSwieC1udmlkaWEtZ3B1LXZiaW9zLXJpbS12ZXJzaW9uLW1hdGNoIjp0cnVlLCJ4LW52aWRpYS1ncHUtdmJpb3MtdmVyc2lvbiI6Ijk2LjAwLkQ5LjAwLjAxIiwieC1udmlkaWEtbWlzbWF0Y2gtbWVhc3VyZW1lbnQtcmVjb3JkcyI6bnVsbH0." + } + ], + "result_code": 0, + "result_message": "Ok" +} From f362bc683556bf80317e0635c6e9db72e4d7885f Mon Sep 17 00:00:00 2001 From: Kevin Wang Date: Thu, 16 Jul 2026 07:01:26 -0700 Subject: [PATCH 07/32] feat(guest): validate every GPU through NVML --- docs/security/security-model.md | 6 +- dstack/Cargo.lock | 89 +++++++- dstack/Cargo.toml | 1 + dstack/dstack-types/src/lib.rs | 15 +- dstack/dstack-util/Cargo.toml | 1 + dstack/dstack-util/src/system_setup.rs | 305 +++++++++++++++++-------- 6 files changed, 314 insertions(+), 103 deletions(-) diff --git a/docs/security/security-model.md b/docs/security/security-model.md index 04ee9165c..61eff6813 100644 --- a/docs/security/security-model.md +++ b/docs/security/security-model.md @@ -61,10 +61,12 @@ An application may additionally set `requirements.gpu_policy` to an object with - `rego` (optional string): a Rego v0 script evaluated with the nvattest output's `claims` array as `input`. It must define the boolean entrypoint `data.policy.nv_match` in `package policy`. - `allow_devtools` (boolean, default `false`): permit NVIDIA DevTools mode. Production applications should leave this disabled because DevTools removes the expected GPU memory-confidentiality guarantee. +- `allow_debug` (boolean, default `false`): permit an attestation claim whose `dbgstat` is `enabled`. +- `allow_insecure_boot` (boolean, default `false`): permit an attestation claim whose GPU `secboot` value is false. After measuring `compose-hash`, dstack enters the GPU setup gate. If GPUs are configured, it attests them, applies field defaults, JCS-canonicalizes the complete `gpu_policy` structure, and measures its SHA-256 digest in a `gpu-policy` event. It then applies the basic settings and optional Rego policy before setting the GPU ready state. A false, undefined, malformed, or non-boolean Rego result stops boot before key provisioning. GPU-less launches return from the gate without measuring or evaluating a GPU policy. `gpu_policy` cannot be combined with `attest_gpu: false`. -Separate fail-closed `nvidia-smi` queries always require the current CC feature to be ON and require DevTools mode to be OFF unless the measured policy explicitly permits it. Only after the default appraisal, optional application policy, and these checks succeed does dstack set the GPU ready state. The dstack CPU/guest boot chain is verified independently through measured boot; a GPU claim named `secboot` refers to the GPU appraisal, not UEFI Secure Boot in the CVM. +For every NVML-enumerated GPU, dstack calls `Device::is_cc_enabled()` and `Device::is_cc_dev_mode_enabled()` and requires the NVML device count to match the expected GPU count. CC must always be ON; DevTools must be OFF unless the measured policy explicitly permits it. The typed claim checks always require `measres == "success"`; by default they also require `dbgstat == "disabled"` and `secboot == true`, with the latter two checks controlled by their explicit opt-ins. Only after the default appraisal, typed claim checks, optional Rego policy, and per-device NVML checks succeed does dstack call `Device::set_confidential_compute_state(true)` to set the GPU ready state. The dstack CPU/guest boot chain is verified independently through measured boot; a GPU claim named `secboot` refers to the GPU appraisal, not UEFI Secure Boot in the CVM. ### Dual Attestation @@ -76,7 +78,7 @@ A verifier must replay the measured event log, require exactly one pre-`system-r The GPU gate assumes a malicious host/VMM and untrusted host-provided PCI topology, while trusting the CPU TEE, the measured dstack guest/kernel, NVIDIA hardware/firmware roots, and the cryptography used by both attestation chains. Availability is out of scope. -The events make the following **boot-time** statement: immediately before key provisioning, all attached VGA/3D PCI functions were NVIDIA devices, their count matched the quote-bound VM configuration, nvattest returned one fresh successfully appraised claim for each device, the measured application policy accepted those claims and GPU state when present, CC was ON, DevTools complied with that policy (OFF by default), and setting the GPU ready state succeeded. This closes these cases: +The events make the following **boot-time** statement: immediately before key provisioning, all attached VGA/3D PCI functions were NVIDIA devices, their count matched the quote-bound VM configuration and NVML inventory, nvattest returned one fresh successfully appraised claim for each device, the measured application policy accepted those claims and GPU state when present, every enumerated GPU passed the CC/DevTools NVML checks, and setting the GPU ready state succeeded. This closes these cases: - A GPU-less launch cannot be presented as a GPU-verified launch: it has `num_gpus == 0` and no `gpu-attestation` event. - A mixed launch cannot attest only its TEE-capable subset. Non-NVIDIA display GPUs are rejected, and the sysfs, `vm_config`, and nvattest claim counts must all agree. A non-CC NVIDIA GPU either prevents evidence collection/appraisal or causes the default appraisal, application policy, or CC-state check to fail. diff --git a/dstack/Cargo.lock b/dstack/Cargo.lock index 873323d91..552652f6a 100644 --- a/dstack/Cargo.lock +++ b/dstack/Cargo.lock @@ -572,7 +572,7 @@ version = "3.9.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "519bd3116aeeb42d5372c29d982d16d0170d3d4a5ed85fc7dd91642ffff3c67c" dependencies = [ - "darling", + "darling 0.23.0", "ident_case", "prettyplease", "proc-macro2", @@ -1252,14 +1252,38 @@ dependencies = [ "syn 2.0.117", ] +[[package]] +name = "darling" +version = "0.20.11" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "fc7f46116c46ff9ab3eb1597a45688b6715c6e628b5c133e288e709a29bcb4ee" +dependencies = [ + "darling_core 0.20.11", + "darling_macro 0.20.11", +] + [[package]] name = "darling" version = "0.23.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "25ae13da2f202d56bd7f91c25fba009e7717a1e4a1cc98a76d844b65ae912e9d" dependencies = [ - "darling_core", - "darling_macro", + "darling_core 0.23.0", + "darling_macro 0.23.0", +] + +[[package]] +name = "darling_core" +version = "0.20.11" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0d00b9596d185e565c2207a0b01f8bd1a135483d02d9b7b0a54b11da8d53412e" +dependencies = [ + "fnv", + "ident_case", + "proc-macro2", + "quote", + "strsim", + "syn 2.0.117", ] [[package]] @@ -1275,13 +1299,24 @@ dependencies = [ "syn 2.0.117", ] +[[package]] +name = "darling_macro" +version = "0.20.11" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "fc34b93ccb385b40dc71c6fceac4b2ad23662c7eeb248cf10d529b7e055b6ead" +dependencies = [ + "darling_core 0.20.11", + "quote", + "syn 2.0.117", +] + [[package]] name = "darling_macro" version = "0.23.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "ac3984ec7bd6cfa798e62b4a642426a5be0e68f9401cfc2a01e3fa9ea2fcdb8d" dependencies = [ - "darling_core", + "darling_core 0.23.0", "quote", "syn 2.0.117", ] @@ -2022,6 +2057,7 @@ dependencies = [ "k256", "libc", "luks2", + "nvml-wrapper", "parity-scale-codec", "ra-rpc", "ra-tls", @@ -3798,6 +3834,16 @@ version = "0.2.186" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "68ab91017fe16c622486840e4c83c9a37afeff978bd239b5293d61ece587de66" +[[package]] +name = "libloading" +version = "0.8.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d7c4b02199fee7c5d21a5ae7d8cfa79a6ef5bb2fc834d6e9058e89c825efdc55" +dependencies = [ + "cfg-if", + "windows-link 0.2.1", +] + [[package]] name = "libm" version = "0.2.16" @@ -4399,6 +4445,29 @@ dependencies = [ "libc", ] +[[package]] +name = "nvml-wrapper" +version = "0.12.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f049ae562349fefb8e837eb15443da1e7c6dcbd8a11f52a228f92220c2e5c85e" +dependencies = [ + "bitflags 2.11.1", + "libloading", + "nvml-wrapper-sys", + "static_assertions", + "thiserror 1.0.69", + "wrapcenum-derive", +] + +[[package]] +name = "nvml-wrapper-sys" +version = "0.9.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6b4d594420fcda43b1c2c4bd44d48974aa3c7a9ab2cbf10dc18e35265767bf0b" +dependencies = [ + "libloading", +] + [[package]] name = "objc2-core-foundation" version = "0.3.2" @@ -8566,6 +8635,18 @@ dependencies = [ "wasmparser", ] +[[package]] +name = "wrapcenum-derive" +version = "0.4.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a76ff259533532054cfbaefb115c613203c73707017459206380f03b3b3f266e" +dependencies = [ + "darling 0.20.11", + "proc-macro2", + "quote", + "syn 2.0.117", +] + [[package]] name = "writeable" version = "0.6.3" diff --git a/dstack/Cargo.toml b/dstack/Cargo.toml index c3c109e24..e5f54d384 100644 --- a/dstack/Cargo.toml +++ b/dstack/Cargo.toml @@ -127,6 +127,7 @@ libc = "0.2.171" log = "0.4.26" moka = { version = "0.12.15", default-features = false, features = ["sync"] } notify = "8.0.0" +nvml-wrapper = "0.12.1" rand = "0.8.5" regorus = { version = "0.4.0", default-features = false, features = ["full-opa"] } tracing = "0.1.40" diff --git a/dstack/dstack-types/src/lib.rs b/dstack/dstack-types/src/lib.rs index dcd728f2e..d7dafd0b4 100644 --- a/dstack/dstack-types/src/lib.rs +++ b/dstack/dstack-types/src/lib.rs @@ -132,6 +132,11 @@ pub struct GpuPolicy { /// disables the GPU memory-confidentiality guarantees expected in /// production. pub allow_devtools: bool, + /// Permit claims whose GPU attestation debug status is `enabled`. Defaults + /// to false. + pub allow_debug: bool, + /// Permit claims that do not assert GPU secure boot. Defaults to false. + pub allow_insecure_boot: bool, } #[derive(Deserialize, Serialize, Debug, Clone, Default, PartialEq, Eq)] @@ -478,7 +483,9 @@ mod app_compose_tests { "launch_token_hash": "9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08", "gpu_policy": { "rego": "package policy\n\ndefault nv_match = false\n", - "allow_devtools": true + "allow_devtools": true, + "allow_debug": true, + "allow_insecure_boot": true } } })) @@ -500,6 +507,8 @@ mod app_compose_tests { Some("package policy\n\ndefault nv_match = false\n") ); assert!(gpu_policy.allow_devtools); + assert!(gpu_policy.allow_debug); + assert!(gpu_policy.allow_insecure_boot); let err = serde_json::from_value::(serde_json::json!({ "manifest_version": "3", @@ -580,6 +589,8 @@ mod app_compose_tests { let gpu_policy = requirements.gpu_policy.as_ref().unwrap(); assert!(gpu_policy.rego.is_some()); assert!(!gpu_policy.allow_devtools); + assert!(!gpu_policy.allow_debug); + assert!(!gpu_policy.allow_insecure_boot); assert!(!requirements.is_empty()); let err = serde_json::from_value::(serde_json::json!({ @@ -589,7 +600,7 @@ mod app_compose_tests { "requirements": { "gpu_policy": { "rego": "package policy", - "allow_debug": true + "allow_debugger": true } } })) diff --git a/dstack/dstack-util/Cargo.toml b/dstack/dstack-util/Cargo.toml index 9a567f6a6..09bce8c4a 100644 --- a/dstack/dstack-util/Cargo.toml +++ b/dstack/dstack-util/Cargo.toml @@ -57,6 +57,7 @@ bollard.workspace = true sodiumbox.workspace = true libc.workspace = true luks2.workspace = true +nvml-wrapper.workspace = true scopeguard.workspace = true tempfile.workspace = true ez-hash.workspace = true diff --git a/dstack/dstack-util/src/system_setup.rs b/dstack/dstack-util/src/system_setup.rs index 8e04de2ee..fc01491ed 100644 --- a/dstack/dstack-util/src/system_setup.rs +++ b/dstack/dstack-util/src/system_setup.rs @@ -1078,17 +1078,15 @@ async fn do_sys_setup(stage0: Stage0<'_>) -> Result<()> { /// Runs before key provisioning so a CVM whose GPU cannot prove it is a /// genuine, CC-enabled NVIDIA TEE never gets its app keys. An optional /// application policy is measured and evaluated after `compose-hash`. The GPU -/// "ready" state (`nvidia-smi conf-compute -srs 1`) is only set from here — -/// nvidia-persistenced deliberately does not set it — so CUDA work cannot be -/// submitted to an unverified GPU either. +/// "ready" state is only set through NVML from here — nvidia-persistenced +/// deliberately does not set it — so CUDA work cannot be submitted to an +/// unverified GPU either. mod gpu { use super::*; const NVATTEST: &str = "/usr/bin/nvattest"; - const NVIDIA_SMI: &str = "/usr/bin/nvidia-smi"; const ATTESTATION_OUTPUT: &str = "/run/nvidia-gpu-attestation/attestation.out"; const ATTESTATION_TIMEOUT: Duration = Duration::from_secs(300); - const NVIDIA_SMI_TIMEOUT: Duration = Duration::from_secs(60); const EVENT_VERSION: u32 = 2; const POLICY_ENTRYPOINT: &str = "data.policy.nv_match"; @@ -1110,6 +1108,7 @@ mod gpu { pub(super) struct GpuAttestationResult { claims: Vec, + parsed_claims: Vec, output: Vec, devices: u32, } @@ -1122,6 +1121,10 @@ mod gpu { pub(super) fn event(&self, devtools: bool) -> Result> { attestation_event(&self.output, self.devices, devtools) } + + pub(super) fn verify_claim_policy(&self, policy: Option<&GpuPolicy>) -> Result<()> { + verify_claim_policy(&self.parsed_claims, policy) + } } #[derive(Deserialize)] @@ -1130,6 +1133,30 @@ mod gpu { claims: Vec, } + #[derive(Debug, Deserialize)] + struct NvidiaGpuClaim { + #[serde(rename = "x-nvidia-device-type")] + device_type: String, + eat_nonce: String, + #[serde(rename = "x-nvidia-gpu-attestation-report-nonce-match")] + nonce_match: bool, + measres: String, + secboot: bool, + dbgstat: NvidiaGpuDebugStatus, + } + + #[derive(Debug, Deserialize, PartialEq, Eq)] + #[serde(rename_all = "lowercase")] + enum NvidiaGpuDebugStatus { + Disabled, + Enabled, + } + + struct ValidatedClaims { + raw: Vec, + parsed: Vec, + } + /// Count passed-through display-class GPUs through sysfs so devices which /// the NVIDIA driver did not bind cannot be hidden from the gate. Reading /// the inventory is fail-closed: a mixed NVIDIA/non-NVIDIA set must not be @@ -1203,36 +1230,6 @@ mod gpu { .with_context(|| format!("failed to run {program}")) } - fn parse_nvidia_smi_state(stdout: &[u8], name: &str) -> Result { - let output = std::str::from_utf8(stdout) - .with_context(|| format!("nvidia-smi returned non-UTF-8 {name}"))?; - let states = output - .split(|ch: char| !ch.is_ascii_alphanumeric()) - .filter_map(|word| match word.to_ascii_lowercase().as_str() { - "on" | "enabled" => Some(true), - "off" | "disabled" => Some(false), - _ => None, - }) - .collect::>(); - if states.len() != 1 { - bail!("unable to parse nvidia-smi {name}"); - } - states.first().copied().context("missing nvidia-smi state") - } - - async fn query_nvidia_smi_state(argument: &str, name: &str) -> Result { - let output = - run_command(NVIDIA_SMI, &["conf-compute", argument], NVIDIA_SMI_TIMEOUT).await?; - if !output.status.success() { - bail!( - "nvidia-smi conf-compute {argument} failed ({}): {}", - output.status, - truncated_lossy(&output.stderr, 512), - ); - } - parse_nvidia_smi_state(&output.stdout, name) - } - fn validate_cc_state(cc_enabled: bool, devtools: bool, allow_devtools: bool) -> Result<()> { if !cc_enabled { bail!("nvidia confidential compute mode is not enabled"); @@ -1243,43 +1240,70 @@ mod gpu { Ok(()) } - /// Require the current system-wide GPU state to use CC. NVAT's NVML - /// collector accepts protected-PCIe as well as CC, so a successful - /// appraisal alone is not a sufficient confidentiality check. DevTools is - /// rejected by default but can be explicitly allowed by the measured app - /// policy. - pub(super) async fn verify_cc_mode(allow_devtools: bool) -> Result { - let cc_enabled = query_nvidia_smi_state("-f", "CC feature state").await?; - let devtools = query_nvidia_smi_state("-d", "DevTools mode").await?; - validate_cc_state(cc_enabled, devtools, allow_devtools)?; - Ok(devtools) + fn init_nvml(expected_devices: u32) -> Result { + let nvml = nvml_wrapper::Nvml::init().context("failed to initialize NVML")?; + let devices = nvml + .device_count() + .context("failed to get NVML GPU count")?; + if devices != expected_devices { + bail!("nvml GPU count mismatch: expected {expected_devices}, got {devices}"); + } + Ok(nvml) } - /// Mark the GPU as ready to accept work. Only meaningful (and only - /// succeeds) when the GPU runs in CC mode. - pub(super) async fn set_gpu_ready_state() -> Result<()> { - let output = run_command( - NVIDIA_SMI, - &["conf-compute", "-srs", "1"], - NVIDIA_SMI_TIMEOUT, - ) - .await?; - if !output.status.success() { - bail!( - "nvidia-smi conf-compute -srs 1 failed ({}): {}", - output.status, - truncated_lossy(&output.stderr, 512), - ); - } + fn set_gpu_ready_state_with_nvml(nvml: &nvml_wrapper::Nvml) -> Result<()> { + // nvml-wrapper exposes nvmlSystemSetConfComputeGpusReadyState through + // Device, but the transition applies to all CC GPUs in the system. + let first = nvml + .device_by_index(0) + .context("failed to get first NVML GPU")?; + first + .set_confidential_compute_state(true) + .context("failed to set GPU ready state")?; info!("GPU ready state set"); Ok(()) } + /// Check the CC and DevTools state through NVML for every expected GPU, + /// then set the system-wide GPU ready state only after every check passes. + /// NVML exposes these settings as system values through Device methods; + /// call them for every handle so every expected device must be enumerable. + pub(super) fn verify_gpu_state_and_set_ready( + expected_devices: u32, + allow_devtools: bool, + ) -> Result { + let nvml = init_nvml(expected_devices)?; + let mut any_devtools = false; + for index in 0..expected_devices { + let device = nvml + .device_by_index(index) + .with_context(|| format!("failed to get NVML GPU at index {index}"))?; + let cc_enabled = device + .is_cc_enabled() + .with_context(|| format!("failed to query CC mode for GPU at index {index}"))?; + let devtools = device.is_cc_dev_mode_enabled().with_context(|| { + format!("failed to query DevTools mode for GPU at index {index}") + })?; + validate_cc_state(cc_enabled, devtools, allow_devtools) + .with_context(|| format!("gpu at index {index} violates the basic policy"))?; + any_devtools |= devtools; + } + set_gpu_ready_state_with_nvml(&nvml)?; + Ok(any_devtools) + } + + /// Set the system-wide GPU ready state without appraisal for the explicit + /// `attest_gpu: false` compatibility path. + pub(super) fn set_gpu_ready_state(expected_devices: u32) -> Result<()> { + let nvml = init_nvml(expected_devices)?; + set_gpu_ready_state_with_nvml(&nvml) + } + fn validate_attestation_output( stdout: &[u8], nonce: &str, expected_devices: u32, - ) -> Result> { + ) -> Result { let output: NvattestOutput = serde_json::from_slice(stdout).context("failed to parse nvattest JSON output")?; if output.result_code != 0 { @@ -1291,24 +1315,39 @@ mod gpu { output.claims.len() ); } + let mut parsed_claims = Vec::with_capacity(output.claims.len()); for (index, claim) in output.claims.iter().enumerate() { - let claim = claim - .as_object() + let claim: NvidiaGpuClaim = serde_json::from_value(claim.clone()) .with_context(|| format!("invalid GPU claim at index {index}"))?; - let string_claim = |name: &str| claim.get(name).and_then(Value::as_str); - if string_claim("x-nvidia-device-type") != Some("gpu") { + if claim.device_type != "gpu" { bail!("gpu claim at index {index} has an invalid device type"); } - if string_claim("eat_nonce") != Some(nonce) - || claim - .get("x-nvidia-gpu-attestation-report-nonce-match") - .and_then(Value::as_bool) - != Some(true) - { + if claim.eat_nonce != nonce || !claim.nonce_match { bail!("gpu claim at index {index} has an invalid nonce"); } + parsed_claims.push(claim); + } + Ok(ValidatedClaims { + raw: output.claims, + parsed: parsed_claims, + }) + } + + fn verify_claim_policy(claims: &[NvidiaGpuClaim], policy: Option<&GpuPolicy>) -> Result<()> { + let allow_debug = policy.is_some_and(|policy| policy.allow_debug); + let allow_insecure_boot = policy.is_some_and(|policy| policy.allow_insecure_boot); + for (index, claim) in claims.iter().enumerate() { + if claim.measres != "success" { + bail!("gpu claim at index {index} has unsuccessful measurements"); + } + if !allow_insecure_boot && !claim.secboot { + bail!("gpu claim at index {index} does not assert secure boot"); + } + if !allow_debug && claim.dbgstat != NvidiaGpuDebugStatus::Disabled { + bail!("gpu claim at index {index} does not disable debug mode"); + } } - Ok(output.claims) + Ok(()) } fn attestation_event(stdout: &[u8], devices: u32, devtools: bool) -> Result> { @@ -1366,7 +1405,8 @@ mod gpu { } let claims = validate_attestation_output(&output.stdout, &nonce, expected_devices)?; Ok(GpuAttestationResult { - claims, + claims: claims.raw, + parsed_claims: claims.parsed, output: output.stdout, devices: expected_devices, }) @@ -1437,7 +1477,10 @@ mod gpu { serde_json::json!({ "x-nvidia-device-type": "gpu", "eat_nonce": nonce, - "x-nvidia-gpu-attestation-report-nonce-match": true + "x-nvidia-gpu-attestation-report-nonce-match": true, + "measres": "success", + "secboot": true, + "dbgstat": "disabled" }) }) .collect::>(); @@ -1486,14 +1529,6 @@ mod gpu { assert_eq!(expected_gpu_count(r#"{"num_gpus":2}"#, nvidia).unwrap(), 2); } - #[test] - fn nvidia_smi_state_parser_is_fail_closed() { - assert!(parse_nvidia_smi_state(b"CC status: ON\n", "CC").unwrap()); - assert!(!parse_nvidia_smi_state(b"DevTools Mode: OFF\n", "DevTools").unwrap()); - assert!(parse_nvidia_smi_state(b"unknown\n", "CC").is_err()); - assert!(parse_nvidia_smi_state(b"current: ON, pending: OFF\n", "CC").is_err()); - } - #[test] fn basic_policy_requires_cc_and_rejects_devtools_by_default() { validate_cc_state(true, false, false).unwrap(); @@ -1516,20 +1551,97 @@ mod gpu { .is_err() ); - // Claim appraisal belongs to nvattest's default policy and the - // optional app policy, not a second policy hard-coded here. + // Basic claim settings are enforced after structural validation. let mut extra_claims: Value = serde_json::from_slice(&valid).unwrap(); extra_claims["claims"][0]["dbgstat"] = Value::String("enabled".to_string()); validate_attestation_output(&serde_json::to_vec(&extra_claims).unwrap(), &nonce, 2) .unwrap(); } + #[test] + fn basic_claim_policy_is_fail_closed_and_honors_opt_ins() { + let nonce = "33".repeat(32); + let output = nvattest_output(&nonce, 1); + let claims = validate_attestation_output(&output, &nonce, 1).unwrap(); + verify_claim_policy(&claims.parsed, None).unwrap(); + + let with_policy = |name: &str, value: Value, policy: &GpuPolicy| { + let mut output: Value = serde_json::from_slice(&output).unwrap(); + output["claims"][0][name] = value; + let output = serde_json::to_vec(&output).unwrap(); + let claims = validate_attestation_output(&output, &nonce, 1).unwrap(); + verify_claim_policy(&claims.parsed, Some(policy)) + }; + + assert!(with_policy("secboot", Value::Bool(false), &GpuPolicy::default()).is_err()); + with_policy( + "secboot", + Value::Bool(false), + &GpuPolicy { + allow_insecure_boot: true, + ..Default::default() + }, + ) + .unwrap(); + + assert!(with_policy( + "dbgstat", + Value::String("enabled".to_string()), + &GpuPolicy::default(), + ) + .is_err()); + with_policy( + "dbgstat", + Value::String("enabled".to_string()), + &GpuPolicy { + allow_debug: true, + ..Default::default() + }, + ) + .unwrap(); + + let mut unknown_debug: Value = serde_json::from_slice(&output).unwrap(); + unknown_debug["claims"][0]["dbgstat"] = Value::String("unknown".to_string()); + assert!(validate_attestation_output( + &serde_json::to_vec(&unknown_debug).unwrap(), + &nonce, + 1, + ) + .is_err()); + + assert!(with_policy( + "measres", + Value::String("failure".to_string()), + &GpuPolicy { + allow_debug: true, + allow_insecure_boot: true, + ..Default::default() + }, + ) + .is_err()); + + for required in ["measres", "secboot", "dbgstat"] { + let mut missing: Value = serde_json::from_slice(&output).unwrap(); + missing["claims"][0] + .as_object_mut() + .unwrap() + .remove(required); + assert!(validate_attestation_output( + &serde_json::to_vec(&missing).unwrap(), + &nonce, + 1, + ) + .is_err()); + } + } + #[test] fn real_h100_attestation_fixture_validates_and_drives_rego() { let nonce = "11".repeat(32); let claims = validate_attestation_output(H100_ATTESTATION_OUTPUT, &nonce, 1).unwrap(); - assert_eq!(claims[0]["hwmodel"], "GH100 A01 GSP BROM"); - assert_eq!(claims[0]["x-nvidia-gpu-claims-version"], "3.0"); + assert_eq!(claims.raw[0]["hwmodel"], "GH100 A01 GSP BROM"); + assert_eq!(claims.raw[0]["x-nvidia-gpu-claims-version"], "3.0"); + verify_claim_policy(&claims.parsed, None).unwrap(); let policy = r#" package policy @@ -1541,7 +1653,7 @@ mod gpu { input[0].measres == "success" } "#; - evaluate_policy(policy, &claims).unwrap(); + evaluate_policy(policy, &claims.raw).unwrap(); } #[test] @@ -1582,6 +1694,7 @@ mod gpu { let policy = GpuPolicy { rego: Some("package policy\n\ndefault nv_match = true\n".to_string()), allow_devtools: true, + ..Default::default() }; let canonical = serde_jcs::to_vec(&policy).unwrap(); assert_eq!(policy_digest(&policy).unwrap(), sha256(&canonical)); @@ -1610,7 +1723,7 @@ impl Stage0<'_> { } warn!("requirements.attest_gpu is false; setting GPU ready state without attestation"); // Best-effort: a GPU with CC mode off has no ready state to set. - if let Err(err) = gpu::set_gpu_ready_state().await { + if let Err(err) = gpu::set_gpu_ready_state(inventory.nvidia) { warn!("failed to set GPU ready state: {err:?}"); } return Ok(()); @@ -1632,15 +1745,17 @@ impl Stage0<'_> { .and_then(|requirements| requirements.gpu_policy.as_ref()); if let Some(policy) = gpu_policy { emit_runtime_event("gpu-policy", &gpu::policy_digest(policy)?)?; - if let Some(rego) = policy.rego.as_deref() { - gpu::evaluate_policy(rego, attestation.claims()) - .context("failed to apply GPU Rego policy")?; - } + } + attestation + .verify_claim_policy(gpu_policy) + .context("failed to apply basic GPU claim policy")?; + if let Some(rego) = gpu_policy.and_then(|policy| policy.rego.as_deref()) { + gpu::evaluate_policy(rego, attestation.claims()) + .context("failed to apply GPU Rego policy")?; } let allow_devtools = gpu_policy.is_some_and(|policy| policy.allow_devtools); - let devtools = gpu::verify_cc_mode(allow_devtools).await?; - gpu::set_gpu_ready_state().await?; + let devtools = gpu::verify_gpu_state_and_set_ready(expected_devices, allow_devtools)?; let event = attestation.event(devtools)?; emit_runtime_event("gpu-attestation", &event) .context("failed to emit GPU attestation event")?; From c89732c1d9e6aa8fdf373fa98f1861102629fcaa Mon Sep 17 00:00:00 2001 From: Kevin Wang Date: Thu, 16 Jul 2026 07:17:31 -0700 Subject: [PATCH 08/32] fix(guest): align GPU hash event naming --- docs/security/security-model.md | 8 ++++---- dstack/dstack-types/src/lib.rs | 2 +- dstack/dstack-util/src/system_setup.rs | 2 +- 3 files changed, 6 insertions(+), 6 deletions(-) diff --git a/docs/security/security-model.md b/docs/security/security-model.md index 61eff6813..2371e5ec8 100644 --- a/docs/security/security-model.md +++ b/docs/security/security-model.md @@ -64,15 +64,15 @@ An application may additionally set `requirements.gpu_policy` to an object with - `allow_debug` (boolean, default `false`): permit an attestation claim whose `dbgstat` is `enabled`. - `allow_insecure_boot` (boolean, default `false`): permit an attestation claim whose GPU `secboot` value is false. -After measuring `compose-hash`, dstack enters the GPU setup gate. If GPUs are configured, it attests them, applies field defaults, JCS-canonicalizes the complete `gpu_policy` structure, and measures its SHA-256 digest in a `gpu-policy` event. It then applies the basic settings and optional Rego policy before setting the GPU ready state. A false, undefined, malformed, or non-boolean Rego result stops boot before key provisioning. GPU-less launches return from the gate without measuring or evaluating a GPU policy. `gpu_policy` cannot be combined with `attest_gpu: false`. +After measuring `compose-hash`, dstack enters the GPU setup gate. If GPUs are configured, it attests them, applies field defaults, JCS-canonicalizes the complete `gpu_policy` structure, and measures its SHA-256 digest in a `gpu-policy-hash` event. It then applies the basic settings and optional Rego policy before setting the GPU ready state. A false, undefined, malformed, or non-boolean Rego result stops boot before key provisioning. GPU-less launches return from the gate without measuring or evaluating a GPU policy. `gpu_policy` cannot be combined with `attest_gpu: false`. For every NVML-enumerated GPU, dstack calls `Device::is_cc_enabled()` and `Device::is_cc_dev_mode_enabled()` and requires the NVML device count to match the expected GPU count. CC must always be ON; DevTools must be OFF unless the measured policy explicitly permits it. The typed claim checks always require `measres == "success"`; by default they also require `dbgstat == "disabled"` and `secboot == true`, with the latter two checks controlled by their explicit opt-ins. Only after the default appraisal, typed claim checks, optional Rego policy, and per-device NVML checks succeed does dstack call `Device::set_confidential_compute_state(true)` to set the GPU ready state. The dstack CPU/guest boot chain is verified independently through measured boot; a GPU claim named `secboot` refers to the GPU appraisal, not UEFI Secure Boot in the CVM. ### Dual Attestation -GPU workloads require verification of both hardware components. The CPU TEE quote verifies the CVM and its measured guest code. NVIDIA-signed evidence, checked against NVIDIA RIMs and certificate status by `nvattest`, verifies the GPU appraisal. After the optional policy and ready-state operations succeed, dstack emits a `gpu-attestation` launch event before `system-ready`. Its versioned payload records the number of appraised devices, asserted CC/DevTools state, and SHA-256 of the complete nvattest JSON output (claims and detached EAT). On TDX, both `gpu-policy` and `gpu-attestation` are append-only RTMR3 events; they are never derived from application-controlled `report_data`. +GPU workloads require verification of both hardware components. The CPU TEE quote verifies the CVM and its measured guest code. NVIDIA-signed evidence, checked against NVIDIA RIMs and certificate status by `nvattest`, verifies the GPU appraisal. After the optional policy and ready-state operations succeed, dstack emits a `gpu-attestation` launch event before `system-ready`. Its versioned payload records the number of appraised devices, asserted CC/DevTools state, and SHA-256 of the complete nvattest JSON output (claims and detached EAT). On TDX, both `gpu-policy-hash` and `gpu-attestation` are append-only RTMR3 events; they are never derived from application-controlled `report_data`. -A verifier must replay the measured event log, require exactly one pre-`system-ready` `gpu-attestation` event when GPU protection is required, and require `devices == vm_config.num_gpus > 0`. If an application policy is required, the verifier must also require exactly one `gpu-policy` event immediately after `compose-hash` and compare its 32-byte payload with the expected policy digest. The raw `attestation.out` file is not trusted by itself; if it is supplied for inspection, its digest must match the `gpu-attestation` event. +A verifier must replay the measured event log, require exactly one pre-`system-ready` `gpu-attestation` event when GPU protection is required, and require `devices == vm_config.num_gpus > 0`. If an application policy is required, the verifier must also require exactly one `gpu-policy-hash` event immediately after `compose-hash` and compare its 32-byte payload with the expected policy digest. The raw `attestation.out` file is not trusted by itself; if it is supplied for inspection, its digest must match the `gpu-attestation` event. ### GPU Threat Model and Lifetime @@ -185,7 +185,7 @@ Use this checklist to verify a workload running in a dstack CVM. **GPU verification (when required):** - [ ] `vm_config.num_gpus` is greater than zero and matches the expected deployment -- [ ] If `requirements.gpu_policy` is required, exactly one `gpu-policy` event follows `compose-hash`, and its payload matches the expected SHA-256 digest +- [ ] If `requirements.gpu_policy` is required, exactly one `gpu-policy-hash` event follows `compose-hash`, and its payload matches the expected SHA-256 digest - [ ] Exactly one `gpu-attestation` event appears before `system-ready` - [ ] The event device count matches `vm_config.num_gpus`, CC is ON, and its DevTools field complies with the measured policy - [ ] The platform binds the event log to a quoted RTMR/PCR (do not accept it from current SEV-SNP evidence) diff --git a/dstack/dstack-types/src/lib.rs b/dstack/dstack-types/src/lib.rs index d7dafd0b4..e0b64a649 100644 --- a/dstack/dstack-types/src/lib.rs +++ b/dstack/dstack-types/src/lib.rs @@ -180,7 +180,7 @@ pub struct Requirements { /// Optional application GPU policy applied before key provisioning. /// /// Its default-expanded, JCS-canonicalized JSON SHA-256 digest is emitted - /// as the `gpu-policy` launch event immediately after `compose-hash`. + /// as the `gpu-policy-hash` launch event immediately after `compose-hash`. #[serde(skip_serializing_if = "Option::is_none")] pub gpu_policy: Option, } diff --git a/dstack/dstack-util/src/system_setup.rs b/dstack/dstack-util/src/system_setup.rs index fc01491ed..a4bce1963 100644 --- a/dstack/dstack-util/src/system_setup.rs +++ b/dstack/dstack-util/src/system_setup.rs @@ -1744,7 +1744,7 @@ impl Stage0<'_> { .as_ref() .and_then(|requirements| requirements.gpu_policy.as_ref()); if let Some(policy) = gpu_policy { - emit_runtime_event("gpu-policy", &gpu::policy_digest(policy)?)?; + emit_runtime_event("gpu-policy-hash", &gpu::policy_digest(policy)?)?; } attestation .verify_claim_policy(gpu_policy) From 1681965c17992a45a5728c71f834a428bb71adfb Mon Sep 17 00:00:00 2001 From: Kevin Wang Date: Thu, 16 Jul 2026 22:20:02 +0800 Subject: [PATCH 09/32] Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> --- dstack/dstack-util/src/system_setup.rs | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/dstack/dstack-util/src/system_setup.rs b/dstack/dstack-util/src/system_setup.rs index a4bce1963..8ebbd96a1 100644 --- a/dstack/dstack-util/src/system_setup.rs +++ b/dstack/dstack-util/src/system_setup.rs @@ -1307,7 +1307,10 @@ mod gpu { let output: NvattestOutput = serde_json::from_slice(stdout).context("failed to parse nvattest JSON output")?; if output.result_code != 0 { - bail!("nvattest JSON result is not successful"); + bail!( + "nvattest JSON result is not successful (result_code={})", + output.result_code + ); } if output.claims.len() != expected_devices as usize { bail!( From fda418673f05f6ffa486ba31c7ce0f4671b692ca Mon Sep 17 00:00:00 2001 From: Kevin Wang Date: Thu, 16 Jul 2026 08:46:26 -0700 Subject: [PATCH 10/32] refactor(guest): unify basic GPU policy checks --- dstack/dstack-types/src/lib.rs | 20 +++- dstack/dstack-util/src/system_setup.rs | 156 +++++++++++++++++-------- 2 files changed, 125 insertions(+), 51 deletions(-) diff --git a/dstack/dstack-types/src/lib.rs b/dstack/dstack-types/src/lib.rs index e0b64a649..3395cd92f 100644 --- a/dstack/dstack-types/src/lib.rs +++ b/dstack/dstack-types/src/lib.rs @@ -139,6 +139,13 @@ pub struct GpuPolicy { pub allow_insecure_boot: bool, } +impl GpuPolicy { + /// Returns true when no application-specific GPU policy setting is set. + pub fn is_default(&self) -> bool { + self == &Self::default() + } +} + #[derive(Deserialize, Serialize, Debug, Clone, Default, PartialEq, Eq)] #[serde(default, deny_unknown_fields)] pub struct Requirements { @@ -181,8 +188,8 @@ pub struct Requirements { /// /// Its default-expanded, JCS-canonicalized JSON SHA-256 digest is emitted /// as the `gpu-policy-hash` launch event immediately after `compose-hash`. - #[serde(skip_serializing_if = "Option::is_none")] - pub gpu_policy: Option, + #[serde(default, skip_serializing_if = "GpuPolicy::is_default")] + pub gpu_policy: GpuPolicy, } impl Requirements { @@ -192,7 +199,7 @@ impl Requirements { && self.tdx_measure_acpi_tables.is_none() && self.launch_token_hash.is_none() && self.attest_gpu.is_none() - && self.gpu_policy.is_none() + && self.gpu_policy.is_default() } } @@ -501,7 +508,7 @@ mod app_compose_tests { requirements.launch_token_hash.as_deref(), Some("9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08") ); - let gpu_policy = requirements.gpu_policy.as_ref().unwrap(); + let gpu_policy = &requirements.gpu_policy; assert_eq!( gpu_policy.rego.as_deref(), Some("package policy\n\ndefault nv_match = false\n") @@ -533,7 +540,10 @@ mod app_compose_tests { .unwrap(); let requirements = omitted.requirements.as_ref().unwrap(); assert_eq!(requirements.platforms, None); + assert!(requirements.gpu_policy.is_default()); assert!(requirements.is_empty()); + let serialized = serde_json::to_value(requirements).unwrap(); + assert!(serialized.get("gpu_policy").is_none()); let explicit_empty: AppCompose = serde_json::from_value(serde_json::json!({ "manifest_version": "3", @@ -586,7 +596,7 @@ mod app_compose_tests { })) .unwrap(); let requirements = gpu_policy.requirements.as_ref().unwrap(); - let gpu_policy = requirements.gpu_policy.as_ref().unwrap(); + let gpu_policy = &requirements.gpu_policy; assert!(gpu_policy.rego.is_some()); assert!(!gpu_policy.allow_devtools); assert!(!gpu_policy.allow_debug); diff --git a/dstack/dstack-util/src/system_setup.rs b/dstack/dstack-util/src/system_setup.rs index 8ebbd96a1..1fb5b38fc 100644 --- a/dstack/dstack-util/src/system_setup.rs +++ b/dstack/dstack-util/src/system_setup.rs @@ -842,7 +842,7 @@ fn verify_manifest_feature_requirements(app_compose: &AppCompose) -> Result<()> .requirements .as_ref() .is_some_and(|requirements| { - requirements.gpu_policy.is_some() && requirements.attest_gpu == Some(false) + !requirements.gpu_policy.is_default() && requirements.attest_gpu == Some(false) }) { bail!("requirements.gpu_policy requires GPU attestation"); @@ -1122,11 +1122,36 @@ mod gpu { attestation_event(&self.output, self.devices, devtools) } - pub(super) fn verify_claim_policy(&self, policy: Option<&GpuPolicy>) -> Result<()> { - verify_claim_policy(&self.parsed_claims, policy) + pub(super) fn verify_claim_policy( + &self, + state: &GpuState, + policy: &GpuPolicy, + ) -> Result<()> { + verify_claim_policy(&self.parsed_claims, &state.devices, policy) } } + pub(super) struct GpuState { + nvml: nvml_wrapper::Nvml, + devices: Vec, + } + + impl GpuState { + pub(super) fn any_devtools(&self) -> bool { + self.devices.iter().any(|device| device.devtools) + } + + pub(super) fn set_ready(&self) -> Result<()> { + set_gpu_ready_state_with_nvml(&self.nvml) + } + } + + #[derive(Debug, Clone, Copy)] + struct GpuDeviceState { + cc_enabled: bool, + devtools: bool, + } + #[derive(Deserialize)] struct NvattestOutput { result_code: i64, @@ -1230,16 +1255,6 @@ mod gpu { .with_context(|| format!("failed to run {program}")) } - fn validate_cc_state(cc_enabled: bool, devtools: bool, allow_devtools: bool) -> Result<()> { - if !cc_enabled { - bail!("nvidia confidential compute mode is not enabled"); - } - if devtools && !allow_devtools { - bail!("nvidia DevTools mode is enabled"); - } - Ok(()) - } - fn init_nvml(expected_devices: u32) -> Result { let nvml = nvml_wrapper::Nvml::init().context("failed to initialize NVML")?; let devices = nvml @@ -1264,16 +1279,12 @@ mod gpu { Ok(()) } - /// Check the CC and DevTools state through NVML for every expected GPU, - /// then set the system-wide GPU ready state only after every check passes. + /// Read the CC and DevTools state through NVML for every expected GPU. /// NVML exposes these settings as system values through Device methods; /// call them for every handle so every expected device must be enumerable. - pub(super) fn verify_gpu_state_and_set_ready( - expected_devices: u32, - allow_devtools: bool, - ) -> Result { + pub(super) fn query_gpu_state(expected_devices: u32) -> Result { let nvml = init_nvml(expected_devices)?; - let mut any_devtools = false; + let mut devices = Vec::with_capacity(expected_devices as usize); for index in 0..expected_devices { let device = nvml .device_by_index(index) @@ -1284,12 +1295,12 @@ mod gpu { let devtools = device.is_cc_dev_mode_enabled().with_context(|| { format!("failed to query DevTools mode for GPU at index {index}") })?; - validate_cc_state(cc_enabled, devtools, allow_devtools) - .with_context(|| format!("gpu at index {index} violates the basic policy"))?; - any_devtools |= devtools; + devices.push(GpuDeviceState { + cc_enabled, + devtools, + }); } - set_gpu_ready_state_with_nvml(&nvml)?; - Ok(any_devtools) + Ok(GpuState { nvml, devices }) } /// Set the system-wide GPU ready state without appraisal for the explicit @@ -1336,17 +1347,29 @@ mod gpu { }) } - fn verify_claim_policy(claims: &[NvidiaGpuClaim], policy: Option<&GpuPolicy>) -> Result<()> { - let allow_debug = policy.is_some_and(|policy| policy.allow_debug); - let allow_insecure_boot = policy.is_some_and(|policy| policy.allow_insecure_boot); + fn verify_claim_policy( + claims: &[NvidiaGpuClaim], + devices: &[GpuDeviceState], + policy: &GpuPolicy, + ) -> Result<()> { + for (index, device) in devices.iter().enumerate() { + if !device.cc_enabled { + bail!("gpu at index {index} does not enable confidential compute mode"); + } + } + for (index, device) in devices.iter().enumerate() { + if device.devtools && !policy.allow_devtools { + bail!("gpu at index {index} enables NVIDIA DevTools mode"); + } + } for (index, claim) in claims.iter().enumerate() { if claim.measres != "success" { bail!("gpu claim at index {index} has unsuccessful measurements"); } - if !allow_insecure_boot && !claim.secboot { + if !policy.allow_insecure_boot && !claim.secboot { bail!("gpu claim at index {index} does not assert secure boot"); } - if !allow_debug && claim.dbgstat != NvidiaGpuDebugStatus::Disabled { + if !policy.allow_debug && claim.dbgstat != NvidiaGpuDebugStatus::Disabled { bail!("gpu claim at index {index} does not disable debug mode"); } } @@ -1534,10 +1557,37 @@ mod gpu { #[test] fn basic_policy_requires_cc_and_rejects_devtools_by_default() { - validate_cc_state(true, false, false).unwrap(); - assert!(validate_cc_state(false, false, false).is_err()); - assert!(validate_cc_state(true, true, false).is_err()); - validate_cc_state(true, true, true).unwrap(); + let nonce = "44".repeat(32); + let output = nvattest_output(&nonce, 1); + let claims = validate_attestation_output(&output, &nonce, 1).unwrap(); + let production = [GpuDeviceState { + cc_enabled: true, + devtools: false, + }]; + verify_claim_policy(&claims.parsed, &production, &GpuPolicy::default()).unwrap(); + + let non_cc = [GpuDeviceState { + cc_enabled: false, + devtools: false, + }]; + let err = + verify_claim_policy(&claims.parsed, &non_cc, &GpuPolicy::default()).unwrap_err(); + assert!(err.to_string().contains("confidential compute mode")); + + let devtools = [GpuDeviceState { + cc_enabled: true, + devtools: true, + }]; + assert!(verify_claim_policy(&claims.parsed, &devtools, &GpuPolicy::default()).is_err()); + verify_claim_policy( + &claims.parsed, + &devtools, + &GpuPolicy { + allow_devtools: true, + ..Default::default() + }, + ) + .unwrap(); } #[test] @@ -1566,14 +1616,18 @@ mod gpu { let nonce = "33".repeat(32); let output = nvattest_output(&nonce, 1); let claims = validate_attestation_output(&output, &nonce, 1).unwrap(); - verify_claim_policy(&claims.parsed, None).unwrap(); + let devices = [GpuDeviceState { + cc_enabled: true, + devtools: false, + }]; + verify_claim_policy(&claims.parsed, &devices, &GpuPolicy::default()).unwrap(); let with_policy = |name: &str, value: Value, policy: &GpuPolicy| { let mut output: Value = serde_json::from_slice(&output).unwrap(); output["claims"][0][name] = value; let output = serde_json::to_vec(&output).unwrap(); let claims = validate_attestation_output(&output, &nonce, 1).unwrap(); - verify_claim_policy(&claims.parsed, Some(policy)) + verify_claim_policy(&claims.parsed, &devices, policy) }; assert!(with_policy("secboot", Value::Bool(false), &GpuPolicy::default()).is_err()); @@ -1644,7 +1698,15 @@ mod gpu { let claims = validate_attestation_output(H100_ATTESTATION_OUTPUT, &nonce, 1).unwrap(); assert_eq!(claims.raw[0]["hwmodel"], "GH100 A01 GSP BROM"); assert_eq!(claims.raw[0]["x-nvidia-gpu-claims-version"], "3.0"); - verify_claim_policy(&claims.parsed, None).unwrap(); + verify_claim_policy( + &claims.parsed, + &[GpuDeviceState { + cc_enabled: true, + devtools: false, + }], + &GpuPolicy::default(), + ) + .unwrap(); let policy = r#" package policy @@ -1745,24 +1807,26 @@ impl Stage0<'_> { .app_compose .requirements .as_ref() - .and_then(|requirements| requirements.gpu_policy.as_ref()); - if let Some(policy) = gpu_policy { - emit_runtime_event("gpu-policy-hash", &gpu::policy_digest(policy)?)?; + .map(|requirements| requirements.gpu_policy.clone()) + .unwrap_or_default(); + if !gpu_policy.is_default() { + emit_runtime_event("gpu-policy-hash", &gpu::policy_digest(&gpu_policy)?)?; } + let gpu_state = gpu::query_gpu_state(expected_devices)?; attestation - .verify_claim_policy(gpu_policy) - .context("failed to apply basic GPU claim policy")?; - if let Some(rego) = gpu_policy.and_then(|policy| policy.rego.as_deref()) { + .verify_claim_policy(&gpu_state, &gpu_policy) + .context("failed to apply basic GPU policy")?; + if let Some(rego) = gpu_policy.rego.as_deref() { gpu::evaluate_policy(rego, attestation.claims()) .context("failed to apply GPU Rego policy")?; } - let allow_devtools = gpu_policy.is_some_and(|policy| policy.allow_devtools); - let devtools = gpu::verify_gpu_state_and_set_ready(expected_devices, allow_devtools)?; + gpu_state.set_ready()?; + let devtools = gpu_state.any_devtools(); let event = attestation.event(devtools)?; emit_runtime_event("gpu-attestation", &event) .context("failed to emit GPU attestation event")?; - if gpu_policy.is_some() { + if !gpu_policy.is_default() { info!("application GPU policy accepted the attestation claims and state"); } info!("GPU TEE attestation succeeded"); From 6a804f0dcf11df833da22e55e2fe15c8d74851d8 Mon Sep 17 00:00:00 2001 From: Kevin Wang Date: Thu, 16 Jul 2026 09:02:34 -0700 Subject: [PATCH 11/32] refactor(guest): allow disabled GPU attestation with policy --- docs/security/security-model.md | 2 +- dstack/dstack-util/src/system_setup.rs | 29 -------------------------- 2 files changed, 1 insertion(+), 30 deletions(-) diff --git a/docs/security/security-model.md b/docs/security/security-model.md index 2371e5ec8..d2af5acfc 100644 --- a/docs/security/security-model.md +++ b/docs/security/security-model.md @@ -64,7 +64,7 @@ An application may additionally set `requirements.gpu_policy` to an object with - `allow_debug` (boolean, default `false`): permit an attestation claim whose `dbgstat` is `enabled`. - `allow_insecure_boot` (boolean, default `false`): permit an attestation claim whose GPU `secboot` value is false. -After measuring `compose-hash`, dstack enters the GPU setup gate. If GPUs are configured, it attests them, applies field defaults, JCS-canonicalizes the complete `gpu_policy` structure, and measures its SHA-256 digest in a `gpu-policy-hash` event. It then applies the basic settings and optional Rego policy before setting the GPU ready state. A false, undefined, malformed, or non-boolean Rego result stops boot before key provisioning. GPU-less launches return from the gate without measuring or evaluating a GPU policy. `gpu_policy` cannot be combined with `attest_gpu: false`. +After measuring `compose-hash`, dstack enters the GPU setup gate. If GPUs are configured, it attests them, applies field defaults, JCS-canonicalizes the complete `gpu_policy` structure, and measures its SHA-256 digest in a `gpu-policy-hash` event. It then applies the basic settings and optional Rego policy before setting the GPU ready state. A false, undefined, malformed, or non-boolean Rego result stops boot before key provisioning. GPU-less launches return from the gate without measuring or evaluating a GPU policy. When `attest_gpu` is false, `gpu_policy` is not measured or evaluated. For every NVML-enumerated GPU, dstack calls `Device::is_cc_enabled()` and `Device::is_cc_dev_mode_enabled()` and requires the NVML device count to match the expected GPU count. CC must always be ON; DevTools must be OFF unless the measured policy explicitly permits it. The typed claim checks always require `measres == "success"`; by default they also require `dbgstat == "disabled"` and `secboot == true`, with the latter two checks controlled by their explicit opt-ins. Only after the default appraisal, typed claim checks, optional Rego policy, and per-device NVML checks succeed does dstack call `Device::set_confidential_compute_state(true)` to set the GPU ready state. The dstack CPU/guest boot chain is verified independently through measured boot; a GPU claim named `secboot` refers to the GPU appraisal, not UEFI Secure Boot in the CVM. diff --git a/dstack/dstack-util/src/system_setup.rs b/dstack/dstack-util/src/system_setup.rs index 1fb5b38fc..8b32d1ba5 100644 --- a/dstack/dstack-util/src/system_setup.rs +++ b/dstack/dstack-util/src/system_setup.rs @@ -838,15 +838,6 @@ fn verify_manifest_feature_requirements(app_compose: &AppCompose) -> Result<()> "requirements requires manifest_version >= {MANIFEST_VERSION_3}; use string manifest_version \"{MANIFEST_VERSION_3}\" so older guests fail closed" ); } - if app_compose - .requirements - .as_ref() - .is_some_and(|requirements| { - !requirements.gpu_policy.is_default() && requirements.attest_gpu == Some(false) - }) - { - bail!("requirements.gpu_policy requires GPU attestation"); - } Ok(()) } @@ -3090,26 +3081,6 @@ fn test_empty_requirements_require_v3_manifest() { assert!(err.to_string().contains("requires manifest_version")); } -#[test] -fn test_gpu_policy_requires_gpu_attestation() { - let app_compose: AppCompose = serde_json::from_value(serde_json::json!({ - "manifest_version": "3", - "name": "test", - "runner": "docker-compose", - "requirements": { - "attest_gpu": false, - "gpu_policy": { - "rego": "package policy\n\ndefault nv_match = true\n" - } - } - })) - .unwrap(); - let err = verify_manifest_feature_requirements(&app_compose).unwrap_err(); - assert!(err - .to_string() - .contains("requirements.gpu_policy requires GPU attestation")); -} - #[test] fn test_platform_requirements_omitted_accepts_any_platform() { let app_compose = test_app_compose(serde_json::json!("3"), None, None); From 5fea6aa7f148c3e2b825f1b308ae2bcf63d0fa94 Mon Sep 17 00:00:00 2001 From: Kevin Wang Date: Thu, 16 Jul 2026 09:04:30 -0700 Subject: [PATCH 12/32] refactor(guest): combine basic GPU state checks --- dstack/dstack-util/src/system_setup.rs | 2 -- 1 file changed, 2 deletions(-) diff --git a/dstack/dstack-util/src/system_setup.rs b/dstack/dstack-util/src/system_setup.rs index 8b32d1ba5..2c8ae897d 100644 --- a/dstack/dstack-util/src/system_setup.rs +++ b/dstack/dstack-util/src/system_setup.rs @@ -1347,8 +1347,6 @@ mod gpu { if !device.cc_enabled { bail!("gpu at index {index} does not enable confidential compute mode"); } - } - for (index, device) in devices.iter().enumerate() { if device.devtools && !policy.allow_devtools { bail!("gpu at index {index} enables NVIDIA DevTools mode"); } From b93a9d050f1f57f0ed599bd739596efa1f69aacc Mon Sep 17 00:00:00 2001 From: Kevin Wang Date: Thu, 16 Jul 2026 09:13:56 -0700 Subject: [PATCH 13/32] refactor(guest): decouple GPU attestation from VM config --- docs/security/security-model.md | 14 ++++++------- dstack/dstack-util/src/system_setup.rs | 27 ++++++-------------------- 2 files changed, 13 insertions(+), 28 deletions(-) diff --git a/docs/security/security-model.md b/docs/security/security-model.md index d2af5acfc..31a10d1b4 100644 --- a/docs/security/security-model.md +++ b/docs/security/security-model.md @@ -55,7 +55,7 @@ dstack supports NVIDIA H100, H200, and B200 GPUs in confidential compute mode fo ### How It Works -GPUs are passed through via VFIO to the TEE-protected CVM. Before key provisioning, `dstack-util setup` inventories every VGA/3D-controller PCI function, compares that count with `vm_config.num_gpus`, and runs NVIDIA's local `nvattest` verifier over every NVIDIA-driver-visible GPU with a fresh nonce. dstack does not ship or pass a custom relying-party policy to this command; nvattest's built-in appraisal still applies. The complete JSON result (`result_code`, `result_message`, `claims`, and `detached_eat`) is saved at `/run/nvidia-gpu-attestation/attestation.out`. +GPUs are passed through via VFIO to the TEE-protected CVM. Before key provisioning, `dstack-util setup` inventories every VGA/3D-controller PCI function, rejects non-NVIDIA display devices, and runs NVIDIA's local `nvattest` verifier over every NVIDIA-driver-visible GPU with a fresh nonce. dstack does not ship or pass a custom relying-party policy to this command; nvattest's built-in appraisal still applies. The complete JSON result (`result_code`, `result_message`, `claims`, and `detached_eat`) is saved at `/run/nvidia-gpu-attestation/attestation.out`. An application may additionally set `requirements.gpu_policy` to an object with the following deny-unknown-fields schema: @@ -72,16 +72,16 @@ For every NVML-enumerated GPU, dstack calls `Device::is_cc_enabled()` and `Devic GPU workloads require verification of both hardware components. The CPU TEE quote verifies the CVM and its measured guest code. NVIDIA-signed evidence, checked against NVIDIA RIMs and certificate status by `nvattest`, verifies the GPU appraisal. After the optional policy and ready-state operations succeed, dstack emits a `gpu-attestation` launch event before `system-ready`. Its versioned payload records the number of appraised devices, asserted CC/DevTools state, and SHA-256 of the complete nvattest JSON output (claims and detached EAT). On TDX, both `gpu-policy-hash` and `gpu-attestation` are append-only RTMR3 events; they are never derived from application-controlled `report_data`. -A verifier must replay the measured event log, require exactly one pre-`system-ready` `gpu-attestation` event when GPU protection is required, and require `devices == vm_config.num_gpus > 0`. If an application policy is required, the verifier must also require exactly one `gpu-policy-hash` event immediately after `compose-hash` and compare its 32-byte payload with the expected policy digest. The raw `attestation.out` file is not trusted by itself; if it is supplied for inspection, its digest must match the `gpu-attestation` event. +A verifier must replay the measured event log, require exactly one pre-`system-ready` `gpu-attestation` event when GPU protection is required, and require `devices > 0` and, when applicable, the expected deployment count. If an application policy is required, the verifier must also require exactly one `gpu-policy-hash` event immediately after `compose-hash` and compare its 32-byte payload with the expected policy digest. The raw `attestation.out` file is not trusted by itself; if it is supplied for inspection, its digest must match the `gpu-attestation` event. ### GPU Threat Model and Lifetime The GPU gate assumes a malicious host/VMM and untrusted host-provided PCI topology, while trusting the CPU TEE, the measured dstack guest/kernel, NVIDIA hardware/firmware roots, and the cryptography used by both attestation chains. Availability is out of scope. -The events make the following **boot-time** statement: immediately before key provisioning, all attached VGA/3D PCI functions were NVIDIA devices, their count matched the quote-bound VM configuration and NVML inventory, nvattest returned one fresh successfully appraised claim for each device, the measured application policy accepted those claims and GPU state when present, every enumerated GPU passed the CC/DevTools NVML checks, and setting the GPU ready state succeeded. This closes these cases: +The events make the following **boot-time** statement: immediately before key provisioning, all attached VGA/3D PCI functions were NVIDIA devices, their count matched the NVML inventory, nvattest returned one fresh successfully appraised claim for each device, the measured application policy accepted those claims and GPU state when present, every enumerated GPU passed the CC/DevTools NVML checks, and setting the GPU ready state succeeded. This closes these cases: -- A GPU-less launch cannot be presented as a GPU-verified launch: it has `num_gpus == 0` and no `gpu-attestation` event. -- A mixed launch cannot attest only its TEE-capable subset. Non-NVIDIA display GPUs are rejected, and the sysfs, `vm_config`, and nvattest claim counts must all agree. A non-CC NVIDIA GPU either prevents evidence collection/appraisal or causes the default appraisal, application policy, or CC-state check to fail. +- A GPU-less launch cannot be presented as a GPU-verified launch because it has no `gpu-attestation` event. +- A mixed launch cannot attest only its TEE-capable subset. Non-NVIDIA display GPUs are rejected, and the sysfs, NVML, and nvattest claim counts must all agree. A non-CC NVIDIA GPU either prevents evidence collection/appraisal or causes the default appraisal, application policy, or CC-state check to fail. - Copying another CVM's result into a file or `report_data` does not work. Only measured pre-application code can place the event before `system-ready`, and event-log replay binds it to the quoted RTMR/PCR value. This is **not a lifetime or physical co-location guarantee**. After `system-ready`, an application with sufficient guest privileges can unload the NVIDIA driver, and a malicious host may attempt PCI hot-remove/replacement or proxy GPU traffic. The boot event remains a true historical statement but does not prove that the same device is still attached. dstack also cannot rule out a live relay/cuckoo attack to a genuine remote GPU: current Hopper/Blackwell deployments do not provide a CPU-TEE-verifiable TEE-I/O/TDISP device binding. Applications that mutate the driver or PCI topology are outside this guarantee; higher-assurance deployments must prevent that behavior and re-attest before using a newly initialized GPU. @@ -184,10 +184,10 @@ Use this checklist to verify a workload running in a dstack CVM. - [ ] reportData contains your challenge (replay protection) **GPU verification (when required):** -- [ ] `vm_config.num_gpus` is greater than zero and matches the expected deployment +- [ ] The `gpu-attestation` device count is greater than zero and matches the expected deployment - [ ] If `requirements.gpu_policy` is required, exactly one `gpu-policy-hash` event follows `compose-hash`, and its payload matches the expected SHA-256 digest - [ ] Exactly one `gpu-attestation` event appears before `system-ready` -- [ ] The event device count matches `vm_config.num_gpus`, CC is ON, and its DevTools field complies with the measured policy +- [ ] CC is ON and the event's DevTools field complies with the measured policy - [ ] The platform binds the event log to a quoted RTMR/PCR (do not accept it from current SEV-SNP evidence) **Key management verification:** diff --git a/dstack/dstack-util/src/system_setup.rs b/dstack/dstack-util/src/system_setup.rs index 2c8ae897d..e47dc9b2f 100644 --- a/dstack/dstack-util/src/system_setup.rs +++ b/dstack/dstack-util/src/system_setup.rs @@ -1206,10 +1206,7 @@ mod gpu { Ok(inventory) } - /// Bind the runtime inventory to `vm_config.num_gpus`. On TDX this config - /// affects the measured launch layout, so a remote verifier can compare the - /// event's device count with the same quote-bound value. - pub(super) fn expected_gpu_count(vm_config: &str, inventory: GpuInventory) -> Result { + pub(super) fn nvidia_gpu_count(inventory: GpuInventory) -> Result { if inventory.total != inventory.nvidia { bail!( "unsupported non-NVIDIA GPU attached: found {} display GPUs, {} NVIDIA", @@ -1217,16 +1214,7 @@ mod gpu { inventory.nvidia ); } - let vm_config: dstack_types::VmConfig = serde_json::from_str(vm_config) - .context("failed to parse vm_config for GPU attestation")?; - let expected = vm_config.num_gpus; - if inventory.total != expected { - bail!( - "gpu count mismatch: vm_config requires {expected}, found {}", - inventory.total - ); - } - Ok(expected) + Ok(inventory.nvidia) } /// Run a GPU tool with a bounded timeout so a wedged driver/GPU cannot @@ -1528,20 +1516,18 @@ mod gpu { } #[test] - fn expected_count_rejects_mixed_or_unmeasured_gpus() { + fn gpu_count_rejects_non_nvidia_gpus() { let mixed = GpuInventory { total: 2, nvidia: 1, }; - assert!(expected_gpu_count(r#"{"num_gpus":2}"#, mixed).is_err()); + assert!(nvidia_gpu_count(mixed).is_err()); let nvidia = GpuInventory { total: 2, nvidia: 2, }; - let err = expected_gpu_count(r#"{"num_gpus":1}"#, nvidia).unwrap_err(); - assert!(err.to_string().contains("gpu count mismatch")); - assert_eq!(expected_gpu_count(r#"{"num_gpus":2}"#, nvidia).unwrap(), 2); + assert_eq!(nvidia_gpu_count(nvidia).unwrap(), 2); } #[test] @@ -1782,8 +1768,7 @@ impl Stage0<'_> { } return Ok(()); } - let expected_devices = - gpu::expected_gpu_count(&self.shared.sys_config.vm_config, inventory)?; + let expected_devices = gpu::nvidia_gpu_count(inventory)?; if expected_devices == 0 { return Ok(()); } From fdf1bc7c3d42ab9c9e2b1a41315c8ec0fbc5d8b5 Mon Sep 17 00:00:00 2001 From: Kevin Wang Date: Thu, 16 Jul 2026 17:16:01 -0700 Subject: [PATCH 14/32] build(yocto): use cmake for aws-lc-sys --- .../meta-dstack/recipes-core/dstack-guest/dstack-guest.bb | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/os/yocto/layers/meta-dstack/recipes-core/dstack-guest/dstack-guest.bb b/os/yocto/layers/meta-dstack/recipes-core/dstack-guest/dstack-guest.bb index 7a31ec53b..daa00f9ed 100644 --- a/os/yocto/layers/meta-dstack/recipes-core/dstack-guest/dstack-guest.bb +++ b/os/yocto/layers/meta-dstack/recipes-core/dstack-guest/dstack-guest.bb @@ -18,6 +18,12 @@ DSTACK_ROOTFS_FILES = "${UNPACKDIR}/repo/os/common/rootfs" RDEPENDS:${PN} += "bash" DEPENDS += "rsync-native tpm2-tss" +DEPENDS += "cmake-native" + +# aws-lc-sys cannot detect Yocto cross builds when the build and target share +# the same Rust target triple, and its cc builder then tries to execute a +# target binary on the build host. Use its supported CMake builder instead. +export AWS_LC_SYS_CMAKE_BUILDER = "1" # Ensure rsync-native is built before unpack runs do_unpack[depends] += "rsync-native:do_populate_sysroot" From 42b283e0a0236c07d8a5ce5172e444ac07b16157 Mon Sep 17 00:00:00 2001 From: Kevin Wang Date: Thu, 16 Jul 2026 17:55:33 -0700 Subject: [PATCH 15/32] fix(guest): evaluate GPU policy without claims --- docs/security/security-model.md | 2 +- dstack/dstack-types/src/lib.rs | 7 ++- dstack/dstack-util/src/system_setup.rs | 72 +++++++++++++++++++------- 3 files changed, 60 insertions(+), 21 deletions(-) diff --git a/docs/security/security-model.md b/docs/security/security-model.md index 31a10d1b4..94f5ae0ce 100644 --- a/docs/security/security-model.md +++ b/docs/security/security-model.md @@ -64,7 +64,7 @@ An application may additionally set `requirements.gpu_policy` to an object with - `allow_debug` (boolean, default `false`): permit an attestation claim whose `dbgstat` is `enabled`. - `allow_insecure_boot` (boolean, default `false`): permit an attestation claim whose GPU `secboot` value is false. -After measuring `compose-hash`, dstack enters the GPU setup gate. If GPUs are configured, it attests them, applies field defaults, JCS-canonicalizes the complete `gpu_policy` structure, and measures its SHA-256 digest in a `gpu-policy-hash` event. It then applies the basic settings and optional Rego policy before setting the GPU ready state. A false, undefined, malformed, or non-boolean Rego result stops boot before key provisioning. GPU-less launches return from the gate without measuring or evaluating a GPU policy. When `attest_gpu` is false, `gpu_policy` is not measured or evaluated. +After measuring `compose-hash`, dstack enters the GPU setup gate, applies field defaults, and, when `gpu_policy` is non-default, JCS-canonicalizes the complete structure and measures its SHA-256 digest in a `gpu-policy-hash` event. If GPU attestation is enabled and NVIDIA GPUs are present, it attests them and applies the basic settings and optional Rego policy before setting the GPU ready state. When no attestation claims are produced—because no GPU is attached or `attest_gpu` is false—Rego is still evaluated with an empty array as `input` before any ready-state transition. This lets an application reject a launch whose attested GPU count is wrong. A false, undefined, malformed, or non-boolean Rego result stops boot before key provisioning. For every NVML-enumerated GPU, dstack calls `Device::is_cc_enabled()` and `Device::is_cc_dev_mode_enabled()` and requires the NVML device count to match the expected GPU count. CC must always be ON; DevTools must be OFF unless the measured policy explicitly permits it. The typed claim checks always require `measres == "success"`; by default they also require `dbgstat == "disabled"` and `secboot == true`, with the latter two checks controlled by their explicit opt-ins. Only after the default appraisal, typed claim checks, optional Rego policy, and per-device NVML checks succeed does dstack call `Device::set_confidential_compute_state(true)` to set the GPU ready state. The dstack CPU/guest boot chain is verified independently through measured boot; a GPU claim named `secboot` refers to the GPU appraisal, not UEFI Secure Boot in the CVM. diff --git a/dstack/dstack-types/src/lib.rs b/dstack/dstack-types/src/lib.rs index 3395cd92f..70cd2225b 100644 --- a/dstack/dstack-types/src/lib.rs +++ b/dstack/dstack-types/src/lib.rs @@ -180,14 +180,17 @@ pub struct Requirements { /// local GPU attestation (nvattest) during system setup and refuses to /// boot — before key provisioning — if the GPU fails to attest (e.g. a /// non-CC GPU or CC mode disabled by the host). `false` skips attestation - /// and sets the GPU ready state directly. Guests without a GPU attached - /// are unaffected either way. + /// and sets the GPU ready state directly. An application Rego GPU policy + /// is still evaluated with an empty claims array when attestation is + /// skipped or no GPU is attached. #[serde(skip_serializing_if = "Option::is_none")] pub attest_gpu: Option, /// Optional application GPU policy applied before key provisioning. /// /// Its default-expanded, JCS-canonicalized JSON SHA-256 digest is emitted /// as the `gpu-policy-hash` launch event immediately after `compose-hash`. + /// Rego receives an empty claims array when no GPU attestation is produced, + /// allowing applications to enforce an expected GPU count. #[serde(default, skip_serializing_if = "GpuPolicy::is_default")] pub gpu_policy: GpuPolicy, } diff --git a/dstack/dstack-util/src/system_setup.rs b/dstack/dstack-util/src/system_setup.rs index e47dc9b2f..011a65a92 100644 --- a/dstack/dstack-util/src/system_setup.rs +++ b/dstack/dstack-util/src/system_setup.rs @@ -1420,6 +1420,13 @@ mod gpu { Ok(sha256(&canonical)) } + pub(super) fn evaluate_rego_policy(policy: &GpuPolicy, claims: &[Value]) -> Result<()> { + let Some(rego) = policy.rego.as_deref() else { + return Ok(()); + }; + evaluate_policy(rego, claims).context("failed to apply GPU Rego policy") + } + /// Evaluate the app-provided Rego v0 policy using the same input shape as /// NVIDIA relying-party policies: the nvattest `claims` JSON array. pub(super) fn evaluate_policy(policy: &str, claims: &[Value]) -> Result<()> { @@ -1722,11 +1729,32 @@ mod gpu { } "#; evaluate_policy(policy, &claims).unwrap(); + assert!(evaluate_policy(policy, &[]).is_err()); let rejected = vec![serde_json::json!({"status": "rejected"})]; assert!(evaluate_policy(policy, &rejected).is_err()); assert!(evaluate_policy("package policy", &claims).is_err()); assert!(evaluate_policy("not valid rego", &claims).is_err()); + + let allow_no_gpus = GpuPolicy { + rego: Some( + r#" + package policy + default nv_match = false + nv_match { count(input) == 0 } + "# + .to_string(), + ), + ..Default::default() + }; + evaluate_rego_policy(&allow_no_gpus, &[]).unwrap(); + + let require_one_gpu = GpuPolicy { + rego: Some(policy.to_string()), + ..Default::default() + }; + assert!(evaluate_rego_policy(&require_one_gpu, &[]).is_err()); + evaluate_rego_policy(&GpuPolicy::default(), &[]).unwrap(); } #[test] @@ -1754,10 +1782,27 @@ mod gpu { impl Stage0<'_> { /// Enforce `requirements.attest_gpu` (default true): attest an attached /// NVIDIA GPU before continuing to key provisioning, or — when explicitly - /// disabled — set the GPU ready state without verification. + /// disabled — set the GPU ready state without verification. The optional + /// Rego policy is always evaluated; when no attestation is performed, its + /// claims-array input is empty. async fn setup_gpu(&self) -> Result<()> { + let gpu_policy = self + .shared + .app_compose + .requirements + .as_ref() + .map(|requirements| requirements.gpu_policy.clone()) + .unwrap_or_default(); + if !gpu_policy.is_default() { + emit_runtime_event("gpu-policy-hash", &gpu::policy_digest(&gpu_policy)?)?; + } + let inventory = gpu::gpu_inventory()?; if !self.shared.app_compose.attest_gpu() { + gpu::evaluate_rego_policy(&gpu_policy, &[])?; + if gpu_policy.rego.is_some() { + info!("application GPU Rego policy accepted an empty claims array"); + } if inventory.nvidia == 0 { return Ok(()); } @@ -1770,39 +1815,30 @@ impl Stage0<'_> { } let expected_devices = gpu::nvidia_gpu_count(inventory)?; if expected_devices == 0 { + gpu::evaluate_rego_policy(&gpu_policy, &[])?; + if gpu_policy.rego.is_some() { + info!("application GPU Rego policy accepted an empty claims array"); + } return Ok(()); } self.vmm.notify_q("boot.progress", "attesting GPU").await; info!("verifying GPU TEE attestation"); let attestation = gpu::attest_gpu(expected_devices).await?; - let gpu_policy = self - .shared - .app_compose - .requirements - .as_ref() - .map(|requirements| requirements.gpu_policy.clone()) - .unwrap_or_default(); - if !gpu_policy.is_default() { - emit_runtime_event("gpu-policy-hash", &gpu::policy_digest(&gpu_policy)?)?; - } let gpu_state = gpu::query_gpu_state(expected_devices)?; attestation .verify_claim_policy(&gpu_state, &gpu_policy) .context("failed to apply basic GPU policy")?; - if let Some(rego) = gpu_policy.rego.as_deref() { - gpu::evaluate_policy(rego, attestation.claims()) - .context("failed to apply GPU Rego policy")?; - } + gpu::evaluate_rego_policy(&gpu_policy, attestation.claims())?; + if !gpu_policy.is_default() { + info!("application GPU policy accepted the attestation claims and state"); + } gpu_state.set_ready()?; let devtools = gpu_state.any_devtools(); let event = attestation.event(devtools)?; emit_runtime_event("gpu-attestation", &event) .context("failed to emit GPU attestation event")?; - if !gpu_policy.is_default() { - info!("application GPU policy accepted the attestation claims and state"); - } info!("GPU TEE attestation succeeded"); Ok(()) } From 27d937a1225e09aecd02b8d3be3fa3e7b80ab767 Mon Sep 17 00:00:00 2001 From: Kevin Wang Date: Thu, 16 Jul 2026 18:05:56 -0700 Subject: [PATCH 16/32] fix(guest): measure raw GPU policy JSON --- docs/security/security-model.md | 2 +- dstack/dstack-types/src/lib.rs | 6 ++- dstack/dstack-util/src/system_setup.rs | 68 ++++++++++++++++++-------- 3 files changed, 53 insertions(+), 23 deletions(-) diff --git a/docs/security/security-model.md b/docs/security/security-model.md index 94f5ae0ce..41eeb147b 100644 --- a/docs/security/security-model.md +++ b/docs/security/security-model.md @@ -64,7 +64,7 @@ An application may additionally set `requirements.gpu_policy` to an object with - `allow_debug` (boolean, default `false`): permit an attestation claim whose `dbgstat` is `enabled`. - `allow_insecure_boot` (boolean, default `false`): permit an attestation claim whose GPU `secboot` value is false. -After measuring `compose-hash`, dstack enters the GPU setup gate, applies field defaults, and, when `gpu_policy` is non-default, JCS-canonicalizes the complete structure and measures its SHA-256 digest in a `gpu-policy-hash` event. If GPU attestation is enabled and NVIDIA GPUs are present, it attests them and applies the basic settings and optional Rego policy before setting the GPU ready state. When no attestation claims are produced—because no GPU is attached or `attest_gpu` is false—Rego is still evaluated with an empty array as `input` before any ready-state transition. This lets an application reject a launch whose attested GPU count is wrong. A false, undefined, malformed, or non-boolean Rego result stops boot before key provisioning. +After measuring `compose-hash`, dstack enters the GPU setup gate and looks for the original `requirements.gpu_policy` JSON value. Whenever the field is present—even as an empty object or with only explicit default values—dstack JCS-canonicalizes that raw value and measures its SHA-256 digest in a `gpu-policy-hash` event. The typed policy used for enforcement still applies omitted-field defaults and rejects unknown fields. If GPU attestation is enabled and NVIDIA GPUs are present, dstack attests them and applies the basic settings and optional Rego policy before setting the GPU ready state. When no attestation claims are produced—because no GPU is attached or `attest_gpu` is false—Rego is still evaluated with an empty array as `input` before any ready-state transition. This lets an application reject a launch whose attested GPU count is wrong. A false, undefined, malformed, or non-boolean Rego result stops boot before key provisioning. For every NVML-enumerated GPU, dstack calls `Device::is_cc_enabled()` and `Device::is_cc_dev_mode_enabled()` and requires the NVML device count to match the expected GPU count. CC must always be ON; DevTools must be OFF unless the measured policy explicitly permits it. The typed claim checks always require `measres == "success"`; by default they also require `dbgstat == "disabled"` and `secboot == true`, with the latter two checks controlled by their explicit opt-ins. Only after the default appraisal, typed claim checks, optional Rego policy, and per-device NVML checks succeed does dstack call `Device::set_confidential_compute_state(true)` to set the GPU ready state. The dstack CPU/guest boot chain is verified independently through measured boot; a GPU claim named `secboot` refers to the GPU appraisal, not UEFI Secure Boot in the CVM. diff --git a/dstack/dstack-types/src/lib.rs b/dstack/dstack-types/src/lib.rs index 70cd2225b..ee2293f03 100644 --- a/dstack/dstack-types/src/lib.rs +++ b/dstack/dstack-types/src/lib.rs @@ -187,8 +187,10 @@ pub struct Requirements { pub attest_gpu: Option, /// Optional application GPU policy applied before key provisioning. /// - /// Its default-expanded, JCS-canonicalized JSON SHA-256 digest is emitted - /// as the `gpu-policy-hash` launch event immediately after `compose-hash`. + /// When this field is present, its original JSON value is JCS-canonicalized + /// and its SHA-256 digest is emitted as the `gpu-policy-hash` launch event + /// immediately after `compose-hash`, including when the object is empty or + /// contains only explicit default values. /// Rego receives an empty claims array when no GPU attestation is produced, /// allowing applications to enforce an expected GPU count. #[serde(default, skip_serializing_if = "GpuPolicy::is_default")] diff --git a/dstack/dstack-util/src/system_setup.rs b/dstack/dstack-util/src/system_setup.rs index 011a65a92..3a423b0e7 100644 --- a/dstack/dstack-util/src/system_setup.rs +++ b/dstack/dstack-util/src/system_setup.rs @@ -1415,9 +1415,25 @@ mod gpu { }) } - pub(super) fn policy_digest(policy: &GpuPolicy) -> Result<[u8; 32]> { - let canonical = serde_jcs::to_vec(policy).context("failed to canonicalize GPU policy")?; - Ok(sha256(&canonical)) + fn gpu_policy_measurement(compose_json: &[u8]) -> Result> { + let compose: Value = + serde_json::from_slice(compose_json).context("failed to parse raw app compose")?; + let Some(policy) = compose.pointer("/requirements/gpu_policy") else { + return Ok(None); + }; + let canonical = + serde_jcs::to_vec(policy).context("failed to canonicalize raw GPU policy")?; + Ok(Some(sha256(&canonical))) + } + + pub(super) fn measure_gpu_policy(compose_path: &Path) -> Result<()> { + let compose_json = fs::read(compose_path) + .with_context(|| format!("failed to read {}", compose_path.display()))?; + let Some(digest) = gpu_policy_measurement(&compose_json)? else { + return Ok(()); + }; + emit_runtime_event("gpu-policy-hash", &digest) + .context("failed to emit GPU policy measurement") } pub(super) fn evaluate_rego_policy(policy: &GpuPolicy, claims: &[Value]) -> Result<()> { @@ -1758,22 +1774,35 @@ mod gpu { } #[test] - fn policy_digest_commits_to_the_canonical_policy_structure() { - let policy = GpuPolicy { - rego: Some("package policy\n\ndefault nv_match = true\n".to_string()), - allow_devtools: true, - ..Default::default() - }; - let canonical = serde_jcs::to_vec(&policy).unwrap(); - assert_eq!(policy_digest(&policy).unwrap(), sha256(&canonical)); + fn gpu_policy_measurement_uses_the_present_raw_json_value() { + let absent = br#"{"requirements": {}}"#; + assert_eq!(gpu_policy_measurement(absent).unwrap(), None); - let production_policy = GpuPolicy { - allow_devtools: false, - ..policy.clone() - }; + let empty = br#"{"requirements": {"gpu_policy": {}}}"#; + assert_eq!(gpu_policy_measurement(empty).unwrap(), Some(sha256(b"{}"))); + + let explicit_default = br#"{"requirements":{"gpu_policy":{"allow_debug":false}}}"#; + let explicit_default_digest = gpu_policy_measurement(explicit_default).unwrap(); assert_ne!( - policy_digest(&policy).unwrap(), - policy_digest(&production_policy).unwrap() + explicit_default_digest, + gpu_policy_measurement(empty).unwrap() + ); + + let reordered = br#" + { + "requirements": { + "gpu_policy": { + "rego": "package policy", + "allow_debug": false + } + } + } + "#; + let canonical_order = + br#"{"requirements":{"gpu_policy":{"allow_debug":false,"rego":"package policy"}}}"#; + assert_eq!( + gpu_policy_measurement(reordered).unwrap(), + gpu_policy_measurement(canonical_order).unwrap() ); } } @@ -1786,6 +1815,8 @@ impl Stage0<'_> { /// Rego policy is always evaluated; when no attestation is performed, its /// claims-array input is empty. async fn setup_gpu(&self) -> Result<()> { + gpu::measure_gpu_policy(&self.shared.dir.app_compose_file())?; + let gpu_policy = self .shared .app_compose @@ -1793,9 +1824,6 @@ impl Stage0<'_> { .as_ref() .map(|requirements| requirements.gpu_policy.clone()) .unwrap_or_default(); - if !gpu_policy.is_default() { - emit_runtime_event("gpu-policy-hash", &gpu::policy_digest(&gpu_policy)?)?; - } let inventory = gpu::gpu_inventory()?; if !self.shared.app_compose.attest_gpu() { From 6453cebcdeed2864962da27ec80103c2f399d1b0 Mon Sep 17 00:00:00 2001 From: Kevin Wang Date: Thu, 16 Jul 2026 18:18:49 -0700 Subject: [PATCH 17/32] refactor(guest): move attestation toggle into GPU policy --- docs/security/security-model.md | 7 +- dstack/dstack-types/src/lib.rs | 103 +++++++++--------- dstack/dstack-util/src/system_setup.rs | 55 +++++----- .../files/10-nvidia-gpu-ordering.conf | 5 +- .../nvidia/files/nvidia-persistenced.service | 3 +- 5 files changed, 89 insertions(+), 84 deletions(-) diff --git a/docs/security/security-model.md b/docs/security/security-model.md index 41eeb147b..627ce0cd2 100644 --- a/docs/security/security-model.md +++ b/docs/security/security-model.md @@ -59,12 +59,13 @@ GPUs are passed through via VFIO to the TEE-protected CVM. Before key provisioni An application may additionally set `requirements.gpu_policy` to an object with the following deny-unknown-fields schema: +- `attest_gpu` (boolean, default `true`): require local NVIDIA GPU attestation before enabling an attached GPU. Setting this to `false` skips attestation and is an explicit reduction in protection. - `rego` (optional string): a Rego v0 script evaluated with the nvattest output's `claims` array as `input`. It must define the boolean entrypoint `data.policy.nv_match` in `package policy`. - `allow_devtools` (boolean, default `false`): permit NVIDIA DevTools mode. Production applications should leave this disabled because DevTools removes the expected GPU memory-confidentiality guarantee. - `allow_debug` (boolean, default `false`): permit an attestation claim whose `dbgstat` is `enabled`. - `allow_insecure_boot` (boolean, default `false`): permit an attestation claim whose GPU `secboot` value is false. -After measuring `compose-hash`, dstack enters the GPU setup gate and looks for the original `requirements.gpu_policy` JSON value. Whenever the field is present—even as an empty object or with only explicit default values—dstack JCS-canonicalizes that raw value and measures its SHA-256 digest in a `gpu-policy-hash` event. The typed policy used for enforcement still applies omitted-field defaults and rejects unknown fields. If GPU attestation is enabled and NVIDIA GPUs are present, dstack attests them and applies the basic settings and optional Rego policy before setting the GPU ready state. When no attestation claims are produced—because no GPU is attached or `attest_gpu` is false—Rego is still evaluated with an empty array as `input` before any ready-state transition. This lets an application reject a launch whose attested GPU count is wrong. A false, undefined, malformed, or non-boolean Rego result stops boot before key provisioning. +After measuring `compose-hash`, dstack enters the GPU setup gate and JCS-canonicalizes the original `requirements.gpu_policy` JSON value, then measures its SHA-256 digest in a `gpu-policy-hash` event. When the field is absent—including when `requirements` itself is absent—both parsing and measurement use the default empty object `{}`. Thus an omitted policy and an explicit `{}` have the same digest, while any explicitly present field, including an explicit default value, changes the digest. The typed policy used for enforcement applies omitted-field defaults and rejects unknown fields. If GPU attestation is enabled and NVIDIA GPUs are present, dstack attests them and applies the basic settings and optional Rego policy before setting the GPU ready state. When no attestation claims are produced—because no GPU is attached or `gpu_policy.attest_gpu` is false—Rego is still evaluated with an empty array as `input` before any ready-state transition. This lets an application reject a launch whose attested GPU count is wrong. A false, undefined, malformed, or non-boolean Rego result stops boot before key provisioning. For every NVML-enumerated GPU, dstack calls `Device::is_cc_enabled()` and `Device::is_cc_dev_mode_enabled()` and requires the NVML device count to match the expected GPU count. CC must always be ON; DevTools must be OFF unless the measured policy explicitly permits it. The typed claim checks always require `measres == "success"`; by default they also require `dbgstat == "disabled"` and `secboot == true`, with the latter two checks controlled by their explicit opt-ins. Only after the default appraisal, typed claim checks, optional Rego policy, and per-device NVML checks succeed does dstack call `Device::set_confidential_compute_state(true)` to set the GPU ready state. The dstack CPU/guest boot chain is verified independently through measured boot; a GPU claim named `secboot` refers to the GPU appraisal, not UEFI Secure Boot in the CVM. @@ -72,7 +73,7 @@ For every NVML-enumerated GPU, dstack calls `Device::is_cc_enabled()` and `Devic GPU workloads require verification of both hardware components. The CPU TEE quote verifies the CVM and its measured guest code. NVIDIA-signed evidence, checked against NVIDIA RIMs and certificate status by `nvattest`, verifies the GPU appraisal. After the optional policy and ready-state operations succeed, dstack emits a `gpu-attestation` launch event before `system-ready`. Its versioned payload records the number of appraised devices, asserted CC/DevTools state, and SHA-256 of the complete nvattest JSON output (claims and detached EAT). On TDX, both `gpu-policy-hash` and `gpu-attestation` are append-only RTMR3 events; they are never derived from application-controlled `report_data`. -A verifier must replay the measured event log, require exactly one pre-`system-ready` `gpu-attestation` event when GPU protection is required, and require `devices > 0` and, when applicable, the expected deployment count. If an application policy is required, the verifier must also require exactly one `gpu-policy-hash` event immediately after `compose-hash` and compare its 32-byte payload with the expected policy digest. The raw `attestation.out` file is not trusted by itself; if it is supplied for inspection, its digest must match the `gpu-attestation` event. +A verifier must replay the measured event log, require exactly one `gpu-policy-hash` event immediately after `compose-hash`, and compare its 32-byte payload with the expected policy digest (`SHA-256(JCS({}))` for the omitted/default policy). When GPU protection is required, it must also require exactly one pre-`system-ready` `gpu-attestation` event with `devices > 0` and, when applicable, the expected deployment count. The raw `attestation.out` file is not trusted by itself; if it is supplied for inspection, its digest must match the `gpu-attestation` event. ### GPU Threat Model and Lifetime @@ -185,7 +186,7 @@ Use this checklist to verify a workload running in a dstack CVM. **GPU verification (when required):** - [ ] The `gpu-attestation` device count is greater than zero and matches the expected deployment -- [ ] If `requirements.gpu_policy` is required, exactly one `gpu-policy-hash` event follows `compose-hash`, and its payload matches the expected SHA-256 digest +- [ ] Exactly one `gpu-policy-hash` event follows `compose-hash`, and its payload matches `SHA-256(JCS(requirements.gpu_policy))` (default `{}` when omitted) - [ ] Exactly one `gpu-attestation` event appears before `system-ready` - [ ] CC is ON and the event's DevTools field complies with the measured policy - [ ] The platform binds the event log to a quoted RTMR/PCR (do not accept it from current SEV-SNP evidence) diff --git a/dstack/dstack-types/src/lib.rs b/dstack/dstack-types/src/lib.rs index ee2293f03..9a5b4c7f1 100644 --- a/dstack/dstack-types/src/lib.rs +++ b/dstack/dstack-types/src/lib.rs @@ -121,9 +121,13 @@ pub struct AppCompose { pub requirements: Option, } -#[derive(Deserialize, Serialize, Debug, Clone, Default, PartialEq, Eq)] +#[derive(Deserialize, Serialize, Debug, Clone, PartialEq, Eq)] #[serde(default, deny_unknown_fields)] pub struct GpuPolicy { + /// Whether an attached GPU must pass local TEE attestation before the + /// guest continues booting. Defaults to true. + #[serde(default = "default_true")] + pub attest_gpu: bool, /// Optional Rego v0 policy evaluated against NVIDIA nvattest's `claims` /// array. It must define the boolean rule `data.policy.nv_match`. #[serde(skip_serializing_if = "Option::is_none")] @@ -139,6 +143,18 @@ pub struct GpuPolicy { pub allow_insecure_boot: bool, } +impl Default for GpuPolicy { + fn default() -> Self { + Self { + attest_gpu: true, + rego: None, + allow_devtools: false, + allow_debug: false, + allow_insecure_boot: false, + } + } +} + impl GpuPolicy { /// Returns true when no application-specific GPU policy setting is set. pub fn is_default(&self) -> bool { @@ -174,23 +190,13 @@ pub struct Requirements { /// (e.g. 32 random alphanumeric characters). #[serde(skip_serializing_if = "Option::is_none")] pub launch_token_hash: Option, - /// GPU TEE attestation requirement, defaults to `true` when omitted. - /// - /// On guests with an NVIDIA GPU attached, `true` means the guest runs - /// local GPU attestation (nvattest) during system setup and refuses to - /// boot — before key provisioning — if the GPU fails to attest (e.g. a - /// non-CC GPU or CC mode disabled by the host). `false` skips attestation - /// and sets the GPU ready state directly. An application Rego GPU policy - /// is still evaluated with an empty claims array when attestation is - /// skipped or no GPU is attached. - #[serde(skip_serializing_if = "Option::is_none")] - pub attest_gpu: Option, - /// Optional application GPU policy applied before key provisioning. + /// Application GPU policy applied before key provisioning. An omitted + /// field is parsed and measured as the default empty policy `{}`. /// - /// When this field is present, its original JSON value is JCS-canonicalized - /// and its SHA-256 digest is emitted as the `gpu-policy-hash` launch event - /// immediately after `compose-hash`, including when the object is empty or - /// contains only explicit default values. + /// Its original JSON value is JCS-canonicalized and its SHA-256 digest is + /// emitted as the `gpu-policy-hash` launch event immediately after + /// `compose-hash`. When the field is absent, `{}` is measured. An explicitly + /// present default-valued field remains part of the raw measurement. /// Rego receives an empty claims array when no GPU attestation is produced, /// allowing applications to enforce an expected GPU count. #[serde(default, skip_serializing_if = "GpuPolicy::is_default")] @@ -203,7 +209,6 @@ impl Requirements { && self.platforms.is_none() && self.tdx_measure_acpi_tables.is_none() && self.launch_token_hash.is_none() - && self.attest_gpu.is_none() && self.gpu_policy.is_default() } } @@ -399,24 +404,6 @@ impl AppCompose { } } } - - /// Whether an attached GPU must pass local TEE attestation before the - /// guest continues booting. Defaults to `true` when - /// `requirements.attest_gpu` is omitted. - /// - /// `requirements` are only valid on manifest_version >= 3 (guests reject - /// older manifests carrying them); the opt-out is additionally ignored on - /// legacy manifests here so a caller that skipped that validation still - /// fails closed. - pub fn attest_gpu(&self) -> bool { - if !matches!(self.manifest_version_u32(), Some(v) if v >= 3) { - return true; - } - self.requirements - .as_ref() - .and_then(|r| r.attest_gpu) - .unwrap_or(true) - } } #[cfg(test)] @@ -514,6 +501,7 @@ mod app_compose_tests { Some("9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08") ); let gpu_policy = &requirements.gpu_policy; + assert!(gpu_policy.attest_gpu); assert_eq!( gpu_policy.rego.as_deref(), Some("package policy\n\ndefault nv_match = false\n") @@ -602,6 +590,7 @@ mod app_compose_tests { .unwrap(); let requirements = gpu_policy.requirements.as_ref().unwrap(); let gpu_policy = &requirements.gpu_policy; + assert!(gpu_policy.attest_gpu); assert!(gpu_policy.rego.is_some()); assert!(!gpu_policy.allow_devtools); assert!(!gpu_policy.allow_debug); @@ -625,50 +614,58 @@ mod app_compose_tests { #[test] fn attest_gpu_defaults_to_true() { - let no_requirements: AppCompose = serde_json::from_value(serde_json::json!({ - "manifest_version": 2, + assert!(GpuPolicy::default().attest_gpu); + + let omitted: AppCompose = serde_json::from_value(serde_json::json!({ + "manifest_version": "3", "name": "test", - "runner": "docker-compose" + "runner": "docker-compose", + "requirements": {} })) .unwrap(); - assert!(no_requirements.attest_gpu()); + let requirements = omitted.requirements.as_ref().unwrap(); + assert!(requirements.gpu_policy.attest_gpu); + assert!(requirements.is_empty()); - let omitted: AppCompose = serde_json::from_value(serde_json::json!({ + let explicit_empty: AppCompose = serde_json::from_value(serde_json::json!({ "manifest_version": "3", "name": "test", "runner": "docker-compose", - "requirements": {} + "requirements": { + "gpu_policy": {} + } })) .unwrap(); - assert!(omitted.attest_gpu()); - assert!(omitted.requirements.as_ref().unwrap().is_empty()); + assert_eq!( + explicit_empty.requirements.unwrap().gpu_policy, + requirements.gpu_policy + ); let disabled: AppCompose = serde_json::from_value(serde_json::json!({ "manifest_version": "3", "name": "test", "runner": "docker-compose", "requirements": { - "attest_gpu": false + "gpu_policy": { + "attest_gpu": false + } } })) .unwrap(); - assert!(!disabled.attest_gpu()); let requirements = disabled.requirements.as_ref().unwrap(); - assert_eq!(requirements.attest_gpu, Some(false)); + assert!(!requirements.gpu_policy.attest_gpu); assert!(!requirements.is_empty()); - // The opt-out is ignored on legacy manifests (requirements are only - // valid on manifest_version >= 3; guests reject such composes anyway). - let legacy_optout: AppCompose = serde_json::from_value(serde_json::json!({ - "manifest_version": 2, + let old_location = serde_json::from_value::(serde_json::json!({ + "manifest_version": "3", "name": "test", "runner": "docker-compose", "requirements": { "attest_gpu": false } })) - .unwrap(); - assert!(legacy_optout.attest_gpu()); + .unwrap_err(); + assert!(old_location.to_string().contains("unknown field")); } #[test] diff --git a/dstack/dstack-util/src/system_setup.rs b/dstack/dstack-util/src/system_setup.rs index 3a423b0e7..d29ef1dee 100644 --- a/dstack/dstack-util/src/system_setup.rs +++ b/dstack/dstack-util/src/system_setup.rs @@ -1064,7 +1064,8 @@ async fn do_sys_setup(stage0: Stage0<'_>) -> Result<()> { stage1.setup().await } -/// GPU TEE attestation gate (`requirements.attest_gpu`, defaults to true). +/// GPU TEE attestation gate (`requirements.gpu_policy.attest_gpu`, defaults to +/// true). /// /// Runs before key provisioning so a CVM whose GPU cannot prove it is a /// genuine, CC-enabled NVIDIA TEE never gets its app keys. An optional @@ -1283,7 +1284,7 @@ mod gpu { } /// Set the system-wide GPU ready state without appraisal for the explicit - /// `attest_gpu: false` compatibility path. + /// `gpu_policy.attest_gpu: false` compatibility path. pub(super) fn set_gpu_ready_state(expected_devices: u32) -> Result<()> { let nvml = init_nvml(expected_devices)?; set_gpu_ready_state_with_nvml(&nvml) @@ -1415,23 +1416,22 @@ mod gpu { }) } - fn gpu_policy_measurement(compose_json: &[u8]) -> Result> { + fn gpu_policy_measurement(compose_json: &[u8]) -> Result<[u8; 32]> { let compose: Value = serde_json::from_slice(compose_json).context("failed to parse raw app compose")?; - let Some(policy) = compose.pointer("/requirements/gpu_policy") else { - return Ok(None); - }; + let default_policy = Value::Object(Default::default()); + let policy = compose + .pointer("/requirements/gpu_policy") + .unwrap_or(&default_policy); let canonical = serde_jcs::to_vec(policy).context("failed to canonicalize raw GPU policy")?; - Ok(Some(sha256(&canonical))) + Ok(sha256(&canonical)) } pub(super) fn measure_gpu_policy(compose_path: &Path) -> Result<()> { let compose_json = fs::read(compose_path) .with_context(|| format!("failed to read {}", compose_path.display()))?; - let Some(digest) = gpu_policy_measurement(&compose_json)? else { - return Ok(()); - }; + let digest = gpu_policy_measurement(&compose_json)?; emit_runtime_event("gpu-policy-hash", &digest) .context("failed to emit GPU policy measurement") } @@ -1774,19 +1774,22 @@ mod gpu { } #[test] - fn gpu_policy_measurement_uses_the_present_raw_json_value() { + fn gpu_policy_measurement_defaults_to_empty_object_and_uses_raw_json() { + let no_requirements = br#"{}"#; let absent = br#"{"requirements": {}}"#; - assert_eq!(gpu_policy_measurement(absent).unwrap(), None); + let empty_digest = sha256(b"{}"); + assert_eq!( + gpu_policy_measurement(no_requirements).unwrap(), + empty_digest + ); + assert_eq!(gpu_policy_measurement(absent).unwrap(), empty_digest); let empty = br#"{"requirements": {"gpu_policy": {}}}"#; - assert_eq!(gpu_policy_measurement(empty).unwrap(), Some(sha256(b"{}"))); + assert_eq!(gpu_policy_measurement(empty).unwrap(), empty_digest); - let explicit_default = br#"{"requirements":{"gpu_policy":{"allow_debug":false}}}"#; + let explicit_default = br#"{"requirements":{"gpu_policy":{"attest_gpu":true}}}"#; let explicit_default_digest = gpu_policy_measurement(explicit_default).unwrap(); - assert_ne!( - explicit_default_digest, - gpu_policy_measurement(empty).unwrap() - ); + assert_ne!(explicit_default_digest, empty_digest); let reordered = br#" { @@ -1809,11 +1812,11 @@ mod gpu { } impl Stage0<'_> { - /// Enforce `requirements.attest_gpu` (default true): attest an attached - /// NVIDIA GPU before continuing to key provisioning, or — when explicitly - /// disabled — set the GPU ready state without verification. The optional - /// Rego policy is always evaluated; when no attestation is performed, its - /// claims-array input is empty. + /// Enforce `requirements.gpu_policy.attest_gpu` (default true): attest an + /// attached NVIDIA GPU before continuing to key provisioning, or — when + /// explicitly disabled — set the GPU ready state without verification. The + /// optional Rego policy is always evaluated; when no attestation is + /// performed, its claims-array input is empty. async fn setup_gpu(&self) -> Result<()> { gpu::measure_gpu_policy(&self.shared.dir.app_compose_file())?; @@ -1826,7 +1829,7 @@ impl Stage0<'_> { .unwrap_or_default(); let inventory = gpu::gpu_inventory()?; - if !self.shared.app_compose.attest_gpu() { + if !gpu_policy.attest_gpu { gpu::evaluate_rego_policy(&gpu_policy, &[])?; if gpu_policy.rego.is_some() { info!("application GPU Rego policy accepted an empty claims array"); @@ -1834,7 +1837,9 @@ impl Stage0<'_> { if inventory.nvidia == 0 { return Ok(()); } - warn!("requirements.attest_gpu is false; setting GPU ready state without attestation"); + warn!( + "requirements.gpu_policy.attest_gpu is false; setting GPU ready state without attestation" + ); // Best-effort: a GPU with CC mode off has no ready state to set. if let Err(err) = gpu::set_gpu_ready_state(inventory.nvidia) { warn!("failed to set GPU ready state: {err:?}"); diff --git a/os/yocto/layers/meta-nvidia/recipes-graphics/nvattest/files/10-nvidia-gpu-ordering.conf b/os/yocto/layers/meta-nvidia/recipes-graphics/nvattest/files/10-nvidia-gpu-ordering.conf index 53923fd4e..83f2d1b5f 100644 --- a/os/yocto/layers/meta-nvidia/recipes-graphics/nvattest/files/10-nvidia-gpu-ordering.conf +++ b/os/yocto/layers/meta-nvidia/recipes-graphics/nvattest/files/10-nvidia-gpu-ordering.conf @@ -1,7 +1,8 @@ # Ordering only, not a hard dependency: on GPU instances, `dstack-util setup` # (run by dstack-prepare.service) performs local GPU TEE attestation per the -# app-compose `requirements.attest_gpu` setting, which needs the nvidia driver -# loaded (nvidia-persistenced) and the NVLink fabric up (nvidia-fabricmanager). +# app-compose `requirements.gpu_policy.attest_gpu` setting, which needs the +# nvidia driver loaded (nvidia-persistenced) and the NVLink fabric up +# (nvidia-fabricmanager). # Both units skip cleanly on GPU-less instances via nvidia-gpu-detect. [Unit] Wants=nvidia-persistenced.service nvidia-fabricmanager.service diff --git a/os/yocto/layers/meta-nvidia/recipes-graphics/nvidia/files/nvidia-persistenced.service b/os/yocto/layers/meta-nvidia/recipes-graphics/nvidia/files/nvidia-persistenced.service index 6c703bef0..74ddc332f 100644 --- a/os/yocto/layers/meta-nvidia/recipes-graphics/nvidia/files/nvidia-persistenced.service +++ b/os/yocto/layers/meta-nvidia/recipes-graphics/nvidia/files/nvidia-persistenced.service @@ -19,7 +19,8 @@ ExecStart=/usr/bin/env nvidia-persistenced --uvm-persistence-mode # NOTE: the GPU ready state (conf-compute -srs 1) is intentionally NOT set # here. It is set by `dstack-util setup` (dstack-prepare.service), only after # local GPU TEE attestation and any application GPU policy succeed -- unless -# the deployment opts out with app-compose requirements.attest_gpu=false. +# the deployment opts out with app-compose +# requirements.gpu_policy.attest_gpu=false. RemainAfterExit=yes [Install] From 23da63248d6b3e08b947e14905d26aca9f39ef80 Mon Sep 17 00:00:00 2001 From: Kevin Wang Date: Thu, 16 Jul 2026 18:31:11 -0700 Subject: [PATCH 18/32] fix(guest): derive default GPU policy from JSON --- dstack/dstack-types/src/lib.rs | 20 +++++++++++--------- dstack/dstack-util/src/system_setup.rs | 7 +++++-- 2 files changed, 16 insertions(+), 11 deletions(-) diff --git a/dstack/dstack-types/src/lib.rs b/dstack/dstack-types/src/lib.rs index 9a5b4c7f1..0098e2c6b 100644 --- a/dstack/dstack-types/src/lib.rs +++ b/dstack/dstack-types/src/lib.rs @@ -121,8 +121,12 @@ pub struct AppCompose { pub requirements: Option, } +/// Canonical source for the policy used when `requirements.gpu_policy` is +/// absent. Both typed defaults and measurement are derived from this JSON. +pub const DEFAULT_GPU_POLICY: &str = "{}"; + #[derive(Deserialize, Serialize, Debug, Clone, PartialEq, Eq)] -#[serde(default, deny_unknown_fields)] +#[serde(deny_unknown_fields)] pub struct GpuPolicy { /// Whether an attached GPU must pass local TEE attestation before the /// guest continues booting. Defaults to true. @@ -130,28 +134,26 @@ pub struct GpuPolicy { pub attest_gpu: bool, /// Optional Rego v0 policy evaluated against NVIDIA nvattest's `claims` /// array. It must define the boolean rule `data.policy.nv_match`. - #[serde(skip_serializing_if = "Option::is_none")] + #[serde(default, skip_serializing_if = "Option::is_none")] pub rego: Option, /// Permit NVIDIA DevTools mode. This defaults to false because DevTools /// disables the GPU memory-confidentiality guarantees expected in /// production. + #[serde(default)] pub allow_devtools: bool, /// Permit claims whose GPU attestation debug status is `enabled`. Defaults /// to false. + #[serde(default)] pub allow_debug: bool, /// Permit claims that do not assert GPU secure boot. Defaults to false. + #[serde(default)] pub allow_insecure_boot: bool, } impl Default for GpuPolicy { fn default() -> Self { - Self { - attest_gpu: true, - rego: None, - allow_devtools: false, - allow_debug: false, - allow_insecure_boot: false, - } + serde_json::from_str(DEFAULT_GPU_POLICY) + .expect("DEFAULT_GPU_POLICY must be a valid GPU policy") } } diff --git a/dstack/dstack-util/src/system_setup.rs b/dstack/dstack-util/src/system_setup.rs index d29ef1dee..82241e19b 100644 --- a/dstack/dstack-util/src/system_setup.rs +++ b/dstack/dstack-util/src/system_setup.rs @@ -20,7 +20,7 @@ use dstack_types::{ APP_COMPOSE, APP_KEYS, DECRYPTED_ENV, DECRYPTED_ENV_JSON, ENCRYPTED_ENV, HOST_SHARED_DIR_NAME, HOST_SHARED_DISK_LABEL, INSTANCE_INFO, SYS_CONFIG, USER_CONFIG, }, - GpuPolicy, KeyProvider, KeyProviderInfo, + GpuPolicy, KeyProvider, KeyProviderInfo, DEFAULT_GPU_POLICY, }; use fs_err as fs; use luks2::{ @@ -1419,7 +1419,8 @@ mod gpu { fn gpu_policy_measurement(compose_json: &[u8]) -> Result<[u8; 32]> { let compose: Value = serde_json::from_slice(compose_json).context("failed to parse raw app compose")?; - let default_policy = Value::Object(Default::default()); + let default_policy: Value = serde_json::from_str(DEFAULT_GPU_POLICY) + .context("failed to parse default GPU policy")?; let policy = compose .pointer("/requirements/gpu_policy") .unwrap_or(&default_policy); @@ -1830,6 +1831,8 @@ impl Stage0<'_> { let inventory = gpu::gpu_inventory()?; if !gpu_policy.attest_gpu { + // Attestation is explicitly disabled, so there are no claims. Rego + // still runs with an empty input before any GPU is made ready. gpu::evaluate_rego_policy(&gpu_policy, &[])?; if gpu_policy.rego.is_some() { info!("application GPU Rego policy accepted an empty claims array"); From 52b0da68695f5d0382b025f8506b53800652539f Mon Sep 17 00:00:00 2001 From: Kevin Wang Date: Thu, 16 Jul 2026 18:38:02 -0700 Subject: [PATCH 19/32] chore(guest): always log accepted GPU policy --- dstack/dstack-util/src/system_setup.rs | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/dstack/dstack-util/src/system_setup.rs b/dstack/dstack-util/src/system_setup.rs index 82241e19b..32b3bf31c 100644 --- a/dstack/dstack-util/src/system_setup.rs +++ b/dstack/dstack-util/src/system_setup.rs @@ -1867,9 +1867,7 @@ impl Stage0<'_> { .context("failed to apply basic GPU policy")?; gpu::evaluate_rego_policy(&gpu_policy, attestation.claims())?; - if !gpu_policy.is_default() { - info!("application GPU policy accepted the attestation claims and state"); - } + info!("application GPU policy accepted the attestation claims and state"); gpu_state.set_ready()?; let devtools = gpu_state.any_devtools(); let event = attestation.event(devtools)?; From bcfd935e7b8312f0b87084332e080f1eb3796f43 Mon Sep 17 00:00:00 2001 From: Kevin Wang Date: Thu, 16 Jul 2026 18:38:59 -0700 Subject: [PATCH 20/32] refactor(guest): rename GPU measurement gate --- dstack/dstack-util/src/system_setup.rs | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/dstack/dstack-util/src/system_setup.rs b/dstack/dstack-util/src/system_setup.rs index 32b3bf31c..0f01ce6e2 100644 --- a/dstack/dstack-util/src/system_setup.rs +++ b/dstack/dstack-util/src/system_setup.rs @@ -1818,7 +1818,7 @@ impl Stage0<'_> { /// explicitly disabled — set the GPU ready state without verification. The /// optional Rego policy is always evaluated; when no attestation is /// performed, its claims-array input is empty. - async fn setup_gpu(&self) -> Result<()> { + async fn measure_gpu(&self) -> Result<()> { gpu::measure_gpu_policy(&self.shared.dir.app_compose_file())?; let gpu_policy = self @@ -2538,7 +2538,7 @@ impl<'a> Stage0<'a> { emit_runtime_event("system-preparing", &[])?; emit_runtime_event("app-id", &instance_info.app_id)?; emit_runtime_event("compose-hash", &compose_hash)?; - self.setup_gpu() + self.measure_gpu() .await .context("failed to verify GPU TEE attestation")?; From 349d906e71ba80edd1b87cd61631eefdd40e7b1b Mon Sep 17 00:00:00 2001 From: Kevin Wang Date: Thu, 16 Jul 2026 18:49:50 -0700 Subject: [PATCH 21/32] feat(guest-agent): expose GPU attestation output --- dstack/dstack-types/src/lib.rs | 3 ++ dstack/dstack-util/src/system_setup.rs | 5 ++- dstack/guest-agent/rpc/proto/agent_rpc.proto | 8 +++++ dstack/guest-agent/src/rpc_service.rs | 38 +++++++++++++++++--- sdk/curl/api.md | 20 +++++++++++ 5 files changed, 66 insertions(+), 8 deletions(-) diff --git a/dstack/dstack-types/src/lib.rs b/dstack/dstack-types/src/lib.rs index 0098e2c6b..635eb7449 100644 --- a/dstack/dstack-types/src/lib.rs +++ b/dstack/dstack-types/src/lib.rs @@ -125,6 +125,9 @@ pub struct AppCompose { /// absent. Both typed defaults and measurement are derived from this JSON. pub const DEFAULT_GPU_POLICY: &str = "{}"; +/// Path containing the complete output of the NVIDIA GPU attestation command. +pub const GPU_ATTESTATION_OUTPUT: &str = "/run/nvidia-gpu-attestation/attestation.out"; + #[derive(Deserialize, Serialize, Debug, Clone, PartialEq, Eq)] #[serde(deny_unknown_fields)] pub struct GpuPolicy { diff --git a/dstack/dstack-util/src/system_setup.rs b/dstack/dstack-util/src/system_setup.rs index 0f01ce6e2..394d46f7c 100644 --- a/dstack/dstack-util/src/system_setup.rs +++ b/dstack/dstack-util/src/system_setup.rs @@ -20,7 +20,7 @@ use dstack_types::{ APP_COMPOSE, APP_KEYS, DECRYPTED_ENV, DECRYPTED_ENV_JSON, ENCRYPTED_ENV, HOST_SHARED_DIR_NAME, HOST_SHARED_DISK_LABEL, INSTANCE_INFO, SYS_CONFIG, USER_CONFIG, }, - GpuPolicy, KeyProvider, KeyProviderInfo, DEFAULT_GPU_POLICY, + GpuPolicy, KeyProvider, KeyProviderInfo, DEFAULT_GPU_POLICY, GPU_ATTESTATION_OUTPUT, }; use fs_err as fs; use luks2::{ @@ -1077,7 +1077,6 @@ mod gpu { use super::*; const NVATTEST: &str = "/usr/bin/nvattest"; - const ATTESTATION_OUTPUT: &str = "/run/nvidia-gpu-attestation/attestation.out"; const ATTESTATION_TIMEOUT: Duration = Duration::from_secs(300); const EVENT_VERSION: u32 = 2; const POLICY_ENTRYPOINT: &str = "data.policy.nv_match"; @@ -1466,7 +1465,7 @@ mod gpu { } fn save_attestation_output(stdout: &[u8]) -> Result<()> { - let output_path = Path::new(ATTESTATION_OUTPUT); + let output_path = Path::new(GPU_ATTESTATION_OUTPUT); if let Some(parent) = output_path.parent() { fs::create_dir_all(parent)?; } diff --git a/dstack/guest-agent/rpc/proto/agent_rpc.proto b/dstack/guest-agent/rpc/proto/agent_rpc.proto index 892cca70b..c14d81d80 100644 --- a/dstack/guest-agent/rpc/proto/agent_rpc.proto +++ b/dstack/guest-agent/rpc/proto/agent_rpc.proto @@ -53,6 +53,9 @@ service DstackGuest { // Get app info rpc Info(google.protobuf.Empty) returns (AppInfo) {} + // Get the complete NVIDIA GPU attestation output produced during boot. + rpc GetGpuAttestation(google.protobuf.Empty) returns (GetGpuAttestationResponse) {} + // Sign a payload rpc Sign(SignRequest) returns (SignResponse) {} @@ -187,6 +190,11 @@ message AttestResponse { bytes attestation = 1; } +message GetGpuAttestationResponse { + // Complete JSON output produced by nvattest. + string attestation = 1; +} + message GetQuoteResponse { // TDX quote (empty on non-TDX platforms such as AMD SEV-SNP) bytes quote = 1; diff --git a/dstack/guest-agent/src/rpc_service.rs b/dstack/guest-agent/src/rpc_service.rs index cb9680353..1cf0539dd 100644 --- a/dstack/guest-agent/src/rpc_service.rs +++ b/dstack/guest-agent/src/rpc_service.rs @@ -2,7 +2,10 @@ // // SPDX-License-Identifier: Apache-2.0 -use std::sync::{Arc, RwLock}; +use std::{ + path::Path, + sync::{Arc, RwLock}, +}; use anyhow::{Context, Result}; use base64::{engine::general_purpose::URL_SAFE_NO_PAD, Engine as _}; @@ -12,11 +15,11 @@ use dstack_guest_agent_rpc::{ tappd_server::{TappdRpc, TappdServer}, worker_server::{WorkerRpc, WorkerServer}, AppInfo, AttestResponse, DeriveK256KeyResponse, DeriveKeyArgs, GetAttestationForAppKeyRequest, - GetKeyArgs, GetKeyResponse, GetQuoteResponse, GetTlsKeyArgs, GetTlsKeyResponse, RawQuoteArgs, - SignRequest, SignResponse, TdxQuoteArgs, TdxQuoteResponse, VerifyRequest, VerifyResponse, - WorkerVersion, + GetGpuAttestationResponse, GetKeyArgs, GetKeyResponse, GetQuoteResponse, GetTlsKeyArgs, + GetTlsKeyResponse, RawQuoteArgs, SignRequest, SignResponse, TdxQuoteArgs, TdxQuoteResponse, + VerifyRequest, VerifyResponse, WorkerVersion, }; -use dstack_types::{AppKeys, SysConfig}; +use dstack_types::{AppKeys, SysConfig, GPU_ATTESTATION_OUTPUT}; use ed25519_dalek::ed25519::signature::hazmat::{PrehashSigner, PrehashVerifier}; use ed25519_dalek::{ Signer as Ed25519Signer, SigningKey as Ed25519SigningKey, Verifier as Ed25519Verifier, @@ -49,6 +52,15 @@ fn read_dmi_file(name: &str) -> String { .unwrap_or_default() } +fn read_gpu_attestation(path: &Path) -> Result { + fs::read_to_string(path).with_context(|| { + format!( + "Failed to read GPU attestation output at {}", + path.display() + ) + }) +} + #[derive(Clone)] pub struct AppState { inner: Arc, @@ -328,6 +340,12 @@ impl DstackGuestRpc for InternalRpcHandler { get_info(&self.state, false).await } + async fn get_gpu_attestation(self) -> Result { + Ok(GetGpuAttestationResponse { + attestation: read_gpu_attestation(Path::new(GPU_ATTESTATION_OUTPUT))?, + }) + } + async fn sign(self, request: SignRequest) -> Result { let algorithm = normalize_algorithm(&request.algorithm); // Use the base algorithm for key derivation (e.g. secp256k1_prehashed -> secp256k1) @@ -668,6 +686,16 @@ mod tests { use std::convert::TryFrom; use std::io::Write; + #[test] + fn reads_gpu_attestation_output_verbatim() { + let mut output = tempfile::NamedTempFile::new().unwrap(); + let attestation = r#"{"result_code":0,"claims":[]}"#; + output.write_all(attestation.as_bytes()).unwrap(); + output.flush().unwrap(); + + assert_eq!(read_gpu_attestation(output.path()).unwrap(), attestation); + } + fn extract_pubkey_from_report_data(report_data: &[u8], prefix: &str) -> Result> { let end = report_data .iter() diff --git a/sdk/curl/api.md b/sdk/curl/api.md index fc7c766c2..c6af8e9d9 100644 --- a/sdk/curl/api.md +++ b/sdk/curl/api.md @@ -285,6 +285,26 @@ curl --unix-socket /var/run/dstack.sock http://dstack/Attest?report_data=0000000 } ``` +### 8. Get GPU Attestation + +Returns the complete JSON output produced by NVIDIA `nvattest` during boot. +The endpoint returns an error when no GPU attestation output is available, for +example on a VM without an NVIDIA GPU or when GPU attestation was disabled. + +**Endpoint:** `/GetGpuAttestation` + +**Example:** +```bash +curl --unix-socket /var/run/dstack.sock http://dstack/GetGpuAttestation +``` + +**Response:** +```json +{ + "attestation": "{\"result_code\": 0, \"claims\": [...]}" +} +``` + ## Error Responses All endpoints may return the following HTTP status codes: From b7910c29f54f7a547217b4a844b5e508f538471b Mon Sep 17 00:00:00 2001 From: Kevin Wang Date: Thu, 16 Jul 2026 19:03:31 -0700 Subject: [PATCH 22/32] feat(sdk): expose GPU attestation output --- dstack/dstack-types/src/lib.rs | 2 +- sdk/go/README.md | 16 +++++++++++++++ sdk/go/dstack/client.go | 19 ++++++++++++++++++ sdk/go/dstack/client_test.go | 23 ++++++++++++++++++++++ sdk/js/README.md | 13 ++++++++++++ sdk/js/src/index.ts | 14 +++++++++++++ sdk/python/README.md | 14 +++++++++++++ sdk/python/src/dstack_sdk/__init__.py | 2 ++ sdk/python/src/dstack_sdk/dstack_client.py | 14 +++++++++++++ sdk/python/tests/test_client.py | 17 ++++++++++++++++ sdk/python/tests/test_typing.py | 4 ++++ sdk/rust/README.md | 13 ++++++++++++ sdk/rust/src/dstack_client.rs | 9 +++++++++ sdk/rust/types/src/dstack.rs | 9 +++++++++ 14 files changed, 168 insertions(+), 1 deletion(-) diff --git a/dstack/dstack-types/src/lib.rs b/dstack/dstack-types/src/lib.rs index 635eb7449..30a3fc023 100644 --- a/dstack/dstack-types/src/lib.rs +++ b/dstack/dstack-types/src/lib.rs @@ -156,7 +156,7 @@ pub struct GpuPolicy { impl Default for GpuPolicy { fn default() -> Self { serde_json::from_str(DEFAULT_GPU_POLICY) - .expect("DEFAULT_GPU_POLICY must be a valid GPU policy") + .or_panic("DEFAULT_GPU_POLICY must be a valid GPU policy") } } diff --git a/sdk/go/README.md b/sdk/go/README.md index e323322e0..1756aac4f 100644 --- a/sdk/go/README.md +++ b/sdk/go/README.md @@ -581,6 +581,22 @@ Generates a TDX attestation quote containing the provided report data. - Cryptographic proof of execution environment - Audit trail generation +##### `GetGpuAttestation(ctx context.Context) (*GetGpuAttestationResponse, error)` + +Returns the complete NVIDIA `nvattest` JSON output produced during boot. + +```go +gpu, err := client.GetGpuAttestation(ctx) +if err != nil { + log.Fatal(err) +} +fmt.Println(gpu.Attestation) +``` + +The call returns an error when no GPU attestation output is available. The raw +output is not trusted by itself; remote verifiers should compare its digest +with the measured `gpu-attestation` runtime event. + ##### `GetTlsKey(ctx context.Context, options TlsKeyOptions) (*GetTlsKeyResponse, error)` Generates a fresh, random TLS key pair with X.509 certificate for TLS/SSL connections. **Important**: This method generates different keys on each call - use `GetKey()` for deterministic keys. diff --git a/sdk/go/dstack/client.go b/sdk/go/dstack/client.go index 98e2dd730..e287ecf21 100644 --- a/sdk/go/dstack/client.go +++ b/sdk/go/dstack/client.go @@ -119,6 +119,11 @@ type AttestResponse struct { Attestation []byte } +// GetGpuAttestationResponse contains the complete NVIDIA GPU attestation JSON. +type GetGpuAttestationResponse struct { + Attestation string `json:"attestation"` +} + // Represents an event log entry in the TCB info type EventLog struct { IMR int `json:"imr"` @@ -579,6 +584,20 @@ func (c *DstackClient) Attest(ctx context.Context, reportData []byte) (*AttestRe return &AttestResponse{Attestation: attestation}, nil } +// GetGpuAttestation returns the complete NVIDIA GPU attestation JSON produced during boot. +func (c *DstackClient) GetGpuAttestation(ctx context.Context) (*GetGpuAttestationResponse, error) { + data, err := c.sendRPCRequest(ctx, "/GetGpuAttestation", map[string]interface{}{}) + if err != nil { + return nil, err + } + + var response GetGpuAttestationResponse + if err := json.Unmarshal(data, &response); err != nil { + return nil, err + } + return &response, nil +} + // Represents the response from a Version request. type VersionResponse struct { Version string `json:"version"` diff --git a/sdk/go/dstack/client_test.go b/sdk/go/dstack/client_test.go index e6072a73e..86dc0f89b 100644 --- a/sdk/go/dstack/client_test.go +++ b/sdk/go/dstack/client_test.go @@ -13,6 +13,8 @@ import ( "encoding/json" "encoding/pem" "fmt" + "net/http" + "net/http/httptest" "strings" "testing" @@ -116,6 +118,27 @@ func TestAttest(t *testing.T) { } } +func TestGetGpuAttestation(t *testing.T) { + const attestation = `{"result_code":0,"claims":[]}` + server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if r.URL.Path != "/GetGpuAttestation" { + t.Fatalf("unexpected path: %s", r.URL.Path) + } + w.Header().Set("Content-Type", "application/json") + _ = json.NewEncoder(w).Encode(map[string]string{"attestation": attestation}) + })) + defer server.Close() + + client := dstack.NewDstackClient(dstack.WithEndpoint(server.URL)) + response, err := client.GetGpuAttestation(context.Background()) + if err != nil { + t.Fatal(err) + } + if response.Attestation != attestation { + t.Fatalf("unexpected attestation: %s", response.Attestation) + } +} + func TestGetTlsKey(t *testing.T) { client := dstack.NewDstackClient() altNames := []string{"localhost"} diff --git a/sdk/js/README.md b/sdk/js/README.md index c1686e24b..7006bb260 100644 --- a/sdk/js/README.md +++ b/sdk/js/README.md @@ -97,6 +97,19 @@ Versioned dstack attestation that works across TDX / GCP / Nitro providers. Pref const { attestation } = await client.attest('app-state-snapshot') ``` +### `getGpuAttestation()` + +Returns the complete NVIDIA `nvattest` JSON output produced during boot. + +```typescript +const gpu = await client.getGpuAttestation() +console.log(gpu.attestation) +``` + +The call throws when no GPU attestation output is available. The raw output is +not trusted by itself; remote verifiers should compare its digest with the +measured `gpu-attestation` runtime event. + ### `info()` App identity and TCB metadata. diff --git a/sdk/js/src/index.ts b/sdk/js/src/index.ts index 6a6d57144..b10e1f02e 100644 --- a/sdk/js/src/index.ts +++ b/sdk/js/src/index.ts @@ -109,6 +109,12 @@ export interface AttestResponse { attestation: Hex } +export interface GetGpuAttestationResponse { + __name__: Readonly<'GetGpuAttestationResponse'> + + attestation: string +} + export interface VersionResponse { __name__: Readonly<'VersionResponse'> @@ -329,6 +335,14 @@ export class DstackClient { }) } + async getGpuAttestation(): Promise { + const result = await send_rpc_request<{ attestation: string }>(this.endpoint, '/GetGpuAttestation', '{}') + return Object.freeze({ + ...result, + __name__: 'GetGpuAttestationResponse', + }) + } + async info(): Promise> { const result = await send_rpc_request, 'tcb_info'> & { tcb_info: string }>(this.endpoint, '/Info', '{}') return Object.freeze({ diff --git a/sdk/python/README.md b/sdk/python/README.md index bb6896076..9f53e3ad6 100644 --- a/sdk/python/README.md +++ b/sdk/python/README.md @@ -101,6 +101,20 @@ print(result.attestation) # hex string print(result.decode_attestation()) # bytes ``` +### Get GPU Attestation + +`get_gpu_attestation()` returns the complete NVIDIA `nvattest` JSON output +produced during boot. + +```python +gpu = client.get_gpu_attestation() +print(gpu.attestation) +``` + +The call fails when no GPU attestation output is available. The raw output is +not trusted by itself; remote verifiers should compare its digest with the +measured `gpu-attestation` runtime event. + ### Get Instance Info ```python diff --git a/sdk/python/src/dstack_sdk/__init__.py b/sdk/python/src/dstack_sdk/__init__.py index f1c20404c..ea4d0f77e 100644 --- a/sdk/python/src/dstack_sdk/__init__.py +++ b/sdk/python/src/dstack_sdk/__init__.py @@ -7,6 +7,7 @@ from .dstack_client import AttestResponse from .dstack_client import DstackClient from .dstack_client import EventLog +from .dstack_client import GetGpuAttestationResponse from .dstack_client import GetKeyResponse from .dstack_client import GetQuoteResponse from .dstack_client import GetTlsKeyResponse @@ -36,6 +37,7 @@ "GetKeyResponse", "GetTlsKeyResponse", "AttestResponse", + "GetGpuAttestationResponse", "GetQuoteResponse", "InfoResponse", "TcbInfo", diff --git a/sdk/python/src/dstack_sdk/dstack_client.py b/sdk/python/src/dstack_sdk/dstack_client.py index 0f9897793..1c5ee000a 100644 --- a/sdk/python/src/dstack_sdk/dstack_client.py +++ b/sdk/python/src/dstack_sdk/dstack_client.py @@ -186,6 +186,10 @@ def decode_attestation(self) -> bytes: return bytes.fromhex(self.attestation) +class GetGpuAttestationResponse(BaseModel): + attestation: str + + class SignResponse(BaseModel): signature: str signature_chain: List[str] @@ -461,6 +465,11 @@ async def attest( result = await self._send_rpc_request("Attest", {"report_data": hex}) return AttestResponse(**result) + async def get_gpu_attestation(self) -> GetGpuAttestationResponse: + """Return the complete NVIDIA GPU attestation JSON produced at boot.""" + result = await self._send_rpc_request("GetGpuAttestation", {}) + return GetGpuAttestationResponse(**result) + async def info(self) -> InfoResponse[TcbInfo]: """Fetch service information including parsed TCB info.""" result = await self._send_rpc_request("Info", {}) @@ -604,6 +613,11 @@ def attest( """Request a versioned attestation for the provided report data.""" raise NotImplementedError + @call_async + def get_gpu_attestation(self) -> GetGpuAttestationResponse: + """Return the complete NVIDIA GPU attestation JSON produced at boot.""" + raise NotImplementedError + @call_async def info(self) -> InfoResponse[TcbInfo]: """Fetch service information including parsed TCB info.""" diff --git a/sdk/python/tests/test_client.py b/sdk/python/tests/test_client.py index 619ddb52d..87846ba11 100644 --- a/sdk/python/tests/test_client.py +++ b/sdk/python/tests/test_client.py @@ -13,6 +13,7 @@ from dstack_sdk import AsyncTappdClient from dstack_sdk import AttestResponse from dstack_sdk import DstackClient +from dstack_sdk import GetGpuAttestationResponse from dstack_sdk import GetKeyResponse from dstack_sdk import GetQuoteResponse from dstack_sdk import GetTlsKeyResponse @@ -120,6 +121,22 @@ async def test_async_client_attest(): assert len(result.attestation) > 0 +@pytest.mark.asyncio +async def test_async_client_get_gpu_attestation(monkeypatch): + attestation = '{"result_code":0,"claims":[]}' + + async def fake_send(self, method, payload): + assert method == "GetGpuAttestation" + assert payload == {} + return {"attestation": attestation} + + monkeypatch.setenv("DSTACK_SIMULATOR_ENDPOINT", "http://localhost:0") + monkeypatch.setattr(AsyncDstackClient, "_send_rpc_request", fake_send) + result = await AsyncDstackClient().get_gpu_attestation() + assert isinstance(result, GetGpuAttestationResponse) + assert result.attestation == attestation + + @pytest.mark.asyncio async def test_async_client_get_tls_key(): client = AsyncDstackClient() diff --git a/sdk/python/tests/test_typing.py b/sdk/python/tests/test_typing.py index 1e5c49467..adcd391a3 100644 --- a/sdk/python/tests/test_typing.py +++ b/sdk/python/tests/test_typing.py @@ -9,6 +9,7 @@ from dstack_sdk import AsyncDstackClient from dstack_sdk import DstackClient +from dstack_sdk import GetGpuAttestationResponse from dstack_sdk import GetKeyResponse from dstack_sdk import GetQuoteResponse from dstack_sdk import GetTlsKeyResponse @@ -52,6 +53,7 @@ def test_all_sync_method_types(): expected_types = { "get_key": GetKeyResponse, + "get_gpu_attestation": GetGpuAttestationResponse, "get_quote": GetQuoteResponse, "get_tls_key": GetTlsKeyResponse, "info": InfoResponse, @@ -93,6 +95,7 @@ def test_async_method_types(): expected_types = { "get_key": GetKeyResponse, + "get_gpu_attestation": GetGpuAttestationResponse, "get_quote": GetQuoteResponse, "get_tls_key": GetTlsKeyResponse, "info": InfoResponse, @@ -126,6 +129,7 @@ def test_method_signature_comparison(): methods_to_check = [ "get_key", + "get_gpu_attestation", "get_quote", "get_tls_key", "info", diff --git a/sdk/rust/README.md b/sdk/rust/README.md index 4c9a32625..c6f49bdf1 100644 --- a/sdk/rust/README.md +++ b/sdk/rust/README.md @@ -105,6 +105,19 @@ println!("{}", info.tcb_info); Generates a versioned attestation with a custom 64-byte payload. - `attestation`: Hex-encoded attestation +#### `get_gpu_attestation() -> GetGpuAttestationResponse` + +Returns the complete NVIDIA `nvattest` JSON output produced during boot. + +```rust +let gpu = client.get_gpu_attestation().await?; +println!("{}", gpu.attestation); +``` + +The call returns an error when no GPU attestation output is available. The raw +output is not trusted by itself; remote verifiers should compare its digest +with the measured `gpu-attestation` runtime event. + ### Generate TLS Certificates `get_tls_key()` creates fresh TLS certificates. Unlike `get_key()`, each call generates a new random key. diff --git a/sdk/rust/src/dstack_client.rs b/sdk/rust/src/dstack_client.rs index 537b58c99..143c3dc2e 100644 --- a/sdk/rust/src/dstack_client.rs +++ b/sdk/rust/src/dstack_client.rs @@ -166,6 +166,15 @@ impl DstackClient { Ok(response) } + /// Returns the complete NVIDIA GPU attestation JSON produced during boot. + pub async fn get_gpu_attestation(&self) -> Result { + let response = self + .send_rpc_request("/GetGpuAttestation", &json!({})) + .await?; + let response = serde_json::from_value::(response)?; + Ok(response) + } + pub async fn info(&self) -> Result { let response = self.send_rpc_request("/Info", &json!({})).await?; Ok(InfoResponse::validated_from_value(response)?) diff --git a/sdk/rust/types/src/dstack.rs b/sdk/rust/types/src/dstack.rs index aa6263fe0..8192bc3ff 100644 --- a/sdk/rust/types/src/dstack.rs +++ b/sdk/rust/types/src/dstack.rs @@ -127,6 +127,15 @@ pub struct AttestResponse { pub attestation: String, } +/// Response containing the complete NVIDIA GPU attestation output. +#[derive(Debug, Serialize, Deserialize)] +#[cfg_attr(feature = "borsh", derive(BorshSerialize, BorshDeserialize))] +#[cfg_attr(feature = "borsh_schema", derive(BorshSchema))] +pub struct GetGpuAttestationResponse { + /// Complete JSON output produced by nvattest during guest boot. + pub attestation: String, +} + impl AttestResponse { pub fn decode_attestation(&self) -> Result, FromHexError> { hex::decode(&self.attestation) From 23df84117d26a89bcc79d5a30cac8c9e0c672959 Mon Sep 17 00:00:00 2001 From: Kevin Wang Date: Thu, 16 Jul 2026 20:23:23 -0700 Subject: [PATCH 23/32] refactor: rename GPU attestation API to GpuInfo --- dstack/guest-agent/rpc/proto/agent_rpc.proto | 6 +++--- dstack/guest-agent/src/rpc_service.rs | 8 ++++---- sdk/curl/api.md | 9 +++++---- sdk/go/README.md | 7 ++++--- sdk/go/dstack/client.go | 12 ++++++------ sdk/go/dstack/client_test.go | 6 +++--- sdk/js/README.md | 7 ++++--- sdk/js/src/index.ts | 10 +++++----- sdk/python/README.md | 8 ++++---- sdk/python/src/dstack_sdk/__init__.py | 4 ++-- sdk/python/src/dstack_sdk/dstack_client.py | 14 +++++++------- sdk/python/tests/test_client.py | 10 +++++----- sdk/python/tests/test_typing.py | 8 ++++---- sdk/rust/README.md | 7 ++++--- sdk/rust/src/dstack_client.rs | 10 ++++------ sdk/rust/types/src/dstack.rs | 2 +- 16 files changed, 65 insertions(+), 63 deletions(-) diff --git a/dstack/guest-agent/rpc/proto/agent_rpc.proto b/dstack/guest-agent/rpc/proto/agent_rpc.proto index c14d81d80..47994bb1d 100644 --- a/dstack/guest-agent/rpc/proto/agent_rpc.proto +++ b/dstack/guest-agent/rpc/proto/agent_rpc.proto @@ -53,8 +53,8 @@ service DstackGuest { // Get app info rpc Info(google.protobuf.Empty) returns (AppInfo) {} - // Get the complete NVIDIA GPU attestation output produced during boot. - rpc GetGpuAttestation(google.protobuf.Empty) returns (GetGpuAttestationResponse) {} + // Get GPU information collected during boot. + rpc GpuInfo(google.protobuf.Empty) returns (GpuInfoResponse) {} // Sign a payload rpc Sign(SignRequest) returns (SignResponse) {} @@ -190,7 +190,7 @@ message AttestResponse { bytes attestation = 1; } -message GetGpuAttestationResponse { +message GpuInfoResponse { // Complete JSON output produced by nvattest. string attestation = 1; } diff --git a/dstack/guest-agent/src/rpc_service.rs b/dstack/guest-agent/src/rpc_service.rs index 1cf0539dd..08e470e05 100644 --- a/dstack/guest-agent/src/rpc_service.rs +++ b/dstack/guest-agent/src/rpc_service.rs @@ -15,8 +15,8 @@ use dstack_guest_agent_rpc::{ tappd_server::{TappdRpc, TappdServer}, worker_server::{WorkerRpc, WorkerServer}, AppInfo, AttestResponse, DeriveK256KeyResponse, DeriveKeyArgs, GetAttestationForAppKeyRequest, - GetGpuAttestationResponse, GetKeyArgs, GetKeyResponse, GetQuoteResponse, GetTlsKeyArgs, - GetTlsKeyResponse, RawQuoteArgs, SignRequest, SignResponse, TdxQuoteArgs, TdxQuoteResponse, + GetKeyArgs, GetKeyResponse, GetQuoteResponse, GetTlsKeyArgs, GetTlsKeyResponse, + GpuInfoResponse, RawQuoteArgs, SignRequest, SignResponse, TdxQuoteArgs, TdxQuoteResponse, VerifyRequest, VerifyResponse, WorkerVersion, }; use dstack_types::{AppKeys, SysConfig, GPU_ATTESTATION_OUTPUT}; @@ -340,8 +340,8 @@ impl DstackGuestRpc for InternalRpcHandler { get_info(&self.state, false).await } - async fn get_gpu_attestation(self) -> Result { - Ok(GetGpuAttestationResponse { + async fn gpu_info(self) -> Result { + Ok(GpuInfoResponse { attestation: read_gpu_attestation(Path::new(GPU_ATTESTATION_OUTPUT))?, }) } diff --git a/sdk/curl/api.md b/sdk/curl/api.md index c6af8e9d9..a2b2d7ea0 100644 --- a/sdk/curl/api.md +++ b/sdk/curl/api.md @@ -285,17 +285,18 @@ curl --unix-socket /var/run/dstack.sock http://dstack/Attest?report_data=0000000 } ``` -### 8. Get GPU Attestation +### 8. GPU Info -Returns the complete JSON output produced by NVIDIA `nvattest` during boot. +Returns GPU information collected during boot. Currently, this includes the +complete JSON output produced by NVIDIA `nvattest`. The endpoint returns an error when no GPU attestation output is available, for example on a VM without an NVIDIA GPU or when GPU attestation was disabled. -**Endpoint:** `/GetGpuAttestation` +**Endpoint:** `/GpuInfo` **Example:** ```bash -curl --unix-socket /var/run/dstack.sock http://dstack/GetGpuAttestation +curl --unix-socket /var/run/dstack.sock http://dstack/GpuInfo ``` **Response:** diff --git a/sdk/go/README.md b/sdk/go/README.md index 1756aac4f..5e628ed71 100644 --- a/sdk/go/README.md +++ b/sdk/go/README.md @@ -581,12 +581,13 @@ Generates a TDX attestation quote containing the provided report data. - Cryptographic proof of execution environment - Audit trail generation -##### `GetGpuAttestation(ctx context.Context) (*GetGpuAttestationResponse, error)` +##### `GpuInfo(ctx context.Context) (*GpuInfoResponse, error)` -Returns the complete NVIDIA `nvattest` JSON output produced during boot. +Returns GPU information collected during boot. Currently, this includes the +complete NVIDIA `nvattest` JSON output. ```go -gpu, err := client.GetGpuAttestation(ctx) +gpu, err := client.GpuInfo(ctx) if err != nil { log.Fatal(err) } diff --git a/sdk/go/dstack/client.go b/sdk/go/dstack/client.go index e287ecf21..4e8fa30e7 100644 --- a/sdk/go/dstack/client.go +++ b/sdk/go/dstack/client.go @@ -119,8 +119,8 @@ type AttestResponse struct { Attestation []byte } -// GetGpuAttestationResponse contains the complete NVIDIA GPU attestation JSON. -type GetGpuAttestationResponse struct { +// GpuInfoResponse contains GPU information collected during boot. +type GpuInfoResponse struct { Attestation string `json:"attestation"` } @@ -584,14 +584,14 @@ func (c *DstackClient) Attest(ctx context.Context, reportData []byte) (*AttestRe return &AttestResponse{Attestation: attestation}, nil } -// GetGpuAttestation returns the complete NVIDIA GPU attestation JSON produced during boot. -func (c *DstackClient) GetGpuAttestation(ctx context.Context) (*GetGpuAttestationResponse, error) { - data, err := c.sendRPCRequest(ctx, "/GetGpuAttestation", map[string]interface{}{}) +// GpuInfo returns GPU information collected during boot. +func (c *DstackClient) GpuInfo(ctx context.Context) (*GpuInfoResponse, error) { + data, err := c.sendRPCRequest(ctx, "/GpuInfo", map[string]interface{}{}) if err != nil { return nil, err } - var response GetGpuAttestationResponse + var response GpuInfoResponse if err := json.Unmarshal(data, &response); err != nil { return nil, err } diff --git a/sdk/go/dstack/client_test.go b/sdk/go/dstack/client_test.go index 86dc0f89b..13ee88c99 100644 --- a/sdk/go/dstack/client_test.go +++ b/sdk/go/dstack/client_test.go @@ -118,10 +118,10 @@ func TestAttest(t *testing.T) { } } -func TestGetGpuAttestation(t *testing.T) { +func TestGpuInfo(t *testing.T) { const attestation = `{"result_code":0,"claims":[]}` server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { - if r.URL.Path != "/GetGpuAttestation" { + if r.URL.Path != "/GpuInfo" { t.Fatalf("unexpected path: %s", r.URL.Path) } w.Header().Set("Content-Type", "application/json") @@ -130,7 +130,7 @@ func TestGetGpuAttestation(t *testing.T) { defer server.Close() client := dstack.NewDstackClient(dstack.WithEndpoint(server.URL)) - response, err := client.GetGpuAttestation(context.Background()) + response, err := client.GpuInfo(context.Background()) if err != nil { t.Fatal(err) } diff --git a/sdk/js/README.md b/sdk/js/README.md index 7006bb260..31dfcad42 100644 --- a/sdk/js/README.md +++ b/sdk/js/README.md @@ -97,12 +97,13 @@ Versioned dstack attestation that works across TDX / GCP / Nitro providers. Pref const { attestation } = await client.attest('app-state-snapshot') ``` -### `getGpuAttestation()` +### `gpuInfo()` -Returns the complete NVIDIA `nvattest` JSON output produced during boot. +Returns GPU information collected during boot. Currently, this includes the +complete NVIDIA `nvattest` JSON output. ```typescript -const gpu = await client.getGpuAttestation() +const gpu = await client.gpuInfo() console.log(gpu.attestation) ``` diff --git a/sdk/js/src/index.ts b/sdk/js/src/index.ts index b10e1f02e..8226cbc27 100644 --- a/sdk/js/src/index.ts +++ b/sdk/js/src/index.ts @@ -109,8 +109,8 @@ export interface AttestResponse { attestation: Hex } -export interface GetGpuAttestationResponse { - __name__: Readonly<'GetGpuAttestationResponse'> +export interface GpuInfoResponse { + __name__: Readonly<'GpuInfoResponse'> attestation: string } @@ -335,11 +335,11 @@ export class DstackClient { }) } - async getGpuAttestation(): Promise { - const result = await send_rpc_request<{ attestation: string }>(this.endpoint, '/GetGpuAttestation', '{}') + async gpuInfo(): Promise { + const result = await send_rpc_request<{ attestation: string }>(this.endpoint, '/GpuInfo', '{}') return Object.freeze({ ...result, - __name__: 'GetGpuAttestationResponse', + __name__: 'GpuInfoResponse', }) } diff --git a/sdk/python/README.md b/sdk/python/README.md index 9f53e3ad6..5c66f766e 100644 --- a/sdk/python/README.md +++ b/sdk/python/README.md @@ -101,13 +101,13 @@ print(result.attestation) # hex string print(result.decode_attestation()) # bytes ``` -### Get GPU Attestation +### GPU Info -`get_gpu_attestation()` returns the complete NVIDIA `nvattest` JSON output -produced during boot. +`gpu_info()` returns GPU information collected during boot. Currently, this +includes the complete NVIDIA `nvattest` JSON output. ```python -gpu = client.get_gpu_attestation() +gpu = client.gpu_info() print(gpu.attestation) ``` diff --git a/sdk/python/src/dstack_sdk/__init__.py b/sdk/python/src/dstack_sdk/__init__.py index ea4d0f77e..e93c6009f 100644 --- a/sdk/python/src/dstack_sdk/__init__.py +++ b/sdk/python/src/dstack_sdk/__init__.py @@ -7,10 +7,10 @@ from .dstack_client import AttestResponse from .dstack_client import DstackClient from .dstack_client import EventLog -from .dstack_client import GetGpuAttestationResponse from .dstack_client import GetKeyResponse from .dstack_client import GetQuoteResponse from .dstack_client import GetTlsKeyResponse +from .dstack_client import GpuInfoResponse from .dstack_client import InfoResponse from .dstack_client import SignResponse from .dstack_client import TappdClient @@ -37,7 +37,7 @@ "GetKeyResponse", "GetTlsKeyResponse", "AttestResponse", - "GetGpuAttestationResponse", + "GpuInfoResponse", "GetQuoteResponse", "InfoResponse", "TcbInfo", diff --git a/sdk/python/src/dstack_sdk/dstack_client.py b/sdk/python/src/dstack_sdk/dstack_client.py index 1c5ee000a..07cfd49bf 100644 --- a/sdk/python/src/dstack_sdk/dstack_client.py +++ b/sdk/python/src/dstack_sdk/dstack_client.py @@ -186,7 +186,7 @@ def decode_attestation(self) -> bytes: return bytes.fromhex(self.attestation) -class GetGpuAttestationResponse(BaseModel): +class GpuInfoResponse(BaseModel): attestation: str @@ -465,10 +465,10 @@ async def attest( result = await self._send_rpc_request("Attest", {"report_data": hex}) return AttestResponse(**result) - async def get_gpu_attestation(self) -> GetGpuAttestationResponse: - """Return the complete NVIDIA GPU attestation JSON produced at boot.""" - result = await self._send_rpc_request("GetGpuAttestation", {}) - return GetGpuAttestationResponse(**result) + async def gpu_info(self) -> GpuInfoResponse: + """Return GPU information collected during boot.""" + result = await self._send_rpc_request("GpuInfo", {}) + return GpuInfoResponse(**result) async def info(self) -> InfoResponse[TcbInfo]: """Fetch service information including parsed TCB info.""" @@ -614,8 +614,8 @@ def attest( raise NotImplementedError @call_async - def get_gpu_attestation(self) -> GetGpuAttestationResponse: - """Return the complete NVIDIA GPU attestation JSON produced at boot.""" + def gpu_info(self) -> GpuInfoResponse: + """Return GPU information collected during boot.""" raise NotImplementedError @call_async diff --git a/sdk/python/tests/test_client.py b/sdk/python/tests/test_client.py index 87846ba11..cec867e72 100644 --- a/sdk/python/tests/test_client.py +++ b/sdk/python/tests/test_client.py @@ -13,10 +13,10 @@ from dstack_sdk import AsyncTappdClient from dstack_sdk import AttestResponse from dstack_sdk import DstackClient -from dstack_sdk import GetGpuAttestationResponse from dstack_sdk import GetKeyResponse from dstack_sdk import GetQuoteResponse from dstack_sdk import GetTlsKeyResponse +from dstack_sdk import GpuInfoResponse from dstack_sdk import SignResponse from dstack_sdk import TappdClient from dstack_sdk import VerifyResponse @@ -122,18 +122,18 @@ async def test_async_client_attest(): @pytest.mark.asyncio -async def test_async_client_get_gpu_attestation(monkeypatch): +async def test_async_client_gpu_info(monkeypatch): attestation = '{"result_code":0,"claims":[]}' async def fake_send(self, method, payload): - assert method == "GetGpuAttestation" + assert method == "GpuInfo" assert payload == {} return {"attestation": attestation} monkeypatch.setenv("DSTACK_SIMULATOR_ENDPOINT", "http://localhost:0") monkeypatch.setattr(AsyncDstackClient, "_send_rpc_request", fake_send) - result = await AsyncDstackClient().get_gpu_attestation() - assert isinstance(result, GetGpuAttestationResponse) + result = await AsyncDstackClient().gpu_info() + assert isinstance(result, GpuInfoResponse) assert result.attestation == attestation diff --git a/sdk/python/tests/test_typing.py b/sdk/python/tests/test_typing.py index adcd391a3..4aa38e469 100644 --- a/sdk/python/tests/test_typing.py +++ b/sdk/python/tests/test_typing.py @@ -9,10 +9,10 @@ from dstack_sdk import AsyncDstackClient from dstack_sdk import DstackClient -from dstack_sdk import GetGpuAttestationResponse from dstack_sdk import GetKeyResponse from dstack_sdk import GetQuoteResponse from dstack_sdk import GetTlsKeyResponse +from dstack_sdk import GpuInfoResponse from dstack_sdk.dstack_client import InfoResponse # Use a test endpoint to avoid socket file not found errors @@ -53,7 +53,7 @@ def test_all_sync_method_types(): expected_types = { "get_key": GetKeyResponse, - "get_gpu_attestation": GetGpuAttestationResponse, + "gpu_info": GpuInfoResponse, "get_quote": GetQuoteResponse, "get_tls_key": GetTlsKeyResponse, "info": InfoResponse, @@ -95,7 +95,7 @@ def test_async_method_types(): expected_types = { "get_key": GetKeyResponse, - "get_gpu_attestation": GetGpuAttestationResponse, + "gpu_info": GpuInfoResponse, "get_quote": GetQuoteResponse, "get_tls_key": GetTlsKeyResponse, "info": InfoResponse, @@ -129,7 +129,7 @@ def test_method_signature_comparison(): methods_to_check = [ "get_key", - "get_gpu_attestation", + "gpu_info", "get_quote", "get_tls_key", "info", diff --git a/sdk/rust/README.md b/sdk/rust/README.md index c6f49bdf1..35d1d98e9 100644 --- a/sdk/rust/README.md +++ b/sdk/rust/README.md @@ -105,12 +105,13 @@ println!("{}", info.tcb_info); Generates a versioned attestation with a custom 64-byte payload. - `attestation`: Hex-encoded attestation -#### `get_gpu_attestation() -> GetGpuAttestationResponse` +#### `gpu_info() -> GpuInfoResponse` -Returns the complete NVIDIA `nvattest` JSON output produced during boot. +Returns GPU information collected during boot. Currently, this includes the +complete NVIDIA `nvattest` JSON output. ```rust -let gpu = client.get_gpu_attestation().await?; +let gpu = client.gpu_info().await?; println!("{}", gpu.attestation); ``` diff --git a/sdk/rust/src/dstack_client.rs b/sdk/rust/src/dstack_client.rs index 143c3dc2e..d33a04e62 100644 --- a/sdk/rust/src/dstack_client.rs +++ b/sdk/rust/src/dstack_client.rs @@ -166,12 +166,10 @@ impl DstackClient { Ok(response) } - /// Returns the complete NVIDIA GPU attestation JSON produced during boot. - pub async fn get_gpu_attestation(&self) -> Result { - let response = self - .send_rpc_request("/GetGpuAttestation", &json!({})) - .await?; - let response = serde_json::from_value::(response)?; + /// Returns GPU information collected during boot. + pub async fn gpu_info(&self) -> Result { + let response = self.send_rpc_request("/GpuInfo", &json!({})).await?; + let response = serde_json::from_value::(response)?; Ok(response) } diff --git a/sdk/rust/types/src/dstack.rs b/sdk/rust/types/src/dstack.rs index 8192bc3ff..d97b169b8 100644 --- a/sdk/rust/types/src/dstack.rs +++ b/sdk/rust/types/src/dstack.rs @@ -131,7 +131,7 @@ pub struct AttestResponse { #[derive(Debug, Serialize, Deserialize)] #[cfg_attr(feature = "borsh", derive(BorshSerialize, BorshDeserialize))] #[cfg_attr(feature = "borsh_schema", derive(BorshSchema))] -pub struct GetGpuAttestationResponse { +pub struct GpuInfoResponse { /// Complete JSON output produced by nvattest during guest boot. pub attestation: String, } From 3679efc4ffffd173d085aca5b195be1b8563784f Mon Sep 17 00:00:00 2001 From: Kevin Wang Date: Thu, 16 Jul 2026 21:06:55 -0700 Subject: [PATCH 24/32] feat: bind GPU policy hash in MrConfigV3 --- docs/security/security-model.md | 5 +- dstack/Cargo.lock | 1 - dstack/dstack-attest/src/v1.rs | 1 + dstack/dstack-attest/tests/sev_snp_verify.rs | 1 + dstack/dstack-mr/src/sev.rs | 15 +++ dstack/dstack-types/src/lib.rs | 15 +++ dstack/dstack-types/src/mr_config.rs | 33 +++++++ dstack/dstack-util/Cargo.toml | 1 - dstack/dstack-util/src/system_setup.rs | 57 +++++------ .../src/system_setup/config_id_verifier.rs | 98 +++++++++++++++++++ dstack/kms/src/main_service.rs | 1 + dstack/kms/src/main_service/amd_attest.rs | 1 + dstack/kms/src/onboard_service.rs | 1 + dstack/vmm/src/app.rs | 35 ++++++- dstack/vmm/src/app/mr_config.rs | 16 ++- dstack/vmm/src/one_shot.rs | 5 +- 16 files changed, 245 insertions(+), 41 deletions(-) diff --git a/docs/security/security-model.md b/docs/security/security-model.md index 627ce0cd2..1e7098601 100644 --- a/docs/security/security-model.md +++ b/docs/security/security-model.md @@ -65,7 +65,7 @@ An application may additionally set `requirements.gpu_policy` to an object with - `allow_debug` (boolean, default `false`): permit an attestation claim whose `dbgstat` is `enabled`. - `allow_insecure_boot` (boolean, default `false`): permit an attestation claim whose GPU `secboot` value is false. -After measuring `compose-hash`, dstack enters the GPU setup gate and JCS-canonicalizes the original `requirements.gpu_policy` JSON value, then measures its SHA-256 digest in a `gpu-policy-hash` event. When the field is absent—including when `requirements` itself is absent—both parsing and measurement use the default empty object `{}`. Thus an omitted policy and an explicit `{}` have the same digest, while any explicitly present field, including an explicit default value, changes the digest. The typed policy used for enforcement applies omitted-field defaults and rejects unknown fields. If GPU attestation is enabled and NVIDIA GPUs are present, dstack attests them and applies the basic settings and optional Rego policy before setting the GPU ready state. When no attestation claims are produced—because no GPU is attached or `gpu_policy.attest_gpu` is false—Rego is still evaluated with an empty array as `input` before any ready-state transition. This lets an application reject a launch whose attested GPU count is wrong. A false, undefined, malformed, or non-boolean Rego result stops boot before key provisioning. +After measuring `compose-hash`, dstack enters the GPU setup gate and JCS-canonicalizes the original `requirements.gpu_policy` JSON value, then measures its SHA-256 digest in a `gpu-policy-hash` event. When the field is absent—including when `requirements` itself is absent—both parsing and measurement use the default empty object `{}`. Thus an omitted policy and an explicit `{}` have the same digest, while any explicitly present field, including an explicit default value, changes the digest. MrConfigV3 GPU launches also carry this digest as the optional `gpu_policy_hash` field; non-GPU launches omit it for compatibility. The guest requires and verifies the field when it observes an attached GPU, and requires it to be absent otherwise. The MrConfigV3 document is bound by TDX `MR_CONFIG_ID` or SEV-SNP `HOST_DATA`, so the host cannot substitute a different GPU policy without changing the platform launch identity. The typed policy used for enforcement applies omitted-field defaults and rejects unknown fields. If GPU attestation is enabled and NVIDIA GPUs are present, dstack attests them and applies the basic settings and optional Rego policy before setting the GPU ready state. When no attestation claims are produced—because no GPU is attached or `gpu_policy.attest_gpu` is false—Rego is still evaluated with an empty array as `input` before any ready-state transition. This lets an application reject a launch whose attested GPU count is wrong. A false, undefined, malformed, or non-boolean Rego result stops boot before key provisioning. For every NVML-enumerated GPU, dstack calls `Device::is_cc_enabled()` and `Device::is_cc_dev_mode_enabled()` and requires the NVML device count to match the expected GPU count. CC must always be ON; DevTools must be OFF unless the measured policy explicitly permits it. The typed claim checks always require `measres == "success"`; by default they also require `dbgstat == "disabled"` and `secboot == true`, with the latter two checks controlled by their explicit opt-ins. Only after the default appraisal, typed claim checks, optional Rego policy, and per-device NVML checks succeed does dstack call `Device::set_confidential_compute_state(true)` to set the GPU ready state. The dstack CPU/guest boot chain is verified independently through measured boot; a GPU claim named `secboot` refers to the GPU appraisal, not UEFI Secure Boot in the CVM. @@ -73,7 +73,7 @@ For every NVML-enumerated GPU, dstack calls `Device::is_cc_enabled()` and `Devic GPU workloads require verification of both hardware components. The CPU TEE quote verifies the CVM and its measured guest code. NVIDIA-signed evidence, checked against NVIDIA RIMs and certificate status by `nvattest`, verifies the GPU appraisal. After the optional policy and ready-state operations succeed, dstack emits a `gpu-attestation` launch event before `system-ready`. Its versioned payload records the number of appraised devices, asserted CC/DevTools state, and SHA-256 of the complete nvattest JSON output (claims and detached EAT). On TDX, both `gpu-policy-hash` and `gpu-attestation` are append-only RTMR3 events; they are never derived from application-controlled `report_data`. -A verifier must replay the measured event log, require exactly one `gpu-policy-hash` event immediately after `compose-hash`, and compare its 32-byte payload with the expected policy digest (`SHA-256(JCS({}))` for the omitted/default policy). When GPU protection is required, it must also require exactly one pre-`system-ready` `gpu-attestation` event with `devices > 0` and, when applicable, the expected deployment count. The raw `attestation.out` file is not trusted by itself; if it is supplied for inspection, its digest must match the `gpu-attestation` event. +A verifier must replay the measured event log, require exactly one `gpu-policy-hash` event immediately after `compose-hash`, and compare its 32-byte payload with the expected policy digest (`SHA-256(JCS({}))` for the omitted/default policy). When a GPU launch uses MrConfigV3, its `gpu_policy_hash` must be present and match the same digest. When GPU protection is required, the verifier must also require exactly one pre-`system-ready` `gpu-attestation` event with `devices > 0` and, when applicable, the expected deployment count. The raw `attestation.out` file is not trusted by itself; if it is supplied for inspection, its digest must match the `gpu-attestation` event. ### GPU Threat Model and Lifetime @@ -187,6 +187,7 @@ Use this checklist to verify a workload running in a dstack CVM. **GPU verification (when required):** - [ ] The `gpu-attestation` device count is greater than zero and matches the expected deployment - [ ] Exactly one `gpu-policy-hash` event follows `compose-hash`, and its payload matches `SHA-256(JCS(requirements.gpu_policy))` (default `{}` when omitted) +- [ ] When a GPU launch uses MrConfigV3, its `gpu_policy_hash` is present and matches the same expected GPU policy digest - [ ] Exactly one `gpu-attestation` event appears before `system-ready` - [ ] CC is ON and the event's DevTools field complies with the measured policy - [ ] The platform binds the event log to a quoted RTMR/PCR (do not accept it from current SEV-SNP evidence) diff --git a/dstack/Cargo.lock b/dstack/Cargo.lock index 552652f6a..b7e63cffa 100644 --- a/dstack/Cargo.lock +++ b/dstack/Cargo.lock @@ -2070,7 +2070,6 @@ dependencies = [ "semver", "serde", "serde-human-bytes", - "serde_jcs", "serde_json", "sha2 0.10.9", "sha3", diff --git a/dstack/dstack-attest/src/v1.rs b/dstack/dstack-attest/src/v1.rs index 7df502218..f201b9c73 100644 --- a/dstack/dstack-attest/src/v1.rs +++ b/dstack/dstack-attest/src/v1.rs @@ -377,6 +377,7 @@ mod tests { MrConfigV3::new( vec![0x11; 20], vec![0x22; 32], + None, dstack_types::KeyProviderKind::None, Vec::new(), vec![0x33; 20], diff --git a/dstack/dstack-attest/tests/sev_snp_verify.rs b/dstack/dstack-attest/tests/sev_snp_verify.rs index 8fa4c3317..6fded651f 100644 --- a/dstack/dstack-attest/tests/sev_snp_verify.rs +++ b/dstack/dstack-attest/tests/sev_snp_verify.rs @@ -352,6 +352,7 @@ fn substituted_mr_config_breaks_host_data_binding() { let evil = MrConfigV3::new( vec![0xab; 20], vec![0xcd; 32], + None, KeyProviderKind::None, Vec::new(), vec![0xef; 20], diff --git a/dstack/dstack-mr/src/sev.rs b/dstack/dstack-mr/src/sev.rs index 157d083bf..34dd2ab53 100644 --- a/dstack/dstack-mr/src/sev.rs +++ b/dstack/dstack-mr/src/sev.rs @@ -913,6 +913,9 @@ pub fn validate_mr_config(mr_config: &MrConfigV3) -> Result<()> { } ensure_len("mr_config.app_id", &mr_config.app_id, 20)?; ensure_len("mr_config.compose_hash", &mr_config.compose_hash, 32)?; + if let Some(gpu_policy_hash) = &mr_config.gpu_policy_hash { + ensure_len("mr_config.gpu_policy_hash", gpu_policy_hash, 32)?; + } if !mr_config.instance_id.is_empty() { ensure_len("mr_config.instance_id", &mr_config.instance_id, 20)?; } @@ -1476,6 +1479,7 @@ mod tests { MrConfigV3::new( vec![0x11; 20], vec![0x22; 32], + None, dstack_types::KeyProviderKind::None, Vec::new(), vec![0x33; 20], @@ -1616,6 +1620,7 @@ mod tests { MrConfigV3::new( vec![0xee; 20], vec![0x22; 32], + None, dstack_types::KeyProviderKind::None, Vec::new(), vec![0x33; 20], @@ -1623,6 +1628,7 @@ mod tests { MrConfigV3::new( vec![0x11; 20], vec![0xee; 32], + None, dstack_types::KeyProviderKind::None, Vec::new(), vec![0x33; 20], @@ -1630,10 +1636,19 @@ mod tests { MrConfigV3::new( vec![0x11; 20], vec![0x22; 32], + None, dstack_types::KeyProviderKind::None, Vec::new(), vec![0xee; 20], ), + MrConfigV3::new( + vec![0x11; 20], + vec![0x22; 32], + Some(vec![0xee; 32]), + dstack_types::KeyProviderKind::None, + Vec::new(), + vec![0x33; 20], + ), ]; for evil in evil_mr_configs { let vm_config = synthetic_vm_config(&input, &evil); diff --git a/dstack/dstack-types/src/lib.rs b/dstack/dstack-types/src/lib.rs index 30a3fc023..87893679a 100644 --- a/dstack/dstack-types/src/lib.rs +++ b/dstack/dstack-types/src/lib.rs @@ -128,6 +128,21 @@ pub const DEFAULT_GPU_POLICY: &str = "{}"; /// Path containing the complete output of the NVIDIA GPU attestation command. pub const GPU_ATTESTATION_OUTPUT: &str = "/run/nvidia-gpu-attestation/attestation.out"; +/// Computes the SHA-256 digest of the JCS-canonicalized raw +/// `requirements.gpu_policy` JSON value. An absent policy is equivalent to +/// the default empty object. +pub fn gpu_policy_hash(compose_json: &[u8]) -> Result<[u8; 32], serde_json::Error> { + use sha2::{Digest, Sha256}; + + let compose: serde_json::Value = serde_json::from_slice(compose_json)?; + let default_policy: serde_json::Value = serde_json::from_str(DEFAULT_GPU_POLICY)?; + let policy = compose + .pointer("/requirements/gpu_policy") + .unwrap_or(&default_policy); + let canonical = serde_jcs::to_vec(policy)?; + Ok(Sha256::digest(canonical).into()) +} + #[derive(Deserialize, Serialize, Debug, Clone, PartialEq, Eq)] #[serde(deny_unknown_fields)] pub struct GpuPolicy { diff --git a/dstack/dstack-types/src/mr_config.rs b/dstack/dstack-types/src/mr_config.rs index f5f8d09de..846f4f746 100644 --- a/dstack/dstack-types/src/mr_config.rs +++ b/dstack/dstack-types/src/mr_config.rs @@ -109,6 +109,10 @@ pub struct MrConfigV3 { pub app_id: Vec, #[serde(with = "hex_bytes")] pub compose_hash: Vec, + /// Hash of the raw application GPU policy. GPU launches populate it; + /// non-GPU and historical v3 launch documents omit it. + #[serde(default, skip_serializing_if = "Option::is_none", with = "hex_bytes")] + pub gpu_policy_hash: Option>, pub key_provider: KeyProviderKind, #[serde(default, with = "hex_bytes")] pub key_provider_id: Vec, @@ -120,6 +124,7 @@ impl MrConfigV3 { pub fn new( app_id: Vec, compose_hash: Vec, + gpu_policy_hash: Option>, key_provider: KeyProviderKind, key_provider_id: Vec, instance_id: Vec, @@ -128,6 +133,7 @@ impl MrConfigV3 { version: mr_config_v3_version(), app_id, compose_hash, + gpu_policy_hash, key_provider, key_provider_id, instance_id, @@ -191,6 +197,7 @@ mod tests { let config = MrConfigV3::new( vec![0x11; 20], vec![0x22; 32], + Some(vec![0x55; 32]), KeyProviderKind::Kms, vec![0x33; 32], vec![0x44; 20], @@ -202,6 +209,30 @@ mod tests { assert_eq!(config.to_snp_host_data().len(), 32); assert_ne!(config.to_tdx_mr_config_id(), changed.to_tdx_mr_config_id()); assert_eq!(config.to_tdx_mr_config_id()[0], 3); + + let mut changed = config.clone(); + if let Some(gpu_policy_hash) = &mut changed.gpu_policy_hash { + gpu_policy_hash[0] ^= 0xff; + } + assert_ne!(config.to_snp_host_data(), changed.to_snp_host_data()); + assert_ne!(config.to_tdx_mr_config_id(), changed.to_tdx_mr_config_id()); + } + + #[test] + fn mr_config_v3_omits_gpu_policy_hash_for_non_gpu_launches() -> Result<(), Box> { + let config = MrConfigV3::new( + vec![0x11; 20], + vec![0x22; 32], + None, + KeyProviderKind::None, + Vec::new(), + vec![0x44; 20], + ); + let document = config.to_canonical_json(); + + assert!(!document.contains("gpu_policy_hash")); + assert_eq!(MrConfigV3::from_document(&document)?, config); + Ok(()) } #[test] @@ -209,6 +240,7 @@ mod tests { let config = MrConfigV3::new( vec![0x11; 20], vec![0x22; 32], + Some(vec![0x55; 32]), KeyProviderKind::Kms, vec![0x33; 32], vec![0x44; 20], @@ -220,6 +252,7 @@ mod tests { concat!( "{\"app_id\":\"1111111111111111111111111111111111111111\",", "\"compose_hash\":\"2222222222222222222222222222222222222222222222222222222222222222\",", + "\"gpu_policy_hash\":\"5555555555555555555555555555555555555555555555555555555555555555\",", "\"instance_id\":\"4444444444444444444444444444444444444444\",", "\"key_provider\":\"kms\",", "\"key_provider_id\":\"3333333333333333333333333333333333333333333333333333333333333333\",", diff --git a/dstack/dstack-util/Cargo.toml b/dstack/dstack-util/Cargo.toml index 09bce8c4a..8f63daaf9 100644 --- a/dstack/dstack-util/Cargo.toml +++ b/dstack/dstack-util/Cargo.toml @@ -25,7 +25,6 @@ serde.workspace = true serde-human-bytes.workspace = true semver.workspace = true serde_json.workspace = true -serde_jcs.workspace = true sha2.workspace = true tokio = { workspace = true, features = ["full"] } tracing.workspace = true diff --git a/dstack/dstack-util/src/system_setup.rs b/dstack/dstack-util/src/system_setup.rs index 394d46f7c..fdab3fb36 100644 --- a/dstack/dstack-util/src/system_setup.rs +++ b/dstack/dstack-util/src/system_setup.rs @@ -16,11 +16,12 @@ use anyhow::{anyhow, bail, Context, Result}; use dstack_attest::emit_runtime_event; use dstack_kms_rpc as rpc; use dstack_types::{ + gpu_policy_hash, shared_filenames::{ APP_COMPOSE, APP_KEYS, DECRYPTED_ENV, DECRYPTED_ENV_JSON, ENCRYPTED_ENV, HOST_SHARED_DIR_NAME, HOST_SHARED_DISK_LABEL, INSTANCE_INFO, SYS_CONFIG, USER_CONFIG, }, - GpuPolicy, KeyProvider, KeyProviderInfo, DEFAULT_GPU_POLICY, GPU_ATTESTATION_OUTPUT, + GpuPolicy, KeyProvider, KeyProviderInfo, GPU_ATTESTATION_OUTPUT, }; use fs_err as fs; use luks2::{ @@ -1415,25 +1416,13 @@ mod gpu { }) } - fn gpu_policy_measurement(compose_json: &[u8]) -> Result<[u8; 32]> { - let compose: Value = - serde_json::from_slice(compose_json).context("failed to parse raw app compose")?; - let default_policy: Value = serde_json::from_str(DEFAULT_GPU_POLICY) - .context("failed to parse default GPU policy")?; - let policy = compose - .pointer("/requirements/gpu_policy") - .unwrap_or(&default_policy); - let canonical = - serde_jcs::to_vec(policy).context("failed to canonicalize raw GPU policy")?; - Ok(sha256(&canonical)) - } - - pub(super) fn measure_gpu_policy(compose_path: &Path) -> Result<()> { + pub(super) fn measure_gpu_policy(compose_path: &Path) -> Result<[u8; 32]> { let compose_json = fs::read(compose_path) .with_context(|| format!("failed to read {}", compose_path.display()))?; - let digest = gpu_policy_measurement(&compose_json)?; + let digest = gpu_policy_hash(&compose_json).context("failed to hash raw GPU policy")?; emit_runtime_event("gpu-policy-hash", &digest) - .context("failed to emit GPU policy measurement") + .context("failed to emit GPU policy measurement")?; + Ok(digest) } pub(super) fn evaluate_rego_policy(policy: &GpuPolicy, claims: &[Value]) -> Result<()> { @@ -1778,17 +1767,14 @@ mod gpu { let no_requirements = br#"{}"#; let absent = br#"{"requirements": {}}"#; let empty_digest = sha256(b"{}"); - assert_eq!( - gpu_policy_measurement(no_requirements).unwrap(), - empty_digest - ); - assert_eq!(gpu_policy_measurement(absent).unwrap(), empty_digest); + assert_eq!(gpu_policy_hash(no_requirements).unwrap(), empty_digest); + assert_eq!(gpu_policy_hash(absent).unwrap(), empty_digest); let empty = br#"{"requirements": {"gpu_policy": {}}}"#; - assert_eq!(gpu_policy_measurement(empty).unwrap(), empty_digest); + assert_eq!(gpu_policy_hash(empty).unwrap(), empty_digest); let explicit_default = br#"{"requirements":{"gpu_policy":{"attest_gpu":true}}}"#; - let explicit_default_digest = gpu_policy_measurement(explicit_default).unwrap(); + let explicit_default_digest = gpu_policy_hash(explicit_default).unwrap(); assert_ne!(explicit_default_digest, empty_digest); let reordered = br#" @@ -1804,8 +1790,8 @@ mod gpu { let canonical_order = br#"{"requirements":{"gpu_policy":{"allow_debug":false,"rego":"package policy"}}}"#; assert_eq!( - gpu_policy_measurement(reordered).unwrap(), - gpu_policy_measurement(canonical_order).unwrap() + gpu_policy_hash(reordered).unwrap(), + gpu_policy_hash(canonical_order).unwrap() ); } } @@ -1817,8 +1803,8 @@ impl Stage0<'_> { /// explicitly disabled — set the GPU ready state without verification. The /// optional Rego policy is always evaluated; when no attestation is /// performed, its claims-array input is empty. - async fn measure_gpu(&self) -> Result<()> { - gpu::measure_gpu_policy(&self.shared.dir.app_compose_file())?; + async fn measure_gpu(&self) -> Result> { + let gpu_policy_hash = gpu::measure_gpu_policy(&self.shared.dir.app_compose_file())?; let gpu_policy = self .shared @@ -1829,6 +1815,7 @@ impl Stage0<'_> { .unwrap_or_default(); let inventory = gpu::gpu_inventory()?; + let gpu_policy_hash = (inventory.total > 0).then_some(gpu_policy_hash); if !gpu_policy.attest_gpu { // Attestation is explicitly disabled, so there are no claims. Rego // still runs with an empty input before any GPU is made ready. @@ -1837,7 +1824,7 @@ impl Stage0<'_> { info!("application GPU Rego policy accepted an empty claims array"); } if inventory.nvidia == 0 { - return Ok(()); + return Ok(gpu_policy_hash); } warn!( "requirements.gpu_policy.attest_gpu is false; setting GPU ready state without attestation" @@ -1846,7 +1833,7 @@ impl Stage0<'_> { if let Err(err) = gpu::set_gpu_ready_state(inventory.nvidia) { warn!("failed to set GPU ready state: {err:?}"); } - return Ok(()); + return Ok(gpu_policy_hash); } let expected_devices = gpu::nvidia_gpu_count(inventory)?; if expected_devices == 0 { @@ -1854,7 +1841,7 @@ impl Stage0<'_> { if gpu_policy.rego.is_some() { info!("application GPU Rego policy accepted an empty claims array"); } - return Ok(()); + return Ok(gpu_policy_hash); } self.vmm.notify_q("boot.progress", "attesting GPU").await; info!("verifying GPU TEE attestation"); @@ -1873,7 +1860,7 @@ impl Stage0<'_> { emit_runtime_event("gpu-attestation", &event) .context("failed to emit GPU attestation event")?; info!("GPU TEE attestation succeeded"); - Ok(()) + Ok(gpu_policy_hash) } } @@ -1922,6 +1909,7 @@ impl AppIdValidator { struct AppInfo { instance_info: InstanceInfo, compose_hash: [u8; 32], + gpu_policy_hash: Option<[u8; 32]>, } struct Stage0<'a> { @@ -2537,7 +2525,8 @@ impl<'a> Stage0<'a> { emit_runtime_event("system-preparing", &[])?; emit_runtime_event("app-id", &instance_info.app_id)?; emit_runtime_event("compose-hash", &compose_hash)?; - self.measure_gpu() + let gpu_policy_hash = self + .measure_gpu() .await .context("failed to verify GPU TEE attestation")?; @@ -2568,12 +2557,14 @@ impl<'a> Stage0<'a> { Ok(AppInfo { instance_info, compose_hash, + gpu_policy_hash, }) } fn verify_app(&self, app_info: &AppInfo, keys: &AppKeys) -> Result<()> { config_id_verifier::verify_mr_config_id( &app_info.compose_hash, + app_info.gpu_policy_hash.as_ref(), &app_info .instance_info .app_id diff --git a/dstack/dstack-util/src/system_setup/config_id_verifier.rs b/dstack/dstack-util/src/system_setup/config_id_verifier.rs index c350b32e6..90a981f51 100644 --- a/dstack/dstack-util/src/system_setup/config_id_verifier.rs +++ b/dstack/dstack-util/src/system_setup/config_id_verifier.rs @@ -14,6 +14,7 @@ use tracing::info; #[derive(Clone, Copy)] struct ExpectedMrConfig<'a> { compose_hash: &'a [u8; 32], + gpu_policy_hash: Option<&'a [u8; 32]>, app_id: &'a [u8; 20], instance_id: &'a [u8], key_provider: KeyProviderKind, @@ -65,6 +66,7 @@ fn read_snp_host_data() -> Result<[u8; 32]> { /// - key_provider_id: [u8] // KMS CA pubkey, local-sgx MR, or empty for none/tpm pub fn verify_mr_config_id( compose_hash: &[u8; 32], + gpu_policy_hash: Option<&[u8; 32]>, app_id: &[u8; 20], instance_id: &[u8], key_provider: KeyProviderKind, @@ -73,6 +75,7 @@ pub fn verify_mr_config_id( let mode = AttestationMode::detect().context("Failed to detect attestation mode")?; let expected = ExpectedMrConfig { compose_hash, + gpu_policy_hash, app_id, instance_id, key_provider, @@ -163,6 +166,10 @@ fn verify_mr_config_v3_document( if mr_config.compose_hash.as_slice() != expected.compose_hash { bail!("Invalid mr_config compose_hash"); } + if mr_config.gpu_policy_hash.as_deref() != expected.gpu_policy_hash.map(|hash| hash.as_slice()) + { + bail!("Invalid mr_config gpu_policy_hash"); + } if mr_config.app_id.as_slice() != expected.app_id { bail!("Invalid mr_config app_id"); } @@ -194,12 +201,14 @@ mod tests { #[test] fn tdx_mr_config_id_v3_accepts_document_value() -> Result<()> { let compose_hash = [0x22u8; 32]; + let gpu_policy_hash = [0x55u8; 32]; let app_id = [0x11u8; 20]; let instance_id = [0x44u8; 20]; let key_provider_id = [0x33u8; 32]; let mr_config = MrConfigV3::new( app_id.to_vec(), compose_hash.to_vec(), + Some(gpu_policy_hash.to_vec()), KeyProviderKind::Kms, key_provider_id.to_vec(), instance_id.to_vec(), @@ -207,6 +216,7 @@ mod tests { let document = mr_config.to_canonical_json(); let expected = ExpectedMrConfig { compose_hash: &compose_hash, + gpu_policy_hash: Some(&gpu_policy_hash), app_id: &app_id, instance_id: &instance_id, key_provider: KeyProviderKind::Kms, @@ -219,12 +229,14 @@ mod tests { #[test] fn mr_config_v3_document_must_match_expected_app_info() { let compose_hash = [0x22u8; 32]; + let gpu_policy_hash = [0x55u8; 32]; let app_id = [0x11u8; 20]; let instance_id = [0x44u8; 20]; let key_provider_id = [0x33u8; 32]; let document = MrConfigV3::new( app_id.to_vec(), compose_hash.to_vec(), + Some(gpu_policy_hash.to_vec()), KeyProviderKind::Kms, key_provider_id.to_vec(), instance_id.to_vec(), @@ -233,6 +245,7 @@ mod tests { let wrong_app_id = [0x12u8; 20]; let expected = ExpectedMrConfig { compose_hash: &compose_hash, + gpu_policy_hash: Some(&gpu_policy_hash), app_id: &wrong_app_id, instance_id: &instance_id, key_provider: KeyProviderKind::Kms, @@ -244,4 +257,89 @@ mod tests { Err(err) => assert!(err.to_string().contains("Invalid mr_config app_id")), } } + + #[test] + fn mr_config_v3_document_must_match_expected_gpu_policy_hash() { + let compose_hash = [0x22u8; 32]; + let gpu_policy_hash = [0x55u8; 32]; + let app_id = [0x11u8; 20]; + let instance_id = [0x44u8; 20]; + let key_provider_id = [0x33u8; 32]; + let document = MrConfigV3::new( + app_id.to_vec(), + compose_hash.to_vec(), + Some(gpu_policy_hash.to_vec()), + KeyProviderKind::Kms, + key_provider_id.to_vec(), + instance_id.to_vec(), + ) + .to_canonical_json(); + let wrong_gpu_policy_hash = [0x56u8; 32]; + let expected = ExpectedMrConfig { + compose_hash: &compose_hash, + gpu_policy_hash: Some(&wrong_gpu_policy_hash), + app_id: &app_id, + instance_id: &instance_id, + key_provider: KeyProviderKind::Kms, + key_provider_id: &key_provider_id, + }; + + match verify_mr_config_v3_document(&document, expected) { + Ok(_) => panic!("mismatched gpu_policy_hash must reject"), + Err(err) => assert!(err + .to_string() + .contains("Invalid mr_config gpu_policy_hash")), + } + + let document_without_gpu_policy_hash = MrConfigV3::new( + app_id.to_vec(), + compose_hash.to_vec(), + None, + KeyProviderKind::Kms, + key_provider_id.to_vec(), + instance_id.to_vec(), + ) + .to_canonical_json(); + let expected = ExpectedMrConfig { + compose_hash: &compose_hash, + gpu_policy_hash: Some(&gpu_policy_hash), + app_id: &app_id, + instance_id: &instance_id, + key_provider: KeyProviderKind::Kms, + key_provider_id: &key_provider_id, + }; + match verify_mr_config_v3_document(&document_without_gpu_policy_hash, expected) { + Ok(_) => panic!("missing gpu_policy_hash on a GPU launch must reject"), + Err(err) => assert!(err + .to_string() + .contains("Invalid mr_config gpu_policy_hash")), + } + } + + #[test] + fn mr_config_v3_allows_missing_gpu_policy_hash_for_non_gpu_launch() -> Result<()> { + let compose_hash = [0x22u8; 32]; + let app_id = [0x11u8; 20]; + let instance_id = [0x44u8; 20]; + let key_provider_id = [0x33u8; 32]; + let mr_config = MrConfigV3::new( + app_id.to_vec(), + compose_hash.to_vec(), + None, + KeyProviderKind::Kms, + key_provider_id.to_vec(), + instance_id.to_vec(), + ); + let document = mr_config.to_canonical_json(); + let expected = ExpectedMrConfig { + compose_hash: &compose_hash, + gpu_policy_hash: None, + app_id: &app_id, + instance_id: &instance_id, + key_provider: KeyProviderKind::Kms, + key_provider_id: &key_provider_id, + }; + + verify_tdx_mr_config_id_value(mr_config.to_tdx_mr_config_id(), Some(&document), expected) + } } diff --git a/dstack/kms/src/main_service.rs b/dstack/kms/src/main_service.rs index 6e3a4694d..77455ca6c 100644 --- a/dstack/kms/src/main_service.rs +++ b/dstack/kms/src/main_service.rs @@ -633,6 +633,7 @@ mod tests { dstack_types::mr_config::MrConfigV3::new( vec![0x11; 20], vec![0x22; 32], + None, dstack_types::KeyProviderKind::None, Vec::new(), vec![0x99; 20], diff --git a/dstack/kms/src/main_service/amd_attest.rs b/dstack/kms/src/main_service/amd_attest.rs index bcbbdbf4d..9e579984f 100644 --- a/dstack/kms/src/main_service/amd_attest.rs +++ b/dstack/kms/src/main_service/amd_attest.rs @@ -193,6 +193,7 @@ fn test_mr_config(app_id: Vec, compose_hash: Vec) -> MrConfigV3 { MrConfigV3::new( app_id, compose_hash, + None, dstack_types::KeyProviderKind::None, Vec::new(), instance_id, diff --git a/dstack/kms/src/onboard_service.rs b/dstack/kms/src/onboard_service.rs index 346e458de..84e01ef93 100644 --- a/dstack/kms/src/onboard_service.rs +++ b/dstack/kms/src/onboard_service.rs @@ -243,6 +243,7 @@ mod tests { dstack_types::mr_config::MrConfigV3::new( vec![0x11; 20], vec![0x22; 32], + None, dstack_types::KeyProviderKind::None, Vec::new(), vec![0x99; 20], diff --git a/dstack/vmm/src/app.rs b/dstack/vmm/src/app.rs index 2ab3aeee9..f4dcd0d12 100644 --- a/dstack/vmm/src/app.rs +++ b/dstack/vmm/src/app.rs @@ -132,6 +132,10 @@ pub struct GpuConfig { } impl GpuConfig { + pub fn has_gpus(&self) -> bool { + !self.gpus.is_empty() + } + pub fn is_empty(&self) -> bool { if self.attach_mode.is_all() { return false; @@ -1175,7 +1179,10 @@ impl App { let mr_config = if use_mr_config_v3 { Some( work_dir - .prepare_mr_config_v3(&app_compose) + .prepare_mr_config_v3( + &app_compose, + manifest.gpus.as_ref().is_some_and(GpuConfig::has_gpus), + ) .context("Failed to prepare mr_config")?, ) } else { @@ -1542,6 +1549,30 @@ mod tests { hex::encode(vec![byte; len]) } + #[test] + fn gpu_config_has_gpus_only_when_resolved_gpu_list_is_non_empty() { + assert!(!GpuConfig::default().has_gpus()); + assert!(!GpuConfig { + attach_mode: AttachMode::All, + ..Default::default() + } + .has_gpus()); + assert!(!GpuConfig { + bridges: vec![GpuSpec { + slot: "0000:01:00.0".into(), + }], + ..Default::default() + } + .has_gpus()); + assert!(GpuConfig { + gpus: vec![GpuSpec { + slot: "0000:02:00.0".into(), + }], + ..Default::default() + } + .has_gpus()); + } + #[test] fn put_manifest_keeps_legacy_networking_for_rollback() -> Result<()> { let temp = std::env::temp_dir().join(format!( @@ -1948,6 +1979,7 @@ mod tests { let mr_config = MrConfigV3::new( vec![0x11; 20], vec![0x22; 32], + None, dstack_types::KeyProviderKind::None, vec![], vec![0x44; 20], @@ -2002,6 +2034,7 @@ mod tests { assert_eq!(parsed_mr_config.app_id, vec![0x11; 20]); assert_eq!(parsed_mr_config.compose_hash, vec![0x22; 32]); + assert_eq!(parsed_mr_config.gpu_policy_hash, None); assert_eq!(vm_config["mr_config"], sys_config["mr_config"]); assert_eq!( vm_config["os_image_hash"] diff --git a/dstack/vmm/src/app/mr_config.rs b/dstack/vmm/src/app/mr_config.rs index f1b908e4f..434d8a1d1 100644 --- a/dstack/vmm/src/app/mr_config.rs +++ b/dstack/vmm/src/app/mr_config.rs @@ -7,7 +7,7 @@ use anyhow::{bail, Context, Result}; use base64::prelude::*; use dstack_types::mr_config::{MrConfig, MrConfigV3}; -use dstack_types::AppCompose; +use dstack_types::{gpu_policy_hash, AppCompose}; use fs_err as fs; use sha2::{Digest, Sha256}; @@ -60,10 +60,21 @@ pub(super) fn snp_host_data(workdir: &VmWorkDir) -> Result { } impl VmWorkDir { - pub fn prepare_mr_config_v3(&self, app_compose: &AppCompose) -> Result { + pub fn prepare_mr_config_v3(&self, app_compose: &AppCompose, has_gpus: bool) -> Result { let compose_hash = self .app_compose_hash() .context("failed to get compose hash")?; + let gpu_policy_hash = if has_gpus { + let compose_json = fs::read(self.app_compose_path()) + .context("failed to read app compose for GPU policy hash")?; + Some( + gpu_policy_hash(&compose_json) + .context("failed to hash raw GPU policy")? + .to_vec(), + ) + } else { + None + }; let mut instance_info = self .instance_info_or_default() .context("failed to get instance info")?; @@ -106,6 +117,7 @@ impl VmWorkDir { Ok(MrConfigV3::new( app_id, compose_hash.to_vec(), + gpu_policy_hash, app_compose.key_provider(), app_compose.key_provider_id.clone(), instance_id, diff --git a/dstack/vmm/src/one_shot.rs b/dstack/vmm/src/one_shot.rs index 3773fbcb2..abcb36ba4 100644 --- a/dstack/vmm/src/one_shot.rs +++ b/dstack/vmm/src/one_shot.rs @@ -247,7 +247,10 @@ Compose file content (first 200 chars): let mr_config = if use_mr_config_v3 { Some( vm_work_dir - .prepare_mr_config_v3(&app_compose) + .prepare_mr_config_v3( + &app_compose, + manifest.gpus.as_ref().is_some_and(|gpus| gpus.has_gpus()), + ) .context("Failed to prepare mr_config")?, ) } else { From ffd12351977b40eb8a5be5bbd8847062677e4a8a Mon Sep 17 00:00:00 2001 From: Kevin Wang Date: Thu, 16 Jul 2026 21:13:04 -0700 Subject: [PATCH 25/32] fix: skip GPU policy hash check when absent --- docs/security/security-model.md | 6 +-- dstack/dstack-util/src/system_setup.rs | 7 ++- .../src/system_setup/config_id_verifier.rs | 46 +++++-------------- 3 files changed, 18 insertions(+), 41 deletions(-) diff --git a/docs/security/security-model.md b/docs/security/security-model.md index 1e7098601..8419d6231 100644 --- a/docs/security/security-model.md +++ b/docs/security/security-model.md @@ -65,7 +65,7 @@ An application may additionally set `requirements.gpu_policy` to an object with - `allow_debug` (boolean, default `false`): permit an attestation claim whose `dbgstat` is `enabled`. - `allow_insecure_boot` (boolean, default `false`): permit an attestation claim whose GPU `secboot` value is false. -After measuring `compose-hash`, dstack enters the GPU setup gate and JCS-canonicalizes the original `requirements.gpu_policy` JSON value, then measures its SHA-256 digest in a `gpu-policy-hash` event. When the field is absent—including when `requirements` itself is absent—both parsing and measurement use the default empty object `{}`. Thus an omitted policy and an explicit `{}` have the same digest, while any explicitly present field, including an explicit default value, changes the digest. MrConfigV3 GPU launches also carry this digest as the optional `gpu_policy_hash` field; non-GPU launches omit it for compatibility. The guest requires and verifies the field when it observes an attached GPU, and requires it to be absent otherwise. The MrConfigV3 document is bound by TDX `MR_CONFIG_ID` or SEV-SNP `HOST_DATA`, so the host cannot substitute a different GPU policy without changing the platform launch identity. The typed policy used for enforcement applies omitted-field defaults and rejects unknown fields. If GPU attestation is enabled and NVIDIA GPUs are present, dstack attests them and applies the basic settings and optional Rego policy before setting the GPU ready state. When no attestation claims are produced—because no GPU is attached or `gpu_policy.attest_gpu` is false—Rego is still evaluated with an empty array as `input` before any ready-state transition. This lets an application reject a launch whose attested GPU count is wrong. A false, undefined, malformed, or non-boolean Rego result stops boot before key provisioning. +After measuring `compose-hash`, dstack enters the GPU setup gate and JCS-canonicalizes the original `requirements.gpu_policy` JSON value, then measures its SHA-256 digest in a `gpu-policy-hash` event. When the field is absent—including when `requirements` itself is absent—both parsing and measurement use the default empty object `{}`. Thus an omitted policy and an explicit `{}` have the same digest, while any explicitly present field, including an explicit default value, changes the digest. MrConfigV3 GPU launches also carry this digest as the optional `gpu_policy_hash` field; non-GPU launches omit it for compatibility. When the field is present, the guest compares it with the digest computed from app-compose; when it is absent, the guest skips this MrConfigV3 check. The MrConfigV3 document is bound by TDX `MR_CONFIG_ID` or SEV-SNP `HOST_DATA`, so the host cannot substitute a different GPU policy when this optional binding is present without changing the platform launch identity. The typed policy used for enforcement applies omitted-field defaults and rejects unknown fields. If GPU attestation is enabled and NVIDIA GPUs are present, dstack attests them and applies the basic settings and optional Rego policy before setting the GPU ready state. When no attestation claims are produced—because no GPU is attached or `gpu_policy.attest_gpu` is false—Rego is still evaluated with an empty array as `input` before any ready-state transition. This lets an application reject a launch whose attested GPU count is wrong. A false, undefined, malformed, or non-boolean Rego result stops boot before key provisioning. For every NVML-enumerated GPU, dstack calls `Device::is_cc_enabled()` and `Device::is_cc_dev_mode_enabled()` and requires the NVML device count to match the expected GPU count. CC must always be ON; DevTools must be OFF unless the measured policy explicitly permits it. The typed claim checks always require `measres == "success"`; by default they also require `dbgstat == "disabled"` and `secboot == true`, with the latter two checks controlled by their explicit opt-ins. Only after the default appraisal, typed claim checks, optional Rego policy, and per-device NVML checks succeed does dstack call `Device::set_confidential_compute_state(true)` to set the GPU ready state. The dstack CPU/guest boot chain is verified independently through measured boot; a GPU claim named `secboot` refers to the GPU appraisal, not UEFI Secure Boot in the CVM. @@ -73,7 +73,7 @@ For every NVML-enumerated GPU, dstack calls `Device::is_cc_enabled()` and `Devic GPU workloads require verification of both hardware components. The CPU TEE quote verifies the CVM and its measured guest code. NVIDIA-signed evidence, checked against NVIDIA RIMs and certificate status by `nvattest`, verifies the GPU appraisal. After the optional policy and ready-state operations succeed, dstack emits a `gpu-attestation` launch event before `system-ready`. Its versioned payload records the number of appraised devices, asserted CC/DevTools state, and SHA-256 of the complete nvattest JSON output (claims and detached EAT). On TDX, both `gpu-policy-hash` and `gpu-attestation` are append-only RTMR3 events; they are never derived from application-controlled `report_data`. -A verifier must replay the measured event log, require exactly one `gpu-policy-hash` event immediately after `compose-hash`, and compare its 32-byte payload with the expected policy digest (`SHA-256(JCS({}))` for the omitted/default policy). When a GPU launch uses MrConfigV3, its `gpu_policy_hash` must be present and match the same digest. When GPU protection is required, the verifier must also require exactly one pre-`system-ready` `gpu-attestation` event with `devices > 0` and, when applicable, the expected deployment count. The raw `attestation.out` file is not trusted by itself; if it is supplied for inspection, its digest must match the `gpu-attestation` event. +A verifier must replay the measured event log, require exactly one `gpu-policy-hash` event immediately after `compose-hash`, and compare its 32-byte payload with the expected policy digest (`SHA-256(JCS({}))` for the omitted/default policy). When MrConfigV3 includes `gpu_policy_hash`, it must match the same digest. When GPU protection is required, the verifier must also require exactly one pre-`system-ready` `gpu-attestation` event with `devices > 0` and, when applicable, the expected deployment count. The raw `attestation.out` file is not trusted by itself; if it is supplied for inspection, its digest must match the `gpu-attestation` event. ### GPU Threat Model and Lifetime @@ -187,7 +187,7 @@ Use this checklist to verify a workload running in a dstack CVM. **GPU verification (when required):** - [ ] The `gpu-attestation` device count is greater than zero and matches the expected deployment - [ ] Exactly one `gpu-policy-hash` event follows `compose-hash`, and its payload matches `SHA-256(JCS(requirements.gpu_policy))` (default `{}` when omitted) -- [ ] When a GPU launch uses MrConfigV3, its `gpu_policy_hash` is present and matches the same expected GPU policy digest +- [ ] When MrConfigV3 includes `gpu_policy_hash`, it matches the same expected GPU policy digest - [ ] Exactly one `gpu-attestation` event appears before `system-ready` - [ ] CC is ON and the event's DevTools field complies with the measured policy - [ ] The platform binds the event log to a quoted RTMR/PCR (do not accept it from current SEV-SNP evidence) diff --git a/dstack/dstack-util/src/system_setup.rs b/dstack/dstack-util/src/system_setup.rs index fdab3fb36..b1fd31d14 100644 --- a/dstack/dstack-util/src/system_setup.rs +++ b/dstack/dstack-util/src/system_setup.rs @@ -1803,7 +1803,7 @@ impl Stage0<'_> { /// explicitly disabled — set the GPU ready state without verification. The /// optional Rego policy is always evaluated; when no attestation is /// performed, its claims-array input is empty. - async fn measure_gpu(&self) -> Result> { + async fn measure_gpu(&self) -> Result<[u8; 32]> { let gpu_policy_hash = gpu::measure_gpu_policy(&self.shared.dir.app_compose_file())?; let gpu_policy = self @@ -1815,7 +1815,6 @@ impl Stage0<'_> { .unwrap_or_default(); let inventory = gpu::gpu_inventory()?; - let gpu_policy_hash = (inventory.total > 0).then_some(gpu_policy_hash); if !gpu_policy.attest_gpu { // Attestation is explicitly disabled, so there are no claims. Rego // still runs with an empty input before any GPU is made ready. @@ -1909,7 +1908,7 @@ impl AppIdValidator { struct AppInfo { instance_info: InstanceInfo, compose_hash: [u8; 32], - gpu_policy_hash: Option<[u8; 32]>, + gpu_policy_hash: [u8; 32], } struct Stage0<'a> { @@ -2564,7 +2563,7 @@ impl<'a> Stage0<'a> { fn verify_app(&self, app_info: &AppInfo, keys: &AppKeys) -> Result<()> { config_id_verifier::verify_mr_config_id( &app_info.compose_hash, - app_info.gpu_policy_hash.as_ref(), + &app_info.gpu_policy_hash, &app_info .instance_info .app_id diff --git a/dstack/dstack-util/src/system_setup/config_id_verifier.rs b/dstack/dstack-util/src/system_setup/config_id_verifier.rs index 90a981f51..c6c79d5a0 100644 --- a/dstack/dstack-util/src/system_setup/config_id_verifier.rs +++ b/dstack/dstack-util/src/system_setup/config_id_verifier.rs @@ -14,7 +14,7 @@ use tracing::info; #[derive(Clone, Copy)] struct ExpectedMrConfig<'a> { compose_hash: &'a [u8; 32], - gpu_policy_hash: Option<&'a [u8; 32]>, + gpu_policy_hash: &'a [u8; 32], app_id: &'a [u8; 20], instance_id: &'a [u8], key_provider: KeyProviderKind, @@ -66,7 +66,7 @@ fn read_snp_host_data() -> Result<[u8; 32]> { /// - key_provider_id: [u8] // KMS CA pubkey, local-sgx MR, or empty for none/tpm pub fn verify_mr_config_id( compose_hash: &[u8; 32], - gpu_policy_hash: Option<&[u8; 32]>, + gpu_policy_hash: &[u8; 32], app_id: &[u8; 20], instance_id: &[u8], key_provider: KeyProviderKind, @@ -166,9 +166,10 @@ fn verify_mr_config_v3_document( if mr_config.compose_hash.as_slice() != expected.compose_hash { bail!("Invalid mr_config compose_hash"); } - if mr_config.gpu_policy_hash.as_deref() != expected.gpu_policy_hash.map(|hash| hash.as_slice()) - { - bail!("Invalid mr_config gpu_policy_hash"); + if let Some(gpu_policy_hash) = mr_config.gpu_policy_hash.as_deref() { + if gpu_policy_hash != expected.gpu_policy_hash { + bail!("Invalid mr_config gpu_policy_hash"); + } } if mr_config.app_id.as_slice() != expected.app_id { bail!("Invalid mr_config app_id"); @@ -216,7 +217,7 @@ mod tests { let document = mr_config.to_canonical_json(); let expected = ExpectedMrConfig { compose_hash: &compose_hash, - gpu_policy_hash: Some(&gpu_policy_hash), + gpu_policy_hash: &gpu_policy_hash, app_id: &app_id, instance_id: &instance_id, key_provider: KeyProviderKind::Kms, @@ -245,7 +246,7 @@ mod tests { let wrong_app_id = [0x12u8; 20]; let expected = ExpectedMrConfig { compose_hash: &compose_hash, - gpu_policy_hash: Some(&gpu_policy_hash), + gpu_policy_hash: &gpu_policy_hash, app_id: &wrong_app_id, instance_id: &instance_id, key_provider: KeyProviderKind::Kms, @@ -277,7 +278,7 @@ mod tests { let wrong_gpu_policy_hash = [0x56u8; 32]; let expected = ExpectedMrConfig { compose_hash: &compose_hash, - gpu_policy_hash: Some(&wrong_gpu_policy_hash), + gpu_policy_hash: &wrong_gpu_policy_hash, app_id: &app_id, instance_id: &instance_id, key_provider: KeyProviderKind::Kms, @@ -290,35 +291,12 @@ mod tests { .to_string() .contains("Invalid mr_config gpu_policy_hash")), } - - let document_without_gpu_policy_hash = MrConfigV3::new( - app_id.to_vec(), - compose_hash.to_vec(), - None, - KeyProviderKind::Kms, - key_provider_id.to_vec(), - instance_id.to_vec(), - ) - .to_canonical_json(); - let expected = ExpectedMrConfig { - compose_hash: &compose_hash, - gpu_policy_hash: Some(&gpu_policy_hash), - app_id: &app_id, - instance_id: &instance_id, - key_provider: KeyProviderKind::Kms, - key_provider_id: &key_provider_id, - }; - match verify_mr_config_v3_document(&document_without_gpu_policy_hash, expected) { - Ok(_) => panic!("missing gpu_policy_hash on a GPU launch must reject"), - Err(err) => assert!(err - .to_string() - .contains("Invalid mr_config gpu_policy_hash")), - } } #[test] - fn mr_config_v3_allows_missing_gpu_policy_hash_for_non_gpu_launch() -> Result<()> { + fn mr_config_v3_skips_gpu_policy_hash_check_when_field_is_missing() -> Result<()> { let compose_hash = [0x22u8; 32]; + let actual_gpu_policy_hash = [0x55u8; 32]; let app_id = [0x11u8; 20]; let instance_id = [0x44u8; 20]; let key_provider_id = [0x33u8; 32]; @@ -333,7 +311,7 @@ mod tests { let document = mr_config.to_canonical_json(); let expected = ExpectedMrConfig { compose_hash: &compose_hash, - gpu_policy_hash: None, + gpu_policy_hash: &actual_gpu_policy_hash, app_id: &app_id, instance_id: &instance_id, key_provider: KeyProviderKind::Kms, From e42defb07583e99fe06c0dd4e460abc2b54530de Mon Sep 17 00:00:00 2001 From: Kevin Wang Date: Thu, 16 Jul 2026 21:16:53 -0700 Subject: [PATCH 26/32] refactor: clarify local mr_config values --- .../src/system_setup/config_id_verifier.rs | 68 +++++++++---------- 1 file changed, 34 insertions(+), 34 deletions(-) diff --git a/dstack/dstack-util/src/system_setup/config_id_verifier.rs b/dstack/dstack-util/src/system_setup/config_id_verifier.rs index c6c79d5a0..82b627846 100644 --- a/dstack/dstack-util/src/system_setup/config_id_verifier.rs +++ b/dstack/dstack-util/src/system_setup/config_id_verifier.rs @@ -12,7 +12,7 @@ use dstack_types::{ use tracing::info; #[derive(Clone, Copy)] -struct ExpectedMrConfig<'a> { +struct LocalMrConfigValues<'a> { compose_hash: &'a [u8; 32], gpu_policy_hash: &'a [u8; 32], app_id: &'a [u8; 20], @@ -52,7 +52,7 @@ fn read_snp_host_data() -> Result<[u8; 32]> { Ok(parsed.host_data) } -/// Verify the mr_config_id matches the expected value +/// Verify the mr_config_id matches values observed locally by the guest. /// /// Configuration ID format /// The mr_config_id is a 48 bytes value in the following format: @@ -73,7 +73,7 @@ pub fn verify_mr_config_id( key_provider_id: &[u8], ) -> Result<()> { let mode = AttestationMode::detect().context("Failed to detect attestation mode")?; - let expected = ExpectedMrConfig { + let local = LocalMrConfigValues { compose_hash, gpu_policy_hash, app_id, @@ -81,24 +81,24 @@ pub fn verify_mr_config_id( key_provider, key_provider_id, }; - verify_mr_config_id_for_mode(mode, expected) + verify_mr_config_id_for_mode(mode, local) } fn verify_mr_config_id_for_mode( mode: AttestationMode, - expected: ExpectedMrConfig<'_>, + local: LocalMrConfigValues<'_>, ) -> Result<()> { match mode { - AttestationMode::DstackAmdSevSnp => verify_snp_mr_config(expected), + AttestationMode::DstackAmdSevSnp => verify_snp_mr_config(local), // AWS PCR8 is computed by the guest from measured reality (MrConfig V2 // in measure_app_info); there is no host-supplied claim to cross-check. // The key_provider_id pin is enforced by verify_key_provider_id. AttestationMode::DstackAwsNitroTpm => Ok(()), - _ => verify_tdx_mr_config_id(expected), + _ => verify_tdx_mr_config_id(local), } } -fn verify_tdx_mr_config_id(expected: ExpectedMrConfig<'_>) -> Result<()> { +fn verify_tdx_mr_config_id(local: LocalMrConfigValues<'_>) -> Result<()> { let read_mr_config_id = read_mr_config_id().context("Failed to read mr_config_id")?; info!("mr_config_id: {}", hex::encode(read_mr_config_id)); let mr_config_document = if read_mr_config_id[0] == 3 { @@ -106,33 +106,33 @@ fn verify_tdx_mr_config_id(expected: ExpectedMrConfig<'_>) -> Result<()> { } else { None }; - verify_tdx_mr_config_id_value(read_mr_config_id, mr_config_document.as_deref(), expected) + verify_tdx_mr_config_id_value(read_mr_config_id, mr_config_document.as_deref(), local) } fn verify_tdx_mr_config_id_value( read_mr_config_id: [u8; 48], mr_config_document: Option<&str>, - expected: ExpectedMrConfig<'_>, + local: LocalMrConfigValues<'_>, ) -> Result<()> { if read_mr_config_id == [0u8; 48] { return Ok(()); } let expected_mr_config_id = match read_mr_config_id[0] { 1 => MrConfig::V1 { - compose_hash: expected.compose_hash, + compose_hash: local.compose_hash, } .to_mr_config_id(), 2 => MrConfig::V2 { - compose_hash: expected.compose_hash, - app_id: expected.app_id, - key_provider: expected.key_provider, - key_provider_id: expected.key_provider_id, + compose_hash: local.compose_hash, + app_id: local.app_id, + key_provider: local.key_provider, + key_provider_id: local.key_provider_id, } .to_mr_config_id(), 3 => { let mr_config_document = mr_config_document.context("mr_config is required for TDX MR_CONFIG_ID v3")?; - verify_mr_config_v3_document(mr_config_document, expected)?; + verify_mr_config_v3_document(mr_config_document, local)?; MrConfigV3::tdx_mr_config_id_from_document(mr_config_document) } _ => bail!("Invalid mr_config_id version"), @@ -143,9 +143,9 @@ fn verify_tdx_mr_config_id_value( Ok(()) } -fn verify_snp_mr_config(expected: ExpectedMrConfig<'_>) -> Result<()> { +fn verify_snp_mr_config(local: LocalMrConfigValues<'_>) -> Result<()> { let mr_config_document = read_mr_config_document().context("Failed to read SNP mr_config")?; - verify_mr_config_v3_document(&mr_config_document, expected)?; + verify_mr_config_v3_document(&mr_config_document, local)?; let read_host_data = read_snp_host_data().context("Failed to read SNP HOST_DATA")?; info!("snp host_data: {}", hex::encode(read_host_data)); if MrConfigV3::snp_host_data_from_document(&mr_config_document) != read_host_data { @@ -156,31 +156,31 @@ fn verify_snp_mr_config(expected: ExpectedMrConfig<'_>) -> Result<()> { fn verify_mr_config_v3_document( mr_config_document: &str, - expected: ExpectedMrConfig<'_>, + local: LocalMrConfigValues<'_>, ) -> Result { let mr_config = MrConfigV3::from_document(mr_config_document).context("Invalid mr_config document")?; if mr_config.version != 3 { bail!("mr_config version must be 3"); } - if mr_config.compose_hash.as_slice() != expected.compose_hash { + if mr_config.compose_hash.as_slice() != local.compose_hash { bail!("Invalid mr_config compose_hash"); } - if let Some(gpu_policy_hash) = mr_config.gpu_policy_hash.as_deref() { - if gpu_policy_hash != expected.gpu_policy_hash { + if let Some(declared_gpu_policy_hash) = mr_config.gpu_policy_hash.as_deref() { + if declared_gpu_policy_hash != local.gpu_policy_hash { bail!("Invalid mr_config gpu_policy_hash"); } } - if mr_config.app_id.as_slice() != expected.app_id { + if mr_config.app_id.as_slice() != local.app_id { bail!("Invalid mr_config app_id"); } - if mr_config.instance_id.as_slice() != expected.instance_id { + if mr_config.instance_id.as_slice() != local.instance_id { bail!("Invalid mr_config instance_id"); } - if mr_config.key_provider != expected.key_provider { + if mr_config.key_provider != local.key_provider { bail!("Invalid mr_config key_provider"); } - if mr_config.key_provider_id.as_slice() != expected.key_provider_id { + if mr_config.key_provider_id.as_slice() != local.key_provider_id { bail!("Invalid mr_config key_provider_id"); } Ok(mr_config) @@ -215,7 +215,7 @@ mod tests { instance_id.to_vec(), ); let document = mr_config.to_canonical_json(); - let expected = ExpectedMrConfig { + let local = LocalMrConfigValues { compose_hash: &compose_hash, gpu_policy_hash: &gpu_policy_hash, app_id: &app_id, @@ -224,7 +224,7 @@ mod tests { key_provider_id: &key_provider_id, }; - verify_tdx_mr_config_id_value(mr_config.to_tdx_mr_config_id(), Some(&document), expected) + verify_tdx_mr_config_id_value(mr_config.to_tdx_mr_config_id(), Some(&document), local) } #[test] @@ -244,7 +244,7 @@ mod tests { ) .to_canonical_json(); let wrong_app_id = [0x12u8; 20]; - let expected = ExpectedMrConfig { + let local = LocalMrConfigValues { compose_hash: &compose_hash, gpu_policy_hash: &gpu_policy_hash, app_id: &wrong_app_id, @@ -253,7 +253,7 @@ mod tests { key_provider_id: &key_provider_id, }; - match verify_mr_config_v3_document(&document, expected) { + match verify_mr_config_v3_document(&document, local) { Ok(_) => panic!("mismatched app_id must reject"), Err(err) => assert!(err.to_string().contains("Invalid mr_config app_id")), } @@ -276,7 +276,7 @@ mod tests { ) .to_canonical_json(); let wrong_gpu_policy_hash = [0x56u8; 32]; - let expected = ExpectedMrConfig { + let local = LocalMrConfigValues { compose_hash: &compose_hash, gpu_policy_hash: &wrong_gpu_policy_hash, app_id: &app_id, @@ -285,7 +285,7 @@ mod tests { key_provider_id: &key_provider_id, }; - match verify_mr_config_v3_document(&document, expected) { + match verify_mr_config_v3_document(&document, local) { Ok(_) => panic!("mismatched gpu_policy_hash must reject"), Err(err) => assert!(err .to_string() @@ -309,7 +309,7 @@ mod tests { instance_id.to_vec(), ); let document = mr_config.to_canonical_json(); - let expected = ExpectedMrConfig { + let local = LocalMrConfigValues { compose_hash: &compose_hash, gpu_policy_hash: &actual_gpu_policy_hash, app_id: &app_id, @@ -318,6 +318,6 @@ mod tests { key_provider_id: &key_provider_id, }; - verify_tdx_mr_config_id_value(mr_config.to_tdx_mr_config_id(), Some(&document), expected) + verify_tdx_mr_config_id_value(mr_config.to_tdx_mr_config_id(), Some(&document), local) } } From 4da381ba6bff0396aec7037469069e9c9c3e870b Mon Sep 17 00:00:00 2001 From: Kevin Wang Date: Thu, 16 Jul 2026 21:31:06 -0700 Subject: [PATCH 27/32] docs: explain GPU policy and evidence verification --- docs/amd-sev-snp.md | 9 +++++ docs/attestation-tdx.md | 6 ++- docs/aws-ec2-production-verifier-runbook.md | 8 ++++ docs/security/security-model.md | 45 +++++++++++++++++++++ docs/tutorials/attestation-verification.md | 9 +++++ sdk/curl/api.md | 27 +++++++++++++ 6 files changed, 103 insertions(+), 1 deletion(-) diff --git a/docs/amd-sev-snp.md b/docs/amd-sev-snp.md index a1320ce98..30da7ed40 100644 --- a/docs/amd-sev-snp.md +++ b/docs/amd-sev-snp.md @@ -82,6 +82,15 @@ Verification is fail-closed and includes: 5. the unified OS image identity, `sha256(sha256sum.txt)`, which must match `digest.txt` and the SNP measurement document. +For a GPU VM, the optional `MrConfigV3.gpu_policy_hash` field contains +`SHA-256(JCS(requirements.gpu_policy))`. The signed SNP report binds the exact +MrConfigV3 document through `HOST_DATA`, so a verifier can validate the report +and document binding and then compare this field with the expected GPU policy +digest. If the field is absent, this optional check is not asserted. This binds +the GPU policy, but not the later `gpu-attestation` runtime event or the +`GpuInfo` output: the current SEV-SNP path has no quote-bound runtime +measurement register. + The verifier supports the AMD Milan, Genoa, and Turin KDS product families. Bergamo and Siena are handled through AMD's canonical Genoa KDS product path. diff --git a/docs/attestation-tdx.md b/docs/attestation-tdx.md index f0f433423..ae6798ca3 100644 --- a/docs/attestation-tdx.md +++ b/docs/attestation-tdx.md @@ -25,12 +25,16 @@ The MR register values indicate the following: - RTMR0: OVMF records CVM's virtual hardware setup, including CPU count, memory size, and device configuration. While dstack uses fixed devices, CPU and memory specifications can vary. RTMR0 can be computed from these specifications. - RTMR1: OVMF records the Linux kernel measurement. - RTMR2: Linux kernel records kernel cmdline (including rootfs hash) and initrd measurements. - - RTMR3: initrd records dstack App details, including compose hash, instance id, app id, and key provider. + - RTMR3: initrd records dstack App details, including compose hash, GPU policy and attestation events, instance id, app id, and key provider. MRTD, RTMR0, RTMR1, and RTMR2 can be pre-calculated from the built image (given CPU+RAM specifications). Compare these with the verified quote's MRs to confirm correct base image code execution. RTMR3 differs as it contains runtime information like compose hash and instance id. Verify this by replaying the event log - if the calculated RTMR3 matches the quote's RTMR3, the event log information is valid. Then verify the compose hash, key provider, and other event log details match expectations. +For a GPU launch, `compose-hash` is followed by `gpu-policy-hash` and, after successful NVIDIA attestation and policy evaluation, `gpu-attestation`. The `gpu-policy-hash` payload is `SHA-256(JCS(requirements.gpu_policy))`, using `{}` when the field is omitted. The `gpu-attestation` payload is JSON containing the verified device count, CC/DevTools state, and `evidence_sha256`. + +The guest-agent `GpuInfo` API returns the complete `nvattest` JSON captured during boot. It is not trustworthy by itself. After verifying the TDX quote and replaying the event log to RTMR3, hash the exact UTF-8 bytes of `GpuInfo.attestation` and require the result to equal the `gpu-attestation` event's `evidence_sha256`. See [GPU Security for AI Workloads](./security/security-model.md#gpu-security-for-ai-workloads) for the event schema, ordering, Rego example, and platform differences. + ### 2.2. Determining expected MRs MRTD, RTMR0, RTMR1, and RTMR2 correspond to the image. dstack OS builds all related software from source. Build the exact image revision you intend to verify. See diff --git a/docs/aws-ec2-production-verifier-runbook.md b/docs/aws-ec2-production-verifier-runbook.md index c5a5e35d3..77e6098cb 100644 --- a/docs/aws-ec2-production-verifier-runbook.md +++ b/docs/aws-ec2-production-verifier-runbook.md @@ -209,6 +209,14 @@ For AWS NitroTPM, the policy must require: `instance-id`, `key-provider`); - a `report_data` challenge for external verifier flows that need liveness. +For GPU workloads, PCR14 also contains `gpu-policy-hash` immediately after +`compose-hash`. Its 32-byte payload must equal +`SHA-256(JCS(requirements.gpu_policy))`, using `{}` when the policy is omitted. +Because the verifier replays the event chain against the signed NitroTPM +Attestation Document, this validates `gpu_policy_hash` on AWS. A successful GPU +launch also adds `gpu-attestation`; its `evidence_sha256` can be compared with +the exact UTF-8 bytes returned by `GpuInfo.attestation` after PCR14 replay. + The guest also extends a `MrConfig` V2 **config commitment** into **PCR8** (`PCR8 = sha384(0^48 || config_id)`). This is **optional** and exists only to let a lightweight third-party verifier confirm `composeHash` + key-provider diff --git a/docs/security/security-model.md b/docs/security/security-model.md index 8419d6231..443b12f36 100644 --- a/docs/security/security-model.md +++ b/docs/security/security-model.md @@ -65,14 +65,59 @@ An application may additionally set `requirements.gpu_policy` to an object with - `allow_debug` (boolean, default `false`): permit an attestation claim whose `dbgstat` is `enabled`. - `allow_insecure_boot` (boolean, default `false`): permit an attestation claim whose GPU `secboot` value is false. +The optional Rego v0 policy can enforce deployment-specific claims. For example, this app-compose fragment requires exactly one H100 whose `hwmodel` matches the value emitted by `nvattest`: + +```json +{ + "requirements": { + "gpu_policy": { + "rego": "package policy\n\ndefault nv_match = false\n\nnv_match {\n count(input) == 1\n input[0].hwmodel == \"GH100 A01 GSP BROM\"\n}" + } + } +} +``` + +The policy must define the boolean rule `data.policy.nv_match`. Its `input` is the complete `claims` array from `nvattest`; when no attestation is performed, `input` is `[]`, so the example also rejects a launch without exactly one attested GPU. + After measuring `compose-hash`, dstack enters the GPU setup gate and JCS-canonicalizes the original `requirements.gpu_policy` JSON value, then measures its SHA-256 digest in a `gpu-policy-hash` event. When the field is absent—including when `requirements` itself is absent—both parsing and measurement use the default empty object `{}`. Thus an omitted policy and an explicit `{}` have the same digest, while any explicitly present field, including an explicit default value, changes the digest. MrConfigV3 GPU launches also carry this digest as the optional `gpu_policy_hash` field; non-GPU launches omit it for compatibility. When the field is present, the guest compares it with the digest computed from app-compose; when it is absent, the guest skips this MrConfigV3 check. The MrConfigV3 document is bound by TDX `MR_CONFIG_ID` or SEV-SNP `HOST_DATA`, so the host cannot substitute a different GPU policy when this optional binding is present without changing the platform launch identity. The typed policy used for enforcement applies omitted-field defaults and rejects unknown fields. If GPU attestation is enabled and NVIDIA GPUs are present, dstack attests them and applies the basic settings and optional Rego policy before setting the GPU ready state. When no attestation claims are produced—because no GPU is attached or `gpu_policy.attest_gpu` is false—Rego is still evaluated with an empty array as `input` before any ready-state transition. This lets an application reject a launch whose attested GPU count is wrong. A false, undefined, malformed, or non-boolean Rego result stops boot before key provisioning. +The policy digest is remotely verifiable on each supported platform, but through different carriers: + +- **TDX:** `gpu-policy-hash` contains the raw 32-byte digest and is measured into RTMR3. Replay the event log and compare the result with the quote's RTMR3, then compare the event payload with the expected digest. When an MrConfigV3 document includes `gpu_policy_hash`, TDX `MR_CONFIG_ID` additionally binds that field. +- **AWS NitroTPM:** the same `gpu-policy-hash` event is extended into non-resettable SHA384 PCR14. Replay the PCR14 event chain against the signed NitroTPM Attestation Document, then compare the payload with the expected digest. +- **AMD SEV-SNP:** there is no quote-bound runtime event register in the current stack. Instead, GPU launches put the digest in optional `MrConfigV3.gpu_policy_hash`, and the signed SNP report's `HOST_DATA` binds the exact MrConfigV3 document. Verify the SNP report and `HOST_DATA` binding, then compare the field with the expected digest. If the optional field is absent, this check is not asserted. + For every NVML-enumerated GPU, dstack calls `Device::is_cc_enabled()` and `Device::is_cc_dev_mode_enabled()` and requires the NVML device count to match the expected GPU count. CC must always be ON; DevTools must be OFF unless the measured policy explicitly permits it. The typed claim checks always require `measres == "success"`; by default they also require `dbgstat == "disabled"` and `secboot == true`, with the latter two checks controlled by their explicit opt-ins. Only after the default appraisal, typed claim checks, optional Rego policy, and per-device NVML checks succeed does dstack call `Device::set_confidential_compute_state(true)` to set the GPU ready state. The dstack CPU/guest boot chain is verified independently through measured boot; a GPU claim named `secboot` refers to the GPU appraisal, not UEFI Secure Boot in the CVM. ### Dual Attestation GPU workloads require verification of both hardware components. The CPU TEE quote verifies the CVM and its measured guest code. NVIDIA-signed evidence, checked against NVIDIA RIMs and certificate status by `nvattest`, verifies the GPU appraisal. After the optional policy and ready-state operations succeed, dstack emits a `gpu-attestation` launch event before `system-ready`. Its versioned payload records the number of appraised devices, asserted CC/DevTools state, and SHA-256 of the complete nvattest JSON output (claims and detached EAT). On TDX, both `gpu-policy-hash` and `gpu-attestation` are append-only RTMR3 events; they are never derived from application-controlled `report_data`. +For a successful TDX GPU launch, the GPU-relevant RTMR3 event order is: + +```text +compose-hash +gpu-policy-hash +gpu-attestation +instance-id +boot-mr-done +``` + +`gpu-policy-hash` is emitted even for a GPU-less launch. `gpu-attestation` is emitted only after an attached GPU passes `nvattest`, the built-in checks, the optional Rego policy, and the NVML state checks. Its UTF-8 JSON payload has this shape: + +```json +{ + "version": 2, + "provider": "nvidia", + "devices": 1, + "cc_mode": "on", + "devtools": false, + "evidence_sha256": "" +} +``` + +`GpuInfo` returns that complete boot-time `nvattest` JSON in its `attestation` string; it does not run a new attestation. To bind the API result to TDX evidence: verify the quote, replay the event log to the quote's RTMR3, require exactly one pre-`system-ready` `gpu-attestation` event, decode its JSON payload, and compare `evidence_sha256` with `SHA-256(UTF-8(GpuInfo.attestation))`. Only after this comparison should the verifier inspect the returned claims. This exact-byte comparison includes any whitespace or trailing newline in the returned string. + A verifier must replay the measured event log, require exactly one `gpu-policy-hash` event immediately after `compose-hash`, and compare its 32-byte payload with the expected policy digest (`SHA-256(JCS({}))` for the omitted/default policy). When MrConfigV3 includes `gpu_policy_hash`, it must match the same digest. When GPU protection is required, the verifier must also require exactly one pre-`system-ready` `gpu-attestation` event with `devices > 0` and, when applicable, the expected deployment count. The raw `attestation.out` file is not trusted by itself; if it is supplied for inspection, its digest must match the `gpu-attestation` event. ### GPU Threat Model and Lifetime diff --git a/docs/tutorials/attestation-verification.md b/docs/tutorials/attestation-verification.md index e39ab8ad0..63f4a6f22 100644 --- a/docs/tutorials/attestation-verification.md +++ b/docs/tutorials/attestation-verification.md @@ -518,6 +518,8 @@ These are the standard events you'll see in the log: | `system-preparing` | System initialization marker | Always present | | `app-id` | Application identifier | Should match your app name | | `compose-hash` | SHA-256 of docker compose config | Should match `tcb_info.compose_hash` | +| `gpu-policy-hash` | SHA-256 of the JCS-canonicalized GPU policy (default `{}`) | Should match the expected `requirements.gpu_policy` digest | +| `gpu-attestation` | Verified GPU state and digest of the boot-time `nvattest` JSON | Required for an attested GPU launch; verify as described below | | `instance-id` | Unique instance identifier | Should match `instance_id` from response | | `boot-mr-done` | Boot measurements complete | Marker event | | `mr-kms` | KMS identity measurement | KMS public key hash | @@ -526,6 +528,13 @@ These are the standard events you'll see in the log: | `storage-fs` | Storage filesystem type | Storage configuration | | `system-ready` | System ready marker | Always present at end | +For a successful GPU launch, the relevant order is `compose-hash`, +`gpu-policy-hash`, `gpu-attestation`, `instance-id`, and `boot-mr-done`. +After replaying the log to the quote's RTMR3, decode the JSON payload of +`gpu-attestation` and compare its `evidence_sha256` with the SHA-256 digest of +the exact UTF-8 `GpuInfo.attestation` string. `GpuInfo` reads the result saved +during boot and does not perform a new attestation. + ### Verify specific event values ```bash diff --git a/sdk/curl/api.md b/sdk/curl/api.md index a2b2d7ea0..6dccf75f2 100644 --- a/sdk/curl/api.md +++ b/sdk/curl/api.md @@ -306,6 +306,33 @@ curl --unix-socket /var/run/dstack.sock http://dstack/GpuInfo } ``` +`GpuInfo.attestation` is the exact UTF-8 `nvattest` output saved during boot; +calling this endpoint does not perform a new attestation. To authenticate it on +TDX, first verify the quote and replay the supplied event log to the quote's +RTMR3. Then decode the `gpu-attestation` event payload and compare its +`evidence_sha256` with the SHA-256 digest of the exact returned string: + +```python +import hashlib +import json + +gpu_info = json.load(open("gpu-info.json")) +quote_response = json.load(open("quote.json")) +events = quote_response["event_log"] +if isinstance(events, str): + events = json.loads(events) + +entry = next(event for event in events if event["event"] == "gpu-attestation") +measured = json.loads(bytes.fromhex(entry["event_payload"])) +actual = hashlib.sha256(gpu_info["attestation"].encode()).hexdigest() +assert actual == measured["evidence_sha256"] +``` + +The comparison above is meaningful only after quote verification and RTMR3 +event-log replay have succeeded. See the +[security model](../../docs/security/security-model.md#gpu-security-for-ai-workloads) +for the event ordering and AWS/AMD SEV-SNP verification paths. + ## Error Responses All endpoints may return the following HTTP status codes: From 73d48573e6a6288d44455d19a5910a3f10e60447 Mon Sep 17 00:00:00 2001 From: Kevin Wang Date: Thu, 16 Jul 2026 21:35:27 -0700 Subject: [PATCH 28/32] docs: show Rego app compose structure --- docs/security/security-model.md | 19 +++++++++++++++++-- 1 file changed, 17 insertions(+), 2 deletions(-) diff --git a/docs/security/security-model.md b/docs/security/security-model.md index 443b12f36..6e80ec640 100644 --- a/docs/security/security-model.md +++ b/docs/security/security-model.md @@ -65,18 +65,33 @@ An application may additionally set `requirements.gpu_policy` to an object with - `allow_debug` (boolean, default `false`): permit an attestation claim whose `dbgstat` is `enabled`. - `allow_insecure_boot` (boolean, default `false`): permit an attestation claim whose GPU `secboot` value is false. -The optional Rego v0 policy can enforce deployment-specific claims. For example, this app-compose fragment requires exactly one H100 whose `hwmodel` matches the value emitted by `nvattest`: +The optional Rego v0 policy can enforce deployment-specific claims. A minimal `app-compose.json` containing one looks like this; replace the placeholder with the policy source: ```json { + "manifest_version": "3", + "name": "gpu-app", + "runner": "docker-compose", "requirements": { "gpu_policy": { - "rego": "package policy\n\ndefault nv_match = false\n\nnv_match {\n count(input) == 1\n input[0].hwmodel == \"GH100 A01 GSP BROM\"\n}" + "rego": "" } } } ``` +For example, the following policy requires exactly one H100 whose `hwmodel` matches the value emitted by `nvattest`: + +```rego +package policy +default nv_match = false + +nv_match { + count(input) == 1 + input[0].hwmodel == "GH100 A01 GSP BROM" +} +``` + The policy must define the boolean rule `data.policy.nv_match`. Its `input` is the complete `claims` array from `nvattest`; when no attestation is performed, `input` is `[]`, so the example also rejects a launch without exactly one attested GPU. After measuring `compose-hash`, dstack enters the GPU setup gate and JCS-canonicalizes the original `requirements.gpu_policy` JSON value, then measures its SHA-256 digest in a `gpu-policy-hash` event. When the field is absent—including when `requirements` itself is absent—both parsing and measurement use the default empty object `{}`. Thus an omitted policy and an explicit `{}` have the same digest, while any explicitly present field, including an explicit default value, changes the digest. MrConfigV3 GPU launches also carry this digest as the optional `gpu_policy_hash` field; non-GPU launches omit it for compatibility. When the field is present, the guest compares it with the digest computed from app-compose; when it is absent, the guest skips this MrConfigV3 check. The MrConfigV3 document is bound by TDX `MR_CONFIG_ID` or SEV-SNP `HOST_DATA`, so the host cannot substitute a different GPU policy when this optional binding is present without changing the platform launch identity. The typed policy used for enforcement applies omitted-field defaults and rejects unknown fields. If GPU attestation is enabled and NVIDIA GPUs are present, dstack attests them and applies the basic settings and optional Rego policy before setting the GPU ready state. When no attestation claims are produced—because no GPU is attached or `gpu_policy.attest_gpu` is false—Rego is still evaluated with an empty array as `input` before any ready-state transition. This lets an application reject a launch whose attested GPU count is wrong. A false, undefined, malformed, or non-boolean Rego result stops boot before key provisioning. From b5f0b9bc6cad3c01e35f18f4ac8404f9246e1c18 Mon Sep 17 00:00:00 2001 From: Kevin Wang Date: Sat, 18 Jul 2026 05:12:33 -0700 Subject: [PATCH 29/32] feat(guest): bound GPU Rego policy evaluation time Upgrade regorus to 0.10.1 and set an execution timer so a runaway application policy cannot hang boot indefinitely. nvattest already runs under a timeout; the Rego evaluation was the remaining unbounded step. The engine needs the arc feature when built without default features. --- dstack/Cargo.lock | 175 ++++++++++++++----------- dstack/Cargo.toml | 2 +- dstack/dstack-util/src/system_setup.rs | 30 +++++ 3 files changed, 129 insertions(+), 78 deletions(-) diff --git a/dstack/Cargo.lock b/dstack/Cargo.lock index b7e63cffa..4cb0b55f1 100644 --- a/dstack/Cargo.lock +++ b/dstack/Cargo.lock @@ -2405,9 +2405,9 @@ dependencies = [ [[package]] name = "fancy-regex" -version = "0.14.0" +version = "0.18.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6e24cb5a94bcae1e5408b0effca5cd7172ea3c5755049c5f3af4cd283a165298" +checksum = "e1e1dacd0d2082dfcf1351c4bdd566bbe89a2b263235a2b50058f1e130a47277" dependencies = [ "bit-set", "regex-automata", @@ -2509,9 +2509,9 @@ dependencies = [ [[package]] name = "fluent-uri" -version = "0.3.2" +version = "0.4.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1918b65d96df47d3591bed19c5cca17e3fa5d0707318e4b5ef2eae01764df7e5" +checksum = "bc74ac4d8359ae70623506d512209619e5cf8f347124910440dbc221714b328e" dependencies = [ "borrow-or-share", "ref-cast", @@ -2819,6 +2819,18 @@ version = "0.3.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "0cc23270f6e1808e30a928bdc84dea0b9b4136a8bc82338574f23baf47bbd280" +[[package]] +name = "globset" +version = "0.4.19" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e47d37d2ae4464254884b60ab7071be2b876a9c35b696bd018ddcc76847309cd" +dependencies = [ + "aho-corasick", + "bstr", + "regex-automata", + "regex-syntax", +] + [[package]] name = "group" version = "0.13.0" @@ -2920,6 +2932,17 @@ dependencies = [ "foldhash 0.1.5", ] +[[package]] +name = "hashbrown" +version = "0.16.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "841d1cc9bed7f9236f321df977030373f4a4163ae1a7dbfe1a51a2c1a51d9100" +dependencies = [ + "allocator-api2", + "equivalent", + "foldhash 0.2.0", +] + [[package]] name = "hashbrown" version = "0.17.1" @@ -3596,15 +3619,6 @@ dependencies = [ "either", ] -[[package]] -name = "itertools" -version = "0.11.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b1c173a5686ce8bfa551b3563d0c2170bf24ca44da99c7ca4bfdab5418c3fe57" -dependencies = [ - "either", -] - [[package]] name = "itertools" version = "0.13.0" @@ -3722,28 +3736,40 @@ dependencies = [ [[package]] name = "jsonschema" -version = "0.29.1" +version = "0.46.10" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "161c33c3ec738cfea3288c5c53dfcdb32fd4fc2954de86ea06f71b5a1a40bfcd" +checksum = "f0a699d3e77675e6aa4bfffe3b907c8b5f7ed3241f9965bffb25475ad4b08d05" dependencies = [ "ahash", - "base64 0.22.1", "bytecount", + "data-encoding", "email_address", "fancy-regex", "fraction", + "getrandom 0.3.4", "idna", "itoa", + "jsonschema-regex", "num-cmp", - "once_cell", + "num-traits", "percent-encoding", "referencing", - "regex-syntax", + "regex", "serde", "serde_json", + "unicode-general-category", "uuid-simd", ] +[[package]] +name = "jsonschema-regex" +version = "0.46.10" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6dbd1086b01b9349fd4ef9a07433965af64c8ce8159abe633a189e4ff817bd13" +dependencies = [ + "regex-syntax", +] + [[package]] name = "k256" version = "0.13.4" @@ -3818,7 +3844,7 @@ version = "1.5.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "bbd2bcb4c963f2ddae06a2efc7e9f3591312473c50c6685e1f298068316e66fe" dependencies = [ - "spin", + "spin 0.9.8", ] [[package]] @@ -3927,6 +3953,12 @@ dependencies = [ "tracing-subscriber", ] +[[package]] +name = "lru" +version = "0.18.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0b6180140927ee907000b0aa540091f6ea512ead4447c92b8fc35bc72788a5a6" + [[package]] name = "lru-slab" version = "0.1.2" @@ -4024,6 +4056,12 @@ dependencies = [ "zeroize", ] +[[package]] +name = "micromap" +version = "0.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c2a86d3146ed3995b5913c414f6664344b9617457320782e64f0bb44afd49d74" + [[package]] name = "mime" version = "0.3.17" @@ -4107,6 +4145,15 @@ dependencies = [ "uuid", ] +[[package]] +name = "msvc_spectre_libs" +version = "0.1.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "29e871a9861f3664f18b7e04e9301d4edd55090c2dadb4b1c602e26ab32b1f5b" +dependencies = [ + "cc", +] + [[package]] name = "multer" version = "3.1.0" @@ -4121,7 +4168,7 @@ dependencies = [ "log", "memchr", "mime", - "spin", + "spin 0.9.8", "tokio", "tokio-util", "version_check", @@ -4924,15 +4971,6 @@ dependencies = [ "universal-hash", ] -[[package]] -name = "pori" -version = "0.0.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a4a63d338dec139f56dacc692ca63ad35a6be6a797442479b55acd611d79e906" -dependencies = [ - "nom", -] - [[package]] name = "portable-atomic" version = "1.13.1" @@ -5577,13 +5615,16 @@ checksum = "09c30c54dffee5b40af088d5d50aa3455c91a0127164b51f0215efc4cb28fb3c" [[package]] name = "referencing" -version = "0.29.1" +version = "0.46.10" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "40a64b3a635fad9000648b4d8a59c8710c523ab61a23d392a7d91d47683f5adc" +checksum = "0fbf332a2f81899f6836f22c03da73dae8a664c32e3016b84692c23cddadc95d" dependencies = [ "ahash", "fluent-uri", - "once_cell", + "getrandom 0.3.4", + "hashbrown 0.16.1", + "itoa", + "micromap", "parking_lot 0.12.5", "percent-encoding", "serde_json", @@ -5620,31 +5661,34 @@ checksum = "dc897dd8d9e8bd1ed8cdad82b5966c3e0ecae09fb1907d58efaa013543185d0a" [[package]] name = "regorus" -version = "0.4.0" +version = "0.10.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "bf70615014ce5c89fe427197fd9b52e6506916838b449e80ee141adc43adec6b" +checksum = "419a0413adeece71e4d4a64fb75adc359cb807496f0dfd10f429517be908b807" dependencies = [ "anyhow", "chrono", "chrono-tz", - "constant_time_eq 0.4.2", "data-encoding", - "hex", - "hmac 0.12.1", + "globset", + "indexmap 2.14.0", + "ipnet", "jsonschema", "lazy_static", - "md-5", - "rand 0.9.4", + "lru", + "msvc_spectre_libs", + "num-bigint", + "num-traits", + "parking_lot 0.12.5", + "rand 0.10.1", "regex", - "scientific", "semver", "serde", "serde_json", "serde_yaml", - "sha2 0.10.9", + "spin 0.10.1", + "thiserror 2.0.18", "url", "uuid", - "wax", ] [[package]] @@ -6325,26 +6369,6 @@ dependencies = [ "zeroize", ] -[[package]] -name = "scientific" -version = "0.5.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "38a4b339a8de779ecb098a772ecbba2ace74e23ed959a5b4f30631d8bf1799a8" -dependencies = [ - "scientific-macro", -] - -[[package]] -name = "scientific-macro" -version = "0.5.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d2ee4885492bb655bfa05d039cd9163eb8fe9f79ddebf00ca23a1637510c2fd2" -dependencies = [ - "proc-macro2", - "quote", - "syn 2.0.117", -] - [[package]] name = "scoped-tls" version = "1.0.1" @@ -6894,6 +6918,12 @@ version = "0.9.8" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "6980e8d7511241f8acf4aebddbb1ff938df5eebe98691418c4468d0b72a96a67" +[[package]] +name = "spin" +version = "0.10.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "023a211cb3138dbc438680b32560ad89f699977624c9f8dbb95a47d5b4c07dd3" + [[package]] name = "spki" version = "0.7.3" @@ -7710,6 +7740,12 @@ version = "2.9.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "dbc4bc3a9f746d862c45cb89d705aa10f187bb96c76001afab07a0d35ce60142" +[[package]] +name = "unicode-general-category" +version = "1.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0b993bddc193ae5bd0d623b49ec06ac3e9312875fdae725a975c51db1cc1677f" + [[package]] name = "unicode-ident" version = "1.0.24" @@ -7812,7 +7848,6 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "23b082222b4f6619906941c17eb2297fff4c2fb96cb60164170522942a200bd8" dependencies = [ "outref", - "uuid", "vsimd", ] @@ -8026,20 +8061,6 @@ dependencies = [ "tracing", ] -[[package]] -name = "wax" -version = "0.6.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8d12a78aa0bab22d2f26ed1a96df7ab58e8a93506a3e20adb47c51a93b4e1357" -dependencies = [ - "const_format", - "itertools 0.11.0", - "nom", - "pori", - "regex", - "thiserror 1.0.69", -] - [[package]] name = "web-sys" version = "0.3.99" diff --git a/dstack/Cargo.toml b/dstack/Cargo.toml index e5f54d384..265eae995 100644 --- a/dstack/Cargo.toml +++ b/dstack/Cargo.toml @@ -129,7 +129,7 @@ moka = { version = "0.12.15", default-features = false, features = ["sync"] } notify = "8.0.0" nvml-wrapper = "0.12.1" rand = "0.8.5" -regorus = { version = "0.4.0", default-features = false, features = ["full-opa"] } +regorus = { version = "0.10.1", default-features = false, features = ["full-opa", "arc"] } tracing = "0.1.40" tracing-subscriber = { version = "0.3.20", features = ["env-filter"] } safe-write = "0.1.2" diff --git a/dstack/dstack-util/src/system_setup.rs b/dstack/dstack-util/src/system_setup.rs index b1fd31d14..ae02dc5db 100644 --- a/dstack/dstack-util/src/system_setup.rs +++ b/dstack/dstack-util/src/system_setup.rs @@ -1081,6 +1081,8 @@ mod gpu { const ATTESTATION_TIMEOUT: Duration = Duration::from_secs(300); const EVENT_VERSION: u32 = 2; const POLICY_ENTRYPOINT: &str = "data.policy.nv_match"; + /// Bound Rego evaluation so a runaway application policy cannot hang boot. + const POLICY_TIMEOUT: Duration = Duration::from_secs(10); #[derive(Debug, Clone, Copy, PartialEq, Eq)] pub(super) struct GpuInventory { @@ -1435,8 +1437,20 @@ mod gpu { /// Evaluate the app-provided Rego v0 policy using the same input shape as /// NVIDIA relying-party policies: the nvattest `claims` JSON array. pub(super) fn evaluate_policy(policy: &str, claims: &[Value]) -> Result<()> { + evaluate_policy_with_timeout(policy, claims, POLICY_TIMEOUT) + } + + fn evaluate_policy_with_timeout( + policy: &str, + claims: &[Value], + timeout: Duration, + ) -> Result<()> { let mut engine = regorus::Engine::new(); engine.set_rego_v0(true); + engine.set_execution_timer_config(regorus::utils::limits::ExecutionTimerConfig { + limit: timeout, + check_interval: std::num::NonZeroU32::new(1024).expect("non-zero check interval"), + }); engine .add_policy("gpu-policy.rego".to_string(), policy.to_string()) .context("failed to load GPU policy")?; @@ -1762,6 +1776,22 @@ mod gpu { evaluate_rego_policy(&GpuPolicy::default(), &[]).unwrap(); } + #[test] + fn rego_policy_evaluation_is_time_bounded() { + let policy = r#" + package policy + default nv_match = false + nv_match { + count([x | + x := numbers.range(0, 5000)[_] + y := numbers.range(0, 5000)[_] + x == y + ]) > 0 + } + "#; + evaluate_policy_with_timeout(policy, &[], Duration::from_millis(50)).unwrap_err(); + } + #[test] fn gpu_policy_measurement_defaults_to_empty_object_and_uses_raw_json() { let no_requirements = br#"{}"#; From e63faf22713e499b3756ab68ad99c2f50a640ca1 Mon Sep 17 00:00:00 2001 From: Kevin Wang Date: Sat, 18 Jul 2026 05:12:33 -0700 Subject: [PATCH 30/32] refactor(guest-agent): return empty GpuInfo without attestation output GpuInfo no longer fails when no boot-time attestation output exists (no GPU attached or attestation disabled); it returns an empty attestation string instead, so callers do not need to treat the GPU-less case as an error. Unexpected read failures are logged and also reported as empty. --- dstack/guest-agent/rpc/proto/agent_rpc.proto | 3 +- dstack/guest-agent/src/rpc_service.rs | 31 +++++++++++++------- sdk/curl/api.md | 4 +-- sdk/go/README.md | 6 ++-- sdk/js/README.md | 6 ++-- sdk/python/README.md | 6 ++-- sdk/rust/README.md | 6 ++-- 7 files changed, 37 insertions(+), 25 deletions(-) diff --git a/dstack/guest-agent/rpc/proto/agent_rpc.proto b/dstack/guest-agent/rpc/proto/agent_rpc.proto index 47994bb1d..cef6ea274 100644 --- a/dstack/guest-agent/rpc/proto/agent_rpc.proto +++ b/dstack/guest-agent/rpc/proto/agent_rpc.proto @@ -191,7 +191,8 @@ message AttestResponse { } message GpuInfoResponse { - // Complete JSON output produced by nvattest. + // Complete JSON output produced by nvattest. Empty when no boot-time GPU + // attestation output is available. string attestation = 1; } diff --git a/dstack/guest-agent/src/rpc_service.rs b/dstack/guest-agent/src/rpc_service.rs index 08e470e05..e20126095 100644 --- a/dstack/guest-agent/src/rpc_service.rs +++ b/dstack/guest-agent/src/rpc_service.rs @@ -39,7 +39,7 @@ use rcgen::KeyPair; use ring::rand::{SecureRandom, SystemRandom}; use serde_json::json; use sha3::{Digest, Keccak256}; -use tracing::error; +use tracing::{error, warn}; use crate::{ backend::{PlatformBackend, RealPlatform}, @@ -52,13 +52,18 @@ fn read_dmi_file(name: &str) -> String { .unwrap_or_default() } -fn read_gpu_attestation(path: &Path) -> Result { - fs::read_to_string(path).with_context(|| { - format!( - "Failed to read GPU attestation output at {}", - path.display() - ) - }) +/// Read the GPU attestation output saved during boot. Returns an empty string +/// when no output is available (e.g. no GPU attached or attestation disabled). +fn read_gpu_attestation(path: &Path) -> String { + match fs::read_to_string(path) { + Ok(attestation) => attestation, + Err(err) => { + if err.kind() != std::io::ErrorKind::NotFound { + warn!("failed to read GPU attestation output: {err:?}"); + } + String::new() + } + } } #[derive(Clone)] @@ -342,7 +347,7 @@ impl DstackGuestRpc for InternalRpcHandler { async fn gpu_info(self) -> Result { Ok(GpuInfoResponse { - attestation: read_gpu_attestation(Path::new(GPU_ATTESTATION_OUTPUT))?, + attestation: read_gpu_attestation(Path::new(GPU_ATTESTATION_OUTPUT)), }) } @@ -693,7 +698,13 @@ mod tests { output.write_all(attestation.as_bytes()).unwrap(); output.flush().unwrap(); - assert_eq!(read_gpu_attestation(output.path()).unwrap(), attestation); + assert_eq!(read_gpu_attestation(output.path()), attestation); + } + + #[test] + fn missing_gpu_attestation_output_reads_as_empty() { + let dir = tempfile::tempdir().unwrap(); + assert_eq!(read_gpu_attestation(&dir.path().join("missing")), ""); } fn extract_pubkey_from_report_data(report_data: &[u8], prefix: &str) -> Result> { diff --git a/sdk/curl/api.md b/sdk/curl/api.md index 6dccf75f2..fb02b9052 100644 --- a/sdk/curl/api.md +++ b/sdk/curl/api.md @@ -289,8 +289,8 @@ curl --unix-socket /var/run/dstack.sock http://dstack/Attest?report_data=0000000 Returns GPU information collected during boot. Currently, this includes the complete JSON output produced by NVIDIA `nvattest`. -The endpoint returns an error when no GPU attestation output is available, for -example on a VM without an NVIDIA GPU or when GPU attestation was disabled. +The `attestation` field is empty when no GPU attestation output is available, +for example on a VM without an NVIDIA GPU or when GPU attestation was disabled. **Endpoint:** `/GpuInfo` diff --git a/sdk/go/README.md b/sdk/go/README.md index 5e628ed71..c5b7f8bac 100644 --- a/sdk/go/README.md +++ b/sdk/go/README.md @@ -594,9 +594,9 @@ if err != nil { fmt.Println(gpu.Attestation) ``` -The call returns an error when no GPU attestation output is available. The raw -output is not trusted by itself; remote verifiers should compare its digest -with the measured `gpu-attestation` runtime event. +The `Attestation` field is empty when no GPU attestation output is available. +The raw output is not trusted by itself; remote verifiers should compare its +digest with the measured `gpu-attestation` runtime event. ##### `GetTlsKey(ctx context.Context, options TlsKeyOptions) (*GetTlsKeyResponse, error)` diff --git a/sdk/js/README.md b/sdk/js/README.md index 31dfcad42..1cc1d9316 100644 --- a/sdk/js/README.md +++ b/sdk/js/README.md @@ -107,9 +107,9 @@ const gpu = await client.gpuInfo() console.log(gpu.attestation) ``` -The call throws when no GPU attestation output is available. The raw output is -not trusted by itself; remote verifiers should compare its digest with the -measured `gpu-attestation` runtime event. +The `attestation` field is empty when no GPU attestation output is available. +The raw output is not trusted by itself; remote verifiers should compare its +digest with the measured `gpu-attestation` runtime event. ### `info()` diff --git a/sdk/python/README.md b/sdk/python/README.md index 5c66f766e..e112f18d4 100644 --- a/sdk/python/README.md +++ b/sdk/python/README.md @@ -111,9 +111,9 @@ gpu = client.gpu_info() print(gpu.attestation) ``` -The call fails when no GPU attestation output is available. The raw output is -not trusted by itself; remote verifiers should compare its digest with the -measured `gpu-attestation` runtime event. +The `attestation` field is empty when no GPU attestation output is available. +The raw output is not trusted by itself; remote verifiers should compare its +digest with the measured `gpu-attestation` runtime event. ### Get Instance Info diff --git a/sdk/rust/README.md b/sdk/rust/README.md index 35d1d98e9..36a54a3ae 100644 --- a/sdk/rust/README.md +++ b/sdk/rust/README.md @@ -115,9 +115,9 @@ let gpu = client.gpu_info().await?; println!("{}", gpu.attestation); ``` -The call returns an error when no GPU attestation output is available. The raw -output is not trusted by itself; remote verifiers should compare its digest -with the measured `gpu-attestation` runtime event. +The `attestation` field is empty when no GPU attestation output is available. +The raw output is not trusted by itself; remote verifiers should compare its +digest with the measured `gpu-attestation` runtime event. ### Generate TLS Certificates From 184df6050fc94a587fb584d02157760eca66ca1f Mon Sep 17 00:00:00 2001 From: Kevin Wang Date: Sat, 18 Jul 2026 05:12:33 -0700 Subject: [PATCH 31/32] chore(yocto): drop stray blank line in nvattest recipe --- .../meta-nvidia/recipes-graphics/nvattest/nvattest_2026.06.09.bb | 1 - 1 file changed, 1 deletion(-) diff --git a/os/yocto/layers/meta-nvidia/recipes-graphics/nvattest/nvattest_2026.06.09.bb b/os/yocto/layers/meta-nvidia/recipes-graphics/nvattest/nvattest_2026.06.09.bb index 121794421..3494a8809 100644 --- a/os/yocto/layers/meta-nvidia/recipes-graphics/nvattest/nvattest_2026.06.09.bb +++ b/os/yocto/layers/meta-nvidia/recipes-graphics/nvattest/nvattest_2026.06.09.bb @@ -100,7 +100,6 @@ do_install() { install -d ${D}${systemd_system_unitdir}/dstack-prepare.service.d install -m 0644 ${UNPACKDIR}/10-nvidia-gpu-ordering.conf \ ${D}${systemd_system_unitdir}/dstack-prepare.service.d/10-nvidia-gpu-ordering.conf - } FILES:${PN} += " \ From 1cfff4449e8e2afaad33dcb933acc29a6dd0e097 Mon Sep 17 00:00:00 2001 From: Kevin Wang Date: Sun, 19 Jul 2026 06:16:16 -0700 Subject: [PATCH 32/32] fix(dstack-util): avoid expect() on NonZeroU32 in GPU policy timer Clippy's -D clippy::expect_used denies expect() on Option, which was breaking rust-checks CI. Use unwrap_or(NonZeroU32::MIN) instead since the literal 1024 is always non-zero. --- dstack/dstack-util/src/system_setup.rs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dstack/dstack-util/src/system_setup.rs b/dstack/dstack-util/src/system_setup.rs index ae02dc5db..225ea4e38 100644 --- a/dstack/dstack-util/src/system_setup.rs +++ b/dstack/dstack-util/src/system_setup.rs @@ -1449,7 +1449,7 @@ mod gpu { engine.set_rego_v0(true); engine.set_execution_timer_config(regorus::utils::limits::ExecutionTimerConfig { limit: timeout, - check_interval: std::num::NonZeroU32::new(1024).expect("non-zero check interval"), + check_interval: std::num::NonZeroU32::new(1024).unwrap_or(std::num::NonZeroU32::MIN), }); engine .add_policy("gpu-policy.rego".to_string(), policy.to_string())