diff --git a/docs/amd-sev-snp.md b/docs/amd-sev-snp.md index a1320ce98..30da7ed40 100644 --- a/docs/amd-sev-snp.md +++ b/docs/amd-sev-snp.md @@ -82,6 +82,15 @@ Verification is fail-closed and includes: 5. the unified OS image identity, `sha256(sha256sum.txt)`, which must match `digest.txt` and the SNP measurement document. +For a GPU VM, the optional `MrConfigV3.gpu_policy_hash` field contains +`SHA-256(JCS(requirements.gpu_policy))`. The signed SNP report binds the exact +MrConfigV3 document through `HOST_DATA`, so a verifier can validate the report +and document binding and then compare this field with the expected GPU policy +digest. If the field is absent, this optional check is not asserted. This binds +the GPU policy, but not the later `gpu-attestation` runtime event or the +`GpuInfo` output: the current SEV-SNP path has no quote-bound runtime +measurement register. + The verifier supports the AMD Milan, Genoa, and Turin KDS product families. Bergamo and Siena are handled through AMD's canonical Genoa KDS product path. diff --git a/docs/attestation-tdx.md b/docs/attestation-tdx.md index f0f433423..ae6798ca3 100644 --- a/docs/attestation-tdx.md +++ b/docs/attestation-tdx.md @@ -25,12 +25,16 @@ The MR register values indicate the following: - RTMR0: OVMF records CVM's virtual hardware setup, including CPU count, memory size, and device configuration. While dstack uses fixed devices, CPU and memory specifications can vary. RTMR0 can be computed from these specifications. - RTMR1: OVMF records the Linux kernel measurement. - RTMR2: Linux kernel records kernel cmdline (including rootfs hash) and initrd measurements. - - RTMR3: initrd records dstack App details, including compose hash, instance id, app id, and key provider. + - RTMR3: initrd records dstack App details, including compose hash, GPU policy and attestation events, instance id, app id, and key provider. MRTD, RTMR0, RTMR1, and RTMR2 can be pre-calculated from the built image (given CPU+RAM specifications). Compare these with the verified quote's MRs to confirm correct base image code execution. RTMR3 differs as it contains runtime information like compose hash and instance id. Verify this by replaying the event log - if the calculated RTMR3 matches the quote's RTMR3, the event log information is valid. Then verify the compose hash, key provider, and other event log details match expectations. +For a GPU launch, `compose-hash` is followed by `gpu-policy-hash` and, after successful NVIDIA attestation and policy evaluation, `gpu-attestation`. The `gpu-policy-hash` payload is `SHA-256(JCS(requirements.gpu_policy))`, using `{}` when the field is omitted. The `gpu-attestation` payload is JSON containing the verified device count, CC/DevTools state, and `evidence_sha256`. + +The guest-agent `GpuInfo` API returns the complete `nvattest` JSON captured during boot. It is not trustworthy by itself. After verifying the TDX quote and replaying the event log to RTMR3, hash the exact UTF-8 bytes of `GpuInfo.attestation` and require the result to equal the `gpu-attestation` event's `evidence_sha256`. See [GPU Security for AI Workloads](./security/security-model.md#gpu-security-for-ai-workloads) for the event schema, ordering, Rego example, and platform differences. + ### 2.2. Determining expected MRs MRTD, RTMR0, RTMR1, and RTMR2 correspond to the image. dstack OS builds all related software from source. Build the exact image revision you intend to verify. See diff --git a/docs/aws-ec2-production-verifier-runbook.md b/docs/aws-ec2-production-verifier-runbook.md index c5a5e35d3..77e6098cb 100644 --- a/docs/aws-ec2-production-verifier-runbook.md +++ b/docs/aws-ec2-production-verifier-runbook.md @@ -209,6 +209,14 @@ For AWS NitroTPM, the policy must require: `instance-id`, `key-provider`); - a `report_data` challenge for external verifier flows that need liveness. +For GPU workloads, PCR14 also contains `gpu-policy-hash` immediately after +`compose-hash`. Its 32-byte payload must equal +`SHA-256(JCS(requirements.gpu_policy))`, using `{}` when the policy is omitted. +Because the verifier replays the event chain against the signed NitroTPM +Attestation Document, this validates `gpu_policy_hash` on AWS. A successful GPU +launch also adds `gpu-attestation`; its `evidence_sha256` can be compared with +the exact UTF-8 bytes returned by `GpuInfo.attestation` after PCR14 replay. + The guest also extends a `MrConfig` V2 **config commitment** into **PCR8** (`PCR8 = sha384(0^48 || config_id)`). This is **optional** and exists only to let a lightweight third-party verifier confirm `composeHash` + key-provider diff --git a/docs/security/security-model.md b/docs/security/security-model.md index b85a83bf3..6e80ec640 100644 --- a/docs/security/security-model.md +++ b/docs/security/security-model.md @@ -8,7 +8,7 @@ This document helps you evaluate whether dstack's security model fits your needs dstack removes the need to trust most infrastructure operators. On TEE platforms such as Intel TDX and AMD SEV-SNP, the cloud or host operator cannot read your protected memory, modify your measured code, or access your secrets without detection. On AWS EC2 NitroTPM attested instances, AWS Nitro is part of the trusted platform, and the untrusted party is the workload AWS account administrator/operator. Network attackers cannot intercept your traffic because TLS terminates inside the attested environment with keys controlled by that environment (Zero Trust HTTPS). Docker registries cannot serve malicious images because the guest verifies SHA256 digests before pulling. -The primary trust input is **attested platform hardware**. Intel TDX is the production TEE path. AMD SEV-SNP is available where the selected dstack OS image and host support it, but it is new and experimental. AWS EC2 NitroTPM attested instances use a different trust root: the AWS Nitro system and AWS NitroTPM attestation PKI. In that mode, the threat model protects against the workload AWS account administrator and EC2 operator actions, but AWS remains trusted. For GPU workloads, you also trust **NVIDIA GPU hardware** and NVIDIA's Remote Attestation Service (NRAS). These are hardware-level trust assumptions. +The primary trust input is **attested platform hardware**. Intel TDX is the production TEE path. AMD SEV-SNP is available where the selected dstack OS image and host support it, but it is new and experimental. AWS EC2 NitroTPM attested instances use a different trust root: the AWS Nitro system and AWS NitroTPM attestation PKI. In that mode, the threat model protects against the workload AWS account administrator and EC2 operator actions, but AWS remains trusted. For GPU workloads, you also trust **NVIDIA GPU hardware**, NVIDIA's attestation PKI/RIMs, and its revocation service (or NRAS when a deployment selects remote verification). These are hardware-level trust assumptions. Everything else is verifiable. @@ -55,11 +55,99 @@ dstack supports NVIDIA H100, H200, and B200 GPUs in confidential compute mode fo ### How It Works -GPUs are passed through via VFIO directly to the TEE-protected CVM. The GPU operates in confidential compute mode, encrypting data during computation. Both the CPU TEE and NVIDIA GPU provide hardware isolation together. If either component fails verification, the security model breaks. +GPUs are passed through via VFIO to the TEE-protected CVM. Before key provisioning, `dstack-util setup` inventories every VGA/3D-controller PCI function, rejects non-NVIDIA display devices, and runs NVIDIA's local `nvattest` verifier over every NVIDIA-driver-visible GPU with a fresh nonce. dstack does not ship or pass a custom relying-party policy to this command; nvattest's built-in appraisal still applies. The complete JSON result (`result_code`, `result_message`, `claims`, and `detached_eat`) is saved at `/run/nvidia-gpu-attestation/attestation.out`. + +An application may additionally set `requirements.gpu_policy` to an object with the following deny-unknown-fields schema: + +- `attest_gpu` (boolean, default `true`): require local NVIDIA GPU attestation before enabling an attached GPU. Setting this to `false` skips attestation and is an explicit reduction in protection. +- `rego` (optional string): a Rego v0 script evaluated with the nvattest output's `claims` array as `input`. It must define the boolean entrypoint `data.policy.nv_match` in `package policy`. +- `allow_devtools` (boolean, default `false`): permit NVIDIA DevTools mode. Production applications should leave this disabled because DevTools removes the expected GPU memory-confidentiality guarantee. +- `allow_debug` (boolean, default `false`): permit an attestation claim whose `dbgstat` is `enabled`. +- `allow_insecure_boot` (boolean, default `false`): permit an attestation claim whose GPU `secboot` value is false. + +The optional Rego v0 policy can enforce deployment-specific claims. A minimal `app-compose.json` containing one looks like this; replace the placeholder with the policy source: + +```json +{ + "manifest_version": "3", + "name": "gpu-app", + "runner": "docker-compose", + "requirements": { + "gpu_policy": { + "rego": "" + } + } +} +``` + +For example, the following policy requires exactly one H100 whose `hwmodel` matches the value emitted by `nvattest`: + +```rego +package policy +default nv_match = false + +nv_match { + count(input) == 1 + input[0].hwmodel == "GH100 A01 GSP BROM" +} +``` + +The policy must define the boolean rule `data.policy.nv_match`. Its `input` is the complete `claims` array from `nvattest`; when no attestation is performed, `input` is `[]`, so the example also rejects a launch without exactly one attested GPU. + +After measuring `compose-hash`, dstack enters the GPU setup gate and JCS-canonicalizes the original `requirements.gpu_policy` JSON value, then measures its SHA-256 digest in a `gpu-policy-hash` event. When the field is absent—including when `requirements` itself is absent—both parsing and measurement use the default empty object `{}`. Thus an omitted policy and an explicit `{}` have the same digest, while any explicitly present field, including an explicit default value, changes the digest. MrConfigV3 GPU launches also carry this digest as the optional `gpu_policy_hash` field; non-GPU launches omit it for compatibility. When the field is present, the guest compares it with the digest computed from app-compose; when it is absent, the guest skips this MrConfigV3 check. The MrConfigV3 document is bound by TDX `MR_CONFIG_ID` or SEV-SNP `HOST_DATA`, so the host cannot substitute a different GPU policy when this optional binding is present without changing the platform launch identity. The typed policy used for enforcement applies omitted-field defaults and rejects unknown fields. If GPU attestation is enabled and NVIDIA GPUs are present, dstack attests them and applies the basic settings and optional Rego policy before setting the GPU ready state. When no attestation claims are produced—because no GPU is attached or `gpu_policy.attest_gpu` is false—Rego is still evaluated with an empty array as `input` before any ready-state transition. This lets an application reject a launch whose attested GPU count is wrong. A false, undefined, malformed, or non-boolean Rego result stops boot before key provisioning. + +The policy digest is remotely verifiable on each supported platform, but through different carriers: + +- **TDX:** `gpu-policy-hash` contains the raw 32-byte digest and is measured into RTMR3. Replay the event log and compare the result with the quote's RTMR3, then compare the event payload with the expected digest. When an MrConfigV3 document includes `gpu_policy_hash`, TDX `MR_CONFIG_ID` additionally binds that field. +- **AWS NitroTPM:** the same `gpu-policy-hash` event is extended into non-resettable SHA384 PCR14. Replay the PCR14 event chain against the signed NitroTPM Attestation Document, then compare the payload with the expected digest. +- **AMD SEV-SNP:** there is no quote-bound runtime event register in the current stack. Instead, GPU launches put the digest in optional `MrConfigV3.gpu_policy_hash`, and the signed SNP report's `HOST_DATA` binds the exact MrConfigV3 document. Verify the SNP report and `HOST_DATA` binding, then compare the field with the expected digest. If the optional field is absent, this check is not asserted. + +For every NVML-enumerated GPU, dstack calls `Device::is_cc_enabled()` and `Device::is_cc_dev_mode_enabled()` and requires the NVML device count to match the expected GPU count. CC must always be ON; DevTools must be OFF unless the measured policy explicitly permits it. The typed claim checks always require `measres == "success"`; by default they also require `dbgstat == "disabled"` and `secboot == true`, with the latter two checks controlled by their explicit opt-ins. Only after the default appraisal, typed claim checks, optional Rego policy, and per-device NVML checks succeed does dstack call `Device::set_confidential_compute_state(true)` to set the GPU ready state. The dstack CPU/guest boot chain is verified independently through measured boot; a GPU claim named `secboot` refers to the GPU appraisal, not UEFI Secure Boot in the CVM. ### Dual Attestation -GPU workloads require verification of both hardware components. The CPU TEE provides the quote that verifies CPU and memory isolation. NVIDIA's Remote Attestation Service (NRAS) independently verifies the GPU is genuine and running in confidential mode. Both attestations must pass for complete verification. +GPU workloads require verification of both hardware components. The CPU TEE quote verifies the CVM and its measured guest code. NVIDIA-signed evidence, checked against NVIDIA RIMs and certificate status by `nvattest`, verifies the GPU appraisal. After the optional policy and ready-state operations succeed, dstack emits a `gpu-attestation` launch event before `system-ready`. Its versioned payload records the number of appraised devices, asserted CC/DevTools state, and SHA-256 of the complete nvattest JSON output (claims and detached EAT). On TDX, both `gpu-policy-hash` and `gpu-attestation` are append-only RTMR3 events; they are never derived from application-controlled `report_data`. + +For a successful TDX GPU launch, the GPU-relevant RTMR3 event order is: + +```text +compose-hash +gpu-policy-hash +gpu-attestation +instance-id +boot-mr-done +``` + +`gpu-policy-hash` is emitted even for a GPU-less launch. `gpu-attestation` is emitted only after an attached GPU passes `nvattest`, the built-in checks, the optional Rego policy, and the NVML state checks. Its UTF-8 JSON payload has this shape: + +```json +{ + "version": 2, + "provider": "nvidia", + "devices": 1, + "cc_mode": "on", + "devtools": false, + "evidence_sha256": "" +} +``` + +`GpuInfo` returns that complete boot-time `nvattest` JSON in its `attestation` string; it does not run a new attestation. To bind the API result to TDX evidence: verify the quote, replay the event log to the quote's RTMR3, require exactly one pre-`system-ready` `gpu-attestation` event, decode its JSON payload, and compare `evidence_sha256` with `SHA-256(UTF-8(GpuInfo.attestation))`. Only after this comparison should the verifier inspect the returned claims. This exact-byte comparison includes any whitespace or trailing newline in the returned string. + +A verifier must replay the measured event log, require exactly one `gpu-policy-hash` event immediately after `compose-hash`, and compare its 32-byte payload with the expected policy digest (`SHA-256(JCS({}))` for the omitted/default policy). When MrConfigV3 includes `gpu_policy_hash`, it must match the same digest. When GPU protection is required, the verifier must also require exactly one pre-`system-ready` `gpu-attestation` event with `devices > 0` and, when applicable, the expected deployment count. The raw `attestation.out` file is not trusted by itself; if it is supplied for inspection, its digest must match the `gpu-attestation` event. + +### GPU Threat Model and Lifetime + +The GPU gate assumes a malicious host/VMM and untrusted host-provided PCI topology, while trusting the CPU TEE, the measured dstack guest/kernel, NVIDIA hardware/firmware roots, and the cryptography used by both attestation chains. Availability is out of scope. + +The events make the following **boot-time** statement: immediately before key provisioning, all attached VGA/3D PCI functions were NVIDIA devices, their count matched the NVML inventory, nvattest returned one fresh successfully appraised claim for each device, the measured application policy accepted those claims and GPU state when present, every enumerated GPU passed the CC/DevTools NVML checks, and setting the GPU ready state succeeded. This closes these cases: + +- A GPU-less launch cannot be presented as a GPU-verified launch because it has no `gpu-attestation` event. +- A mixed launch cannot attest only its TEE-capable subset. Non-NVIDIA display GPUs are rejected, and the sysfs, NVML, and nvattest claim counts must all agree. A non-CC NVIDIA GPU either prevents evidence collection/appraisal or causes the default appraisal, application policy, or CC-state check to fail. +- Copying another CVM's result into a file or `report_data` does not work. Only measured pre-application code can place the event before `system-ready`, and event-log replay binds it to the quoted RTMR/PCR value. + +This is **not a lifetime or physical co-location guarantee**. After `system-ready`, an application with sufficient guest privileges can unload the NVIDIA driver, and a malicious host may attempt PCI hot-remove/replacement or proxy GPU traffic. The boot event remains a true historical statement but does not prove that the same device is still attached. dstack also cannot rule out a live relay/cuckoo attack to a genuine remote GPU: current Hopper/Blackwell deployments do not provide a CPU-TEE-verifiable TEE-I/O/TDISP device binding. Applications that mutate the driver or PCI topology are outside this guarantee; higher-assurance deployments must prevent that behavior and re-attest before using a newly initialized GPU. + +AMD SEV-SNP has no runtime measurement register in the current dstack stack. The local boot gate can still fail closed, but a `gpu-attestation` event carried beside an SNP report is not remotely bound to that report and must not be accepted as dual-attestation evidence. SNP needs a measured vTPM/PCR channel before it can provide the same remote binding. ### AI Workload Protection @@ -156,6 +244,14 @@ Use this checklist to verify a workload running in a dstack CVM. - [ ] Config commitment matches the expected app/config target (on AWS: PCR14 replay; PCR8 is an optional shortcut — see the [AWS verifier runbook](../aws-ec2-production-verifier-runbook.md)) - [ ] reportData contains your challenge (replay protection) +**GPU verification (when required):** +- [ ] The `gpu-attestation` device count is greater than zero and matches the expected deployment +- [ ] Exactly one `gpu-policy-hash` event follows `compose-hash`, and its payload matches `SHA-256(JCS(requirements.gpu_policy))` (default `{}` when omitted) +- [ ] When MrConfigV3 includes `gpu_policy_hash`, it matches the same expected GPU policy digest +- [ ] Exactly one `gpu-attestation` event appears before `system-ready` +- [ ] CC is ON and the event's DevTools field complies with the measured policy +- [ ] The platform binds the event log to a quoted RTMR/PCR (do not accept it from current SEV-SNP evidence) + **Key management verification:** - [ ] key-provider matches expected KMS identity - [ ] KMS attestation is valid diff --git a/docs/tutorials/attestation-verification.md b/docs/tutorials/attestation-verification.md index e39ab8ad0..63f4a6f22 100644 --- a/docs/tutorials/attestation-verification.md +++ b/docs/tutorials/attestation-verification.md @@ -518,6 +518,8 @@ These are the standard events you'll see in the log: | `system-preparing` | System initialization marker | Always present | | `app-id` | Application identifier | Should match your app name | | `compose-hash` | SHA-256 of docker compose config | Should match `tcb_info.compose_hash` | +| `gpu-policy-hash` | SHA-256 of the JCS-canonicalized GPU policy (default `{}`) | Should match the expected `requirements.gpu_policy` digest | +| `gpu-attestation` | Verified GPU state and digest of the boot-time `nvattest` JSON | Required for an attested GPU launch; verify as described below | | `instance-id` | Unique instance identifier | Should match `instance_id` from response | | `boot-mr-done` | Boot measurements complete | Marker event | | `mr-kms` | KMS identity measurement | KMS public key hash | @@ -526,6 +528,13 @@ These are the standard events you'll see in the log: | `storage-fs` | Storage filesystem type | Storage configuration | | `system-ready` | System ready marker | Always present at end | +For a successful GPU launch, the relevant order is `compose-hash`, +`gpu-policy-hash`, `gpu-attestation`, `instance-id`, and `boot-mr-done`. +After replaying the log to the quote's RTMR3, decode the JSON payload of +`gpu-attestation` and compare its `evidence_sha256` with the SHA-256 digest of +the exact UTF-8 `GpuInfo.attestation` string. `GpuInfo` reads the result saved +during boot and does not perform a new attestation. + ### Verify specific event values ```bash diff --git a/dstack/Cargo.lock b/dstack/Cargo.lock index f12355b98..4cb0b55f1 100644 --- a/dstack/Cargo.lock +++ b/dstack/Cargo.lock @@ -55,6 +55,20 @@ dependencies = [ "subtle", ] +[[package]] +name = "ahash" +version = "0.8.12" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5a15f179cd60c4584b8a8c596927aadc462e27f2ca70c04e0071964a73ba7a75" +dependencies = [ + "cfg-if", + "getrandom 0.3.4", + "once_cell", + "serde", + "version_check", + "zerocopy", +] + [[package]] name = "aho-corasick" version = "1.1.4" @@ -401,6 +415,21 @@ dependencies = [ "syn 2.0.117", ] +[[package]] +name = "bit-set" +version = "0.8.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "08807e080ed7f9d5433fa9b275196cfc35414f66a0c79d864dc51a0d825231a3" +dependencies = [ + "bit-vec", +] + +[[package]] +name = "bit-vec" +version = "0.8.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5e764a1d40d510daf35e07be9eb06e75770908c27d411ee6c92109c9840eaaf7" + [[package]] name = "bitfield" version = "0.15.0" @@ -543,7 +572,7 @@ version = "3.9.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "519bd3116aeeb42d5372c29d982d16d0170d3d4a5ed85fc7dd91642ffff3c67c" dependencies = [ - "darling", + "darling 0.23.0", "ident_case", "prettyplease", "proc-macro2", @@ -552,6 +581,12 @@ dependencies = [ "syn 2.0.117", ] +[[package]] +name = "borrow-or-share" +version = "0.2.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "dc0b364ead1874514c8c2855ab558056ebfeb775653e7ae45ff72f28f8f3166c" + [[package]] name = "borsh" version = "1.6.1" @@ -608,6 +643,12 @@ version = "1.2.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "7575182f7272186991736b70173b0ea045398f984bf5ebbb3804736ce1330c9d" +[[package]] +name = "bytecount" +version = "0.6.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "175812e0be2bccb6abe50bb8d566126198344f707e304f45c648fd8f2cc0365e" + [[package]] name = "bytemuck" version = "1.25.0" @@ -755,6 +796,16 @@ dependencies = [ "windows-link 0.2.1", ] +[[package]] +name = "chrono-tz" +version = "0.10.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a6139a8597ed92cf816dfb33f5dd6cf0bb93a6adc938f11039f371bc5bcd26c3" +dependencies = [ + "chrono", + "phf", +] + [[package]] name = "ciborium" version = "0.2.2" @@ -1201,14 +1252,38 @@ dependencies = [ "syn 2.0.117", ] +[[package]] +name = "darling" +version = "0.20.11" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "fc7f46116c46ff9ab3eb1597a45688b6715c6e628b5c133e288e709a29bcb4ee" +dependencies = [ + "darling_core 0.20.11", + "darling_macro 0.20.11", +] + [[package]] name = "darling" version = "0.23.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "25ae13da2f202d56bd7f91c25fba009e7717a1e4a1cc98a76d844b65ae912e9d" dependencies = [ - "darling_core", - "darling_macro", + "darling_core 0.23.0", + "darling_macro 0.23.0", +] + +[[package]] +name = "darling_core" +version = "0.20.11" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0d00b9596d185e565c2207a0b01f8bd1a135483d02d9b7b0a54b11da8d53412e" +dependencies = [ + "fnv", + "ident_case", + "proc-macro2", + "quote", + "strsim", + "syn 2.0.117", ] [[package]] @@ -1224,13 +1299,24 @@ dependencies = [ "syn 2.0.117", ] +[[package]] +name = "darling_macro" +version = "0.20.11" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "fc34b93ccb385b40dc71c6fceac4b2ad23662c7eeb248cf10d529b7e055b6ead" +dependencies = [ + "darling_core 0.20.11", + "quote", + "syn 2.0.117", +] + [[package]] name = "darling_macro" version = "0.23.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "ac3984ec7bd6cfa798e62b4a642426a5be0e68f9401cfc2a01e3fa9ea2fcdb8d" dependencies = [ - "darling_core", + "darling_core 0.23.0", "quote", "syn 2.0.117", ] @@ -1971,11 +2057,13 @@ dependencies = [ "k256", "libc", "luks2", + "nvml-wrapper", "parity-scale-codec", "ra-rpc", "ra-tls", "rand 0.8.6", "regex", + "regorus", "safe-write", "schnorrkel", "scopeguard", @@ -2196,6 +2284,15 @@ dependencies = [ "zeroize", ] +[[package]] +name = "email_address" +version = "0.2.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e079f19b08ca6239f47f8ba8509c11cf3ea30095831f7fed61441475edd8c449" +dependencies = [ + "serde", +] + [[package]] name = "encode_unicode" version = "1.0.0" @@ -2306,6 +2403,17 @@ dependencies = [ "winapi", ] +[[package]] +name = "fancy-regex" +version = "0.18.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e1e1dacd0d2082dfcf1351c4bdd566bbe89a2b263235a2b50058f1e130a47277" +dependencies = [ + "bit-set", + "regex-automata", + "regex-syntax", +] + [[package]] name = "fastrand" version = "2.4.1" @@ -2399,6 +2507,17 @@ dependencies = [ "miniz_oxide", ] +[[package]] +name = "fluent-uri" +version = "0.4.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "bc74ac4d8359ae70623506d512209619e5cf8f347124910440dbc221714b328e" +dependencies = [ + "borrow-or-share", + "ref-cast", + "serde", +] + [[package]] name = "fnv" version = "1.0.7" @@ -2426,6 +2545,16 @@ dependencies = [ "percent-encoding", ] +[[package]] +name = "fraction" +version = "0.15.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e076045bb43dac435333ed5f04caf35c7463631d0dae2deb2638d94dd0a5b872" +dependencies = [ + "lazy_static", + "num", +] + [[package]] name = "fs-err" version = "3.3.0" @@ -2690,6 +2819,18 @@ version = "0.3.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "0cc23270f6e1808e30a928bdc84dea0b9b4136a8bc82338574f23baf47bbd280" +[[package]] +name = "globset" +version = "0.4.19" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e47d37d2ae4464254884b60ab7071be2b876a9c35b696bd018ddcc76847309cd" +dependencies = [ + "aho-corasick", + "bstr", + "regex-automata", + "regex-syntax", +] + [[package]] name = "group" version = "0.13.0" @@ -2791,6 +2932,17 @@ dependencies = [ "foldhash 0.1.5", ] +[[package]] +name = "hashbrown" +version = "0.16.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "841d1cc9bed7f9236f321df977030373f4a4163ae1a7dbfe1a51a2c1a51d9100" +dependencies = [ + "allocator-api2", + "equivalent", + "foldhash 0.2.0", +] + [[package]] name = "hashbrown" version = "0.17.1" @@ -3582,6 +3734,42 @@ dependencies = [ "wasm-bindgen", ] +[[package]] +name = "jsonschema" +version = "0.46.10" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f0a699d3e77675e6aa4bfffe3b907c8b5f7ed3241f9965bffb25475ad4b08d05" +dependencies = [ + "ahash", + "bytecount", + "data-encoding", + "email_address", + "fancy-regex", + "fraction", + "getrandom 0.3.4", + "idna", + "itoa", + "jsonschema-regex", + "num-cmp", + "num-traits", + "percent-encoding", + "referencing", + "regex", + "serde", + "serde_json", + "unicode-general-category", + "uuid-simd", +] + +[[package]] +name = "jsonschema-regex" +version = "0.46.10" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6dbd1086b01b9349fd4ef9a07433965af64c8ce8159abe633a189e4ff817bd13" +dependencies = [ + "regex-syntax", +] + [[package]] name = "k256" version = "0.13.4" @@ -3656,7 +3844,7 @@ version = "1.5.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "bbd2bcb4c963f2ddae06a2efc7e9f3591312473c50c6685e1f298068316e66fe" dependencies = [ - "spin", + "spin 0.9.8", ] [[package]] @@ -3671,6 +3859,16 @@ version = "0.2.186" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "68ab91017fe16c622486840e4c83c9a37afeff978bd239b5293d61ece587de66" +[[package]] +name = "libloading" +version = "0.8.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d7c4b02199fee7c5d21a5ae7d8cfa79a6ef5bb2fc834d6e9058e89c825efdc55" +dependencies = [ + "cfg-if", + "windows-link 0.2.1", +] + [[package]] name = "libm" version = "0.2.16" @@ -3755,6 +3953,12 @@ dependencies = [ "tracing-subscriber", ] +[[package]] +name = "lru" +version = "0.18.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0b6180140927ee907000b0aa540091f6ea512ead4447c92b8fc35bc72788a5a6" + [[package]] name = "lru-slab" version = "0.1.2" @@ -3852,6 +4056,12 @@ dependencies = [ "zeroize", ] +[[package]] +name = "micromap" +version = "0.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c2a86d3146ed3995b5913c414f6664344b9617457320782e64f0bb44afd49d74" + [[package]] name = "mime" version = "0.3.17" @@ -3935,6 +4145,15 @@ dependencies = [ "uuid", ] +[[package]] +name = "msvc_spectre_libs" +version = "0.1.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "29e871a9861f3664f18b7e04e9301d4edd55090c2dadb4b1c602e26ab32b1f5b" +dependencies = [ + "cc", +] + [[package]] name = "multer" version = "3.1.0" @@ -3949,7 +4168,7 @@ dependencies = [ "log", "memchr", "mime", - "spin", + "spin 0.9.8", "tokio", "tokio-util", "version_check", @@ -4160,6 +4379,20 @@ dependencies = [ "windows-sys 0.61.2", ] +[[package]] +name = "num" +version = "0.4.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "35bd024e8b2ff75562e5f34e7f4905839deb4b22955ef5e73d2fea1b9813cb23" +dependencies = [ + "num-bigint", + "num-complex", + "num-integer", + "num-iter", + "num-rational", + "num-traits", +] + [[package]] name = "num-bigint" version = "0.4.6" @@ -4186,6 +4419,21 @@ dependencies = [ "zeroize", ] +[[package]] +name = "num-cmp" +version = "0.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "63335b2e2c34fae2fb0aa2cecfd9f0832a1e24b3b32ecec612c3426d46dc8aaa" + +[[package]] +name = "num-complex" +version = "0.4.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "73f88a1307638156682bada9d7604135552957b7818057dcef22705b4d509495" +dependencies = [ + "num-traits", +] + [[package]] name = "num-conv" version = "0.2.2" @@ -4218,6 +4466,7 @@ version = "0.4.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "f83d14da390562dca69fc84082e73e548e1ad308d24accdedd2720017cb37824" dependencies = [ + "num-bigint", "num-integer", "num-traits", ] @@ -4242,6 +4491,29 @@ dependencies = [ "libc", ] +[[package]] +name = "nvml-wrapper" +version = "0.12.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f049ae562349fefb8e837eb15443da1e7c6dcbd8a11f52a228f92220c2e5c85e" +dependencies = [ + "bitflags 2.11.1", + "libloading", + "nvml-wrapper-sys", + "static_assertions", + "thiserror 1.0.69", + "wrapcenum-derive", +] + +[[package]] +name = "nvml-wrapper-sys" +version = "0.9.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6b4d594420fcda43b1c2c4bd44d48974aa3c7a9ab2cbf10dc18e35265767bf0b" +dependencies = [ + "libloading", +] + [[package]] name = "objc2-core-foundation" version = "0.3.2" @@ -4342,6 +4614,12 @@ dependencies = [ "windows-sys 0.61.2", ] +[[package]] +name = "outref" +version = "0.5.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1a80800c0488c3a21695ea981a54918fbb37abf04f4d0720c453632255e2ff0e" + [[package]] name = "p256" version = "0.13.2" @@ -5335,6 +5613,23 @@ version = "0.1.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "09c30c54dffee5b40af088d5d50aa3455c91a0127164b51f0215efc4cb28fb3c" +[[package]] +name = "referencing" +version = "0.46.10" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0fbf332a2f81899f6836f22c03da73dae8a664c32e3016b84692c23cddadc95d" +dependencies = [ + "ahash", + "fluent-uri", + "getrandom 0.3.4", + "hashbrown 0.16.1", + "itoa", + "micromap", + "parking_lot 0.12.5", + "percent-encoding", + "serde_json", +] + [[package]] name = "regex" version = "1.12.3" @@ -5364,6 +5659,38 @@ version = "0.8.10" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "dc897dd8d9e8bd1ed8cdad82b5966c3e0ecae09fb1907d58efaa013543185d0a" +[[package]] +name = "regorus" +version = "0.10.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "419a0413adeece71e4d4a64fb75adc359cb807496f0dfd10f429517be908b807" +dependencies = [ + "anyhow", + "chrono", + "chrono-tz", + "data-encoding", + "globset", + "indexmap 2.14.0", + "ipnet", + "jsonschema", + "lazy_static", + "lru", + "msvc_spectre_libs", + "num-bigint", + "num-traits", + "parking_lot 0.12.5", + "rand 0.10.1", + "regex", + "semver", + "serde", + "serde_json", + "serde_yaml", + "spin 0.10.1", + "thiserror 2.0.18", + "url", + "uuid", +] + [[package]] name = "reqwest" version = "0.13.4" @@ -6299,6 +6626,19 @@ dependencies = [ "time", ] +[[package]] +name = "serde_yaml" +version = "0.9.34+deprecated" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6a8b1a1a2ebf674015cc02edccce75287f1a0130d394307b36743c2f5d504b47" +dependencies = [ + "indexmap 2.14.0", + "itoa", + "ryu", + "serde", + "unsafe-libyaml", +] + [[package]] name = "sev" version = "6.0.0" @@ -6578,6 +6918,12 @@ version = "0.9.8" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "6980e8d7511241f8acf4aebddbb1ff938df5eebe98691418c4468d0b72a96a67" +[[package]] +name = "spin" +version = "0.10.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "023a211cb3138dbc438680b32560ad89f699977624c9f8dbb95a47d5b4c07dd3" + [[package]] name = "spki" version = "0.7.3" @@ -7394,6 +7740,12 @@ version = "2.9.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "dbc4bc3a9f746d862c45cb89d705aa10f187bb96c76001afab07a0d35ce60142" +[[package]] +name = "unicode-general-category" +version = "1.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0b993bddc193ae5bd0d623b49ec06ac3e9312875fdae725a975c51db1cc1677f" + [[package]] name = "unicode-ident" version = "1.0.24" @@ -7422,6 +7774,12 @@ dependencies = [ "subtle", ] +[[package]] +name = "unsafe-libyaml" +version = "0.2.11" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "673aac59facbab8a9007c7f6108d11f63b603f7cabff99fabf650fea5c32b861" + [[package]] name = "untrusted" version = "0.7.1" @@ -7478,10 +7836,21 @@ checksum = "d258b83ceec21034727ecee8c382cfa6c3e133699b0742c64571814fb420c9f7" dependencies = [ "getrandom 0.4.2", "js-sys", + "rand 0.10.1", "serde_core", "wasm-bindgen", ] +[[package]] +name = "uuid-simd" +version = "0.8.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "23b082222b4f6619906941c17eb2297fff4c2fb96cb60164170522942a200bd8" +dependencies = [ + "outref", + "vsimd", +] + [[package]] name = "valuable" version = "0.1.1" @@ -7506,6 +7875,12 @@ version = "1.0.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "6a02e4885ed3bc0f2de90ea6dd45ebcbb66dacffe03547fadbb0eeae2770887d" +[[package]] +name = "vsimd" +version = "0.8.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5c3082ca00d5a5ef149bb8b555a72ae84c9c59f7250f013ac822ac2e49b19c64" + [[package]] name = "vsock" version = "0.5.4" @@ -8280,6 +8655,18 @@ dependencies = [ "wasmparser", ] +[[package]] +name = "wrapcenum-derive" +version = "0.4.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a76ff259533532054cfbaefb115c613203c73707017459206380f03b3b3f266e" +dependencies = [ + "darling 0.20.11", + "proc-macro2", + "quote", + "syn 2.0.117", +] + [[package]] name = "writeable" version = "0.6.3" diff --git a/dstack/Cargo.toml b/dstack/Cargo.toml index eb6f8f6d6..265eae995 100644 --- a/dstack/Cargo.toml +++ b/dstack/Cargo.toml @@ -127,7 +127,9 @@ libc = "0.2.171" log = "0.4.26" moka = { version = "0.12.15", default-features = false, features = ["sync"] } notify = "8.0.0" +nvml-wrapper = "0.12.1" rand = "0.8.5" +regorus = { version = "0.10.1", default-features = false, features = ["full-opa", "arc"] } tracing = "0.1.40" tracing-subscriber = { version = "0.3.20", features = ["env-filter"] } safe-write = "0.1.2" diff --git a/dstack/dstack-attest/src/v1.rs b/dstack/dstack-attest/src/v1.rs index 7df502218..f201b9c73 100644 --- a/dstack/dstack-attest/src/v1.rs +++ b/dstack/dstack-attest/src/v1.rs @@ -377,6 +377,7 @@ mod tests { MrConfigV3::new( vec![0x11; 20], vec![0x22; 32], + None, dstack_types::KeyProviderKind::None, Vec::new(), vec![0x33; 20], diff --git a/dstack/dstack-attest/tests/sev_snp_verify.rs b/dstack/dstack-attest/tests/sev_snp_verify.rs index 8fa4c3317..6fded651f 100644 --- a/dstack/dstack-attest/tests/sev_snp_verify.rs +++ b/dstack/dstack-attest/tests/sev_snp_verify.rs @@ -352,6 +352,7 @@ fn substituted_mr_config_breaks_host_data_binding() { let evil = MrConfigV3::new( vec![0xab; 20], vec![0xcd; 32], + None, KeyProviderKind::None, Vec::new(), vec![0xef; 20], diff --git a/dstack/dstack-mr/src/sev.rs b/dstack/dstack-mr/src/sev.rs index 157d083bf..34dd2ab53 100644 --- a/dstack/dstack-mr/src/sev.rs +++ b/dstack/dstack-mr/src/sev.rs @@ -913,6 +913,9 @@ pub fn validate_mr_config(mr_config: &MrConfigV3) -> Result<()> { } ensure_len("mr_config.app_id", &mr_config.app_id, 20)?; ensure_len("mr_config.compose_hash", &mr_config.compose_hash, 32)?; + if let Some(gpu_policy_hash) = &mr_config.gpu_policy_hash { + ensure_len("mr_config.gpu_policy_hash", gpu_policy_hash, 32)?; + } if !mr_config.instance_id.is_empty() { ensure_len("mr_config.instance_id", &mr_config.instance_id, 20)?; } @@ -1476,6 +1479,7 @@ mod tests { MrConfigV3::new( vec![0x11; 20], vec![0x22; 32], + None, dstack_types::KeyProviderKind::None, Vec::new(), vec![0x33; 20], @@ -1616,6 +1620,7 @@ mod tests { MrConfigV3::new( vec![0xee; 20], vec![0x22; 32], + None, dstack_types::KeyProviderKind::None, Vec::new(), vec![0x33; 20], @@ -1623,6 +1628,7 @@ mod tests { MrConfigV3::new( vec![0x11; 20], vec![0xee; 32], + None, dstack_types::KeyProviderKind::None, Vec::new(), vec![0x33; 20], @@ -1630,10 +1636,19 @@ mod tests { MrConfigV3::new( vec![0x11; 20], vec![0x22; 32], + None, dstack_types::KeyProviderKind::None, Vec::new(), vec![0xee; 20], ), + MrConfigV3::new( + vec![0x11; 20], + vec![0x22; 32], + Some(vec![0xee; 32]), + dstack_types::KeyProviderKind::None, + Vec::new(), + vec![0x33; 20], + ), ]; for evil in evil_mr_configs { let vm_config = synthetic_vm_config(&input, &evil); diff --git a/dstack/dstack-types/src/lib.rs b/dstack/dstack-types/src/lib.rs index a6fe1901f..87893679a 100644 --- a/dstack/dstack-types/src/lib.rs +++ b/dstack/dstack-types/src/lib.rs @@ -121,6 +121,67 @@ pub struct AppCompose { pub requirements: Option, } +/// Canonical source for the policy used when `requirements.gpu_policy` is +/// absent. Both typed defaults and measurement are derived from this JSON. +pub const DEFAULT_GPU_POLICY: &str = "{}"; + +/// Path containing the complete output of the NVIDIA GPU attestation command. +pub const GPU_ATTESTATION_OUTPUT: &str = "/run/nvidia-gpu-attestation/attestation.out"; + +/// Computes the SHA-256 digest of the JCS-canonicalized raw +/// `requirements.gpu_policy` JSON value. An absent policy is equivalent to +/// the default empty object. +pub fn gpu_policy_hash(compose_json: &[u8]) -> Result<[u8; 32], serde_json::Error> { + use sha2::{Digest, Sha256}; + + let compose: serde_json::Value = serde_json::from_slice(compose_json)?; + let default_policy: serde_json::Value = serde_json::from_str(DEFAULT_GPU_POLICY)?; + let policy = compose + .pointer("/requirements/gpu_policy") + .unwrap_or(&default_policy); + let canonical = serde_jcs::to_vec(policy)?; + Ok(Sha256::digest(canonical).into()) +} + +#[derive(Deserialize, Serialize, Debug, Clone, PartialEq, Eq)] +#[serde(deny_unknown_fields)] +pub struct GpuPolicy { + /// Whether an attached GPU must pass local TEE attestation before the + /// guest continues booting. Defaults to true. + #[serde(default = "default_true")] + pub attest_gpu: bool, + /// Optional Rego v0 policy evaluated against NVIDIA nvattest's `claims` + /// array. It must define the boolean rule `data.policy.nv_match`. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub rego: Option, + /// Permit NVIDIA DevTools mode. This defaults to false because DevTools + /// disables the GPU memory-confidentiality guarantees expected in + /// production. + #[serde(default)] + pub allow_devtools: bool, + /// Permit claims whose GPU attestation debug status is `enabled`. Defaults + /// to false. + #[serde(default)] + pub allow_debug: bool, + /// Permit claims that do not assert GPU secure boot. Defaults to false. + #[serde(default)] + pub allow_insecure_boot: bool, +} + +impl Default for GpuPolicy { + fn default() -> Self { + serde_json::from_str(DEFAULT_GPU_POLICY) + .or_panic("DEFAULT_GPU_POLICY must be a valid GPU policy") + } +} + +impl GpuPolicy { + /// Returns true when no application-specific GPU policy setting is set. + pub fn is_default(&self) -> bool { + self == &Self::default() + } +} + #[derive(Deserialize, Serialize, Debug, Clone, Default, PartialEq, Eq)] #[serde(default, deny_unknown_fields)] pub struct Requirements { @@ -149,16 +210,17 @@ pub struct Requirements { /// (e.g. 32 random alphanumeric characters). #[serde(skip_serializing_if = "Option::is_none")] pub launch_token_hash: Option, - /// GPU TEE attestation requirement, defaults to `true` when omitted. + /// Application GPU policy applied before key provisioning. An omitted + /// field is parsed and measured as the default empty policy `{}`. /// - /// On guests with an NVIDIA GPU attached, `true` means the guest runs - /// local GPU attestation (nvattest) during system setup and refuses to - /// boot — before key provisioning — if the GPU fails to attest (e.g. a - /// non-CC GPU or CC mode disabled by the host). `false` skips attestation - /// and sets the GPU ready state directly. Guests without a GPU attached - /// are unaffected either way. - #[serde(skip_serializing_if = "Option::is_none")] - pub attest_gpu: Option, + /// Its original JSON value is JCS-canonicalized and its SHA-256 digest is + /// emitted as the `gpu-policy-hash` launch event immediately after + /// `compose-hash`. When the field is absent, `{}` is measured. An explicitly + /// present default-valued field remains part of the raw measurement. + /// Rego receives an empty claims array when no GPU attestation is produced, + /// allowing applications to enforce an expected GPU count. + #[serde(default, skip_serializing_if = "GpuPolicy::is_default")] + pub gpu_policy: GpuPolicy, } impl Requirements { @@ -167,7 +229,7 @@ impl Requirements { && self.platforms.is_none() && self.tdx_measure_acpi_tables.is_none() && self.launch_token_hash.is_none() - && self.attest_gpu.is_none() + && self.gpu_policy.is_default() } } @@ -362,24 +424,6 @@ impl AppCompose { } } } - - /// Whether an attached GPU must pass local TEE attestation before the - /// guest continues booting. Defaults to `true` when - /// `requirements.attest_gpu` is omitted. - /// - /// `requirements` are only valid on manifest_version >= 3 (guests reject - /// older manifests carrying them); the opt-out is additionally ignored on - /// legacy manifests here so a caller that skipped that validation still - /// fails closed. - pub fn attest_gpu(&self) -> bool { - if !matches!(self.manifest_version_u32(), Some(v) if v >= 3) { - return true; - } - self.requirements - .as_ref() - .and_then(|r| r.attest_gpu) - .unwrap_or(true) - } } #[cfg(test)] @@ -455,7 +499,13 @@ mod app_compose_tests { "os_version": ">=0.6.1", "platforms": ["dstack-gcp-tdx", "dstack-tdx"], "tdx_measure_acpi_tables": true, - "launch_token_hash": "9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08" + "launch_token_hash": "9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08", + "gpu_policy": { + "rego": "package policy\n\ndefault nv_match = false\n", + "allow_devtools": true, + "allow_debug": true, + "allow_insecure_boot": true + } } })) .unwrap(); @@ -470,6 +520,15 @@ mod app_compose_tests { requirements.launch_token_hash.as_deref(), Some("9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08") ); + let gpu_policy = &requirements.gpu_policy; + assert!(gpu_policy.attest_gpu); + assert_eq!( + gpu_policy.rego.as_deref(), + Some("package policy\n\ndefault nv_match = false\n") + ); + assert!(gpu_policy.allow_devtools); + assert!(gpu_policy.allow_debug); + assert!(gpu_policy.allow_insecure_boot); let err = serde_json::from_value::(serde_json::json!({ "manifest_version": "3", @@ -494,7 +553,10 @@ mod app_compose_tests { .unwrap(); let requirements = omitted.requirements.as_ref().unwrap(); assert_eq!(requirements.platforms, None); + assert!(requirements.gpu_policy.is_default()); assert!(requirements.is_empty()); + let serialized = serde_json::to_value(requirements).unwrap(); + assert!(serialized.get("gpu_policy").is_none()); let explicit_empty: AppCompose = serde_json::from_value(serde_json::json!({ "manifest_version": "3", @@ -534,54 +596,96 @@ mod app_compose_tests { let requirements = launch_token.requirements.as_ref().unwrap(); assert!(requirements.launch_token_hash.is_some()); assert!(!requirements.is_empty()); + + let gpu_policy: AppCompose = serde_json::from_value(serde_json::json!({ + "manifest_version": "3", + "name": "test", + "runner": "docker-compose", + "requirements": { + "gpu_policy": { + "rego": "package policy\n\ndefault nv_match = false\n" + } + } + })) + .unwrap(); + let requirements = gpu_policy.requirements.as_ref().unwrap(); + let gpu_policy = &requirements.gpu_policy; + assert!(gpu_policy.attest_gpu); + assert!(gpu_policy.rego.is_some()); + assert!(!gpu_policy.allow_devtools); + assert!(!gpu_policy.allow_debug); + assert!(!gpu_policy.allow_insecure_boot); + assert!(!requirements.is_empty()); + + let err = serde_json::from_value::(serde_json::json!({ + "manifest_version": "3", + "name": "test", + "runner": "docker-compose", + "requirements": { + "gpu_policy": { + "rego": "package policy", + "allow_debugger": true + } + } + })) + .unwrap_err(); + assert!(err.to_string().contains("unknown field")); } #[test] fn attest_gpu_defaults_to_true() { - let no_requirements: AppCompose = serde_json::from_value(serde_json::json!({ - "manifest_version": 2, + assert!(GpuPolicy::default().attest_gpu); + + let omitted: AppCompose = serde_json::from_value(serde_json::json!({ + "manifest_version": "3", "name": "test", - "runner": "docker-compose" + "runner": "docker-compose", + "requirements": {} })) .unwrap(); - assert!(no_requirements.attest_gpu()); + let requirements = omitted.requirements.as_ref().unwrap(); + assert!(requirements.gpu_policy.attest_gpu); + assert!(requirements.is_empty()); - let omitted: AppCompose = serde_json::from_value(serde_json::json!({ + let explicit_empty: AppCompose = serde_json::from_value(serde_json::json!({ "manifest_version": "3", "name": "test", "runner": "docker-compose", - "requirements": {} + "requirements": { + "gpu_policy": {} + } })) .unwrap(); - assert!(omitted.attest_gpu()); - assert!(omitted.requirements.as_ref().unwrap().is_empty()); + assert_eq!( + explicit_empty.requirements.unwrap().gpu_policy, + requirements.gpu_policy + ); let disabled: AppCompose = serde_json::from_value(serde_json::json!({ "manifest_version": "3", "name": "test", "runner": "docker-compose", "requirements": { - "attest_gpu": false + "gpu_policy": { + "attest_gpu": false + } } })) .unwrap(); - assert!(!disabled.attest_gpu()); let requirements = disabled.requirements.as_ref().unwrap(); - assert_eq!(requirements.attest_gpu, Some(false)); + assert!(!requirements.gpu_policy.attest_gpu); assert!(!requirements.is_empty()); - // The opt-out is ignored on legacy manifests (requirements are only - // valid on manifest_version >= 3; guests reject such composes anyway). - let legacy_optout: AppCompose = serde_json::from_value(serde_json::json!({ - "manifest_version": 2, + let old_location = serde_json::from_value::(serde_json::json!({ + "manifest_version": "3", "name": "test", "runner": "docker-compose", "requirements": { "attest_gpu": false } })) - .unwrap(); - assert!(legacy_optout.attest_gpu()); + .unwrap_err(); + assert!(old_location.to_string().contains("unknown field")); } #[test] diff --git a/dstack/dstack-types/src/mr_config.rs b/dstack/dstack-types/src/mr_config.rs index f5f8d09de..846f4f746 100644 --- a/dstack/dstack-types/src/mr_config.rs +++ b/dstack/dstack-types/src/mr_config.rs @@ -109,6 +109,10 @@ pub struct MrConfigV3 { pub app_id: Vec, #[serde(with = "hex_bytes")] pub compose_hash: Vec, + /// Hash of the raw application GPU policy. GPU launches populate it; + /// non-GPU and historical v3 launch documents omit it. + #[serde(default, skip_serializing_if = "Option::is_none", with = "hex_bytes")] + pub gpu_policy_hash: Option>, pub key_provider: KeyProviderKind, #[serde(default, with = "hex_bytes")] pub key_provider_id: Vec, @@ -120,6 +124,7 @@ impl MrConfigV3 { pub fn new( app_id: Vec, compose_hash: Vec, + gpu_policy_hash: Option>, key_provider: KeyProviderKind, key_provider_id: Vec, instance_id: Vec, @@ -128,6 +133,7 @@ impl MrConfigV3 { version: mr_config_v3_version(), app_id, compose_hash, + gpu_policy_hash, key_provider, key_provider_id, instance_id, @@ -191,6 +197,7 @@ mod tests { let config = MrConfigV3::new( vec![0x11; 20], vec![0x22; 32], + Some(vec![0x55; 32]), KeyProviderKind::Kms, vec![0x33; 32], vec![0x44; 20], @@ -202,6 +209,30 @@ mod tests { assert_eq!(config.to_snp_host_data().len(), 32); assert_ne!(config.to_tdx_mr_config_id(), changed.to_tdx_mr_config_id()); assert_eq!(config.to_tdx_mr_config_id()[0], 3); + + let mut changed = config.clone(); + if let Some(gpu_policy_hash) = &mut changed.gpu_policy_hash { + gpu_policy_hash[0] ^= 0xff; + } + assert_ne!(config.to_snp_host_data(), changed.to_snp_host_data()); + assert_ne!(config.to_tdx_mr_config_id(), changed.to_tdx_mr_config_id()); + } + + #[test] + fn mr_config_v3_omits_gpu_policy_hash_for_non_gpu_launches() -> Result<(), Box> { + let config = MrConfigV3::new( + vec![0x11; 20], + vec![0x22; 32], + None, + KeyProviderKind::None, + Vec::new(), + vec![0x44; 20], + ); + let document = config.to_canonical_json(); + + assert!(!document.contains("gpu_policy_hash")); + assert_eq!(MrConfigV3::from_document(&document)?, config); + Ok(()) } #[test] @@ -209,6 +240,7 @@ mod tests { let config = MrConfigV3::new( vec![0x11; 20], vec![0x22; 32], + Some(vec![0x55; 32]), KeyProviderKind::Kms, vec![0x33; 32], vec![0x44; 20], @@ -220,6 +252,7 @@ mod tests { concat!( "{\"app_id\":\"1111111111111111111111111111111111111111\",", "\"compose_hash\":\"2222222222222222222222222222222222222222222222222222222222222222\",", + "\"gpu_policy_hash\":\"5555555555555555555555555555555555555555555555555555555555555555\",", "\"instance_id\":\"4444444444444444444444444444444444444444\",", "\"key_provider\":\"kms\",", "\"key_provider_id\":\"3333333333333333333333333333333333333333333333333333333333333333\",", diff --git a/dstack/dstack-util/Cargo.toml b/dstack/dstack-util/Cargo.toml index e70b58fd0..8f63daaf9 100644 --- a/dstack/dstack-util/Cargo.toml +++ b/dstack/dstack-util/Cargo.toml @@ -46,6 +46,7 @@ dcap-qvl.workspace = true k256 = { workspace = true, features = ["ecdsa"] } dstack-types.workspace = true rand.workspace = true +regorus.workspace = true sha3.workspace = true dstack-attest.workspace = true cert-client.workspace = true @@ -55,6 +56,7 @@ bollard.workspace = true sodiumbox.workspace = true libc.workspace = true luks2.workspace = true +nvml-wrapper.workspace = true scopeguard.workspace = true tempfile.workspace = true ez-hash.workspace = true diff --git a/dstack/dstack-util/src/system_setup.rs b/dstack/dstack-util/src/system_setup.rs index 73823d254..225ea4e38 100644 --- a/dstack/dstack-util/src/system_setup.rs +++ b/dstack/dstack-util/src/system_setup.rs @@ -16,11 +16,12 @@ use anyhow::{anyhow, bail, Context, Result}; use dstack_attest::emit_runtime_event; use dstack_kms_rpc as rpc; use dstack_types::{ + gpu_policy_hash, shared_filenames::{ APP_COMPOSE, APP_KEYS, DECRYPTED_ENV, DECRYPTED_ENV_JSON, ENCRYPTED_ENV, HOST_SHARED_DIR_NAME, HOST_SHARED_DISK_LABEL, INSTANCE_INFO, SYS_CONFIG, USER_CONFIG, }, - KeyProvider, KeyProviderInfo, + GpuPolicy, KeyProvider, KeyProviderInfo, GPU_ATTESTATION_OUTPUT, }; use fs_err as fs; use luks2::{ @@ -1060,48 +1061,163 @@ async fn do_sys_setup(stage0: Stage0<'_>) -> Result<()> { } else { info!("System time will be synchronized by chronyd in background"); } - stage0 - .setup_gpu() - .await - .context("Failed to verify GPU TEE attestation")?; let stage1 = stage0.setup_fs().await?; stage1.setup().await } -/// GPU TEE attestation gate (`requirements.attest_gpu`, defaults to true). +/// GPU TEE attestation gate (`requirements.gpu_policy.attest_gpu`, defaults to +/// true). /// /// Runs before key provisioning so a CVM whose GPU cannot prove it is a -/// genuine, CC-enabled NVIDIA TEE never gets its app keys. The GPU -/// "ready" state (`nvidia-smi conf-compute -srs 1`) is only set from here — -/// nvidia-persistenced deliberately does not set it — so CUDA work cannot be -/// submitted to an unverified GPU either. +/// genuine, CC-enabled NVIDIA TEE never gets its app keys. An optional +/// application policy is measured and evaluated after `compose-hash`. The GPU +/// "ready" state is only set through NVML from here — nvidia-persistenced +/// deliberately does not set it — so CUDA work cannot be submitted to an +/// unverified GPU either. mod gpu { use super::*; const NVATTEST: &str = "/usr/bin/nvattest"; - const NVIDIA_SMI: &str = "/usr/bin/nvidia-smi"; - const ATTESTATION_OUTPUT: &str = "/run/nvidia-gpu-attestation/attestation.out"; const ATTESTATION_TIMEOUT: Duration = Duration::from_secs(300); - const NVIDIA_SMI_TIMEOUT: Duration = Duration::from_secs(60); - - /// True if a passed-through NVIDIA GPU is present, detected via sysfs PCI - /// (vendor 0x10de, class VGA 0x0300xx or 3D controller 0x0302xx) so it - /// works even before the nvidia driver is loaded. Fails (rather than - /// reporting "no GPU") when the PCI bus cannot be enumerated, so a broken - /// /sys cannot bypass the attestation gate. - pub(super) fn nvidia_gpu_present() -> Result { - let entries = - fs::read_dir("/sys/bus/pci/devices").context("failed to enumerate PCI devices")?; - Ok(entries.filter_map(|e| e.ok()).any(|dev| { - let read = |name: &str| { - fs::read_to_string(dev.path().join(name)) - .unwrap_or_default() - .trim() - .to_string() - }; - read("vendor") == "0x10de" - && matches!(read("class").get(..6), Some("0x0300") | Some("0x0302")) - })) + const EVENT_VERSION: u32 = 2; + const POLICY_ENTRYPOINT: &str = "data.policy.nv_match"; + /// Bound Rego evaluation so a runaway application policy cannot hang boot. + const POLICY_TIMEOUT: Duration = Duration::from_secs(10); + + #[derive(Debug, Clone, Copy, PartialEq, Eq)] + pub(super) struct GpuInventory { + pub(super) total: u32, + pub(super) nvidia: u32, + } + + #[derive(Debug, Serialize)] + struct GpuAttestationEvent { + version: u32, + provider: &'static str, + devices: u32, + cc_mode: &'static str, + devtools: bool, + evidence_sha256: String, + } + + pub(super) struct GpuAttestationResult { + claims: Vec, + parsed_claims: Vec, + output: Vec, + devices: u32, + } + + impl GpuAttestationResult { + pub(super) fn claims(&self) -> &[Value] { + &self.claims + } + + pub(super) fn event(&self, devtools: bool) -> Result> { + attestation_event(&self.output, self.devices, devtools) + } + + pub(super) fn verify_claim_policy( + &self, + state: &GpuState, + policy: &GpuPolicy, + ) -> Result<()> { + verify_claim_policy(&self.parsed_claims, &state.devices, policy) + } + } + + pub(super) struct GpuState { + nvml: nvml_wrapper::Nvml, + devices: Vec, + } + + impl GpuState { + pub(super) fn any_devtools(&self) -> bool { + self.devices.iter().any(|device| device.devtools) + } + + pub(super) fn set_ready(&self) -> Result<()> { + set_gpu_ready_state_with_nvml(&self.nvml) + } + } + + #[derive(Debug, Clone, Copy)] + struct GpuDeviceState { + cc_enabled: bool, + devtools: bool, + } + + #[derive(Deserialize)] + struct NvattestOutput { + result_code: i64, + claims: Vec, + } + + #[derive(Debug, Deserialize)] + struct NvidiaGpuClaim { + #[serde(rename = "x-nvidia-device-type")] + device_type: String, + eat_nonce: String, + #[serde(rename = "x-nvidia-gpu-attestation-report-nonce-match")] + nonce_match: bool, + measres: String, + secboot: bool, + dbgstat: NvidiaGpuDebugStatus, + } + + #[derive(Debug, Deserialize, PartialEq, Eq)] + #[serde(rename_all = "lowercase")] + enum NvidiaGpuDebugStatus { + Disabled, + Enabled, + } + + struct ValidatedClaims { + raw: Vec, + parsed: Vec, + } + + /// Count passed-through display-class GPUs through sysfs so devices which + /// the NVIDIA driver did not bind cannot be hidden from the gate. Reading + /// the inventory is fail-closed: a mixed NVIDIA/non-NVIDIA set must not be + /// represented by an attestation result for only the NVIDIA subset. + pub(super) fn gpu_inventory() -> Result { + gpu_inventory_at(Path::new("/sys/bus/pci/devices")) + } + + fn gpu_inventory_at(devices_path: &Path) -> Result { + let entries = fs::read_dir(devices_path).context("failed to enumerate PCI devices")?; + let mut inventory = GpuInventory { + total: 0, + nvidia: 0, + }; + for entry in entries { + let device = entry.context("failed to read PCI device entry")?; + let class_path = device.path().join("class"); + let class = fs::read_to_string(&class_path) + .with_context(|| format!("failed to read {}", class_path.display()))?; + if !matches!(class.trim().get(..6), Some("0x0300") | Some("0x0302")) { + continue; + } + inventory.total += 1; + let vendor_path = device.path().join("vendor"); + let vendor = fs::read_to_string(&vendor_path) + .with_context(|| format!("failed to read {}", vendor_path.display()))?; + if vendor.trim() == "0x10de" { + inventory.nvidia += 1; + } + } + Ok(inventory) + } + + pub(super) fn nvidia_gpu_count(inventory: GpuInventory) -> Result { + if inventory.total != inventory.nvidia { + bail!( + "unsupported non-NVIDIA GPU attached: found {} display GPUs, {} NVIDIA", + inventory.total, + inventory.nvidia + ); + } + Ok(inventory.nvidia) } /// Run a GPU tool with a bounded timeout so a wedged driver/GPU cannot @@ -1121,31 +1237,142 @@ mod gpu { .with_context(|| format!("failed to run {program}")) } - /// Mark the GPU as ready to accept work. Only meaningful (and only - /// succeeds) when the GPU runs in CC mode. - pub(super) async fn set_gpu_ready_state() -> Result<()> { - let output = run_command( - NVIDIA_SMI, - &["conf-compute", "-srs", "1"], - NVIDIA_SMI_TIMEOUT, - ) - .await?; - if !output.status.success() { + fn init_nvml(expected_devices: u32) -> Result { + let nvml = nvml_wrapper::Nvml::init().context("failed to initialize NVML")?; + let devices = nvml + .device_count() + .context("failed to get NVML GPU count")?; + if devices != expected_devices { + bail!("nvml GPU count mismatch: expected {expected_devices}, got {devices}"); + } + Ok(nvml) + } + + fn set_gpu_ready_state_with_nvml(nvml: &nvml_wrapper::Nvml) -> Result<()> { + // nvml-wrapper exposes nvmlSystemSetConfComputeGpusReadyState through + // Device, but the transition applies to all CC GPUs in the system. + let first = nvml + .device_by_index(0) + .context("failed to get first NVML GPU")?; + first + .set_confidential_compute_state(true) + .context("failed to set GPU ready state")?; + info!("GPU ready state set"); + Ok(()) + } + + /// Read the CC and DevTools state through NVML for every expected GPU. + /// NVML exposes these settings as system values through Device methods; + /// call them for every handle so every expected device must be enumerable. + pub(super) fn query_gpu_state(expected_devices: u32) -> Result { + let nvml = init_nvml(expected_devices)?; + let mut devices = Vec::with_capacity(expected_devices as usize); + for index in 0..expected_devices { + let device = nvml + .device_by_index(index) + .with_context(|| format!("failed to get NVML GPU at index {index}"))?; + let cc_enabled = device + .is_cc_enabled() + .with_context(|| format!("failed to query CC mode for GPU at index {index}"))?; + let devtools = device.is_cc_dev_mode_enabled().with_context(|| { + format!("failed to query DevTools mode for GPU at index {index}") + })?; + devices.push(GpuDeviceState { + cc_enabled, + devtools, + }); + } + Ok(GpuState { nvml, devices }) + } + + /// Set the system-wide GPU ready state without appraisal for the explicit + /// `gpu_policy.attest_gpu: false` compatibility path. + pub(super) fn set_gpu_ready_state(expected_devices: u32) -> Result<()> { + let nvml = init_nvml(expected_devices)?; + set_gpu_ready_state_with_nvml(&nvml) + } + + fn validate_attestation_output( + stdout: &[u8], + nonce: &str, + expected_devices: u32, + ) -> Result { + let output: NvattestOutput = + serde_json::from_slice(stdout).context("failed to parse nvattest JSON output")?; + if output.result_code != 0 { bail!( - "nvidia-smi conf-compute -srs 1 failed ({}): {}", - output.status, - truncated_lossy(&output.stderr, 512), + "nvattest JSON result is not successful (result_code={})", + output.result_code ); } - info!("GPU ready state set"); + if output.claims.len() != expected_devices as usize { + bail!( + "gpu attestation count mismatch: expected {expected_devices}, got {}", + output.claims.len() + ); + } + let mut parsed_claims = Vec::with_capacity(output.claims.len()); + for (index, claim) in output.claims.iter().enumerate() { + let claim: NvidiaGpuClaim = serde_json::from_value(claim.clone()) + .with_context(|| format!("invalid GPU claim at index {index}"))?; + if claim.device_type != "gpu" { + bail!("gpu claim at index {index} has an invalid device type"); + } + if claim.eat_nonce != nonce || !claim.nonce_match { + bail!("gpu claim at index {index} has an invalid nonce"); + } + parsed_claims.push(claim); + } + Ok(ValidatedClaims { + raw: output.claims, + parsed: parsed_claims, + }) + } + + fn verify_claim_policy( + claims: &[NvidiaGpuClaim], + devices: &[GpuDeviceState], + policy: &GpuPolicy, + ) -> Result<()> { + for (index, device) in devices.iter().enumerate() { + if !device.cc_enabled { + bail!("gpu at index {index} does not enable confidential compute mode"); + } + if device.devtools && !policy.allow_devtools { + bail!("gpu at index {index} enables NVIDIA DevTools mode"); + } + } + for (index, claim) in claims.iter().enumerate() { + if claim.measres != "success" { + bail!("gpu claim at index {index} has unsuccessful measurements"); + } + if !policy.allow_insecure_boot && !claim.secboot { + bail!("gpu claim at index {index} does not assert secure boot"); + } + if !policy.allow_debug && claim.dbgstat != NvidiaGpuDebugStatus::Disabled { + bail!("gpu claim at index {index} does not disable debug mode"); + } + } Ok(()) } - /// Run local GPU attestation via nvattest with a fresh nonce, keeping the - /// verifier output in /run for debugging. Fails on any non-zero exit — - /// including a GPU that cannot produce an attestation report at all (a - /// non-CC GPU, or CC mode left off by the host). - pub(super) async fn attest_gpu() -> Result<()> { + fn attestation_event(stdout: &[u8], devices: u32, devtools: bool) -> Result> { + let event = GpuAttestationEvent { + version: EVENT_VERSION, + provider: "nvidia", + devices, + cc_mode: "on", + devtools, + evidence_sha256: hex::encode(sha256(stdout)), + }; + serde_json::to_vec(&event).context("failed to serialize GPU attestation event") + } + + /// Run local GPU attestation via nvattest with a fresh nonce and no custom + /// relying-party policy. nvattest's built-in appraisal still applies. The + /// complete JSON output is preserved under /run and retained with its + /// claims for the application policy and versioned RTMR3 summary. + pub(super) async fn attest_gpu(expected_devices: u32) -> Result { if !Path::new(NVATTEST).exists() { bail!("nvattest is not available in this image"); } @@ -1165,6 +1392,8 @@ mod gpu { "local", "--nonce", &nonce, + "--format", + "json", ], ATTESTATION_TIMEOUT, ) @@ -1172,6 +1401,7 @@ mod gpu { if !output.stderr.is_empty() { info!("nvattest: {}", truncated_lossy(&output.stderr, 2048)); } + save_attestation_output(&output.stdout).context("failed to save GPU attestation output")?; if !output.status.success() { bail!( "nvattest exited with {}: {}", @@ -1179,14 +1409,66 @@ mod gpu { truncated_lossy(&output.stderr, 512), ); } - if let Err(err) = save_attestation_output(&output.stdout) { - warn!("failed to save GPU attestation output: {err:?}"); + let claims = validate_attestation_output(&output.stdout, &nonce, expected_devices)?; + Ok(GpuAttestationResult { + claims: claims.raw, + parsed_claims: claims.parsed, + output: output.stdout, + devices: expected_devices, + }) + } + + pub(super) fn measure_gpu_policy(compose_path: &Path) -> Result<[u8; 32]> { + let compose_json = fs::read(compose_path) + .with_context(|| format!("failed to read {}", compose_path.display()))?; + let digest = gpu_policy_hash(&compose_json).context("failed to hash raw GPU policy")?; + emit_runtime_event("gpu-policy-hash", &digest) + .context("failed to emit GPU policy measurement")?; + Ok(digest) + } + + pub(super) fn evaluate_rego_policy(policy: &GpuPolicy, claims: &[Value]) -> Result<()> { + let Some(rego) = policy.rego.as_deref() else { + return Ok(()); + }; + evaluate_policy(rego, claims).context("failed to apply GPU Rego policy") + } + + /// Evaluate the app-provided Rego v0 policy using the same input shape as + /// NVIDIA relying-party policies: the nvattest `claims` JSON array. + pub(super) fn evaluate_policy(policy: &str, claims: &[Value]) -> Result<()> { + evaluate_policy_with_timeout(policy, claims, POLICY_TIMEOUT) + } + + fn evaluate_policy_with_timeout( + policy: &str, + claims: &[Value], + timeout: Duration, + ) -> Result<()> { + let mut engine = regorus::Engine::new(); + engine.set_rego_v0(true); + engine.set_execution_timer_config(regorus::utils::limits::ExecutionTimerConfig { + limit: timeout, + check_interval: std::num::NonZeroU32::new(1024).unwrap_or(std::num::NonZeroU32::MIN), + }); + engine + .add_policy("gpu-policy.rego".to_string(), policy.to_string()) + .context("failed to load GPU policy")?; + let input = serde_json::to_string(claims).context("failed to serialize GPU claims")?; + engine + .set_input_json(&input) + .context("failed to set GPU policy input")?; + if !engine + .eval_bool_query(POLICY_ENTRYPOINT.to_string(), false) + .context("failed to evaluate GPU policy")? + { + bail!("gpu policy rejected the attestation claims"); } Ok(()) } fn save_attestation_output(stdout: &[u8]) -> Result<()> { - let output_path = Path::new(ATTESTATION_OUTPUT); + let output_path = Path::new(GPU_ATTESTATION_OUTPUT); if let Some(parent) = output_path.parent() { fs::create_dir_all(parent)?; } @@ -1206,30 +1488,408 @@ mod gpu { None => text.to_string(), } } + + #[cfg(test)] + mod tests { + use super::*; + + fn add_pci_device(root: &Path, name: &str, vendor: &str, class: &str) { + let device = root.join(name); + fs::create_dir_all(&device).unwrap(); + fs::write(device.join("vendor"), vendor).unwrap(); + fs::write(device.join("class"), class).unwrap(); + } + + fn nvattest_output(nonce: &str, claims: usize) -> Vec { + let claims = (0..claims) + .map(|_| { + serde_json::json!({ + "x-nvidia-device-type": "gpu", + "eat_nonce": nonce, + "x-nvidia-gpu-attestation-report-nonce-match": true, + "measres": "success", + "secboot": true, + "dbgstat": "disabled" + }) + }) + .collect::>(); + serde_json::to_vec(&serde_json::json!({ + "result_code": 0, + "claims": claims, + "detached_eat": {} + })) + .unwrap() + } + + // Captured with the pinned nvattest SDK on an Ubuntu 22.04 GCP A3 TDX + // VM with an H100 and the NVIDIA 580 open kernel driver. + const H100_ATTESTATION_OUTPUT: &[u8] = + include_bytes!("../tests/fixtures/gpu_attestation_h100.json"); + + #[test] + fn inventory_counts_nvidia_and_non_nvidia_gpus() { + let root = tempfile::tempdir().unwrap(); + add_pci_device(root.path(), "0000:01:00.0", "0x10de\n", "0x030200\n"); + add_pci_device(root.path(), "0000:02:00.0", "0x1234\n", "0x030000\n"); + add_pci_device(root.path(), "0000:03:00.0", "0x1af4\n", "0x020000\n"); + assert_eq!( + gpu_inventory_at(root.path()).unwrap(), + GpuInventory { + total: 2, + nvidia: 1 + } + ); + } + + #[test] + fn gpu_count_rejects_non_nvidia_gpus() { + let mixed = GpuInventory { + total: 2, + nvidia: 1, + }; + assert!(nvidia_gpu_count(mixed).is_err()); + + let nvidia = GpuInventory { + total: 2, + nvidia: 2, + }; + assert_eq!(nvidia_gpu_count(nvidia).unwrap(), 2); + } + + #[test] + fn basic_policy_requires_cc_and_rejects_devtools_by_default() { + let nonce = "44".repeat(32); + let output = nvattest_output(&nonce, 1); + let claims = validate_attestation_output(&output, &nonce, 1).unwrap(); + let production = [GpuDeviceState { + cc_enabled: true, + devtools: false, + }]; + verify_claim_policy(&claims.parsed, &production, &GpuPolicy::default()).unwrap(); + + let non_cc = [GpuDeviceState { + cc_enabled: false, + devtools: false, + }]; + let err = + verify_claim_policy(&claims.parsed, &non_cc, &GpuPolicy::default()).unwrap_err(); + assert!(err.to_string().contains("confidential compute mode")); + + let devtools = [GpuDeviceState { + cc_enabled: true, + devtools: true, + }]; + assert!(verify_claim_policy(&claims.parsed, &devtools, &GpuPolicy::default()).is_err()); + verify_claim_policy( + &claims.parsed, + &devtools, + &GpuPolicy { + allow_devtools: true, + ..Default::default() + }, + ) + .unwrap(); + } + + #[test] + fn nvattest_output_requires_every_expected_gpu_and_fresh_nonce() { + let nonce = "11".repeat(32); + let valid = nvattest_output(&nonce, 2); + validate_attestation_output(&valid, &nonce, 2).unwrap(); + assert!(validate_attestation_output(&valid, &nonce, 1).is_err()); + + let mut invalid: Value = serde_json::from_slice(&valid).unwrap(); + invalid["claims"][1]["eat_nonce"] = Value::String("stale".to_string()); + assert!( + validate_attestation_output(&serde_json::to_vec(&invalid).unwrap(), &nonce, 2) + .is_err() + ); + + // Basic claim settings are enforced after structural validation. + let mut extra_claims: Value = serde_json::from_slice(&valid).unwrap(); + extra_claims["claims"][0]["dbgstat"] = Value::String("enabled".to_string()); + validate_attestation_output(&serde_json::to_vec(&extra_claims).unwrap(), &nonce, 2) + .unwrap(); + } + + #[test] + fn basic_claim_policy_is_fail_closed_and_honors_opt_ins() { + let nonce = "33".repeat(32); + let output = nvattest_output(&nonce, 1); + let claims = validate_attestation_output(&output, &nonce, 1).unwrap(); + let devices = [GpuDeviceState { + cc_enabled: true, + devtools: false, + }]; + verify_claim_policy(&claims.parsed, &devices, &GpuPolicy::default()).unwrap(); + + let with_policy = |name: &str, value: Value, policy: &GpuPolicy| { + let mut output: Value = serde_json::from_slice(&output).unwrap(); + output["claims"][0][name] = value; + let output = serde_json::to_vec(&output).unwrap(); + let claims = validate_attestation_output(&output, &nonce, 1).unwrap(); + verify_claim_policy(&claims.parsed, &devices, policy) + }; + + assert!(with_policy("secboot", Value::Bool(false), &GpuPolicy::default()).is_err()); + with_policy( + "secboot", + Value::Bool(false), + &GpuPolicy { + allow_insecure_boot: true, + ..Default::default() + }, + ) + .unwrap(); + + assert!(with_policy( + "dbgstat", + Value::String("enabled".to_string()), + &GpuPolicy::default(), + ) + .is_err()); + with_policy( + "dbgstat", + Value::String("enabled".to_string()), + &GpuPolicy { + allow_debug: true, + ..Default::default() + }, + ) + .unwrap(); + + let mut unknown_debug: Value = serde_json::from_slice(&output).unwrap(); + unknown_debug["claims"][0]["dbgstat"] = Value::String("unknown".to_string()); + assert!(validate_attestation_output( + &serde_json::to_vec(&unknown_debug).unwrap(), + &nonce, + 1, + ) + .is_err()); + + assert!(with_policy( + "measres", + Value::String("failure".to_string()), + &GpuPolicy { + allow_debug: true, + allow_insecure_boot: true, + ..Default::default() + }, + ) + .is_err()); + + for required in ["measres", "secboot", "dbgstat"] { + let mut missing: Value = serde_json::from_slice(&output).unwrap(); + missing["claims"][0] + .as_object_mut() + .unwrap() + .remove(required); + assert!(validate_attestation_output( + &serde_json::to_vec(&missing).unwrap(), + &nonce, + 1, + ) + .is_err()); + } + } + + #[test] + fn real_h100_attestation_fixture_validates_and_drives_rego() { + let nonce = "11".repeat(32); + let claims = validate_attestation_output(H100_ATTESTATION_OUTPUT, &nonce, 1).unwrap(); + assert_eq!(claims.raw[0]["hwmodel"], "GH100 A01 GSP BROM"); + assert_eq!(claims.raw[0]["x-nvidia-gpu-claims-version"], "3.0"); + verify_claim_policy( + &claims.parsed, + &[GpuDeviceState { + cc_enabled: true, + devtools: false, + }], + &GpuPolicy::default(), + ) + .unwrap(); + + let policy = r#" + package policy + default nv_match = false + nv_match { + count(input) == 1 + input[0].secboot == true + input[0].dbgstat == "disabled" + input[0].measres == "success" + } + "#; + evaluate_policy(policy, &claims.raw).unwrap(); + } + + #[test] + fn event_commits_to_complete_nvattest_output() { + let nonce = "22".repeat(32); + let output = nvattest_output(&nonce, 1); + let event: Value = + serde_json::from_slice(&attestation_event(&output, 1, true).unwrap()).unwrap(); + assert_eq!(event["version"], EVENT_VERSION); + assert_eq!(event["devices"], 1); + assert!(event.get("policy").is_none()); + assert_eq!(event["cc_mode"], "on"); + assert_eq!(event["devtools"], true); + assert_eq!(event["evidence_sha256"], hex::encode(sha256(&output))); + } + + #[test] + fn app_policy_receives_claims_array_and_must_return_true() { + let claims = vec![serde_json::json!({"status": "accepted"})]; + let policy = r#" + package policy + default nv_match = false + nv_match { + count(input) == 1 + input[0].status == "accepted" + } + "#; + evaluate_policy(policy, &claims).unwrap(); + assert!(evaluate_policy(policy, &[]).is_err()); + + let rejected = vec![serde_json::json!({"status": "rejected"})]; + assert!(evaluate_policy(policy, &rejected).is_err()); + assert!(evaluate_policy("package policy", &claims).is_err()); + assert!(evaluate_policy("not valid rego", &claims).is_err()); + + let allow_no_gpus = GpuPolicy { + rego: Some( + r#" + package policy + default nv_match = false + nv_match { count(input) == 0 } + "# + .to_string(), + ), + ..Default::default() + }; + evaluate_rego_policy(&allow_no_gpus, &[]).unwrap(); + + let require_one_gpu = GpuPolicy { + rego: Some(policy.to_string()), + ..Default::default() + }; + assert!(evaluate_rego_policy(&require_one_gpu, &[]).is_err()); + evaluate_rego_policy(&GpuPolicy::default(), &[]).unwrap(); + } + + #[test] + fn rego_policy_evaluation_is_time_bounded() { + let policy = r#" + package policy + default nv_match = false + nv_match { + count([x | + x := numbers.range(0, 5000)[_] + y := numbers.range(0, 5000)[_] + x == y + ]) > 0 + } + "#; + evaluate_policy_with_timeout(policy, &[], Duration::from_millis(50)).unwrap_err(); + } + + #[test] + fn gpu_policy_measurement_defaults_to_empty_object_and_uses_raw_json() { + let no_requirements = br#"{}"#; + let absent = br#"{"requirements": {}}"#; + let empty_digest = sha256(b"{}"); + assert_eq!(gpu_policy_hash(no_requirements).unwrap(), empty_digest); + assert_eq!(gpu_policy_hash(absent).unwrap(), empty_digest); + + let empty = br#"{"requirements": {"gpu_policy": {}}}"#; + assert_eq!(gpu_policy_hash(empty).unwrap(), empty_digest); + + let explicit_default = br#"{"requirements":{"gpu_policy":{"attest_gpu":true}}}"#; + let explicit_default_digest = gpu_policy_hash(explicit_default).unwrap(); + assert_ne!(explicit_default_digest, empty_digest); + + let reordered = br#" + { + "requirements": { + "gpu_policy": { + "rego": "package policy", + "allow_debug": false + } + } + } + "#; + let canonical_order = + br#"{"requirements":{"gpu_policy":{"allow_debug":false,"rego":"package policy"}}}"#; + assert_eq!( + gpu_policy_hash(reordered).unwrap(), + gpu_policy_hash(canonical_order).unwrap() + ); + } + } } impl Stage0<'_> { - /// Enforce `requirements.attest_gpu` (default true): attest an attached - /// NVIDIA GPU before continuing to key provisioning, or — when explicitly - /// disabled — set the GPU ready state without verification. - async fn setup_gpu(&self) -> Result<()> { - if !gpu::nvidia_gpu_present()? { - return Ok(()); - } - if !self.shared.app_compose.attest_gpu() { - warn!("requirements.attest_gpu is false; setting GPU ready state without attestation"); + /// Enforce `requirements.gpu_policy.attest_gpu` (default true): attest an + /// attached NVIDIA GPU before continuing to key provisioning, or — when + /// explicitly disabled — set the GPU ready state without verification. The + /// optional Rego policy is always evaluated; when no attestation is + /// performed, its claims-array input is empty. + async fn measure_gpu(&self) -> Result<[u8; 32]> { + let gpu_policy_hash = gpu::measure_gpu_policy(&self.shared.dir.app_compose_file())?; + + let gpu_policy = self + .shared + .app_compose + .requirements + .as_ref() + .map(|requirements| requirements.gpu_policy.clone()) + .unwrap_or_default(); + + let inventory = gpu::gpu_inventory()?; + if !gpu_policy.attest_gpu { + // Attestation is explicitly disabled, so there are no claims. Rego + // still runs with an empty input before any GPU is made ready. + gpu::evaluate_rego_policy(&gpu_policy, &[])?; + if gpu_policy.rego.is_some() { + info!("application GPU Rego policy accepted an empty claims array"); + } + if inventory.nvidia == 0 { + return Ok(gpu_policy_hash); + } + warn!( + "requirements.gpu_policy.attest_gpu is false; setting GPU ready state without attestation" + ); // Best-effort: a GPU with CC mode off has no ready state to set. - if let Err(err) = gpu::set_gpu_ready_state().await { + if let Err(err) = gpu::set_gpu_ready_state(inventory.nvidia) { warn!("failed to set GPU ready state: {err:?}"); } - return Ok(()); + return Ok(gpu_policy_hash); + } + let expected_devices = gpu::nvidia_gpu_count(inventory)?; + if expected_devices == 0 { + gpu::evaluate_rego_policy(&gpu_policy, &[])?; + if gpu_policy.rego.is_some() { + info!("application GPU Rego policy accepted an empty claims array"); + } + return Ok(gpu_policy_hash); } self.vmm.notify_q("boot.progress", "attesting GPU").await; info!("verifying GPU TEE attestation"); - gpu::attest_gpu().await?; - gpu::set_gpu_ready_state().await?; + let attestation = gpu::attest_gpu(expected_devices).await?; + + let gpu_state = gpu::query_gpu_state(expected_devices)?; + attestation + .verify_claim_policy(&gpu_state, &gpu_policy) + .context("failed to apply basic GPU policy")?; + gpu::evaluate_rego_policy(&gpu_policy, attestation.claims())?; + + info!("application GPU policy accepted the attestation claims and state"); + gpu_state.set_ready()?; + let devtools = gpu_state.any_devtools(); + let event = attestation.event(devtools)?; + emit_runtime_event("gpu-attestation", &event) + .context("failed to emit GPU attestation event")?; info!("GPU TEE attestation succeeded"); - Ok(()) + Ok(gpu_policy_hash) } } @@ -1278,6 +1938,7 @@ impl AppIdValidator { struct AppInfo { instance_info: InstanceInfo, compose_hash: [u8; 32], + gpu_policy_hash: [u8; 32], } struct Stage0<'a> { @@ -1841,7 +2502,7 @@ impl<'a> Stage0<'a> { Ok(()) } - fn measure_app_info(&self) -> Result { + async fn measure_app_info(&self) -> Result { let compose_hash = sha256_file(self.shared.dir.app_compose_file())?; let truncated_compose_hash = truncate(&compose_hash, 20); let key_provider = self.shared.app_compose.key_provider(); @@ -1893,6 +2554,11 @@ impl<'a> Stage0<'a> { emit_runtime_event("system-preparing", &[])?; emit_runtime_event("app-id", &instance_info.app_id)?; emit_runtime_event("compose-hash", &compose_hash)?; + let gpu_policy_hash = self + .measure_gpu() + .await + .context("failed to verify GPU TEE attestation")?; + emit_runtime_event("instance-id", &instance_id)?; emit_runtime_event("boot-mr-done", &[])?; @@ -1920,12 +2586,14 @@ impl<'a> Stage0<'a> { Ok(AppInfo { instance_info, compose_hash, + gpu_policy_hash, }) } fn verify_app(&self, app_info: &AppInfo, keys: &AppKeys) -> Result<()> { config_id_verifier::verify_mr_config_id( &app_info.compose_hash, + &app_info.gpu_policy_hash, &app_info .instance_info .app_id @@ -1957,6 +2625,7 @@ impl<'a> Stage0<'a> { async fn setup_fs(self) -> Result> { let app_info = self .measure_app_info() + .await .context("Failed to measure app info")?; if self.shared.app_compose.key_provider().is_kms() { cmd_show_mrs()?; diff --git a/dstack/dstack-util/src/system_setup/config_id_verifier.rs b/dstack/dstack-util/src/system_setup/config_id_verifier.rs index c350b32e6..82b627846 100644 --- a/dstack/dstack-util/src/system_setup/config_id_verifier.rs +++ b/dstack/dstack-util/src/system_setup/config_id_verifier.rs @@ -12,8 +12,9 @@ use dstack_types::{ use tracing::info; #[derive(Clone, Copy)] -struct ExpectedMrConfig<'a> { +struct LocalMrConfigValues<'a> { compose_hash: &'a [u8; 32], + gpu_policy_hash: &'a [u8; 32], app_id: &'a [u8; 20], instance_id: &'a [u8], key_provider: KeyProviderKind, @@ -51,7 +52,7 @@ fn read_snp_host_data() -> Result<[u8; 32]> { Ok(parsed.host_data) } -/// Verify the mr_config_id matches the expected value +/// Verify the mr_config_id matches values observed locally by the guest. /// /// Configuration ID format /// The mr_config_id is a 48 bytes value in the following format: @@ -65,37 +66,39 @@ fn read_snp_host_data() -> Result<[u8; 32]> { /// - key_provider_id: [u8] // KMS CA pubkey, local-sgx MR, or empty for none/tpm pub fn verify_mr_config_id( compose_hash: &[u8; 32], + gpu_policy_hash: &[u8; 32], app_id: &[u8; 20], instance_id: &[u8], key_provider: KeyProviderKind, key_provider_id: &[u8], ) -> Result<()> { let mode = AttestationMode::detect().context("Failed to detect attestation mode")?; - let expected = ExpectedMrConfig { + let local = LocalMrConfigValues { compose_hash, + gpu_policy_hash, app_id, instance_id, key_provider, key_provider_id, }; - verify_mr_config_id_for_mode(mode, expected) + verify_mr_config_id_for_mode(mode, local) } fn verify_mr_config_id_for_mode( mode: AttestationMode, - expected: ExpectedMrConfig<'_>, + local: LocalMrConfigValues<'_>, ) -> Result<()> { match mode { - AttestationMode::DstackAmdSevSnp => verify_snp_mr_config(expected), + AttestationMode::DstackAmdSevSnp => verify_snp_mr_config(local), // AWS PCR8 is computed by the guest from measured reality (MrConfig V2 // in measure_app_info); there is no host-supplied claim to cross-check. // The key_provider_id pin is enforced by verify_key_provider_id. AttestationMode::DstackAwsNitroTpm => Ok(()), - _ => verify_tdx_mr_config_id(expected), + _ => verify_tdx_mr_config_id(local), } } -fn verify_tdx_mr_config_id(expected: ExpectedMrConfig<'_>) -> Result<()> { +fn verify_tdx_mr_config_id(local: LocalMrConfigValues<'_>) -> Result<()> { let read_mr_config_id = read_mr_config_id().context("Failed to read mr_config_id")?; info!("mr_config_id: {}", hex::encode(read_mr_config_id)); let mr_config_document = if read_mr_config_id[0] == 3 { @@ -103,33 +106,33 @@ fn verify_tdx_mr_config_id(expected: ExpectedMrConfig<'_>) -> Result<()> { } else { None }; - verify_tdx_mr_config_id_value(read_mr_config_id, mr_config_document.as_deref(), expected) + verify_tdx_mr_config_id_value(read_mr_config_id, mr_config_document.as_deref(), local) } fn verify_tdx_mr_config_id_value( read_mr_config_id: [u8; 48], mr_config_document: Option<&str>, - expected: ExpectedMrConfig<'_>, + local: LocalMrConfigValues<'_>, ) -> Result<()> { if read_mr_config_id == [0u8; 48] { return Ok(()); } let expected_mr_config_id = match read_mr_config_id[0] { 1 => MrConfig::V1 { - compose_hash: expected.compose_hash, + compose_hash: local.compose_hash, } .to_mr_config_id(), 2 => MrConfig::V2 { - compose_hash: expected.compose_hash, - app_id: expected.app_id, - key_provider: expected.key_provider, - key_provider_id: expected.key_provider_id, + compose_hash: local.compose_hash, + app_id: local.app_id, + key_provider: local.key_provider, + key_provider_id: local.key_provider_id, } .to_mr_config_id(), 3 => { let mr_config_document = mr_config_document.context("mr_config is required for TDX MR_CONFIG_ID v3")?; - verify_mr_config_v3_document(mr_config_document, expected)?; + verify_mr_config_v3_document(mr_config_document, local)?; MrConfigV3::tdx_mr_config_id_from_document(mr_config_document) } _ => bail!("Invalid mr_config_id version"), @@ -140,9 +143,9 @@ fn verify_tdx_mr_config_id_value( Ok(()) } -fn verify_snp_mr_config(expected: ExpectedMrConfig<'_>) -> Result<()> { +fn verify_snp_mr_config(local: LocalMrConfigValues<'_>) -> Result<()> { let mr_config_document = read_mr_config_document().context("Failed to read SNP mr_config")?; - verify_mr_config_v3_document(&mr_config_document, expected)?; + verify_mr_config_v3_document(&mr_config_document, local)?; let read_host_data = read_snp_host_data().context("Failed to read SNP HOST_DATA")?; info!("snp host_data: {}", hex::encode(read_host_data)); if MrConfigV3::snp_host_data_from_document(&mr_config_document) != read_host_data { @@ -153,26 +156,31 @@ fn verify_snp_mr_config(expected: ExpectedMrConfig<'_>) -> Result<()> { fn verify_mr_config_v3_document( mr_config_document: &str, - expected: ExpectedMrConfig<'_>, + local: LocalMrConfigValues<'_>, ) -> Result { let mr_config = MrConfigV3::from_document(mr_config_document).context("Invalid mr_config document")?; if mr_config.version != 3 { bail!("mr_config version must be 3"); } - if mr_config.compose_hash.as_slice() != expected.compose_hash { + if mr_config.compose_hash.as_slice() != local.compose_hash { bail!("Invalid mr_config compose_hash"); } - if mr_config.app_id.as_slice() != expected.app_id { + if let Some(declared_gpu_policy_hash) = mr_config.gpu_policy_hash.as_deref() { + if declared_gpu_policy_hash != local.gpu_policy_hash { + bail!("Invalid mr_config gpu_policy_hash"); + } + } + if mr_config.app_id.as_slice() != local.app_id { bail!("Invalid mr_config app_id"); } - if mr_config.instance_id.as_slice() != expected.instance_id { + if mr_config.instance_id.as_slice() != local.instance_id { bail!("Invalid mr_config instance_id"); } - if mr_config.key_provider != expected.key_provider { + if mr_config.key_provider != local.key_provider { bail!("Invalid mr_config key_provider"); } - if mr_config.key_provider_id.as_slice() != expected.key_provider_id { + if mr_config.key_provider_id.as_slice() != local.key_provider_id { bail!("Invalid mr_config key_provider_id"); } Ok(mr_config) @@ -194,54 +202,122 @@ mod tests { #[test] fn tdx_mr_config_id_v3_accepts_document_value() -> Result<()> { let compose_hash = [0x22u8; 32]; + let gpu_policy_hash = [0x55u8; 32]; let app_id = [0x11u8; 20]; let instance_id = [0x44u8; 20]; let key_provider_id = [0x33u8; 32]; let mr_config = MrConfigV3::new( app_id.to_vec(), compose_hash.to_vec(), + Some(gpu_policy_hash.to_vec()), KeyProviderKind::Kms, key_provider_id.to_vec(), instance_id.to_vec(), ); let document = mr_config.to_canonical_json(); - let expected = ExpectedMrConfig { + let local = LocalMrConfigValues { compose_hash: &compose_hash, + gpu_policy_hash: &gpu_policy_hash, app_id: &app_id, instance_id: &instance_id, key_provider: KeyProviderKind::Kms, key_provider_id: &key_provider_id, }; - verify_tdx_mr_config_id_value(mr_config.to_tdx_mr_config_id(), Some(&document), expected) + verify_tdx_mr_config_id_value(mr_config.to_tdx_mr_config_id(), Some(&document), local) } #[test] fn mr_config_v3_document_must_match_expected_app_info() { let compose_hash = [0x22u8; 32]; + let gpu_policy_hash = [0x55u8; 32]; let app_id = [0x11u8; 20]; let instance_id = [0x44u8; 20]; let key_provider_id = [0x33u8; 32]; let document = MrConfigV3::new( app_id.to_vec(), compose_hash.to_vec(), + Some(gpu_policy_hash.to_vec()), KeyProviderKind::Kms, key_provider_id.to_vec(), instance_id.to_vec(), ) .to_canonical_json(); let wrong_app_id = [0x12u8; 20]; - let expected = ExpectedMrConfig { + let local = LocalMrConfigValues { compose_hash: &compose_hash, + gpu_policy_hash: &gpu_policy_hash, app_id: &wrong_app_id, instance_id: &instance_id, key_provider: KeyProviderKind::Kms, key_provider_id: &key_provider_id, }; - match verify_mr_config_v3_document(&document, expected) { + match verify_mr_config_v3_document(&document, local) { Ok(_) => panic!("mismatched app_id must reject"), Err(err) => assert!(err.to_string().contains("Invalid mr_config app_id")), } } + + #[test] + fn mr_config_v3_document_must_match_expected_gpu_policy_hash() { + let compose_hash = [0x22u8; 32]; + let gpu_policy_hash = [0x55u8; 32]; + let app_id = [0x11u8; 20]; + let instance_id = [0x44u8; 20]; + let key_provider_id = [0x33u8; 32]; + let document = MrConfigV3::new( + app_id.to_vec(), + compose_hash.to_vec(), + Some(gpu_policy_hash.to_vec()), + KeyProviderKind::Kms, + key_provider_id.to_vec(), + instance_id.to_vec(), + ) + .to_canonical_json(); + let wrong_gpu_policy_hash = [0x56u8; 32]; + let local = LocalMrConfigValues { + compose_hash: &compose_hash, + gpu_policy_hash: &wrong_gpu_policy_hash, + app_id: &app_id, + instance_id: &instance_id, + key_provider: KeyProviderKind::Kms, + key_provider_id: &key_provider_id, + }; + + match verify_mr_config_v3_document(&document, local) { + Ok(_) => panic!("mismatched gpu_policy_hash must reject"), + Err(err) => assert!(err + .to_string() + .contains("Invalid mr_config gpu_policy_hash")), + } + } + + #[test] + fn mr_config_v3_skips_gpu_policy_hash_check_when_field_is_missing() -> Result<()> { + let compose_hash = [0x22u8; 32]; + let actual_gpu_policy_hash = [0x55u8; 32]; + let app_id = [0x11u8; 20]; + let instance_id = [0x44u8; 20]; + let key_provider_id = [0x33u8; 32]; + let mr_config = MrConfigV3::new( + app_id.to_vec(), + compose_hash.to_vec(), + None, + KeyProviderKind::Kms, + key_provider_id.to_vec(), + instance_id.to_vec(), + ); + let document = mr_config.to_canonical_json(); + let local = LocalMrConfigValues { + compose_hash: &compose_hash, + gpu_policy_hash: &actual_gpu_policy_hash, + app_id: &app_id, + instance_id: &instance_id, + key_provider: KeyProviderKind::Kms, + key_provider_id: &key_provider_id, + }; + + verify_tdx_mr_config_id_value(mr_config.to_tdx_mr_config_id(), Some(&document), local) + } } diff --git a/dstack/dstack-util/tests/fixtures/gpu_attestation_h100.json b/dstack/dstack-util/tests/fixtures/gpu_attestation_h100.json new file mode 100644 index 000000000..c8e52e018 --- /dev/null +++ b/dstack/dstack-util/tests/fixtures/gpu_attestation_h100.json @@ -0,0 +1,67 @@ +{ + "claims": [ + { + "dbgstat": "disabled", + "eat_nonce": "1111111111111111111111111111111111111111111111111111111111111111", + "hwmodel": "GH100 A01 GSP BROM", + "measres": "success", + "oemid": "5703", + "secboot": true, + "ueid": "457723480349051719100670602112458195243836813799", + "x-nvidia-device-type": "gpu", + "x-nvidia-gpu-arch-check": true, + "x-nvidia-gpu-attestation-report-cert-chain": { + "x-nvidia-cert-expiration-date": "9999-12-31T23:59:59Z", + "x-nvidia-cert-ocsp-nonce-matches": true, + "x-nvidia-cert-ocsp-response-valid": true, + "x-nvidia-cert-ocsp-status": "good", + "x-nvidia-cert-revocation-reason": null, + "x-nvidia-cert-status": "valid" + }, + "x-nvidia-gpu-attestation-report-cert-chain-fwid-match": true, + "x-nvidia-gpu-attestation-report-nonce-match": true, + "x-nvidia-gpu-attestation-report-parsed": true, + "x-nvidia-gpu-attestation-report-signature-verified": true, + "x-nvidia-gpu-claims-version": "3.0", + "x-nvidia-gpu-driver-rim-cert-chain": { + "x-nvidia-cert-expiration-date": "2028-04-23T05:31:21Z", + "x-nvidia-cert-ocsp-nonce-matches": true, + "x-nvidia-cert-ocsp-response-valid": true, + "x-nvidia-cert-ocsp-status": "good", + "x-nvidia-cert-revocation-reason": null, + "x-nvidia-cert-status": "valid" + }, + "x-nvidia-gpu-driver-rim-fetched": true, + "x-nvidia-gpu-driver-rim-measurements-available": true, + "x-nvidia-gpu-driver-rim-signature-verified": true, + "x-nvidia-gpu-driver-rim-version-match": true, + "x-nvidia-gpu-driver-version": "580.159.03", + "x-nvidia-gpu-vbios-index-no-conflict": true, + "x-nvidia-gpu-vbios-rim-cert-chain": { + "x-nvidia-cert-expiration-date": "2027-04-30T23:59:59Z", + "x-nvidia-cert-ocsp-nonce-matches": true, + "x-nvidia-cert-ocsp-response-valid": true, + "x-nvidia-cert-ocsp-status": "good", + "x-nvidia-cert-revocation-reason": null, + "x-nvidia-cert-status": "valid" + }, + "x-nvidia-gpu-vbios-rim-fetched": true, + "x-nvidia-gpu-vbios-rim-measurements-available": true, + "x-nvidia-gpu-vbios-rim-signature-verified": true, + "x-nvidia-gpu-vbios-rim-version-match": true, + "x-nvidia-gpu-vbios-version": "96.00.D9.00.01", + "x-nvidia-mismatch-measurement-records": null + } + ], + "detached_eat": [ + [ + "JWT", + "eyJhbGciOiJub25lIn0.eyJlYXRfbm9uY2UiOiIxMTExMTExMTExMTExMTExMTExMTExMTExMTExMTExMTExMTExMTExMTExMTExMTExMTExMTExMTExMTExMTExIiwiZXhwIjoxNzg0MjExNDYyLCJpYXQiOjE3ODQyMDc4NjIsImlzcyI6Ik5WQVQtTE9DQUwtVkVSSUZJRVIiLCJqdGkiOiI2MjcwZTgxMTVhNWMxOTExODNjNjg0ZWY4YzFkYWRmYzYxYzYzMjQyZGUwOGY0MjQzOGU3M2NmMjNjODBkMTg1Iiwic3ViIjoiTlZJRElBLVBMQVRGT1JNLUFUVEVTVEFUSU9OIiwic3VibW9kcyI6eyJHUFUtMCI6WyJESUdFU1QiLFsiU0hBMjU2IiwiMDNkZTczYjliMTU5OTM2OTNkZmMyMzY3Y2QyZjNmMDE3ZWVkYjQ2ZWZhNzY3Yzc5ZGJkODY3ODZlMGQ3OWZlYSJdXX0sIngtbnZpZGlhLW92ZXJhbGwtYXR0LXJlc3VsdCI6dHJ1ZSwieC1udmlkaWEtdmVyIjoiMy4wIn0." + ], + { + "GPU-0": "eyJhbGciOiJub25lIn0.eyJkYmdzdGF0IjoiZGlzYWJsZWQiLCJlYXRfbm9uY2UiOiIxMTExMTExMTExMTExMTExMTExMTExMTExMTExMTExMTExMTExMTExMTExMTExMTExMTExMTExMTExMTExMTExIiwiZXhwIjoxNzg0MjExNDYyLCJod21vZGVsIjoiR0gxMDAgQTAxIEdTUCBCUk9NIiwiaWF0IjoxNzg0MjA3ODYyLCJpc3MiOiJOVkFULUxPQ0FMLVZFUklGSUVSIiwianRpIjoiYTU5ZWQ5ODAwYThhNjFhZDc5NjVlNmI1MGRmMjEyODVlNjMxZTg5ZWNlZTM5NmMwZjc1ZDkwNjIwNWI2MDUzYiIsIm1lYXNyZXMiOiJzdWNjZXNzIiwib2VtaWQiOiI1NzAzIiwic2VjYm9vdCI6dHJ1ZSwidWVpZCI6IjQ1NzcyMzQ4MDM0OTA1MTcxOTEwMDY3MDYwMjExMjQ1ODE5NTI0MzgzNjgxMzc5OSIsIngtbnZpZGlhLWRldmljZS10eXBlIjoiZ3B1IiwieC1udmlkaWEtZ3B1LWFyY2gtY2hlY2siOnRydWUsIngtbnZpZGlhLWdwdS1hdHRlc3RhdGlvbi1yZXBvcnQtY2VydC1jaGFpbiI6eyJ4LW52aWRpYS1jZXJ0LWV4cGlyYXRpb24tZGF0ZSI6Ijk5OTktMTItMzFUMjM6NTk6NTlaIiwieC1udmlkaWEtY2VydC1vY3NwLW5vbmNlLW1hdGNoZXMiOnRydWUsIngtbnZpZGlhLWNlcnQtb2NzcC1yZXNwb25zZS12YWxpZCI6dHJ1ZSwieC1udmlkaWEtY2VydC1vY3NwLXN0YXR1cyI6Imdvb2QiLCJ4LW52aWRpYS1jZXJ0LXJldm9jYXRpb24tcmVhc29uIjpudWxsLCJ4LW52aWRpYS1jZXJ0LXN0YXR1cyI6InZhbGlkIn0sIngtbnZpZGlhLWdwdS1hdHRlc3RhdGlvbi1yZXBvcnQtY2VydC1jaGFpbi1md2lkLW1hdGNoIjp0cnVlLCJ4LW52aWRpYS1ncHUtYXR0ZXN0YXRpb24tcmVwb3J0LW5vbmNlLW1hdGNoIjp0cnVlLCJ4LW52aWRpYS1ncHUtYXR0ZXN0YXRpb24tcmVwb3J0LXBhcnNlZCI6dHJ1ZSwieC1udmlkaWEtZ3B1LWF0dGVzdGF0aW9uLXJlcG9ydC1zaWduYXR1cmUtdmVyaWZpZWQiOnRydWUsIngtbnZpZGlhLWdwdS1jbGFpbXMtdmVyc2lvbiI6IjMuMCIsIngtbnZpZGlhLWdwdS1kcml2ZXItcmltLWNlcnQtY2hhaW4iOnsieC1udmlkaWEtY2VydC1leHBpcmF0aW9uLWRhdGUiOiIyMDI4LTA0LTIzVDA1OjMxOjIxWiIsIngtbnZpZGlhLWNlcnQtb2NzcC1ub25jZS1tYXRjaGVzIjp0cnVlLCJ4LW52aWRpYS1jZXJ0LW9jc3AtcmVzcG9uc2UtdmFsaWQiOnRydWUsIngtbnZpZGlhLWNlcnQtb2NzcC1zdGF0dXMiOiJnb29kIiwieC1udmlkaWEtY2VydC1yZXZvY2F0aW9uLXJlYXNvbiI6bnVsbCwieC1udmlkaWEtY2VydC1zdGF0dXMiOiJ2YWxpZCJ9LCJ4LW52aWRpYS1ncHUtZHJpdmVyLXJpbS1mZXRjaGVkIjp0cnVlLCJ4LW52aWRpYS1ncHUtZHJpdmVyLXJpbS1tZWFzdXJlbWVudHMtYXZhaWxhYmxlIjp0cnVlLCJ4LW52aWRpYS1ncHUtZHJpdmVyLXJpbS1zaWduYXR1cmUtdmVyaWZpZWQiOnRydWUsIngtbnZpZGlhLWdwdS1kcml2ZXItcmltLXZlcnNpb24tbWF0Y2giOnRydWUsIngtbnZpZGlhLWdwdS1kcml2ZXItdmVyc2lvbiI6IjU4MC4xNTkuMDMiLCJ4LW52aWRpYS1ncHUtdmJpb3MtaW5kZXgtbm8tY29uZmxpY3QiOnRydWUsIngtbnZpZGlhLWdwdS12Ymlvcy1yaW0tY2VydC1jaGFpbiI6eyJ4LW52aWRpYS1jZXJ0LWV4cGlyYXRpb24tZGF0ZSI6IjIwMjctMDQtMzBUMjM6NTk6NTlaIiwieC1udmlkaWEtY2VydC1vY3NwLW5vbmNlLW1hdGNoZXMiOnRydWUsIngtbnZpZGlhLWNlcnQtb2NzcC1yZXNwb25zZS12YWxpZCI6dHJ1ZSwieC1udmlkaWEtY2VydC1vY3NwLXN0YXR1cyI6Imdvb2QiLCJ4LW52aWRpYS1jZXJ0LXJldm9jYXRpb24tcmVhc29uIjpudWxsLCJ4LW52aWRpYS1jZXJ0LXN0YXR1cyI6InZhbGlkIn0sIngtbnZpZGlhLWdwdS12Ymlvcy1yaW0tZmV0Y2hlZCI6dHJ1ZSwieC1udmlkaWEtZ3B1LXZiaW9zLXJpbS1tZWFzdXJlbWVudHMtYXZhaWxhYmxlIjp0cnVlLCJ4LW52aWRpYS1ncHUtdmJpb3MtcmltLXNpZ25hdHVyZS12ZXJpZmllZCI6dHJ1ZSwieC1udmlkaWEtZ3B1LXZiaW9zLXJpbS12ZXJzaW9uLW1hdGNoIjp0cnVlLCJ4LW52aWRpYS1ncHUtdmJpb3MtdmVyc2lvbiI6Ijk2LjAwLkQ5LjAwLjAxIiwieC1udmlkaWEtbWlzbWF0Y2gtbWVhc3VyZW1lbnQtcmVjb3JkcyI6bnVsbH0." + } + ], + "result_code": 0, + "result_message": "Ok" +} diff --git a/dstack/guest-agent/rpc/proto/agent_rpc.proto b/dstack/guest-agent/rpc/proto/agent_rpc.proto index 892cca70b..cef6ea274 100644 --- a/dstack/guest-agent/rpc/proto/agent_rpc.proto +++ b/dstack/guest-agent/rpc/proto/agent_rpc.proto @@ -53,6 +53,9 @@ service DstackGuest { // Get app info rpc Info(google.protobuf.Empty) returns (AppInfo) {} + // Get GPU information collected during boot. + rpc GpuInfo(google.protobuf.Empty) returns (GpuInfoResponse) {} + // Sign a payload rpc Sign(SignRequest) returns (SignResponse) {} @@ -187,6 +190,12 @@ message AttestResponse { bytes attestation = 1; } +message GpuInfoResponse { + // Complete JSON output produced by nvattest. Empty when no boot-time GPU + // attestation output is available. + string attestation = 1; +} + message GetQuoteResponse { // TDX quote (empty on non-TDX platforms such as AMD SEV-SNP) bytes quote = 1; diff --git a/dstack/guest-agent/src/rpc_service.rs b/dstack/guest-agent/src/rpc_service.rs index cb9680353..e20126095 100644 --- a/dstack/guest-agent/src/rpc_service.rs +++ b/dstack/guest-agent/src/rpc_service.rs @@ -2,7 +2,10 @@ // // SPDX-License-Identifier: Apache-2.0 -use std::sync::{Arc, RwLock}; +use std::{ + path::Path, + sync::{Arc, RwLock}, +}; use anyhow::{Context, Result}; use base64::{engine::general_purpose::URL_SAFE_NO_PAD, Engine as _}; @@ -12,11 +15,11 @@ use dstack_guest_agent_rpc::{ tappd_server::{TappdRpc, TappdServer}, worker_server::{WorkerRpc, WorkerServer}, AppInfo, AttestResponse, DeriveK256KeyResponse, DeriveKeyArgs, GetAttestationForAppKeyRequest, - GetKeyArgs, GetKeyResponse, GetQuoteResponse, GetTlsKeyArgs, GetTlsKeyResponse, RawQuoteArgs, - SignRequest, SignResponse, TdxQuoteArgs, TdxQuoteResponse, VerifyRequest, VerifyResponse, - WorkerVersion, + GetKeyArgs, GetKeyResponse, GetQuoteResponse, GetTlsKeyArgs, GetTlsKeyResponse, + GpuInfoResponse, RawQuoteArgs, SignRequest, SignResponse, TdxQuoteArgs, TdxQuoteResponse, + VerifyRequest, VerifyResponse, WorkerVersion, }; -use dstack_types::{AppKeys, SysConfig}; +use dstack_types::{AppKeys, SysConfig, GPU_ATTESTATION_OUTPUT}; use ed25519_dalek::ed25519::signature::hazmat::{PrehashSigner, PrehashVerifier}; use ed25519_dalek::{ Signer as Ed25519Signer, SigningKey as Ed25519SigningKey, Verifier as Ed25519Verifier, @@ -36,7 +39,7 @@ use rcgen::KeyPair; use ring::rand::{SecureRandom, SystemRandom}; use serde_json::json; use sha3::{Digest, Keccak256}; -use tracing::error; +use tracing::{error, warn}; use crate::{ backend::{PlatformBackend, RealPlatform}, @@ -49,6 +52,20 @@ fn read_dmi_file(name: &str) -> String { .unwrap_or_default() } +/// Read the GPU attestation output saved during boot. Returns an empty string +/// when no output is available (e.g. no GPU attached or attestation disabled). +fn read_gpu_attestation(path: &Path) -> String { + match fs::read_to_string(path) { + Ok(attestation) => attestation, + Err(err) => { + if err.kind() != std::io::ErrorKind::NotFound { + warn!("failed to read GPU attestation output: {err:?}"); + } + String::new() + } + } +} + #[derive(Clone)] pub struct AppState { inner: Arc, @@ -328,6 +345,12 @@ impl DstackGuestRpc for InternalRpcHandler { get_info(&self.state, false).await } + async fn gpu_info(self) -> Result { + Ok(GpuInfoResponse { + attestation: read_gpu_attestation(Path::new(GPU_ATTESTATION_OUTPUT)), + }) + } + async fn sign(self, request: SignRequest) -> Result { let algorithm = normalize_algorithm(&request.algorithm); // Use the base algorithm for key derivation (e.g. secp256k1_prehashed -> secp256k1) @@ -668,6 +691,22 @@ mod tests { use std::convert::TryFrom; use std::io::Write; + #[test] + fn reads_gpu_attestation_output_verbatim() { + let mut output = tempfile::NamedTempFile::new().unwrap(); + let attestation = r#"{"result_code":0,"claims":[]}"#; + output.write_all(attestation.as_bytes()).unwrap(); + output.flush().unwrap(); + + assert_eq!(read_gpu_attestation(output.path()), attestation); + } + + #[test] + fn missing_gpu_attestation_output_reads_as_empty() { + let dir = tempfile::tempdir().unwrap(); + assert_eq!(read_gpu_attestation(&dir.path().join("missing")), ""); + } + fn extract_pubkey_from_report_data(report_data: &[u8], prefix: &str) -> Result> { let end = report_data .iter() diff --git a/dstack/kms/src/main_service.rs b/dstack/kms/src/main_service.rs index 6e3a4694d..77455ca6c 100644 --- a/dstack/kms/src/main_service.rs +++ b/dstack/kms/src/main_service.rs @@ -633,6 +633,7 @@ mod tests { dstack_types::mr_config::MrConfigV3::new( vec![0x11; 20], vec![0x22; 32], + None, dstack_types::KeyProviderKind::None, Vec::new(), vec![0x99; 20], diff --git a/dstack/kms/src/main_service/amd_attest.rs b/dstack/kms/src/main_service/amd_attest.rs index bcbbdbf4d..9e579984f 100644 --- a/dstack/kms/src/main_service/amd_attest.rs +++ b/dstack/kms/src/main_service/amd_attest.rs @@ -193,6 +193,7 @@ fn test_mr_config(app_id: Vec, compose_hash: Vec) -> MrConfigV3 { MrConfigV3::new( app_id, compose_hash, + None, dstack_types::KeyProviderKind::None, Vec::new(), instance_id, diff --git a/dstack/kms/src/onboard_service.rs b/dstack/kms/src/onboard_service.rs index 346e458de..84e01ef93 100644 --- a/dstack/kms/src/onboard_service.rs +++ b/dstack/kms/src/onboard_service.rs @@ -243,6 +243,7 @@ mod tests { dstack_types::mr_config::MrConfigV3::new( vec![0x11; 20], vec![0x22; 32], + None, dstack_types::KeyProviderKind::None, Vec::new(), vec![0x99; 20], diff --git a/dstack/vmm/src/app.rs b/dstack/vmm/src/app.rs index 2ab3aeee9..f4dcd0d12 100644 --- a/dstack/vmm/src/app.rs +++ b/dstack/vmm/src/app.rs @@ -132,6 +132,10 @@ pub struct GpuConfig { } impl GpuConfig { + pub fn has_gpus(&self) -> bool { + !self.gpus.is_empty() + } + pub fn is_empty(&self) -> bool { if self.attach_mode.is_all() { return false; @@ -1175,7 +1179,10 @@ impl App { let mr_config = if use_mr_config_v3 { Some( work_dir - .prepare_mr_config_v3(&app_compose) + .prepare_mr_config_v3( + &app_compose, + manifest.gpus.as_ref().is_some_and(GpuConfig::has_gpus), + ) .context("Failed to prepare mr_config")?, ) } else { @@ -1542,6 +1549,30 @@ mod tests { hex::encode(vec![byte; len]) } + #[test] + fn gpu_config_has_gpus_only_when_resolved_gpu_list_is_non_empty() { + assert!(!GpuConfig::default().has_gpus()); + assert!(!GpuConfig { + attach_mode: AttachMode::All, + ..Default::default() + } + .has_gpus()); + assert!(!GpuConfig { + bridges: vec![GpuSpec { + slot: "0000:01:00.0".into(), + }], + ..Default::default() + } + .has_gpus()); + assert!(GpuConfig { + gpus: vec![GpuSpec { + slot: "0000:02:00.0".into(), + }], + ..Default::default() + } + .has_gpus()); + } + #[test] fn put_manifest_keeps_legacy_networking_for_rollback() -> Result<()> { let temp = std::env::temp_dir().join(format!( @@ -1948,6 +1979,7 @@ mod tests { let mr_config = MrConfigV3::new( vec![0x11; 20], vec![0x22; 32], + None, dstack_types::KeyProviderKind::None, vec![], vec![0x44; 20], @@ -2002,6 +2034,7 @@ mod tests { assert_eq!(parsed_mr_config.app_id, vec![0x11; 20]); assert_eq!(parsed_mr_config.compose_hash, vec![0x22; 32]); + assert_eq!(parsed_mr_config.gpu_policy_hash, None); assert_eq!(vm_config["mr_config"], sys_config["mr_config"]); assert_eq!( vm_config["os_image_hash"] diff --git a/dstack/vmm/src/app/mr_config.rs b/dstack/vmm/src/app/mr_config.rs index f1b908e4f..434d8a1d1 100644 --- a/dstack/vmm/src/app/mr_config.rs +++ b/dstack/vmm/src/app/mr_config.rs @@ -7,7 +7,7 @@ use anyhow::{bail, Context, Result}; use base64::prelude::*; use dstack_types::mr_config::{MrConfig, MrConfigV3}; -use dstack_types::AppCompose; +use dstack_types::{gpu_policy_hash, AppCompose}; use fs_err as fs; use sha2::{Digest, Sha256}; @@ -60,10 +60,21 @@ pub(super) fn snp_host_data(workdir: &VmWorkDir) -> Result { } impl VmWorkDir { - pub fn prepare_mr_config_v3(&self, app_compose: &AppCompose) -> Result { + pub fn prepare_mr_config_v3(&self, app_compose: &AppCompose, has_gpus: bool) -> Result { let compose_hash = self .app_compose_hash() .context("failed to get compose hash")?; + let gpu_policy_hash = if has_gpus { + let compose_json = fs::read(self.app_compose_path()) + .context("failed to read app compose for GPU policy hash")?; + Some( + gpu_policy_hash(&compose_json) + .context("failed to hash raw GPU policy")? + .to_vec(), + ) + } else { + None + }; let mut instance_info = self .instance_info_or_default() .context("failed to get instance info")?; @@ -106,6 +117,7 @@ impl VmWorkDir { Ok(MrConfigV3::new( app_id, compose_hash.to_vec(), + gpu_policy_hash, app_compose.key_provider(), app_compose.key_provider_id.clone(), instance_id, diff --git a/dstack/vmm/src/one_shot.rs b/dstack/vmm/src/one_shot.rs index 3773fbcb2..abcb36ba4 100644 --- a/dstack/vmm/src/one_shot.rs +++ b/dstack/vmm/src/one_shot.rs @@ -247,7 +247,10 @@ Compose file content (first 200 chars): let mr_config = if use_mr_config_v3 { Some( vm_work_dir - .prepare_mr_config_v3(&app_compose) + .prepare_mr_config_v3( + &app_compose, + manifest.gpus.as_ref().is_some_and(|gpus| gpus.has_gpus()), + ) .context("Failed to prepare mr_config")?, ) } else { diff --git a/os/yocto/layers/meta-dstack/recipes-core/dstack-guest/dstack-guest.bb b/os/yocto/layers/meta-dstack/recipes-core/dstack-guest/dstack-guest.bb index 7a31ec53b..daa00f9ed 100644 --- a/os/yocto/layers/meta-dstack/recipes-core/dstack-guest/dstack-guest.bb +++ b/os/yocto/layers/meta-dstack/recipes-core/dstack-guest/dstack-guest.bb @@ -18,6 +18,12 @@ DSTACK_ROOTFS_FILES = "${UNPACKDIR}/repo/os/common/rootfs" RDEPENDS:${PN} += "bash" DEPENDS += "rsync-native tpm2-tss" +DEPENDS += "cmake-native" + +# aws-lc-sys cannot detect Yocto cross builds when the build and target share +# the same Rust target triple, and its cc builder then tries to execute a +# target binary on the build host. Use its supported CMake builder instead. +export AWS_LC_SYS_CMAKE_BUILDER = "1" # Ensure rsync-native is built before unpack runs do_unpack[depends] += "rsync-native:do_populate_sysroot" diff --git a/os/yocto/layers/meta-nvidia/recipes-graphics/nvattest/files/10-nvidia-gpu-ordering.conf b/os/yocto/layers/meta-nvidia/recipes-graphics/nvattest/files/10-nvidia-gpu-ordering.conf index 50e0b7d1f..83f2d1b5f 100644 --- a/os/yocto/layers/meta-nvidia/recipes-graphics/nvattest/files/10-nvidia-gpu-ordering.conf +++ b/os/yocto/layers/meta-nvidia/recipes-graphics/nvattest/files/10-nvidia-gpu-ordering.conf @@ -1,7 +1,8 @@ # Ordering only, not a hard dependency: on GPU instances, `dstack-util setup` # (run by dstack-prepare.service) performs local GPU TEE attestation per the -# app-compose `requirements.verify_gpu` policy, which needs the nvidia driver -# loaded (nvidia-persistenced) and the NVLink fabric up (nvidia-fabricmanager). +# app-compose `requirements.gpu_policy.attest_gpu` setting, which needs the +# nvidia driver loaded (nvidia-persistenced) and the NVLink fabric up +# (nvidia-fabricmanager). # Both units skip cleanly on GPU-less instances via nvidia-gpu-detect. [Unit] Wants=nvidia-persistenced.service nvidia-fabricmanager.service diff --git a/os/yocto/layers/meta-nvidia/recipes-graphics/nvattest/nvattest_2026.06.09.bb b/os/yocto/layers/meta-nvidia/recipes-graphics/nvattest/nvattest_2026.06.09.bb index 7d907d69d..3494a8809 100644 --- a/os/yocto/layers/meta-nvidia/recipes-graphics/nvattest/nvattest_2026.06.09.bb +++ b/os/yocto/layers/meta-nvidia/recipes-graphics/nvattest/nvattest_2026.06.09.bb @@ -1,5 +1,5 @@ SUMMARY = "NVIDIA GPU attestation CLI" -DESCRIPTION = "Builds NVIDIA's nvattest CLI. dstack-util setup runs it at boot to gate readiness on local GPU TEE attestation (app-compose requirements.verify_gpu)." +DESCRIPTION = "Builds NVIDIA's nvattest CLI. dstack-util setup runs it at boot to gate readiness on local GPU TEE attestation and an optional application policy." HOMEPAGE = "https://github.com/NVIDIA/attestation-sdk" LICENSE = "Apache-2.0" LIC_FILES_CHKSUM = "file://LICENSE;md5=e620fc90e76c4aa0c3efdd1673ca0b3b" diff --git a/os/yocto/layers/meta-nvidia/recipes-graphics/nvidia/files/nvidia-persistenced.service b/os/yocto/layers/meta-nvidia/recipes-graphics/nvidia/files/nvidia-persistenced.service index f40aef461..74ddc332f 100644 --- a/os/yocto/layers/meta-nvidia/recipes-graphics/nvidia/files/nvidia-persistenced.service +++ b/os/yocto/layers/meta-nvidia/recipes-graphics/nvidia/files/nvidia-persistenced.service @@ -18,8 +18,9 @@ ExecStartPre=/usr/bin/env modprobe nvidia-uvm ExecStart=/usr/bin/env nvidia-persistenced --uvm-persistence-mode # NOTE: the GPU ready state (conf-compute -srs 1) is intentionally NOT set # here. It is set by `dstack-util setup` (dstack-prepare.service), only after -# local GPU TEE attestation via nvattest succeeds -- unless the deployment -# opts out with app-compose requirements.verify_gpu=false. +# local GPU TEE attestation and any application GPU policy succeed -- unless +# the deployment opts out with app-compose +# requirements.gpu_policy.attest_gpu=false. RemainAfterExit=yes [Install] diff --git a/sdk/curl/api.md b/sdk/curl/api.md index fc7c766c2..fb02b9052 100644 --- a/sdk/curl/api.md +++ b/sdk/curl/api.md @@ -285,6 +285,54 @@ curl --unix-socket /var/run/dstack.sock http://dstack/Attest?report_data=0000000 } ``` +### 8. GPU Info + +Returns GPU information collected during boot. Currently, this includes the +complete JSON output produced by NVIDIA `nvattest`. +The `attestation` field is empty when no GPU attestation output is available, +for example on a VM without an NVIDIA GPU or when GPU attestation was disabled. + +**Endpoint:** `/GpuInfo` + +**Example:** +```bash +curl --unix-socket /var/run/dstack.sock http://dstack/GpuInfo +``` + +**Response:** +```json +{ + "attestation": "{\"result_code\": 0, \"claims\": [...]}" +} +``` + +`GpuInfo.attestation` is the exact UTF-8 `nvattest` output saved during boot; +calling this endpoint does not perform a new attestation. To authenticate it on +TDX, first verify the quote and replay the supplied event log to the quote's +RTMR3. Then decode the `gpu-attestation` event payload and compare its +`evidence_sha256` with the SHA-256 digest of the exact returned string: + +```python +import hashlib +import json + +gpu_info = json.load(open("gpu-info.json")) +quote_response = json.load(open("quote.json")) +events = quote_response["event_log"] +if isinstance(events, str): + events = json.loads(events) + +entry = next(event for event in events if event["event"] == "gpu-attestation") +measured = json.loads(bytes.fromhex(entry["event_payload"])) +actual = hashlib.sha256(gpu_info["attestation"].encode()).hexdigest() +assert actual == measured["evidence_sha256"] +``` + +The comparison above is meaningful only after quote verification and RTMR3 +event-log replay have succeeded. See the +[security model](../../docs/security/security-model.md#gpu-security-for-ai-workloads) +for the event ordering and AWS/AMD SEV-SNP verification paths. + ## Error Responses All endpoints may return the following HTTP status codes: diff --git a/sdk/go/README.md b/sdk/go/README.md index e323322e0..c5b7f8bac 100644 --- a/sdk/go/README.md +++ b/sdk/go/README.md @@ -581,6 +581,23 @@ Generates a TDX attestation quote containing the provided report data. - Cryptographic proof of execution environment - Audit trail generation +##### `GpuInfo(ctx context.Context) (*GpuInfoResponse, error)` + +Returns GPU information collected during boot. Currently, this includes the +complete NVIDIA `nvattest` JSON output. + +```go +gpu, err := client.GpuInfo(ctx) +if err != nil { + log.Fatal(err) +} +fmt.Println(gpu.Attestation) +``` + +The `Attestation` field is empty when no GPU attestation output is available. +The raw output is not trusted by itself; remote verifiers should compare its +digest with the measured `gpu-attestation` runtime event. + ##### `GetTlsKey(ctx context.Context, options TlsKeyOptions) (*GetTlsKeyResponse, error)` Generates a fresh, random TLS key pair with X.509 certificate for TLS/SSL connections. **Important**: This method generates different keys on each call - use `GetKey()` for deterministic keys. diff --git a/sdk/go/dstack/client.go b/sdk/go/dstack/client.go index 98e2dd730..4e8fa30e7 100644 --- a/sdk/go/dstack/client.go +++ b/sdk/go/dstack/client.go @@ -119,6 +119,11 @@ type AttestResponse struct { Attestation []byte } +// GpuInfoResponse contains GPU information collected during boot. +type GpuInfoResponse struct { + Attestation string `json:"attestation"` +} + // Represents an event log entry in the TCB info type EventLog struct { IMR int `json:"imr"` @@ -579,6 +584,20 @@ func (c *DstackClient) Attest(ctx context.Context, reportData []byte) (*AttestRe return &AttestResponse{Attestation: attestation}, nil } +// GpuInfo returns GPU information collected during boot. +func (c *DstackClient) GpuInfo(ctx context.Context) (*GpuInfoResponse, error) { + data, err := c.sendRPCRequest(ctx, "/GpuInfo", map[string]interface{}{}) + if err != nil { + return nil, err + } + + var response GpuInfoResponse + if err := json.Unmarshal(data, &response); err != nil { + return nil, err + } + return &response, nil +} + // Represents the response from a Version request. type VersionResponse struct { Version string `json:"version"` diff --git a/sdk/go/dstack/client_test.go b/sdk/go/dstack/client_test.go index e6072a73e..13ee88c99 100644 --- a/sdk/go/dstack/client_test.go +++ b/sdk/go/dstack/client_test.go @@ -13,6 +13,8 @@ import ( "encoding/json" "encoding/pem" "fmt" + "net/http" + "net/http/httptest" "strings" "testing" @@ -116,6 +118,27 @@ func TestAttest(t *testing.T) { } } +func TestGpuInfo(t *testing.T) { + const attestation = `{"result_code":0,"claims":[]}` + server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if r.URL.Path != "/GpuInfo" { + t.Fatalf("unexpected path: %s", r.URL.Path) + } + w.Header().Set("Content-Type", "application/json") + _ = json.NewEncoder(w).Encode(map[string]string{"attestation": attestation}) + })) + defer server.Close() + + client := dstack.NewDstackClient(dstack.WithEndpoint(server.URL)) + response, err := client.GpuInfo(context.Background()) + if err != nil { + t.Fatal(err) + } + if response.Attestation != attestation { + t.Fatalf("unexpected attestation: %s", response.Attestation) + } +} + func TestGetTlsKey(t *testing.T) { client := dstack.NewDstackClient() altNames := []string{"localhost"} diff --git a/sdk/js/README.md b/sdk/js/README.md index c1686e24b..1cc1d9316 100644 --- a/sdk/js/README.md +++ b/sdk/js/README.md @@ -97,6 +97,20 @@ Versioned dstack attestation that works across TDX / GCP / Nitro providers. Pref const { attestation } = await client.attest('app-state-snapshot') ``` +### `gpuInfo()` + +Returns GPU information collected during boot. Currently, this includes the +complete NVIDIA `nvattest` JSON output. + +```typescript +const gpu = await client.gpuInfo() +console.log(gpu.attestation) +``` + +The `attestation` field is empty when no GPU attestation output is available. +The raw output is not trusted by itself; remote verifiers should compare its +digest with the measured `gpu-attestation` runtime event. + ### `info()` App identity and TCB metadata. diff --git a/sdk/js/src/index.ts b/sdk/js/src/index.ts index 6a6d57144..8226cbc27 100644 --- a/sdk/js/src/index.ts +++ b/sdk/js/src/index.ts @@ -109,6 +109,12 @@ export interface AttestResponse { attestation: Hex } +export interface GpuInfoResponse { + __name__: Readonly<'GpuInfoResponse'> + + attestation: string +} + export interface VersionResponse { __name__: Readonly<'VersionResponse'> @@ -329,6 +335,14 @@ export class DstackClient { }) } + async gpuInfo(): Promise { + const result = await send_rpc_request<{ attestation: string }>(this.endpoint, '/GpuInfo', '{}') + return Object.freeze({ + ...result, + __name__: 'GpuInfoResponse', + }) + } + async info(): Promise> { const result = await send_rpc_request, 'tcb_info'> & { tcb_info: string }>(this.endpoint, '/Info', '{}') return Object.freeze({ diff --git a/sdk/python/README.md b/sdk/python/README.md index bb6896076..e112f18d4 100644 --- a/sdk/python/README.md +++ b/sdk/python/README.md @@ -101,6 +101,20 @@ print(result.attestation) # hex string print(result.decode_attestation()) # bytes ``` +### GPU Info + +`gpu_info()` returns GPU information collected during boot. Currently, this +includes the complete NVIDIA `nvattest` JSON output. + +```python +gpu = client.gpu_info() +print(gpu.attestation) +``` + +The `attestation` field is empty when no GPU attestation output is available. +The raw output is not trusted by itself; remote verifiers should compare its +digest with the measured `gpu-attestation` runtime event. + ### Get Instance Info ```python diff --git a/sdk/python/src/dstack_sdk/__init__.py b/sdk/python/src/dstack_sdk/__init__.py index f1c20404c..e93c6009f 100644 --- a/sdk/python/src/dstack_sdk/__init__.py +++ b/sdk/python/src/dstack_sdk/__init__.py @@ -10,6 +10,7 @@ from .dstack_client import GetKeyResponse from .dstack_client import GetQuoteResponse from .dstack_client import GetTlsKeyResponse +from .dstack_client import GpuInfoResponse from .dstack_client import InfoResponse from .dstack_client import SignResponse from .dstack_client import TappdClient @@ -36,6 +37,7 @@ "GetKeyResponse", "GetTlsKeyResponse", "AttestResponse", + "GpuInfoResponse", "GetQuoteResponse", "InfoResponse", "TcbInfo", diff --git a/sdk/python/src/dstack_sdk/dstack_client.py b/sdk/python/src/dstack_sdk/dstack_client.py index 0f9897793..07cfd49bf 100644 --- a/sdk/python/src/dstack_sdk/dstack_client.py +++ b/sdk/python/src/dstack_sdk/dstack_client.py @@ -186,6 +186,10 @@ def decode_attestation(self) -> bytes: return bytes.fromhex(self.attestation) +class GpuInfoResponse(BaseModel): + attestation: str + + class SignResponse(BaseModel): signature: str signature_chain: List[str] @@ -461,6 +465,11 @@ async def attest( result = await self._send_rpc_request("Attest", {"report_data": hex}) return AttestResponse(**result) + async def gpu_info(self) -> GpuInfoResponse: + """Return GPU information collected during boot.""" + result = await self._send_rpc_request("GpuInfo", {}) + return GpuInfoResponse(**result) + async def info(self) -> InfoResponse[TcbInfo]: """Fetch service information including parsed TCB info.""" result = await self._send_rpc_request("Info", {}) @@ -604,6 +613,11 @@ def attest( """Request a versioned attestation for the provided report data.""" raise NotImplementedError + @call_async + def gpu_info(self) -> GpuInfoResponse: + """Return GPU information collected during boot.""" + raise NotImplementedError + @call_async def info(self) -> InfoResponse[TcbInfo]: """Fetch service information including parsed TCB info.""" diff --git a/sdk/python/tests/test_client.py b/sdk/python/tests/test_client.py index 619ddb52d..cec867e72 100644 --- a/sdk/python/tests/test_client.py +++ b/sdk/python/tests/test_client.py @@ -16,6 +16,7 @@ from dstack_sdk import GetKeyResponse from dstack_sdk import GetQuoteResponse from dstack_sdk import GetTlsKeyResponse +from dstack_sdk import GpuInfoResponse from dstack_sdk import SignResponse from dstack_sdk import TappdClient from dstack_sdk import VerifyResponse @@ -120,6 +121,22 @@ async def test_async_client_attest(): assert len(result.attestation) > 0 +@pytest.mark.asyncio +async def test_async_client_gpu_info(monkeypatch): + attestation = '{"result_code":0,"claims":[]}' + + async def fake_send(self, method, payload): + assert method == "GpuInfo" + assert payload == {} + return {"attestation": attestation} + + monkeypatch.setenv("DSTACK_SIMULATOR_ENDPOINT", "http://localhost:0") + monkeypatch.setattr(AsyncDstackClient, "_send_rpc_request", fake_send) + result = await AsyncDstackClient().gpu_info() + assert isinstance(result, GpuInfoResponse) + assert result.attestation == attestation + + @pytest.mark.asyncio async def test_async_client_get_tls_key(): client = AsyncDstackClient() diff --git a/sdk/python/tests/test_typing.py b/sdk/python/tests/test_typing.py index 1e5c49467..4aa38e469 100644 --- a/sdk/python/tests/test_typing.py +++ b/sdk/python/tests/test_typing.py @@ -12,6 +12,7 @@ from dstack_sdk import GetKeyResponse from dstack_sdk import GetQuoteResponse from dstack_sdk import GetTlsKeyResponse +from dstack_sdk import GpuInfoResponse from dstack_sdk.dstack_client import InfoResponse # Use a test endpoint to avoid socket file not found errors @@ -52,6 +53,7 @@ def test_all_sync_method_types(): expected_types = { "get_key": GetKeyResponse, + "gpu_info": GpuInfoResponse, "get_quote": GetQuoteResponse, "get_tls_key": GetTlsKeyResponse, "info": InfoResponse, @@ -93,6 +95,7 @@ def test_async_method_types(): expected_types = { "get_key": GetKeyResponse, + "gpu_info": GpuInfoResponse, "get_quote": GetQuoteResponse, "get_tls_key": GetTlsKeyResponse, "info": InfoResponse, @@ -126,6 +129,7 @@ def test_method_signature_comparison(): methods_to_check = [ "get_key", + "gpu_info", "get_quote", "get_tls_key", "info", diff --git a/sdk/rust/README.md b/sdk/rust/README.md index 4c9a32625..36a54a3ae 100644 --- a/sdk/rust/README.md +++ b/sdk/rust/README.md @@ -105,6 +105,20 @@ println!("{}", info.tcb_info); Generates a versioned attestation with a custom 64-byte payload. - `attestation`: Hex-encoded attestation +#### `gpu_info() -> GpuInfoResponse` + +Returns GPU information collected during boot. Currently, this includes the +complete NVIDIA `nvattest` JSON output. + +```rust +let gpu = client.gpu_info().await?; +println!("{}", gpu.attestation); +``` + +The `attestation` field is empty when no GPU attestation output is available. +The raw output is not trusted by itself; remote verifiers should compare its +digest with the measured `gpu-attestation` runtime event. + ### Generate TLS Certificates `get_tls_key()` creates fresh TLS certificates. Unlike `get_key()`, each call generates a new random key. diff --git a/sdk/rust/src/dstack_client.rs b/sdk/rust/src/dstack_client.rs index 537b58c99..d33a04e62 100644 --- a/sdk/rust/src/dstack_client.rs +++ b/sdk/rust/src/dstack_client.rs @@ -166,6 +166,13 @@ impl DstackClient { Ok(response) } + /// Returns GPU information collected during boot. + pub async fn gpu_info(&self) -> Result { + let response = self.send_rpc_request("/GpuInfo", &json!({})).await?; + let response = serde_json::from_value::(response)?; + Ok(response) + } + pub async fn info(&self) -> Result { let response = self.send_rpc_request("/Info", &json!({})).await?; Ok(InfoResponse::validated_from_value(response)?) diff --git a/sdk/rust/types/src/dstack.rs b/sdk/rust/types/src/dstack.rs index aa6263fe0..d97b169b8 100644 --- a/sdk/rust/types/src/dstack.rs +++ b/sdk/rust/types/src/dstack.rs @@ -127,6 +127,15 @@ pub struct AttestResponse { pub attestation: String, } +/// Response containing the complete NVIDIA GPU attestation output. +#[derive(Debug, Serialize, Deserialize)] +#[cfg_attr(feature = "borsh", derive(BorshSerialize, BorshDeserialize))] +#[cfg_attr(feature = "borsh_schema", derive(BorshSchema))] +pub struct GpuInfoResponse { + /// Complete JSON output produced by nvattest during guest boot. + pub attestation: String, +} + impl AttestResponse { pub fn decode_attestation(&self) -> Result, FromHexError> { hex::decode(&self.attestation)