feat: caching OCSP/RIM proxy for GPU attestation collateral#790
feat: caching OCSP/RIM proxy for GPU attestation collateral#790kvinwang wants to merge 5 commits into
Conversation
There was a problem hiding this comment.
Pull request overview
This PR adds a host-side caching proxy (gpu-attest-proxy) for NVIDIA GPU attestation collateral (OCSP + RIM) and wires it through VMM sys-config into the guest so nvattest can route collateral requests via the proxy and use a nonce-less relying-party policy when appropriate. The goal is to prevent GPU CVM boot from failing closed due to NVIDIA endpoint reachability/outages (boot DoS risk) while keeping guest-side signature/validity verification.
Changes:
- Introduces
dstack/gpu-attest-proxy(Rocket + reqwest) with OCSP-by-CertID caching, RIM caching with stale-serve window, and a background refresh loop. - Adds sys-config plumbing for
gpu_attest_proxy_urlfromvmm.toml→ VMM config →SysConfig→ guest boot. - Updates guest GPU attestation flow to optionally pass
--ocsp-url/--rim-urland a packaged NVIDIA “outpost” policy; updates Yocto recipe to include that policy; adds docs.
Reviewed changes
Copilot reviewed 15 out of 16 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| os/yocto/layers/meta-nvidia/recipes-graphics/nvattest/nvattest_2026.06.09.bb | Packages NVIDIA’s allow_trust_outpost_ocsp.rego into the image for proxy mode. |
| dstack/vmm/vmm.toml | Documents the new [cvm] gpu_attest_proxy_url setting. |
| dstack/vmm/src/config.rs | Adds gpu_attest_proxy_url to VMM CVM configuration parsing. |
| dstack/vmm/src/app.rs | Passes gpu_attest_proxy_url through to generated sys-config JSON. |
| dstack/gpu-attest-proxy/src/proxy.rs | Implements OCSP/RIM proxy endpoints plus health/info and refresh loop. |
| dstack/gpu-attest-proxy/src/main.rs | CLI + config loading + Rocket launch + refresh task spawn. |
| dstack/gpu-attest-proxy/src/lib.rs | Exposes proxy/cache/DER modules. |
| dstack/gpu-attest-proxy/src/der.rs | Minimal DER walkers for OCSP request CertID extraction and response validity parsing. |
| dstack/gpu-attest-proxy/src/cache.rs | File-backed JSON cache with stale retention and basic stats. |
| dstack/gpu-attest-proxy/gpu-attest-proxy.toml | Default runtime configuration (Rocket + upstreams + TTLs). |
| dstack/gpu-attest-proxy/Cargo.toml | New crate manifest and dependencies. |
| dstack/dstack-util/src/system_setup.rs | Guest boot now optionally routes nvattest OCSP/RIM through proxy + uses outpost policy. |
| dstack/dstack-types/src/lib.rs | Extends SysConfig with gpu_attest_proxy_url. |
| dstack/Cargo.toml | Adds gpu-attest-proxy to workspace members. |
| dstack/Cargo.lock | Locks dependencies for the new crate. |
| docs/gpu-attestation-proxy.md | Documents proxy behavior, trust trade-offs, and configuration. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| fn nvattest_args(nonce: &str, proxy_url: Option<&str>) -> Result<Vec<String>> { | ||
| let proxy_base = match proxy_url { | ||
| Some(url) => { | ||
| let base = url.trim_end_matches('/'); | ||
| if base.is_empty() { | ||
| bail!("sys-config gpu_attest_proxy_url is empty"); | ||
| } | ||
| Some(base) | ||
| } | ||
| None => None, | ||
| }; | ||
| if proxy_base.is_some() && !Path::new(OUTPOST_POLICY).exists() { | ||
| bail!( | ||
| "sys-config gpu_attest_proxy_url is set but {} is not packaged in this image", | ||
| OUTPOST_POLICY | ||
| ); | ||
| } | ||
| Ok(nvattest_cmdline(nonce, proxy_base)) | ||
| } |
There was a problem hiding this comment.
Good catch. This is still valid in the current branch: an empty or whitespace-only value is documented as disabling the proxy, so it should be normalized to None rather than failing GPU attestation. We will align this with the existing pccs_url behavior.
| if let Some(base) = proxy_url { | ||
| info!( | ||
| "routing GPU attestation collateral through proxy: {}", | ||
| base.trim_end_matches('/') | ||
| ); | ||
| } |
There was a problem hiding this comment.
Agreed. This remains valid and is the logging side of the same normalization issue. The log should use the normalized non-empty proxy value, not the original optional input.
| // NVIDIA's SDK queries one certificate per request. | ||
| let key = hex::encode(&cert_ids[0]); | ||
| let now = unix_now(); |
There was a problem hiding this comment.
Agreed. This remains valid: the cache key only represents the first CertID, so accepting a multi-CertID request is unsafe even if NVIDIA currently sends one CertID. We should require exactly one CertID and return 400 otherwise.
| let content_type = resp | ||
| .headers() | ||
| .get(reqwest::header::CONTENT_TYPE) | ||
| .and_then(|value| value.to_str().ok()) | ||
| .and_then(|value| value.parse::<ContentType>().ok()) | ||
| .unwrap_or(ContentType::Binary); |
There was a problem hiding this comment.
Agreed. This remains valid. Since this endpoint specifically proxies OCSP responses, a missing or unparsable upstream Content-Type should default to application/ocsp-response, not application/octet-stream.
| let text = std::str::from_utf8(bytes).context("GeneralizedTime is not ASCII")?; | ||
| let digits = text.strip_suffix('Z').unwrap_or(text); | ||
| if digits.len() != 14 || !digits.bytes().all(|b| b.is_ascii_digit()) { | ||
| bail!("unsupported GeneralizedTime format: {text}"); | ||
| } | ||
| let num = |range: std::ops::Range<usize>| digits[range].parse::<i64>(); | ||
| let (Ok(year), Ok(month), Ok(day), Ok(hour), Ok(min), Ok(sec)) = ( | ||
| num(0..4), | ||
| num(4..6), | ||
| num(6..8), | ||
| num(8..10), | ||
| num(10..12), | ||
| num(12..14), | ||
| ) else { | ||
| bail!("unparseable GeneralizedTime: {text}"); | ||
| }; | ||
| if !(1..=12).contains(&month) || !(1..=31).contains(&day) { | ||
| bail!("out-of-range GeneralizedTime: {text}"); | ||
| } |
There was a problem hiding this comment.
Agreed. This remains valid because the parsed value controls cache expiry. The parser should require the trailing Z and validate the complete UTC date/time, including real month lengths and hour/minute/second ranges.
Plumb an optional NVIDIA GPU attestation collateral proxy URL from the VMM config through sys-config, following the existing pccs_url pattern. Empty (the default) keeps guests talking to NVIDIA's endpoints directly.
NVIDIA's sample relying-party policy keeps every built-in appraisal check except the OCSP nonce match. dstack-util passes it to nvattest when a caching collateral proxy is configured, because a cached OCSP response can never echo the per-request nonce.
When sys-config carries gpu_attest_proxy_url, run nvattest with --ocsp-url/--rim-url pointing at the proxy and --relying-party-policy allow_trust_outpost_ocsp.rego so cached OCSP responses (which cannot match the request nonce) are still appraised for signature, validity window and measurements. Fail closed if the policy file is missing.
A PCCS-style cache for NVIDIA GPU attestation collateral, run on the
host or at site level:
- POST /ocsp relays requests byte-for-byte and caches successful
responses keyed by CertID until their nextUpdate, so reboots and
fleets stop depending on the responder at every boot.
- GET /v1/rim/{id} caches RIM documents with a TTL and serves stale
within a bounded window when the upstream is unreachable.
- A background refresher renews entries before expiry, letting a warm
cache ride through upstream outages with close to a full validity
window.
- The proxy holds no secrets: guests still verify OCSP signatures,
validity windows and RIM signatures themselves.
The minimal DER walker extracts CertIDs from requests (cache key) and
thisUpdate/nextUpdate from responses (expiry). Includes unit tests and
a documented default configuration.
Covers the motivation (issue #778: OCSP as a boot DoS), the nonce relaxation trade-off, setup via vmm.toml, and operational notes.
9035db0 to
0f15ec9
Compare
|
In favor of #791 |
Summary
Addresses the third follow-up in #778 ("Don't let OCSP turn into a boot DoS"), stacked on #789.
Local GPU attestation calls NVIDIA's OCSP responder and RIM service live at every boot, and the gate fails closed — so egress-restricted networks, air-gapped deployments, or NVIDIA downtime stop every GPU CVM from booting. This PR adds a PCCS-style caching proxy, in the same spirit as the existing
pccs_urlplumbing for TDX:gpu-attest-proxy(new host-side crate): relays and caches NVIDIA attestation collateral.POST /ocsp— byte-transparent relay; successful responses cached keyed by CertID (parsed out of the request with a minimal DER walker) until theirnextUpdate(parsed from the response; falls back to thisUpdate + 1h, matching the SDK).GET /v1/rim/{id}— RIM documents cached by id with a TTL; stale entries served within a bounded window when the upstream is unreachable./health+/infofor ops.[cvm] gpu_attest_proxy_urlin vmm.toml →SysConfig.gpu_attest_proxy_url→ guest.dstack-utilruns nvattest with--ocsp-url {base}/ocsp,--rim-url {base}, and--relying-party-policy allow_trust_outpost_ocsp.rego.allow_trust_outpost_ocsp.regoat/usr/share/nvattest/policies/.Why the policy swap is safe
The SDK's built-in appraisal requires the OCSP response to echo the request nonce — which a cached response can never do. The packaged NVIDIA sample policy keeps every other check (cert chains valid, OCSP status good, response signature + validity window verified guest-side by
OCSP_basic_verify, measurements match). Freshness degrades from per-request nonce to the response validity window — the same trade-off the SDK's ownNvHttpOcspCacheClientmakes. The proxy holds no secrets and cannot forge collateral; worst case it withholds service, which fails closed (no worse than blocking NVIDIA directly). Guests without the config keep the default policy + nonce check.Test plan
cargo test -p gpu-attest-proxy— DER extraction (CertID, validity window, GeneralizedTime), cache persistence/stalenesscargo test -p dstack-util -p dstack-types -p dstack-vmm— incl. newnvattest_cmdlineproxy-arg testcargo clippyclean on touched cratesNV_GPU_DRIVER_GH100_580.159.03through the proxy fromrim.attestation.nvidia.comgpu_attest_proxy_urlset (needs GPU host)Notes
nextUpdatewindow while the proxy is configured; documented indocs/gpu-attestation-proxy.md.relying_party_policy_examples/allow_trust_outpost_ocsp.rego@9d12801.