feat: caching OCSP/RIM proxy for GPU attestation collateral

docs/gpu-attestation-proxy.md

@@ -0,0 +1,90 @@
+# GPU Attestation Collateral Proxy
+
+Local GPU attestation (`nvattest --verifier local`) makes live calls to two
+NVIDIA services at every CVM boot: the OCSP responder
+(`ocsp.ndis.nvidia.com`, one request per certificate in three chains) and the
+RIM service (`rim.attestation.nvidia.com`, driver + VBIOS RIM documents).
+Because the GPU attestation gate fails closed, anything that breaks those
+calls — an egress-restricted or air-gapped network, an upstream outage, or a
+middlebox — keeps every GPU CVM from booting.
+
+`gpu-attest-proxy` is a small host-side (or site-level) caching proxy that
+removes the guest's direct dependency on those services, in the same spirit
+as Intel's PCCS for TDX collateral:
+
+- **OCSP** requests are relayed byte-for-byte; successful responses are
+  cached keyed by CertID until their `nextUpdate`, so repeat boots and
+  fleets of CVMs sharing GPUs stop hitting the responder.
+- **RIM** documents are cached by RIM id and re-fetched on a TTL; when the
+  upstream is unreachable, a stale document keeps being served within a
+  bounded window.
+- A background refresher renews entries before they expire, so a warm cache
+  rides through upstream outages with close to a full validity window.
+
+Everything the proxy serves is signed collateral that the guest verifies
+itself (OCSP response signatures and validity windows via
+`OCSP_basic_verify`, RIM signatures and certificate chains by the attestation
+SDK). The proxy needs no secrets and cannot forge responses; its worst
+misbehavior is withholding service, which fails closed — the same outcome as
+blocking NVIDIA's endpoints directly.
+
+## Why the OCSP nonce check is relaxed
+
+The attestation SDK adds a random nonce to every OCSP request and its
+built-in appraisal policy requires the response to echo it
+(`x-nvidia-cert-ocsp-nonce-matches`). A response cached for another request
+can never echo this request's nonce, so caching is only possible if the
+guest relaxes that single check.
+
+When `gpu_attest_proxy_url` is set, `dstack-util` runs nvattest with
+`--relying-party-policy` pointing at NVIDIA's
+`allow_trust_outpost_ocsp.rego` (packaged in the image at
+`/usr/share/nvattest/policies/`), which keeps every built-in check except
+the nonce match. Freshness is then bounded by the response validity window
+instead of the nonce: a revoked certificate may keep passing until the
+cached response's `nextUpdate`. This is the same trade-off the SDK's own
+in-process OCSP cache makes. Guests without the proxy configured keep the
+default policy and the nonce check.
+
+## Setup
+
+Run the proxy somewhere every CVM can reach (typically on each host, next to
+dstack-vmm):
+
+```console
+$ gpu-attest-proxy -c /etc/gpu-attest-proxy/gpu-attest-proxy.toml
+```
+
+See `dstack/gpu-attest-proxy/gpu-attest-proxy.toml` for the default
+configuration (upstreams, cache directory, TTLs). The cache persists as one
+JSON file per entry under `cache_dir`, so restarts keep it warm.
+
+Point guests at it through `vmm.toml`:
+
+```toml
+[cvm]
+gpu_attest_proxy_url = "http://<host-bridge-ip>:8090"
+```
+
+The VMM passes the URL to the guest via sys-config. At boot, `dstack-util`
+derives the SDK endpoints from it:
+
+- `--ocsp-url {base}/ocsp`
+- `--rim-url {base}` (the SDK appends `/v1/rim/{rim_id}` itself)
+- `--relying-party-policy /usr/share/nvattest/policies/allow_trust_outpost_ocsp.rego`
+
+When the value is empty (default), guests talk to NVIDIA directly and the
+built-in appraisal policy applies unchanged.
+
+## Operational notes
+
+- `GET /health` and `GET /info` expose liveness and cache statistics.
+- A cold cache cannot help: the first boot after an outage begins still
+  needs a reachable upstream. Keep the proxy running so entries stay warm;
+  `refresh_margin` controls how far ahead of expiry entries are renewed.
+- The guest-side clock check is unchanged: OCSP validity windows are
+  verified by the guest, so an expired cached response never passes.
+- Revocation visibility degrades from "immediate" (nonce) to the OCSP
+  response validity window (`nextUpdate`). Measure the actual window for
+  your certificate chains — it determines both how long an outage the cache
+  can absorb and how long a revocation can go unnoticed.

dstack/Cargo.toml

@@ -59,6 +59,7 @@ members = [
     "dstack-mr",
     "dstack-mr/cli",
     "verifier",
+    "gpu-attest-proxy",
     "size-parser",
     "port-forward",
     "crates/dstack-cli-core",

dstack/dstack-types/src/lib.rs

@@ -710,6 +710,12 @@ pub struct SysConfig {
     #[serde(default, alias = "tproxy_urls")]
     pub gateway_urls: Vec<String>,
     pub pccs_url: Option<String>,
+    /// Base URL of an NVIDIA GPU attestation collateral proxy (OCSP + RIM
+    /// cache, e.g. gpu-attest-proxy). When set, the guest routes nvattest's
+    /// OCSP and RIM requests through it instead of NVIDIA's endpoints and
+    /// relaxes the OCSP nonce check via a packaged relying-party policy.
+    #[serde(default)]
+    pub gpu_attest_proxy_url: Option<String>,
     pub docker_registry: Option<String>,
     pub host_api_url: Option<String>,
     /// MrConfigV3 document string for platform app/config binding.

dstack/dstack-util/src/system_setup.rs

@@ -1083,6 +1083,10 @@ mod gpu {
     const POLICY_ENTRYPOINT: &str = "data.policy.nv_match";
     /// Bound Rego evaluation so a runaway application policy cannot hang boot.
     const POLICY_TIMEOUT: Duration = Duration::from_secs(10);
+    /// NVIDIA's sample relying-party policy packaged by the nvattest recipe.
+    /// It keeps every built-in appraisal check except the OCSP nonce match,
+    /// which a caching collateral proxy can never satisfy.
+    const OUTPOST_POLICY: &str = "/usr/share/nvattest/policies/allow_trust_outpost_ocsp.rego";
 
     #[derive(Debug, Clone, Copy, PartialEq, Eq)]
     pub(super) struct GpuInventory {
@@ -1368,11 +1372,70 @@ mod gpu {
         serde_json::to_vec(&event).context("failed to serialize GPU attestation event")
     }
 
-    /// Run local GPU attestation via nvattest with a fresh nonce and no custom
-    /// relying-party policy. nvattest's built-in appraisal still applies. The
-    /// complete JSON output is preserved under /run and retained with its
-    /// claims for the application policy and versioned RTMR3 summary.
-    pub(super) async fn attest_gpu(expected_devices: u32) -> Result<GpuAttestationResult> {
+    /// Build the nvattest command line. When `proxy_url` is set, OCSP and RIM
+    /// requests are routed through the caching collateral proxy and the
+    /// packaged outpost policy replaces nvattest's built-in appraisal (a
+    /// cached OCSP response can never echo the per-request nonce). The policy
+    /// file must exist — it is packaged by the nvattest recipe; fail closed
+    /// with a clear error rather than silently skipping appraisal.
+    fn nvattest_args(nonce: &str, proxy_url: Option<&str>) -> Result<Vec<String>> {
+        let proxy_base = match proxy_url {
+            Some(url) => {
+                let base = url.trim_end_matches('/');
+                if base.is_empty() {
+                    bail!("sys-config gpu_attest_proxy_url is empty");
+                }
+                Some(base)
+            }
+            None => None,
+        };
+        if proxy_base.is_some() && !Path::new(OUTPOST_POLICY).exists() {
+            bail!(
+                "sys-config gpu_attest_proxy_url is set but {} is not packaged in this image",
+                OUTPOST_POLICY
+            );
+        }
+        Ok(nvattest_cmdline(nonce, proxy_base))
+    }
+
+    fn nvattest_cmdline(nonce: &str, proxy_base: Option<&str>) -> Vec<String> {
+        let mut args = vec![
+            "attest".to_string(),
+            "--device".to_string(),
+            "gpu".to_string(),
+            "--verifier".to_string(),
+            "local".to_string(),
+            "--nonce".to_string(),
+            nonce.to_string(),
+            "--format".to_string(),
+            "json".to_string(),
+        ];
+        if let Some(base) = proxy_base.map(|url| url.trim_end_matches('/')) {
+            args.extend([
+                "--ocsp-url".to_string(),
+                format!("{base}/ocsp"),
+                "--rim-url".to_string(),
+                base.to_string(),
+                "--relying-party-policy".to_string(),
+                OUTPOST_POLICY.to_string(),
+            ]);
+        }
+        args
+    }
+
+    /// Run local GPU attestation via nvattest with a fresh nonce. Without a
+    /// collateral proxy no custom relying-party policy is given, so nvattest's
+    /// built-in appraisal applies. With `proxy_url` (sys-config
+    /// `gpu_attest_proxy_url`), OCSP and RIM requests go through the caching
+    /// proxy and the packaged outpost policy replaces the built-in one: a
+    /// cached OCSP response can never echo the request nonce, while signature,
+    /// validity-window and measurement checks are unchanged. The complete JSON
+    /// output is preserved under /run and retained with its claims for the
+    /// application policy and versioned RTMR3 summary.
+    pub(super) async fn attest_gpu(
+        expected_devices: u32,
+        proxy_url: Option<&str>,
+    ) -> Result<GpuAttestationResult> {
         if !Path::new(NVATTEST).exists() {
             bail!("nvattest is not available in this image");
         }
@@ -1382,22 +1445,15 @@ mod gpu {
             warn!("failed to step system clock: {err:?}");
         }
         let nonce = hex::encode(rand::thread_rng().gen::<[u8; 32]>());
-        let output = run_command(
-            NVATTEST,
-            &[
-                "attest",
-                "--device",
-                "gpu",
-                "--verifier",
-                "local",
-                "--nonce",
-                &nonce,
-                "--format",
-                "json",
-            ],
-            ATTESTATION_TIMEOUT,
-        )
-        .await?;
+        let args = nvattest_args(&nonce, proxy_url)?;
+        if let Some(base) = proxy_url {
+            info!(
+                "routing GPU attestation collateral through proxy: {}",
+                base.trim_end_matches('/')
+            );
+        }
+        let arg_refs = args.iter().map(String::as_str).collect::<Vec<_>>();
+        let output = run_command(NVATTEST, &arg_refs, ATTESTATION_TIMEOUT).await?;
         if !output.stderr.is_empty() {
             info!("nvattest: {}", truncated_lossy(&output.stderr, 2048));
         }
@@ -1556,6 +1612,20 @@ mod gpu {
             assert_eq!(nvidia_gpu_count(nvidia).unwrap(), 2);
         }
 
+        #[test]
+        fn nvattest_cmdline_routes_collateral_through_proxy() {
+            let args = nvattest_cmdline("aa", Some("http://10.0.2.1:8090/"));
+            let joined = args.join(" ");
+            assert!(joined.contains("--ocsp-url http://10.0.2.1:8090/ocsp"));
+            assert!(joined.contains("--rim-url http://10.0.2.1:8090"));
+            assert!(joined.contains(&format!("--relying-party-policy {OUTPOST_POLICY}")));
+
+            let direct = nvattest_cmdline("aa", None).join(" ");
+            assert!(!direct.contains("--ocsp-url"));
+            assert!(!direct.contains("--rim-url"));
+            assert!(!direct.contains("--relying-party-policy"));
+        }
+
         #[test]
         fn basic_policy_requires_cc_and_rejects_devtools_by_default() {
             let nonce = "44".repeat(32);
@@ -1874,7 +1944,11 @@ impl Stage0<'_> {
         }
         self.vmm.notify_q("boot.progress", "attesting GPU").await;
         info!("verifying GPU TEE attestation");
-        let attestation = gpu::attest_gpu(expected_devices).await?;
+        let attestation = gpu::attest_gpu(
+            expected_devices,
+            self.shared.sys_config.gpu_attest_proxy_url.as_deref(),
+        )
+        .await?;
 
         let gpu_state = gpu::query_gpu_state(expected_devices)?;
         attestation

dstack/gpu-attest-proxy/Cargo.toml

@@ -0,0 +1,33 @@
+# SPDX-FileCopyrightText: © 2026 Phala Network <dstack@phala.network>
+#
+# SPDX-License-Identifier: Apache-2.0
+
+[package]
+name = "gpu-attest-proxy"
+version.workspace = true
+authors.workspace = true
+edition.workspace = true
+license.workspace = true
+
+[dependencies]
+anyhow.workspace = true
+base64.workspace = true
+clap = { workspace = true, features = ["derive", "string"] }
+dashmap.workspace = true
+fs-err.workspace = true
+git-version.workspace = true
+hex = { workspace = true, features = ["alloc"] }
+load_config.workspace = true
+or-panic.workspace = true
+reqwest.workspace = true
+rocket = { workspace = true, features = ["json"] }
+serde = { workspace = true, features = ["derive"] }
+serde-duration.workspace = true
+serde_json.workspace = true
+sha2.workspace = true
+tokio = { workspace = true, features = ["macros", "rt-multi-thread", "time"] }
+tracing.workspace = true
+tracing-subscriber.workspace = true
+
+[dev-dependencies]
+tempfile.workspace = true

dstack/gpu-attest-proxy/gpu-attest-proxy.toml

@@ -0,0 +1,25 @@
+# SPDX-FileCopyrightText: © 2026 Phala Network <dstack@phala.network>
+#
+# SPDX-License-Identifier: Apache-2.0
+
+address = "0.0.0.0"
+port = 8090
+log_level = "info"
+
+# NVIDIA attestation collateral upstreams.
+upstream_ocsp_url = "https://ocsp.ndis.nvidia.com"
+upstream_rim_url = "https://rim.attestation.nvidia.com"
+
+# Persistent cache location: one JSON file per entry under ocsp/ and rim/.
+cache_dir = "/var/lib/dstack/gpu-attest-proxy"
+
+# RIM documents are re-fetched after rim_ttl; when the upstream is
+# unreachable a stale document keeps being served for up to rim_max_stale.
+rim_ttl = "24h"
+rim_max_stale = "7d"
+
+# OCSP responses are cached until their nextUpdate, bounded by ocsp_max_ttl;
+# entries within refresh_margin of expiry are refreshed in the background so
+# a warm cache rides through upstream outages.
+ocsp_max_ttl = "7d"
+refresh_margin = "24h"