Prove Kubernetes TLS identity rotation and rollback under load

README.md

@@ -14,8 +14,8 @@ The default posture is conservative: API-key authentication is selected, proxyin
 - Fail-closed reviewed-staging qualification binding prior/candidate registry digests, observed deployment state, and
   loaded rollout/rollback, plus a deployment-equivalent capacity staircase bound to the exact candidate and
   per-replica telemetry; CI-gated Compose and live two-zone Kubernetes proofs cover distribution, candidate abort,
-  content-distinct image rollout and baseline rollback under load, pod-identity turnover, endpoint continuity, replica
-  loss, planned worker
+  content-distinct image rollout and baseline rollback under load, immutable inbound-TLS identity rotation and rollback,
+  pod-identity turnover, endpoint continuity, replica loss, planned worker
   removal, operator-remediated abrupt worker loss, degraded service, and worker recovery.
 - API-key and OAuth2 resource-server modes with deny-by-default API classification.
 - Actuator health/readiness, optional Prometheus metrics, and optional OTLP metrics export with endpoint validation.
@@ -44,9 +44,11 @@ deployment-capacity lane binds the reviewed forecast to the exact staging-profil
 fresh replica identities and hash-pinned per-replica telemetry at every repeat, recomputes the saturation envelope,
 and restores the prior digest. The Kubernetes adapter compiler supplies rollout, rollback, fault, reset, restart,
 certificate-rotation, deployment-inspection, and capacity-sampling executables. A disposable two-worker/two-zone kind
-lane now deploys the restricted production image and proves live Service distribution, a metadata-only content-distinct
-candidate rollout and baseline rollback with complete pod-UID turnover and endpoint continuity, per-replica traffic,
-worker drain/stop, degraded traffic, and
+  lane now deploys the restricted production image and proves live Service distribution, a metadata-only content-distinct
+  candidate rollout and baseline rollback, then rotates between independently rooted certificates through versioned
+  immutable Secrets and restores the original identity. Both transitions require complete pod-UID turnover, unchanged
+  runtime image identity, endpoint continuity, and traffic through both replicas and backends before the lane exercises
+  worker drain/stop, degraded traffic, and
 operator-remediated no-drain worker loss and recovery. The next action remains to compile the adapters from the reviewed
 staging
 cluster identity, freeze the observed configuration/ingress hashes into the profiles, then run staging qualification

deploy/kubernetes/qualification.yaml

@@ -328,7 +328,7 @@ spec:
                 path: loadbalancerpro.api.key
         - name: server-tls
           secret:
-            secretName: loadbalancerpro-server-tls
+            secretName: loadbalancerpro-server-tls-a
             items:
               - key: tls.crt
                 path: certificate.pem

docs/DEPLOYMENT.md

@@ -107,7 +107,12 @@ Secret/ConfigMap mounts. Its image remains a deliberately non-resolving digest p
 [`../scripts/bench/proxy-kubernetes-topology.sh`](../scripts/bench/proxy-kubernetes-topology.sh) lane applies the
 separate loopback qualification workload and proves a metadata-only content-distinct candidate rollout and baseline
 rollback under continuous traffic, complete pod-UID turnover in both directions, runtime-image identity transition and
-restoration, ready-endpoint continuity, two-zone Service distribution, planned worker removal, and
+restoration, then performs a versioned immutable TLS Secret rotation and identity rollback under two additional
+continuous traffic windows. The TLS exercise uses independently generated roots, a dual-CA rollover bundle, served-leaf
+fingerprints, and single-CA positive/negative checks; it also requires fresh pod UIDs, unchanged runtime image identity,
+ready-endpoint continuity, and traffic through both replicas and backends in both directions. It proves application
+server TLS termination behind the loopback NodePort, not an ingress controller, external issuer, or trust-distribution
+system. The lane then proves two-zone Service distribution, planned worker removal, and
 operator-remediated no-drain worker loss and recovery. The abrupt-loss exercise forcibly stops the kind worker,
 confirms its container is down, applies the out-of-service `NoExecute` taint, and force-removes the three exact stateless
 qualification pods from the API. The disposable cluster also pins immediate EndpointSlice-triggered iptables updates

docs/LOAD_BALANCER_BUILD_OUT.md

@@ -160,7 +160,12 @@ replicas and both backends, a zero-unavailable content-distinct candidate rollou
 dropping below
 two ready Service endpoints, both candidate replicas and both backends must serve post-rollout traffic, and a baseline
 rollback must repeat the continuity and pod-turnover proof, restore the initial runtime image identity, and serve traffic
-through both restored replicas and backends. One worker is then
+through both restored replicas and backends. The restored image then rotates from an immutable baseline TLS Secret to
+an independently rooted immutable candidate Secret and back under continuous traffic. Single-CA rejection checks and
+served-leaf fingerprints prove both identity transitions; fresh pods, unchanged runtime image identity, two-zone
+continuity, and positive traffic deltas on both replicas and backends are required in both directions. This exercises
+application server TLS behind the loopback NodePort, not an ingress controller, external issuer, or client
+trust-distribution system. One worker is then
 drained and stopped under load, degraded traffic must continue through the remaining replica, and the stopped worker
 and second replica must recover inside the bound, and both recovered replicas and backends must serve new traffic. It
 then forcibly stops that recovered worker without a drain, confirms the container is down, and applies the documented
@@ -193,7 +198,9 @@ The disposable Kubernetes lane executes that zero-unavailable strategy in both d
 repeatedly samples ready pods and Service endpoints with a one-second pause between queries, proves complete pod-UID
 turnover and a content-distinct runtime image transition, then proves another complete pod turnover and restoration of
 the initial runtime image identity. It restores two-zone placement and requires positive post-transition traffic deltas
-on both candidate/restored replicas and both backends.
+on both candidate/restored replicas and both backends. It applies the same zero-unavailable and distribution checks to
+versioned immutable TLS Secret rotation and rollback while proving the served leaf fingerprint changes and returns and
+the runtime image identity remains fixed.
 
 Use an immutable image digest and begin with a small, explicitly approved traffic slice. During every step, compare
 client success/latency, upstream health, proxy p95/p99, in-flight work, retries, sheds, cooldown trips, CPU, memory, GC,

scripts/bench/README.md

@@ -124,7 +124,12 @@ replicas and both backends served requests, promotes a metadata-only content-dis
 traffic, samples pod and Service endpoint continuity, proves complete pod-UID and runtime-image identity transition, requires both
 candidate replicas and both backends to serve new traffic, then rolls back under a second continuous load window and
 proves fresh pod identities, restoration of the initial runtime image identity, and traffic through both restored
-replicas. It then drains and stops one worker under load,
+replicas. It next creates two independently rooted one-day server identities as versioned immutable Kubernetes TLS
+Secrets, rotates the Deployment's TLS Secret reference under continuous close-per-request traffic using a dual-CA
+rollover bundle, and restores the baseline Secret under a second load window. Positive and negative single-CA checks
+plus repeated served-leaf SHA-256 fingerprint checks prove the identity changed and returned; both directions also
+require complete pod turnover, an unchanged runtime image ID, two-zone endpoint continuity, and post-transition traffic
+through both replicas and backends. It then drains and stops one worker under load,
 tests the one-replica degraded service, requires both recovered replicas and backends to serve new traffic, then
 forcibly stops that recovered worker without a drain. After confirming the worker container is down, it applies
 Kubernetes' out-of-service `NoExecute` remediation and force-removes the three exact stateless workload pods from the
@@ -138,9 +143,11 @@ bash scripts/bench/kubernetes-topology-contract-test.sh
 bash scripts/bench/proxy-kubernetes-topology.sh --mode smoke
 ```
 
-Smoke mode requires Docker, kind 0.31.0, kubectl 1.34.3, Vegeta, jq, OpenSSL, and curl. TLS keys and the API key live
-only in a temporary directory; redacted reports are written beneath `target/kubernetes/`. The result proves
-disposable Kubernetes content-addressed transition, rollback, and worker-loss mechanics. Because the local candidate
+Smoke mode requires Docker, kind 0.31.0, kubectl 1.34.3, Vegeta, jq, OpenSSL, and curl. TLS private keys and the API key
+live only in a temporary directory; evidence contains only redacted Secret metadata and generated leaf fingerprints
+and is written beneath `target/kubernetes/`. The result proves disposable Kubernetes content-addressed image,
+inbound-server TLS Secret transition/rollback, and worker-loss mechanics. It does not deploy or test an ingress
+controller, external certificate authority, or client trust-distribution system. Because the local candidate
 changes immutable proof metadata but not application layers, it does not prove application-layer release compatibility,
 registry integrity,
 deployment capacity, external ingress behavior, automatic infrastructure-failure detection, or an authorized staging

scripts/bench/kubernetes-topology-contract-test.sh

@@ -14,7 +14,7 @@ trap cleanup EXIT
 
 bash -n "$runner"
 bash "$runner" --mode validate --profile "$profile" > "$work_dir/valid.log"
-grep -Fq 'service-distribution per-replica-metrics content-distinct-rollout endpoint-continuity candidate-pod-identity-turnover post-rollout-distribution baseline-rollback rollback-endpoint-continuity rollback-pod-identity-turnover post-rollback-distribution planned-worker-drain stopped-worker degraded-service worker-recovery abrupt-worker-stop out-of-service-remediation abrupt-endpoint-withdrawal abrupt-recovery' \
+grep -Fq 'service-distribution per-replica-metrics content-distinct-rollout endpoint-continuity candidate-pod-identity-turnover post-rollout-distribution baseline-rollback rollback-endpoint-continuity rollback-pod-identity-turnover post-rollback-distribution immutable-certificate-secrets certificate-identity-transition certificate-rotation-continuity certificate-pod-identity-turnover post-certificate-rotation-distribution certificate-identity-rollback certificate-rollback-continuity certificate-rollback-pod-identity-turnover post-certificate-rollback-distribution planned-worker-drain stopped-worker degraded-service worker-recovery abrupt-worker-stop out-of-service-remediation abrupt-endpoint-withdrawal abrupt-recovery' \
     "$work_dir/valid.log"
 
 assert_rejected() {
@@ -30,7 +30,8 @@ assert_rejected() {
 
 assert_rejected production-status '.review.status = "reviewed"'
 assert_rejected legacy-schema '.schemaVersion = 1'
-assert_rejected previous-schema '.schemaVersion = 3'
+assert_rejected older-schema '.schemaVersion = 3'
+assert_rejected previous-schema '.schemaVersion = 4'
 assert_rejected skewed-kubectl '.cluster.kubectlVersion = "v1.32.2"'
 assert_rejected mutable-node-image '.cluster.nodeImageDigest = "sha256:" + ("f" * 64)'
 assert_rejected one-worker '.cluster.workers = 1'
@@ -45,6 +46,12 @@ assert_rejected rollout-window-too-short '.workload.rolloutSeconds = .objectives
 assert_rejected short-rollback '.workload.rollbackSeconds = 10'
 assert_rejected short-post-rollback '.workload.postRollbackSeconds = 1'
 assert_rejected rollback-window-too-short '.workload.rollbackSeconds = .objectives.maximumRollbackSeconds'
+assert_rejected short-certificate-rotation '.workload.certificateRotationSeconds = 10'
+assert_rejected short-post-certificate-rotation '.workload.postCertificateRotationSeconds = 1'
+assert_rejected certificate-rotation-window-too-short '.workload.certificateRotationSeconds = .objectives.maximumCertificateRotationSeconds'
+assert_rejected short-certificate-rollback '.workload.certificateRollbackSeconds = 10'
+assert_rejected short-post-certificate-rollback '.workload.postCertificateRollbackSeconds = 1'
+assert_rejected certificate-rollback-window-too-short '.workload.certificateRollbackSeconds = .objectives.maximumCertificateRollbackSeconds'
 assert_rejected short-transition '.workload.transitionSeconds = 5'
 assert_rejected short-abrupt-transition '.workload.abruptTransitionSeconds = 5'
 assert_rejected short-abrupt-degraded '.workload.abruptDegradedSeconds = 1'
@@ -54,8 +61,14 @@ assert_rejected weak-rollout-objective '.objectives.minimumRolloutSuccessRatio =
 assert_rejected weak-post-rollout-objective '.objectives.minimumPostRolloutSuccessRatio = 0.5'
 assert_rejected weak-rollback-objective '.objectives.minimumRollbackSuccessRatio = 0.5'
 assert_rejected weak-post-rollback-objective '.objectives.minimumPostRollbackSuccessRatio = 0.5'
+assert_rejected weak-certificate-rotation-objective '.objectives.minimumCertificateRotationSuccessRatio = 0.5'
+assert_rejected weak-post-certificate-rotation-objective '.objectives.minimumPostCertificateRotationSuccessRatio = 0.5'
+assert_rejected weak-certificate-rollback-objective '.objectives.minimumCertificateRollbackSuccessRatio = 0.5'
+assert_rejected weak-post-certificate-rollback-objective '.objectives.minimumPostCertificateRollbackSuccessRatio = 0.5'
 assert_rejected long-rollout '.objectives.maximumRolloutSeconds = 180'
 assert_rejected long-rollback '.objectives.maximumRollbackSeconds = 180'
+assert_rejected long-certificate-rotation '.objectives.maximumCertificateRotationSeconds = 180'
+assert_rejected long-certificate-rollback '.objectives.maximumCertificateRollbackSeconds = 180'
 assert_rejected weak-transition-objective '.objectives.minimumTransitionSuccessRatio = 0.5'
 assert_rejected weak-abrupt-transition-objective '.objectives.minimumAbruptTransitionSuccessRatio = 0.5'
 assert_rejected weak-abrupt-degraded-objective '.objectives.minimumAbruptDegradedSuccessRatio = 0.5'
@@ -64,5 +77,8 @@ assert_rejected long-abrupt-transition-p99 '.objectives.maximumAbruptTransitionP
 assert_rejected long-abrupt-endpoint-withdrawal '.objectives.maximumAbruptEndpointWithdrawalSeconds = 60'
 assert_rejected long-recovery '.objectives.maximumRecoverySeconds = 600'
 assert_rejected long-abrupt-recovery '.objectives.maximumAbruptRecoverySeconds = 600'
+assert_rejected wrong-tls-hostname '.tlsRotation.hostname = "production.example.com"'
+assert_rejected wrong-baseline-tls-secret '.tlsRotation.baselineSecret = "production-server-tls"'
+assert_rejected unchanged-tls-secret '.tlsRotation.candidateSecret = .tlsRotation.baselineSecret'
 
-printf 'Kubernetes topology contract rejected 36 unsafe profiles without creating a cluster.\n'
+printf 'Kubernetes topology contract rejected 52 unsafe profiles without creating a cluster.\n'

scripts/bench/kubernetes-topology-profile.example.json

@@ -1,5 +1,5 @@
 {
-  "schemaVersion": 4,
+  "schemaVersion": 5,
   "profileId": "local-kind-two-zone-smoke",
   "review": {
     "status": "example"
@@ -23,6 +23,10 @@
     "postRolloutSeconds": 8,
     "rollbackSeconds": 90,
     "postRollbackSeconds": 8,
+    "certificateRotationSeconds": 90,
+    "postCertificateRotationSeconds": 8,
+    "certificateRollbackSeconds": 90,
+    "postCertificateRollbackSeconds": 8,
     "transitionSeconds": 20,
     "degradedSeconds": 8,
     "recoveredSeconds": 8,
@@ -36,6 +40,10 @@
     "minimumPostRolloutSuccessRatio": 0.999,
     "minimumRollbackSuccessRatio": 0.999,
     "minimumPostRollbackSuccessRatio": 0.999,
+    "minimumCertificateRotationSuccessRatio": 0.999,
+    "minimumPostCertificateRotationSuccessRatio": 0.999,
+    "minimumCertificateRollbackSuccessRatio": 0.999,
+    "minimumPostCertificateRollbackSuccessRatio": 0.999,
     "minimumTransitionSuccessRatio": 0.97,
     "minimumDegradedSuccessRatio": 0.999,
     "minimumRecoveredSuccessRatio": 0.999,
@@ -46,8 +54,15 @@
     "maximumAbruptTransitionP99Millis": 5500,
     "maximumRolloutSeconds": 75,
     "maximumRollbackSeconds": 75,
+    "maximumCertificateRotationSeconds": 75,
+    "maximumCertificateRollbackSeconds": 75,
     "maximumRecoverySeconds": 180,
     "maximumAbruptEndpointWithdrawalSeconds": 15,
     "maximumAbruptRecoverySeconds": 180
+  },
+  "tlsRotation": {
+    "hostname": "lbp-kubernetes.local",
+    "baselineSecret": "loadbalancerpro-server-tls-a",
+    "candidateSecret": "loadbalancerpro-server-tls-b"
   }
 }