Prove Kubernetes image rollout and rollback under load

.github/workflows/ci.yml

@@ -537,6 +537,7 @@ jobs:
           LBP_KUBERNETES_CLUSTER: lbp-k8s-${{ github.run_id }}-${{ github.run_attempt }}
           LBP_KUBERNETES_OUTPUT_DIR: target/kubernetes/ci-smoke
           LBP_KUBERNETES_PROXY_SOURCE_IMAGE: loadbalancerpro:ci
+          LBP_KUBERNETES_CANDIDATE_SOURCE_IMAGE: loadbalancerpro:immutable-candidate-ci
           LBP_KUBERNETES_FIXTURE_SOURCE_IMAGE: loadbalancerpro:proxy-prod-fixture-ci
         run: |
           set -euo pipefail

README.md

@@ -14,7 +14,8 @@ The default posture is conservative: API-key authentication is selected, proxyin
 - Fail-closed reviewed-staging qualification binding prior/candidate registry digests, observed deployment state, and
   loaded rollout/rollback, plus a deployment-equivalent capacity staircase bound to the exact candidate and
   per-replica telemetry; CI-gated Compose and live two-zone Kubernetes proofs cover distribution, candidate abort,
-  rolling pod replacement under load, pod-identity turnover, endpoint continuity, replica loss, planned worker
+  content-distinct image rollout and baseline rollback under load, pod-identity turnover, endpoint continuity, replica
+  loss, planned worker
   removal, operator-remediated abrupt worker loss, degraded service, and worker recovery.
 - API-key and OAuth2 resource-server modes with deny-by-default API classification.
 - Actuator health/readiness, optional Prometheus metrics, and optional OTLP metrics export with endpoint validation.
@@ -43,8 +44,9 @@ deployment-capacity lane binds the reviewed forecast to the exact staging-profil
 fresh replica identities and hash-pinned per-replica telemetry at every repeat, recomputes the saturation envelope,
 and restores the prior digest. The Kubernetes adapter compiler supplies rollout, rollback, fault, reset, restart,
 certificate-rotation, deployment-inspection, and capacity-sampling executables. A disposable two-worker/two-zone kind
-lane now deploys the restricted production image and proves live Service distribution, same-image rolling replacement
-with complete pod-UID turnover and endpoint continuity, per-replica traffic, worker drain/stop, degraded traffic, and
+lane now deploys the restricted production image and proves live Service distribution, a metadata-only content-distinct
+candidate rollout and baseline rollback with complete pod-UID turnover and endpoint continuity, per-replica traffic,
+worker drain/stop, degraded traffic, and
 operator-remediated no-drain worker loss and recovery. The next action remains to compile the adapters from the reviewed
 staging
 cluster identity, freeze the observed configuration/ingress hashes into the profiles, then run staging qualification

docs/DEPLOYMENT.md

@@ -105,13 +105,15 @@ surge pod, preferred host spreading, a one-replica disruption budget, startup/re
 preStop delay, a 40-second termination window, a token-free service account, numeric non-root execution, and external
 Secret/ConfigMap mounts. Its image remains a deliberately non-resolving digest placeholder. The disposable
 [`../scripts/bench/proxy-kubernetes-topology.sh`](../scripts/bench/proxy-kubernetes-topology.sh) lane applies the
-separate loopback qualification workload and proves same-image rolling pod replacement under continuous traffic,
-complete pod-UID turnover, ready-endpoint continuity, two-zone Service distribution, planned worker removal, and
+separate loopback qualification workload and proves a metadata-only content-distinct candidate rollout and baseline
+rollback under continuous traffic, complete pod-UID turnover in both directions, runtime-image identity transition and
+restoration, ready-endpoint continuity, two-zone Service distribution, planned worker removal, and
 operator-remediated no-drain worker loss and recovery. The abrupt-loss exercise forcibly stops the kind worker,
 confirms its container is down, applies the out-of-service `NoExecute` taint, and force-removes the three exact stateless
 qualification pods from the API. The disposable cluster also pins immediate EndpointSlice-triggered iptables updates
 and a one-second kube-proxy cleanup sync; operators must review the equivalent setting or managed-ingress
-behavior for their environment. It does not prove automatic deployment failure detection. The reviewed staging runner
+behavior for their environment. The candidate preserves the baseline application layers, so this does not prove
+application-layer release compatibility or automatic deployment failure detection. The reviewed staging runner
 separately validates the external target's digest, replicas, zones, resources,
 configuration, ingress, metrics, drain, and transitions; local proof does not establish registry integrity, deployment
 capacity, external ingress behavior, or production readiness.

docs/LOAD_BALANCER_BUILD_OUT.md

@@ -156,18 +156,22 @@ metrics, replica loss, and recovery under load.
 
 [`proxy-kubernetes-topology.sh`](../scripts/bench/proxy-kubernetes-topology.sh) adds a live disposable Kubernetes proof:
 two restricted proxy replicas are scheduled across two labeled worker zones, Kubernetes Service traffic must reach both
-replicas and both backends, a zero-unavailable rolling replacement must turn over both pod UIDs without dropping below
-two ready Service endpoints, both replacement replicas and both backends must serve post-rollout traffic, one worker is
+replicas and both backends, a zero-unavailable content-distinct candidate rollout must turn over both pod UIDs without
+dropping below
+two ready Service endpoints, both candidate replicas and both backends must serve post-rollout traffic, and a baseline
+rollback must repeat the continuity and pod-turnover proof, restore the initial runtime image identity, and serve traffic
+through both restored replicas and backends. One worker is then
 drained and stopped under load, degraded traffic must continue through the remaining replica, and the stopped worker
 and second replica must recover inside the bound, and both recovered replicas and backends must serve new traffic. It
 then forcibly stops that recovered worker without a drain, confirms the container is down, and applies the documented
 out-of-service `NoExecute` remediation. It force-removes the three exact stateless workload pods from the API, bounds
 endpoint withdrawal, proves degraded traffic, rejects the failed pod identity after recovery, and requires both
 recovered replicas and backends to serve new traffic. The disposable cluster pins iptables-mode kube-proxy
 to immediate EndpointSlice-triggered updates and a one-second cleanup sync; deployment environments must review the
-equivalent Service/ingress failure-detection and reconciliation behavior. The replacement reuses one exact local image
-content ID, so it proves Kubernetes mechanics
-rather than compatibility between releases. The reviewed deployment ingress, deployment-equivalent resources, registry
+equivalent Service/ingress failure-detection and reconciliation behavior. The candidate has a distinct local image
+content ID but preserves the baseline application layers, so it proves Kubernetes transition and rollback mechanics
+rather than compatibility between application releases. The reviewed deployment ingress, deployment-equivalent
+resources, registry
 digest transition, and automatic infrastructure-failure detection remain staging gates.
 
 ### 4. Stage The Rollout And Rollback
@@ -185,10 +189,11 @@ history/progress, required zone separation, preferred host separation, and a one
 adapter compiler supplies the matching rollout/rollback, fault, reset, inspection, and telemetry executables. They
 remain unapplied until an authorized staging environment supplies the reviewed cluster and workload profiles.
 
-The disposable Kubernetes lane executes that zero-unavailable strategy under continuous traffic, repeatedly samples
-ready pods and Service endpoints with a one-second pause between queries, proves complete pod-UID turnover and unchanged
-runtime image identity, restores two-zone placement, and requires positive post-rollout traffic deltas on both
-replacement replicas and both backends.
+The disposable Kubernetes lane executes that zero-unavailable strategy in both directions under continuous traffic,
+repeatedly samples ready pods and Service endpoints with a one-second pause between queries, proves complete pod-UID
+turnover and a content-distinct runtime image transition, then proves another complete pod turnover and restoration of
+the initial runtime image identity. It restores two-zone placement and requires positive post-transition traffic deltas
+on both candidate/restored replicas and both backends.
 
 Use an immutable image digest and begin with a small, explicitly approved traffic slice. During every step, compare
 client success/latency, upstream health, proxy p95/p99, in-flight work, retries, sheds, cooldown trips, CPU, memory, GC,

scripts/bench/README.md

@@ -120,9 +120,11 @@ images. The local candidate changes proof metadata only; its Docker content ID i
 `proxy-kubernetes-topology.sh` creates an isolated kind cluster from a digest-pinned Kubernetes node image, loads the
 numeric-non-root proxy and fixture images, and deploys two proxy replicas and redundant backends across two workers and
 zones. It sends TLS/API-key-protected connection-churn traffic through a loopback-only NodePort, proves both proxy
-replicas and both backends served requests, performs a same-image rolling pod replacement under continuous traffic,
-samples pod and Service endpoint continuity, proves complete pod-UID turnover and unchanged runtime image identity,
-requires both replacement replicas and both backends to serve new traffic, drains and stops one worker under load,
+replicas and both backends served requests, promotes a metadata-only content-distinct candidate under continuous
+traffic, samples pod and Service endpoint continuity, proves complete pod-UID and runtime-image identity transition, requires both
+candidate replicas and both backends to serve new traffic, then rolls back under a second continuous load window and
+proves fresh pod identities, restoration of the initial runtime image identity, and traffic through both restored
+replicas. It then drains and stops one worker under load,
 tests the one-replica degraded service, requires both recovered replicas and backends to serve new traffic, then
 forcibly stops that recovered worker without a drain. After confirming the worker container is down, it applies
 Kubernetes' out-of-service `NoExecute` remediation and force-removes the three exact stateless workload pods from the
@@ -138,7 +140,9 @@ bash scripts/bench/proxy-kubernetes-topology.sh --mode smoke
 
 Smoke mode requires Docker, kind 0.31.0, kubectl 1.34.3, Vegeta, jq, OpenSSL, and curl. TLS keys and the API key live
 only in a temporary directory; redacted reports are written beneath `target/kubernetes/`. The result proves
-disposable Kubernetes replacement and worker-loss mechanics, not release compatibility, registry integrity,
+disposable Kubernetes content-addressed transition, rollback, and worker-loss mechanics. Because the local candidate
+changes immutable proof metadata but not application layers, it does not prove application-layer release compatibility,
+registry integrity,
 deployment capacity, external ingress behavior, automatic infrastructure-failure detection, or an authorized staging
 environment.
 

scripts/bench/kubernetes-topology-contract-test.sh

@@ -14,7 +14,7 @@ trap cleanup EXIT
 
 bash -n "$runner"
 bash "$runner" --mode validate --profile "$profile" > "$work_dir/valid.log"
-grep -Fq 'service-distribution per-replica-metrics rolling-replacement endpoint-continuity pod-identity-turnover post-rollout-distribution planned-worker-drain stopped-worker degraded-service worker-recovery abrupt-worker-stop out-of-service-remediation abrupt-endpoint-withdrawal abrupt-recovery' \
+grep -Fq 'service-distribution per-replica-metrics content-distinct-rollout endpoint-continuity candidate-pod-identity-turnover post-rollout-distribution baseline-rollback rollback-endpoint-continuity rollback-pod-identity-turnover post-rollback-distribution planned-worker-drain stopped-worker degraded-service worker-recovery abrupt-worker-stop out-of-service-remediation abrupt-endpoint-withdrawal abrupt-recovery' \
     "$work_dir/valid.log"
 
 assert_rejected() {
@@ -30,7 +30,7 @@ assert_rejected() {
 
 assert_rejected production-status '.review.status = "reviewed"'
 assert_rejected legacy-schema '.schemaVersion = 1'
-assert_rejected previous-schema '.schemaVersion = 2'
+assert_rejected previous-schema '.schemaVersion = 3'
 assert_rejected skewed-kubectl '.cluster.kubectlVersion = "v1.32.2"'
 assert_rejected mutable-node-image '.cluster.nodeImageDigest = "sha256:" + ("f" * 64)'
 assert_rejected one-worker '.cluster.workers = 1'
@@ -42,14 +42,20 @@ assert_rejected low-rate '.workload.ratePerSecond = 1'
 assert_rejected short-rollout '.workload.rolloutSeconds = 10'
 assert_rejected short-post-rollout '.workload.postRolloutSeconds = 1'
 assert_rejected rollout-window-too-short '.workload.rolloutSeconds = .objectives.maximumRolloutSeconds'
+assert_rejected short-rollback '.workload.rollbackSeconds = 10'
+assert_rejected short-post-rollback '.workload.postRollbackSeconds = 1'
+assert_rejected rollback-window-too-short '.workload.rollbackSeconds = .objectives.maximumRollbackSeconds'
 assert_rejected short-transition '.workload.transitionSeconds = 5'
 assert_rejected short-abrupt-transition '.workload.abruptTransitionSeconds = 5'
 assert_rejected short-abrupt-degraded '.workload.abruptDegradedSeconds = 1'
 assert_rejected short-abrupt-recovered '.workload.abruptRecoveredSeconds = 1'
 assert_rejected abrupt-window-too-short '.workload.abruptTransitionSeconds = .objectives.maximumAbruptEndpointWithdrawalSeconds'
 assert_rejected weak-rollout-objective '.objectives.minimumRolloutSuccessRatio = 0.5'
 assert_rejected weak-post-rollout-objective '.objectives.minimumPostRolloutSuccessRatio = 0.5'
+assert_rejected weak-rollback-objective '.objectives.minimumRollbackSuccessRatio = 0.5'
+assert_rejected weak-post-rollback-objective '.objectives.minimumPostRollbackSuccessRatio = 0.5'
 assert_rejected long-rollout '.objectives.maximumRolloutSeconds = 180'
+assert_rejected long-rollback '.objectives.maximumRollbackSeconds = 180'
 assert_rejected weak-transition-objective '.objectives.minimumTransitionSuccessRatio = 0.5'
 assert_rejected weak-abrupt-transition-objective '.objectives.minimumAbruptTransitionSuccessRatio = 0.5'
 assert_rejected weak-abrupt-degraded-objective '.objectives.minimumAbruptDegradedSuccessRatio = 0.5'
@@ -59,4 +65,4 @@ assert_rejected long-abrupt-endpoint-withdrawal '.objectives.maximumAbruptEndpoi
 assert_rejected long-recovery '.objectives.maximumRecoverySeconds = 600'
 assert_rejected long-abrupt-recovery '.objectives.maximumAbruptRecoverySeconds = 600'
 
-printf 'Kubernetes topology contract rejected 30 unsafe profiles without creating a cluster.\n'
+printf 'Kubernetes topology contract rejected 36 unsafe profiles without creating a cluster.\n'

scripts/bench/kubernetes-topology-profile.example.json

@@ -1,5 +1,5 @@
 {
-  "schemaVersion": 3,
+  "schemaVersion": 4,
   "profileId": "local-kind-two-zone-smoke",
   "review": {
     "status": "example"
@@ -21,6 +21,8 @@
     "baselineSeconds": 8,
     "rolloutSeconds": 90,
     "postRolloutSeconds": 8,
+    "rollbackSeconds": 90,
+    "postRollbackSeconds": 8,
     "transitionSeconds": 20,
     "degradedSeconds": 8,
     "recoveredSeconds": 8,
@@ -32,6 +34,8 @@
     "minimumBaselineSuccessRatio": 0.999,
     "minimumRolloutSuccessRatio": 0.999,
     "minimumPostRolloutSuccessRatio": 0.999,
+    "minimumRollbackSuccessRatio": 0.999,
+    "minimumPostRollbackSuccessRatio": 0.999,
     "minimumTransitionSuccessRatio": 0.97,
     "minimumDegradedSuccessRatio": 0.999,
     "minimumRecoveredSuccessRatio": 0.999,
@@ -41,6 +45,7 @@
     "maximumP99Millis": 1500,
     "maximumAbruptTransitionP99Millis": 5500,
     "maximumRolloutSeconds": 75,
+    "maximumRollbackSeconds": 75,
     "maximumRecoverySeconds": 180,
     "maximumAbruptEndpointWithdrawalSeconds": 15,
     "maximumAbruptRecoverySeconds": 180