diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 73f8cda1..7c7a8d99 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -537,6 +537,7 @@ jobs: LBP_KUBERNETES_CLUSTER: lbp-k8s-${{ github.run_id }}-${{ github.run_attempt }} LBP_KUBERNETES_OUTPUT_DIR: target/kubernetes/ci-smoke LBP_KUBERNETES_PROXY_SOURCE_IMAGE: loadbalancerpro:ci + LBP_KUBERNETES_CANDIDATE_SOURCE_IMAGE: loadbalancerpro:immutable-candidate-ci LBP_KUBERNETES_FIXTURE_SOURCE_IMAGE: loadbalancerpro:proxy-prod-fixture-ci run: | set -euo pipefail diff --git a/README.md b/README.md index 25746f07..d99df1f4 100644 --- a/README.md +++ b/README.md @@ -14,7 +14,8 @@ The default posture is conservative: API-key authentication is selected, proxyin - Fail-closed reviewed-staging qualification binding prior/candidate registry digests, observed deployment state, and loaded rollout/rollback, plus a deployment-equivalent capacity staircase bound to the exact candidate and per-replica telemetry; CI-gated Compose and live two-zone Kubernetes proofs cover distribution, candidate abort, - rolling pod replacement under load, pod-identity turnover, endpoint continuity, replica loss, planned worker + content-distinct image rollout and baseline rollback under load, pod-identity turnover, endpoint continuity, replica + loss, planned worker removal, operator-remediated abrupt worker loss, degraded service, and worker recovery. - API-key and OAuth2 resource-server modes with deny-by-default API classification. - Actuator health/readiness, optional Prometheus metrics, and optional OTLP metrics export with endpoint validation. @@ -43,8 +44,9 @@ deployment-capacity lane binds the reviewed forecast to the exact staging-profil fresh replica identities and hash-pinned per-replica telemetry at every repeat, recomputes the saturation envelope, and restores the prior digest. The Kubernetes adapter compiler supplies rollout, rollback, fault, reset, restart, certificate-rotation, deployment-inspection, and capacity-sampling executables. A disposable two-worker/two-zone kind -lane now deploys the restricted production image and proves live Service distribution, same-image rolling replacement -with complete pod-UID turnover and endpoint continuity, per-replica traffic, worker drain/stop, degraded traffic, and +lane now deploys the restricted production image and proves live Service distribution, a metadata-only content-distinct +candidate rollout and baseline rollback with complete pod-UID turnover and endpoint continuity, per-replica traffic, +worker drain/stop, degraded traffic, and operator-remediated no-drain worker loss and recovery. The next action remains to compile the adapters from the reviewed staging cluster identity, freeze the observed configuration/ingress hashes into the profiles, then run staging qualification diff --git a/docs/DEPLOYMENT.md b/docs/DEPLOYMENT.md index 0a7e9959..e841a03b 100644 --- a/docs/DEPLOYMENT.md +++ b/docs/DEPLOYMENT.md @@ -105,13 +105,15 @@ surge pod, preferred host spreading, a one-replica disruption budget, startup/re preStop delay, a 40-second termination window, a token-free service account, numeric non-root execution, and external Secret/ConfigMap mounts. Its image remains a deliberately non-resolving digest placeholder. The disposable [`../scripts/bench/proxy-kubernetes-topology.sh`](../scripts/bench/proxy-kubernetes-topology.sh) lane applies the -separate loopback qualification workload and proves same-image rolling pod replacement under continuous traffic, -complete pod-UID turnover, ready-endpoint continuity, two-zone Service distribution, planned worker removal, and +separate loopback qualification workload and proves a metadata-only content-distinct candidate rollout and baseline +rollback under continuous traffic, complete pod-UID turnover in both directions, runtime-image identity transition and +restoration, ready-endpoint continuity, two-zone Service distribution, planned worker removal, and operator-remediated no-drain worker loss and recovery. The abrupt-loss exercise forcibly stops the kind worker, confirms its container is down, applies the out-of-service `NoExecute` taint, and force-removes the three exact stateless qualification pods from the API. The disposable cluster also pins immediate EndpointSlice-triggered iptables updates and a one-second kube-proxy cleanup sync; operators must review the equivalent setting or managed-ingress -behavior for their environment. It does not prove automatic deployment failure detection. The reviewed staging runner +behavior for their environment. The candidate preserves the baseline application layers, so this does not prove +application-layer release compatibility or automatic deployment failure detection. The reviewed staging runner separately validates the external target's digest, replicas, zones, resources, configuration, ingress, metrics, drain, and transitions; local proof does not establish registry integrity, deployment capacity, external ingress behavior, or production readiness. diff --git a/docs/LOAD_BALANCER_BUILD_OUT.md b/docs/LOAD_BALANCER_BUILD_OUT.md index bc7a32af..d3974c18 100644 --- a/docs/LOAD_BALANCER_BUILD_OUT.md +++ b/docs/LOAD_BALANCER_BUILD_OUT.md @@ -156,8 +156,11 @@ metrics, replica loss, and recovery under load. [`proxy-kubernetes-topology.sh`](../scripts/bench/proxy-kubernetes-topology.sh) adds a live disposable Kubernetes proof: two restricted proxy replicas are scheduled across two labeled worker zones, Kubernetes Service traffic must reach both -replicas and both backends, a zero-unavailable rolling replacement must turn over both pod UIDs without dropping below -two ready Service endpoints, both replacement replicas and both backends must serve post-rollout traffic, one worker is +replicas and both backends, a zero-unavailable content-distinct candidate rollout must turn over both pod UIDs without +dropping below +two ready Service endpoints, both candidate replicas and both backends must serve post-rollout traffic, and a baseline +rollback must repeat the continuity and pod-turnover proof, restore the initial runtime image identity, and serve traffic +through both restored replicas and backends. One worker is then drained and stopped under load, degraded traffic must continue through the remaining replica, and the stopped worker and second replica must recover inside the bound, and both recovered replicas and backends must serve new traffic. It then forcibly stops that recovered worker without a drain, confirms the container is down, and applies the documented @@ -165,9 +168,10 @@ out-of-service `NoExecute` remediation. It force-removes the three exact statele endpoint withdrawal, proves degraded traffic, rejects the failed pod identity after recovery, and requires both recovered replicas and backends to serve new traffic. The disposable cluster pins iptables-mode kube-proxy to immediate EndpointSlice-triggered updates and a one-second cleanup sync; deployment environments must review the -equivalent Service/ingress failure-detection and reconciliation behavior. The replacement reuses one exact local image -content ID, so it proves Kubernetes mechanics -rather than compatibility between releases. The reviewed deployment ingress, deployment-equivalent resources, registry +equivalent Service/ingress failure-detection and reconciliation behavior. The candidate has a distinct local image +content ID but preserves the baseline application layers, so it proves Kubernetes transition and rollback mechanics +rather than compatibility between application releases. The reviewed deployment ingress, deployment-equivalent +resources, registry digest transition, and automatic infrastructure-failure detection remain staging gates. ### 4. Stage The Rollout And Rollback @@ -185,10 +189,11 @@ history/progress, required zone separation, preferred host separation, and a one adapter compiler supplies the matching rollout/rollback, fault, reset, inspection, and telemetry executables. They remain unapplied until an authorized staging environment supplies the reviewed cluster and workload profiles. -The disposable Kubernetes lane executes that zero-unavailable strategy under continuous traffic, repeatedly samples -ready pods and Service endpoints with a one-second pause between queries, proves complete pod-UID turnover and unchanged -runtime image identity, restores two-zone placement, and requires positive post-rollout traffic deltas on both -replacement replicas and both backends. +The disposable Kubernetes lane executes that zero-unavailable strategy in both directions under continuous traffic, +repeatedly samples ready pods and Service endpoints with a one-second pause between queries, proves complete pod-UID +turnover and a content-distinct runtime image transition, then proves another complete pod turnover and restoration of +the initial runtime image identity. It restores two-zone placement and requires positive post-transition traffic deltas +on both candidate/restored replicas and both backends. Use an immutable image digest and begin with a small, explicitly approved traffic slice. During every step, compare client success/latency, upstream health, proxy p95/p99, in-flight work, retries, sheds, cooldown trips, CPU, memory, GC, diff --git a/scripts/bench/README.md b/scripts/bench/README.md index a8779f16..0488c815 100644 --- a/scripts/bench/README.md +++ b/scripts/bench/README.md @@ -120,9 +120,11 @@ images. The local candidate changes proof metadata only; its Docker content ID i `proxy-kubernetes-topology.sh` creates an isolated kind cluster from a digest-pinned Kubernetes node image, loads the numeric-non-root proxy and fixture images, and deploys two proxy replicas and redundant backends across two workers and zones. It sends TLS/API-key-protected connection-churn traffic through a loopback-only NodePort, proves both proxy -replicas and both backends served requests, performs a same-image rolling pod replacement under continuous traffic, -samples pod and Service endpoint continuity, proves complete pod-UID turnover and unchanged runtime image identity, -requires both replacement replicas and both backends to serve new traffic, drains and stops one worker under load, +replicas and both backends served requests, promotes a metadata-only content-distinct candidate under continuous +traffic, samples pod and Service endpoint continuity, proves complete pod-UID and runtime-image identity transition, requires both +candidate replicas and both backends to serve new traffic, then rolls back under a second continuous load window and +proves fresh pod identities, restoration of the initial runtime image identity, and traffic through both restored +replicas. It then drains and stops one worker under load, tests the one-replica degraded service, requires both recovered replicas and backends to serve new traffic, then forcibly stops that recovered worker without a drain. After confirming the worker container is down, it applies Kubernetes' out-of-service `NoExecute` remediation and force-removes the three exact stateless workload pods from the @@ -138,7 +140,9 @@ bash scripts/bench/proxy-kubernetes-topology.sh --mode smoke Smoke mode requires Docker, kind 0.31.0, kubectl 1.34.3, Vegeta, jq, OpenSSL, and curl. TLS keys and the API key live only in a temporary directory; redacted reports are written beneath `target/kubernetes/`. The result proves -disposable Kubernetes replacement and worker-loss mechanics, not release compatibility, registry integrity, +disposable Kubernetes content-addressed transition, rollback, and worker-loss mechanics. Because the local candidate +changes immutable proof metadata but not application layers, it does not prove application-layer release compatibility, +registry integrity, deployment capacity, external ingress behavior, automatic infrastructure-failure detection, or an authorized staging environment. diff --git a/scripts/bench/kubernetes-topology-contract-test.sh b/scripts/bench/kubernetes-topology-contract-test.sh index 7bc41a2f..56fa3b37 100644 --- a/scripts/bench/kubernetes-topology-contract-test.sh +++ b/scripts/bench/kubernetes-topology-contract-test.sh @@ -14,7 +14,7 @@ trap cleanup EXIT bash -n "$runner" bash "$runner" --mode validate --profile "$profile" > "$work_dir/valid.log" -grep -Fq 'service-distribution per-replica-metrics rolling-replacement endpoint-continuity pod-identity-turnover post-rollout-distribution planned-worker-drain stopped-worker degraded-service worker-recovery abrupt-worker-stop out-of-service-remediation abrupt-endpoint-withdrawal abrupt-recovery' \ +grep -Fq 'service-distribution per-replica-metrics content-distinct-rollout endpoint-continuity candidate-pod-identity-turnover post-rollout-distribution baseline-rollback rollback-endpoint-continuity rollback-pod-identity-turnover post-rollback-distribution planned-worker-drain stopped-worker degraded-service worker-recovery abrupt-worker-stop out-of-service-remediation abrupt-endpoint-withdrawal abrupt-recovery' \ "$work_dir/valid.log" assert_rejected() { @@ -30,7 +30,7 @@ assert_rejected() { assert_rejected production-status '.review.status = "reviewed"' assert_rejected legacy-schema '.schemaVersion = 1' -assert_rejected previous-schema '.schemaVersion = 2' +assert_rejected previous-schema '.schemaVersion = 3' assert_rejected skewed-kubectl '.cluster.kubectlVersion = "v1.32.2"' assert_rejected mutable-node-image '.cluster.nodeImageDigest = "sha256:" + ("f" * 64)' assert_rejected one-worker '.cluster.workers = 1' @@ -42,6 +42,9 @@ assert_rejected low-rate '.workload.ratePerSecond = 1' assert_rejected short-rollout '.workload.rolloutSeconds = 10' assert_rejected short-post-rollout '.workload.postRolloutSeconds = 1' assert_rejected rollout-window-too-short '.workload.rolloutSeconds = .objectives.maximumRolloutSeconds' +assert_rejected short-rollback '.workload.rollbackSeconds = 10' +assert_rejected short-post-rollback '.workload.postRollbackSeconds = 1' +assert_rejected rollback-window-too-short '.workload.rollbackSeconds = .objectives.maximumRollbackSeconds' assert_rejected short-transition '.workload.transitionSeconds = 5' assert_rejected short-abrupt-transition '.workload.abruptTransitionSeconds = 5' assert_rejected short-abrupt-degraded '.workload.abruptDegradedSeconds = 1' @@ -49,7 +52,10 @@ assert_rejected short-abrupt-recovered '.workload.abruptRecoveredSeconds = 1' assert_rejected abrupt-window-too-short '.workload.abruptTransitionSeconds = .objectives.maximumAbruptEndpointWithdrawalSeconds' assert_rejected weak-rollout-objective '.objectives.minimumRolloutSuccessRatio = 0.5' assert_rejected weak-post-rollout-objective '.objectives.minimumPostRolloutSuccessRatio = 0.5' +assert_rejected weak-rollback-objective '.objectives.minimumRollbackSuccessRatio = 0.5' +assert_rejected weak-post-rollback-objective '.objectives.minimumPostRollbackSuccessRatio = 0.5' assert_rejected long-rollout '.objectives.maximumRolloutSeconds = 180' +assert_rejected long-rollback '.objectives.maximumRollbackSeconds = 180' assert_rejected weak-transition-objective '.objectives.minimumTransitionSuccessRatio = 0.5' assert_rejected weak-abrupt-transition-objective '.objectives.minimumAbruptTransitionSuccessRatio = 0.5' assert_rejected weak-abrupt-degraded-objective '.objectives.minimumAbruptDegradedSuccessRatio = 0.5' @@ -59,4 +65,4 @@ assert_rejected long-abrupt-endpoint-withdrawal '.objectives.maximumAbruptEndpoi assert_rejected long-recovery '.objectives.maximumRecoverySeconds = 600' assert_rejected long-abrupt-recovery '.objectives.maximumAbruptRecoverySeconds = 600' -printf 'Kubernetes topology contract rejected 30 unsafe profiles without creating a cluster.\n' +printf 'Kubernetes topology contract rejected 36 unsafe profiles without creating a cluster.\n' diff --git a/scripts/bench/kubernetes-topology-profile.example.json b/scripts/bench/kubernetes-topology-profile.example.json index e628451d..af9b13ef 100644 --- a/scripts/bench/kubernetes-topology-profile.example.json +++ b/scripts/bench/kubernetes-topology-profile.example.json @@ -1,5 +1,5 @@ { - "schemaVersion": 3, + "schemaVersion": 4, "profileId": "local-kind-two-zone-smoke", "review": { "status": "example" @@ -21,6 +21,8 @@ "baselineSeconds": 8, "rolloutSeconds": 90, "postRolloutSeconds": 8, + "rollbackSeconds": 90, + "postRollbackSeconds": 8, "transitionSeconds": 20, "degradedSeconds": 8, "recoveredSeconds": 8, @@ -32,6 +34,8 @@ "minimumBaselineSuccessRatio": 0.999, "minimumRolloutSuccessRatio": 0.999, "minimumPostRolloutSuccessRatio": 0.999, + "minimumRollbackSuccessRatio": 0.999, + "minimumPostRollbackSuccessRatio": 0.999, "minimumTransitionSuccessRatio": 0.97, "minimumDegradedSuccessRatio": 0.999, "minimumRecoveredSuccessRatio": 0.999, @@ -41,6 +45,7 @@ "maximumP99Millis": 1500, "maximumAbruptTransitionP99Millis": 5500, "maximumRolloutSeconds": 75, + "maximumRollbackSeconds": 75, "maximumRecoverySeconds": 180, "maximumAbruptEndpointWithdrawalSeconds": 15, "maximumAbruptRecoverySeconds": 180 diff --git a/scripts/bench/proxy-kubernetes-topology.sh b/scripts/bench/proxy-kubernetes-topology.sh index 7aa35743..9d140d8c 100644 --- a/scripts/bench/proxy-kubernetes-topology.sh +++ b/scripts/bench/proxy-kubernetes-topology.sh @@ -6,6 +6,7 @@ repo_root="$(cd "$script_dir/../.." && pwd)" cluster_config="$repo_root/deploy/kubernetes/kind-cluster.yaml" workload_manifest="$repo_root/deploy/kubernetes/qualification.yaml" example_profile="$script_dir/kubernetes-topology-profile.example.json" +candidate_dockerfile="$repo_root/deploy/topology/RolloutCandidate.Dockerfile" mode=validate profile="$example_profile" @@ -19,12 +20,12 @@ while [[ $# -gt 0 ]]; do done case "$mode" in validate|smoke) ;; *) echo "Mode must be validate or smoke" >&2; exit 2 ;; esac command -v jq >/dev/null 2>&1 || { echo "jq is required" >&2; exit 2; } -for required_file in "$cluster_config" "$workload_manifest" "$profile"; do +for required_file in "$cluster_config" "$workload_manifest" "$candidate_dockerfile" "$profile"; do [[ -f "$required_file" ]] || { echo "Missing required file: $required_file" >&2; exit 2; } done jq -e ' - .schemaVersion == 3 + .schemaVersion == 4 and (.profileId | type == "string" and test("^[a-z0-9][a-z0-9._-]{0,62}$")) and .review.status == "example" and .cluster.kindVersion == "v0.31.0" @@ -41,6 +42,8 @@ jq -e ' and (.workload.baselineSeconds | type == "number" and . >= 5 and . <= 60 and floor == .) and (.workload.rolloutSeconds | type == "number" and . >= 20 and . <= 180 and floor == .) and (.workload.postRolloutSeconds | type == "number" and . >= 5 and . <= 60 and floor == .) + and (.workload.rollbackSeconds | type == "number" and . >= 20 and . <= 180 and floor == .) + and (.workload.postRollbackSeconds | type == "number" and . >= 5 and . <= 60 and floor == .) and (.workload.transitionSeconds | type == "number" and . >= 15 and . <= 120 and floor == .) and (.workload.degradedSeconds | type == "number" and . >= 5 and . <= 60 and floor == .) and (.workload.recoveredSeconds | type == "number" and . >= 5 and . <= 60 and floor == .) @@ -50,6 +53,8 @@ jq -e ' and (.objectives.minimumBaselineSuccessRatio | type == "number" and . >= 0.95 and . <= 1) and (.objectives.minimumRolloutSuccessRatio | type == "number" and . >= 0.95 and . <= 1) and (.objectives.minimumPostRolloutSuccessRatio | type == "number" and . >= 0.95 and . <= 1) + and (.objectives.minimumRollbackSuccessRatio | type == "number" and . >= 0.95 and . <= 1) + and (.objectives.minimumPostRollbackSuccessRatio | type == "number" and . >= 0.95 and . <= 1) and (.objectives.minimumTransitionSuccessRatio | type == "number" and . >= 0.90 and . <= 1) and (.objectives.minimumDegradedSuccessRatio | type == "number" and . >= 0.95 and . <= 1) and (.objectives.minimumRecoveredSuccessRatio | type == "number" and . >= 0.95 and . <= 1) @@ -59,10 +64,12 @@ jq -e ' and (.objectives.maximumP99Millis | type == "number" and . >= 100 and . <= 5000 and floor == .) and (.objectives.maximumAbruptTransitionP99Millis | type == "number" and . >= 1000 and . <= 6000 and floor == .) and (.objectives.maximumRolloutSeconds | type == "number" and . >= 20 and . <= 120 and floor == .) + and (.objectives.maximumRollbackSeconds | type == "number" and . >= 20 and . <= 120 and floor == .) and (.objectives.maximumRecoverySeconds | type == "number" and . >= 30 and . <= 300 and floor == .) and (.objectives.maximumAbruptEndpointWithdrawalSeconds | type == "number" and . >= 5 and . <= 30 and floor == .) and (.objectives.maximumAbruptRecoverySeconds | type == "number" and . >= 30 and . <= 300 and floor == .) and .workload.rolloutSeconds >= (.objectives.maximumRolloutSeconds + 5) + and .workload.rollbackSeconds >= (.objectives.maximumRollbackSeconds + 5) and .workload.abruptTransitionSeconds >= (.objectives.maximumAbruptEndpointWithdrawalSeconds + 5) ' "$profile" >/dev/null || { echo "Kubernetes topology profile does not satisfy the executable contract" >&2; exit 2; } @@ -97,7 +104,7 @@ fi if [[ "$mode" == "validate" ]]; then printf 'Validated disposable two-worker/two-zone Kubernetes topology contract %s.\n' "$(jq -r '.profileId' "$profile")" - printf 'Validated proof cases: service-distribution per-replica-metrics rolling-replacement endpoint-continuity pod-identity-turnover post-rollout-distribution planned-worker-drain stopped-worker degraded-service worker-recovery abrupt-worker-stop out-of-service-remediation abrupt-endpoint-withdrawal abrupt-recovery\n' + printf 'Validated proof cases: service-distribution per-replica-metrics content-distinct-rollout endpoint-continuity candidate-pod-identity-turnover post-rollout-distribution baseline-rollback rollback-endpoint-continuity rollback-pod-identity-turnover post-rollback-distribution planned-worker-drain stopped-worker degraded-service worker-recovery abrupt-worker-stop out-of-service-remediation abrupt-endpoint-withdrawal abrupt-recovery\n' exit 0 fi @@ -164,15 +171,32 @@ cleanup() { trap cleanup EXIT proxy_image=loadbalancerpro:kubernetes-proxy +candidate_image=loadbalancerpro:kubernetes-candidate fixture_image=loadbalancerpro:kubernetes-fixture proxy_source="${LBP_KUBERNETES_PROXY_SOURCE_IMAGE:-}" +candidate_source="${LBP_KUBERNETES_CANDIDATE_SOURCE_IMAGE:-}" fixture_source="${LBP_KUBERNETES_FIXTURE_SOURCE_IMAGE:-}" +source_revision="$(git -C "$repo_root" rev-parse HEAD)" +[[ "$source_revision" =~ ^[0-9a-f]{40}$ ]] || { echo "Unable to bind source revision" >&2; exit 1; } +candidate_release_id="kubernetes-${source_revision:0:12}" if [[ -n "$proxy_source" ]]; then docker image inspect "$proxy_source" >/dev/null docker tag "$proxy_source" "$proxy_image" else docker build --tag "$proxy_image" "$repo_root" fi +if [[ -n "$candidate_source" ]]; then + docker image inspect "$candidate_source" >/dev/null + docker tag "$candidate_source" "$candidate_image" + candidate_release_id="$(docker image inspect --format \ + '{{index .Config.Labels "com.richeyworks.loadbalancerpro.rollout.release-id"}}' "$candidate_image")" + [[ "$candidate_release_id" =~ ^[a-z0-9][a-z0-9._-]{0,127}$ ]] || { + echo "Provided Kubernetes rollout candidate has no bounded release label" >&2; exit 1; + } +else + docker build --quiet --file "$candidate_dockerfile" --build-arg "BASE_IMAGE=$proxy_image" \ + --build-arg "ROLLOUT_RELEASE_ID=$candidate_release_id" --tag "$candidate_image" "$repo_root" >/dev/null +fi if [[ -n "$fixture_source" ]]; then docker image inspect "$fixture_source" >/dev/null docker tag "$fixture_source" "$fixture_image" @@ -180,10 +204,28 @@ else docker build --file "$repo_root/deploy/fixture/Dockerfile" --tag "$fixture_image" "$repo_root" fi proxy_image_id="$(docker image inspect --format '{{.Id}}' "$proxy_image")" +candidate_image_id="$(docker image inspect --format '{{.Id}}' "$candidate_image")" fixture_image_id="$(docker image inspect --format '{{.Id}}' "$fixture_image")" -for image_id in "$proxy_image_id" "$fixture_image_id"; do +for image_id in "$proxy_image_id" "$candidate_image_id" "$fixture_image_id"; do [[ "$image_id" =~ ^sha256:[0-9a-f]{64}$ ]] || { echo "Qualification image has no exact local content ID" >&2; exit 1; } done +[[ "$candidate_image_id" != "$proxy_image_id" ]] || { + echo "Kubernetes rollout candidate is not content-distinct from the baseline image" >&2; exit 1; +} +baseline_layers="$(docker image inspect --format '{{json .RootFS.Layers}}' "$proxy_image_id")" +[[ "$(docker image inspect --format '{{json .RootFS.Layers}}' "$candidate_image_id")" == "$baseline_layers" ]] || { + echo "Kubernetes rollout candidate changed application layers" >&2; exit 1; +} +for inherited_config in '{{json .Config.Cmd}}' '{{json .Config.Entrypoint}}' '{{json .Config.User}}'; do + [[ "$(docker image inspect --format "$inherited_config" "$candidate_image_id")" == \ + "$(docker image inspect --format "$inherited_config" "$proxy_image_id")" ]] || { + echo "Kubernetes rollout candidate changed inherited runtime configuration" >&2; exit 1; + } +done +[[ "$(docker image inspect --format '{{index .Config.Labels "com.richeyworks.loadbalancerpro.rollout.release-id"}}' \ + "$candidate_image_id")" == "$candidate_release_id" ]] || { + echo "Kubernetes rollout candidate release label is missing or stale" >&2; exit 1; +} if kind get clusters | grep -Fxq "$cluster_name"; then echo "Refusing to reuse or delete an existing kind cluster named $cluster_name" >&2 exit 2 @@ -210,7 +252,7 @@ mapfile -t workers < <(kind get nodes --name "$cluster_name" | grep -- '-worker' kubectl label node "${workers[0]}" loadbalancerpro.io/qualification-worker=true topology.kubernetes.io/zone=zone-a --overwrite kubectl label node "${workers[1]}" loadbalancerpro.io/qualification-worker=true topology.kubernetes.io/zone=zone-b --overwrite -kind load docker-image "$proxy_image" "$fixture_image" --name "$cluster_name" +kind load docker-image "$proxy_image" "$candidate_image" "$fixture_image" --name "$cluster_name" mkdir -p "$tls_dir" openssl rand -hex 24 > "$api_key_file" ca_key="$work_dir/ca-key.pem" @@ -240,8 +282,6 @@ kubectl create secret generic loadbalancerpro-server-tls --namespace "$namespace --from-file=tls.crt="$tls_dir/certificate.pem" \ --from-file=tls.key="$tls_dir/private-key.pem" \ --from-file=ca.crt="$tls_dir/ca.pem" -source_revision="$(git -C "$repo_root" rev-parse HEAD)" -[[ "$source_revision" =~ ^[0-9a-f]{40}$ ]] || { echo "Unable to bind source revision" >&2; exit 1; } kubectl patch deployment loadbalancerpro --namespace "$namespace" --type merge \ -p "{\"spec\":{\"template\":{\"metadata\":{\"annotations\":{\"loadbalancerpro.io/source-revision\":\"$source_revision\"}}}}}" for deployment in backend-a backend-b loadbalancerpro; do @@ -290,8 +330,9 @@ wait_for_count() { return 1 } -sample_rollout_continuity() { - local output="$output_dir/rollout-continuity.csv" +sample_transition_continuity() { + local phase="$1" + local output="$output_dir/${phase}-continuity.csv" printf 'epoch_seconds,ready_proxy_pods,ready_service_endpoints,total_proxy_pods\n' > "$output" while [[ ! -f "$rollout_stop_file" ]]; do local ready_pods ready_endpoints total_pods @@ -301,7 +342,7 @@ sample_rollout_continuity() { -l app.kubernetes.io/name=loadbalancerpro -o json | jq '.items | length')" printf '%s,%s,%s,%s\n' "$(date +%s)" "$ready_pods" "$ready_endpoints" "$total_pods" >> "$output" if (( ready_pods < 2 || ready_endpoints < 2 )); then - echo "Rolling replacement dropped below two ready proxy pods or Service endpoints" >&2 + echo "$phase image transition dropped below two ready proxy pods or Service endpoints" >&2 return 1 fi sleep 1 @@ -438,10 +479,10 @@ collect_distribution baseline rollout_duration_seconds="$(jq -r '.workload.rolloutSeconds' "$profile")" maximum_rollout_seconds="$(jq -r '.objectives.maximumRolloutSeconds' "$profile")" -rollout_token="$(printf '%s\n' "$source_revision|$default_run_id|rolling-replacement" \ +rollout_token="$(printf '%s\n' "$source_revision|$candidate_image_id|$default_run_id|candidate-rollout" \ | sha256sum | awk '{print $1}')" rm -f -- "$rollout_stop_file" -sample_rollout_continuity & +sample_transition_continuity rollout & rollout_sampler_pid=$! vegeta attack -duration="${rollout_duration_seconds}s" -rate="${rate}/s" -timeout=5s \ -keepalive=false -http2=false -root-certs="$tls_dir/ca.pem" -targets="$targets" \ @@ -449,8 +490,12 @@ vegeta attack -duration="${rollout_duration_seconds}s" -rate="${rate}/s" -timeou attack_pid=$! sleep 3 rollout_started_epoch="$(date +%s)" -kubectl patch deployment loadbalancerpro --namespace "$namespace" --type merge \ - -p "{\"spec\":{\"template\":{\"metadata\":{\"annotations\":{\"loadbalancerpro.io/qualification-rollout\":\"$rollout_token\"}}}}}" +rollout_patch="$(jq -cn --arg image "$candidate_image" --arg token "$rollout_token" \ + --arg release "$candidate_release_id" \ + '{spec:{template:{metadata:{annotations:{"loadbalancerpro.io/qualification-rollout":$token, + "loadbalancerpro.io/qualification-release":$release}}, + spec:{containers:[{name:"loadbalancerpro",image:$image}]}}}}')" +kubectl patch deployment loadbalancerpro --namespace "$namespace" --type strategic -p "$rollout_patch" kubectl rollout status deployment/loadbalancerpro --namespace "$namespace" \ --timeout="${maximum_rollout_seconds}s" rollout_elapsed_seconds=$(( $(date +%s) - rollout_started_epoch )) @@ -492,6 +537,11 @@ replacement_ready_proxy_pods_json="$(jq --arg token "$rollout_token" '[.items[] [[ "$(jq 'length' <<< "$replacement_ready_proxy_pods_json")" == 2 ]] || { echo "Rolling replacement did not converge to two ready replacement pods" >&2; exit 1; } +[[ "$(jq --arg image "$candidate_image" '[.[] | any(.spec.containers[]?; + .name == "loadbalancerpro" and .image == $image)] | all' \ + <<< "$replacement_ready_proxy_pods_json")" == true ]] || { + echo "Candidate proxy pods do not reference the expected local candidate image" >&2; exit 1; +} [[ "$(ready_proxy_count)" == 2 && "$(ready_endpoint_count)" == 2 ]] || { echo "Rolling replacement did not restore exactly two ready pods and Service endpoints" >&2; exit 1; } @@ -505,8 +555,11 @@ rollout_old_uid_overlap="$(jq -n --argjson prior "$initial_proxy_uids_json" \ replacement_proxy_runtime_image_ids_json="$(jq '[.[].status.containerStatuses[]? | select(.name == "loadbalancerpro") | .imageID] | unique | sort' \ <<< "$replacement_ready_proxy_pods_json")" -[[ "$replacement_proxy_runtime_image_ids_json" == "$initial_proxy_runtime_image_ids_json" ]] || { - echo "Rolling replacement changed the immutable runtime image ID" >&2; exit 1; +[[ "$(jq 'length' <<< "$replacement_proxy_runtime_image_ids_json")" == 1 ]] || { + echo "Candidate proxy pods did not converge to one immutable runtime image ID" >&2; exit 1; +} +[[ "$replacement_proxy_runtime_image_ids_json" != "$initial_proxy_runtime_image_ids_json" ]] || { + echo "Candidate rollout did not change the immutable runtime image ID" >&2; exit 1; } [[ "$(jq '[.[].spec.nodeName] | unique | length' <<< "$replacement_ready_proxy_pods_json")" == 2 ]] || { echo "Replacement proxy pods were not restored to distinct workers" >&2; exit 1; @@ -547,7 +600,130 @@ jq -e '(.pods | length) == 2 echo "Both replacement proxies and both backends must serve post-rollout traffic" >&2; exit 1; } -failed_node="$(jq -r '.[0].spec.nodeName' <<< "$replacement_ready_proxy_pods_json")" +rollback_duration_seconds="$(jq -r '.workload.rollbackSeconds' "$profile")" +maximum_rollback_seconds="$(jq -r '.objectives.maximumRollbackSeconds' "$profile")" +rollback_token="$(printf '%s\n' "$source_revision|$proxy_image_id|$default_run_id|baseline-rollback" \ + | sha256sum | awk '{print $1}')" +baseline_release_id="baseline-${source_revision:0:12}" +rm -f -- "$rollout_stop_file" +sample_transition_continuity rollback & +rollout_sampler_pid=$! +vegeta attack -duration="${rollback_duration_seconds}s" -rate="${rate}/s" -timeout=5s \ + -keepalive=false -http2=false -root-certs="$tls_dir/ca.pem" -targets="$targets" \ + > "$work_dir/rollback.bin" & +attack_pid=$! +sleep 3 +rollback_started_epoch="$(date +%s)" +rollback_patch="$(jq -cn --arg image "$proxy_image" --arg token "$rollback_token" \ + --arg release "$baseline_release_id" \ + '{spec:{template:{metadata:{annotations:{"loadbalancerpro.io/qualification-rollout":$token, + "loadbalancerpro.io/qualification-release":$release}}, + spec:{containers:[{name:"loadbalancerpro",image:$image}]}}}}')" +kubectl patch deployment loadbalancerpro --namespace "$namespace" --type strategic -p "$rollback_patch" +kubectl rollout status deployment/loadbalancerpro --namespace "$namespace" \ + --timeout="${maximum_rollback_seconds}s" +rollback_elapsed_seconds=$(( $(date +%s) - rollback_started_epoch )) +(( rollback_elapsed_seconds <= maximum_rollback_seconds )) || { + echo "Baseline rollback exceeded the rollback objective" >&2; exit 1; +} +if ! wait "$attack_pid"; then + attack_pid="" + echo "Baseline rollback traffic attack failed" >&2 + exit 1 +fi +attack_pid="" +: > "$rollout_stop_file" +if ! wait "$rollout_sampler_pid"; then + rollout_sampler_pid="" + echo "Baseline rollback endpoint-continuity sampler failed" >&2 + exit 1 +fi +rollout_sampler_pid="" +report_attack rollback "$(jq -r '.objectives.minimumRollbackSuccessRatio' "$profile")" +rollback_sample_count="$(awk -F, 'NR > 1 { count++ } END { print count + 0 }' \ + "$output_dir/rollback-continuity.csv")" +rollback_min_ready_pods="$(awk -F, 'NR > 1 && (minimum == "" || $2 < minimum) { minimum = $2 } + END { print minimum + 0 }' "$output_dir/rollback-continuity.csv")" +rollback_min_ready_endpoints="$(awk -F, 'NR > 1 && (minimum == "" || $3 < minimum) { minimum = $3 } + END { print minimum + 0 }' "$output_dir/rollback-continuity.csv")" +(( rollback_sample_count >= 5 && rollback_min_ready_pods >= 2 && rollback_min_ready_endpoints >= 2 )) || { + echo "Baseline rollback continuity evidence was incomplete" >&2; exit 1; +} + +rollback_proxy_pods_json="$(kubectl get pod --namespace "$namespace" \ + -l app.kubernetes.io/name=loadbalancerpro -o json)" +rollback_ready_proxy_pods_json="$(jq --arg token "$rollback_token" '[.items[] + | select(.metadata.deletionTimestamp == null) + | select(.status.phase == "Running") + | select(any(.status.conditions[]?; .type == "Ready" and .status == "True")) + | select(.metadata.annotations["loadbalancerpro.io/qualification-rollout"] == $token)]' \ + <<< "$rollback_proxy_pods_json")" +[[ "$(jq 'length' <<< "$rollback_ready_proxy_pods_json")" == 2 ]] || { + echo "Baseline rollback did not converge to two ready restored pods" >&2; exit 1; +} +[[ "$(jq --arg image "$proxy_image" '[.[] | any(.spec.containers[]?; + .name == "loadbalancerpro" and .image == $image)] | all' \ + <<< "$rollback_ready_proxy_pods_json")" == true ]] || { + echo "Restored proxy pods do not reference the expected baseline image" >&2; exit 1; +} +[[ "$(ready_proxy_count)" == 2 && "$(ready_endpoint_count)" == 2 ]] || { + echo "Baseline rollback did not restore exactly two ready pods and Service endpoints" >&2; exit 1; +} +rollback_proxy_uids_json="$(jq '[.[].metadata.uid] | sort' <<< "$rollback_ready_proxy_pods_json")" +rollback_candidate_uid_overlap="$(jq -n --argjson candidate "$replacement_proxy_uids_json" \ + --argjson restored "$rollback_proxy_uids_json" \ + '[ $candidate[] as $uid | $restored[] | select(. == $uid) ] | length')" +rollback_initial_uid_overlap="$(jq -n --argjson initial "$initial_proxy_uids_json" \ + --argjson restored "$rollback_proxy_uids_json" \ + '[ $initial[] as $uid | $restored[] | select(. == $uid) ] | length')" +[[ "$rollback_candidate_uid_overlap" == 0 && "$rollback_initial_uid_overlap" == 0 ]] || { + echo "Baseline rollback retained a prior proxy pod UID" >&2; exit 1; +} +rollback_proxy_runtime_image_ids_json="$(jq '[.[].status.containerStatuses[]? + | select(.name == "loadbalancerpro") | .imageID] | unique | sort' \ + <<< "$rollback_ready_proxy_pods_json")" +[[ "$rollback_proxy_runtime_image_ids_json" == "$initial_proxy_runtime_image_ids_json" ]] || { + echo "Baseline rollback did not restore the initial immutable runtime image ID" >&2; exit 1; +} +[[ "$(jq '[.[].spec.nodeName] | unique | length' <<< "$rollback_ready_proxy_pods_json")" == 2 ]] || { + echo "Restored baseline proxy pods were not placed on distinct workers" >&2; exit 1; +} +rollback_proxy_zones="$(jq -r '.[].spec.nodeName' <<< "$rollback_ready_proxy_pods_json" \ + | while read -r node; do kubectl get node "$node" \ + -o jsonpath='{.metadata.labels.topology\.kubernetes\.io/zone}{"\n"}'; done \ + | sort -u | wc -l | tr -d ' ')" +[[ "$rollback_proxy_zones" == 2 ]] || { + echo "Restored baseline proxy pods were not placed in distinct zones" >&2; exit 1; +} +mapfile -t rollback_proxy_pods < <(jq -r '.[].metadata.name' <<< "$rollback_ready_proxy_pods_json" | sort) +for pod in "${rollback_proxy_pods[@]}"; do + [[ "$(kubectl exec --namespace "$namespace" "$pod" -- id -u)" == 10001 ]] || { + echo "$pod restored baseline is not running with UID 10001" >&2; exit 1; + } +done +capture_state post-rollback +collect_distribution post-rollback-before +run_attack post-rollback "$(jq -r '.workload.postRollbackSeconds' "$profile")" \ + "$(jq -r '.objectives.minimumPostRollbackSuccessRatio' "$profile")" +collect_distribution post-rollback +jq -n --slurpfile before "$output_dir/post-rollback-before-distribution.json" \ + --slurpfile after "$output_dir/post-rollback-distribution.json" ' + ($before[0]) as $before | ($after[0]) as $after | + {phase: "post-rollback", bothRestoredBaselineProxyReplicasServed: true, + backendARequestDelta: ($after.backendARequests - $before.backendARequests), + backendBRequestDelta: ($after.backendBRequests - $before.backendBRequests), + pods: [$after.pods[] as $current + | ($before.pods[] | select(.pod == $current.pod)) as $prior + | {pod: $current.pod, requestDelta: ($current.requests - $prior.requests)}]} + ' > "$output_dir/post-rollback-distribution-delta.json" +jq -e '(.pods | length) == 2 + and all(.pods[]; .requestDelta > 0) + and .backendARequestDelta > 0 + and .backendBRequestDelta > 0' "$output_dir/post-rollback-distribution-delta.json" >/dev/null || { + echo "Both restored baseline proxies and both backends must serve post-rollback traffic" >&2; exit 1; +} + +failed_node="$(jq -r '.[0].spec.nodeName' <<< "$rollback_ready_proxy_pods_json")" [[ "$failed_node" == "${cluster_name}-worker" || "$failed_node" == "${cluster_name}-worker2" ]] || { echo "Refusing to drain unexpected node $failed_node" >&2; exit 1; } @@ -730,52 +906,82 @@ jq -e '(.pods | length) == 2 kubectl version -o json > "$output_dir/kubernetes-version.json" kind version > "$output_dir/kind-version.txt" -sha256sum "$profile" "$cluster_config" "$workload_manifest" > "$output_dir/input-sha256.txt" +docker image inspect "$proxy_image_id" > "$output_dir/proxy-baseline-image.json" +docker image inspect "$candidate_image_id" > "$output_dir/proxy-candidate-image.json" +sha256sum "$profile" "$cluster_config" "$workload_manifest" "$candidate_dockerfile" \ + > "$output_dir/input-sha256.txt" baseline_distribution_json="$(<"$output_dir/baseline-distribution.json")" post_rollout_distribution_delta_json="$(<"$output_dir/post-rollout-distribution-delta.json")" +post_rollback_distribution_delta_json="$(<"$output_dir/post-rollback-distribution-delta.json")" recovered_distribution_delta_json="$(<"$output_dir/recovered-distribution-delta.json")" abrupt_recovered_distribution_delta_json="$(<"$output_dir/abrupt-recovered-distribution-delta.json")" jq -n \ --arg profileId "$profile_id" \ --arg sourceRevision "$source_revision" \ --arg proxyImageId "$proxy_image_id" \ + --arg candidateImageId "$candidate_image_id" \ --arg fixtureImageId "$fixture_image_id" \ + --arg candidateReleaseId "$candidate_release_id" \ --arg rolloutToken "$rollout_token" \ + --arg rollbackToken "$rollback_token" \ --arg drainedWorker "$failed_node" \ --arg abruptWorker "$abrupt_node" \ --arg abruptFailedProxyUid "$abrupt_failed_proxy_uid" \ --argjson abruptForcedPodNames "$abrupt_forced_pod_names_json" \ --argjson priorPodUids "$initial_proxy_uids_json" \ - --argjson replacementPodUids "$replacement_proxy_uids_json" \ - --argjson runtimeImageIds "$replacement_proxy_runtime_image_ids_json" \ + --argjson candidatePodUids "$replacement_proxy_uids_json" \ + --argjson restoredPodUids "$rollback_proxy_uids_json" \ + --argjson baselineRuntimeImageIds "$initial_proxy_runtime_image_ids_json" \ + --argjson candidateRuntimeImageIds "$replacement_proxy_runtime_image_ids_json" \ + --argjson restoredRuntimeImageIds "$rollback_proxy_runtime_image_ids_json" \ --argjson rolloutSeconds "$rollout_elapsed_seconds" \ --argjson rolloutSamples "$rollout_sample_count" \ - --argjson minimumReadyPods "$rollout_min_ready_pods" \ - --argjson minimumReadyEndpoints "$rollout_min_ready_endpoints" \ + --argjson rolloutMinimumReadyPods "$rollout_min_ready_pods" \ + --argjson rolloutMinimumReadyEndpoints "$rollout_min_ready_endpoints" \ + --argjson rollbackSeconds "$rollback_elapsed_seconds" \ + --argjson rollbackSamples "$rollback_sample_count" \ + --argjson rollbackMinimumReadyPods "$rollback_min_ready_pods" \ + --argjson rollbackMinimumReadyEndpoints "$rollback_min_ready_endpoints" \ --argjson recoverySeconds "$recovery_seconds" \ --argjson abruptEndpointWithdrawalSeconds "$abrupt_endpoint_withdrawal_seconds" \ --argjson abruptRecoverySeconds "$abrupt_recovery_seconds" \ --argjson baselineDistribution "$baseline_distribution_json" \ --argjson postRolloutDistribution "$post_rollout_distribution_delta_json" \ + --argjson postRollbackDistribution "$post_rollback_distribution_delta_json" \ --argjson recoveredDistribution "$recovered_distribution_delta_json" \ --argjson abruptRecoveredDistribution "$abrupt_recovered_distribution_delta_json" \ - '{schemaVersion: 3, result: "pass", evidenceBoundary: "disposable loopback kind same-image replacement, planned worker loss, and operator-remediated abrupt worker-container loss; not automatic infrastructure-failure detection, release compatibility, registry/source binding, deployment-ingress, or deployment-capacity proof", + '{schemaVersion: 4, result: "pass", evidenceBoundary: "disposable loopback kind metadata-only content-distinct image rollout and baseline rollback, planned worker loss, and operator-remediated abrupt worker-container loss; not automatic infrastructure-failure detection, application-layer release compatibility, registry/source binding, deployment-ingress, or deployment-capacity proof", profileId: $profileId, repositoryRevision: $sourceRevision, - images: {proxyContentId: $proxyImageId, fixtureContentId: $fixtureImageId}, + images: {identityType: "local Docker content-addressed image ID", + baseline: {contentId: $proxyImageId}, + candidate: {releaseId: $candidateReleaseId, contentId: $candidateImageId}, + fixtureContentId: $fixtureImageId, applicationLayersIdentical: true}, topology: {workers: 2, zones: 2, initialProxyReplicas: 2, postRolloutProxyReplicas: 2, + postRollbackProxyReplicas: 2, degradedProxyReplicas: 1, recoveredProxyReplicas: 2, abruptDegradedProxyReplicas: 1, abruptRecoveredProxyReplicas: 2}, traffic: {bothProxyReplicasServed: true, baseline: $baselineDistribution, rollout: "pass", postRollout: $postRolloutDistribution, + rollback: "pass", postRollback: $postRollbackDistribution, drainTransition: "pass", degraded: "pass", recovered: $recoveredDistribution, abruptTransition: "pass", abruptDegraded: "pass", abruptRecovered: $abruptRecoveredDistribution}, - rolloutExercise: {triggerAnnotation: $rolloutToken, sameRuntimeImageId: true, - runtimeImageIds: $runtimeImageIds, priorPodUids: $priorPodUids, - replacementPodUids: $replacementPodUids, retainedPriorPodUids: 0, + rolloutExercise: {triggerAnnotation: $rolloutToken, contentDistinctRuntimeImageId: true, + fromRuntimeImageIds: $baselineRuntimeImageIds, + toRuntimeImageIds: $candidateRuntimeImageIds, + priorPodUids: $priorPodUids, candidatePodUids: $candidatePodUids, + retainedPriorPodUids: 0, rolloutSeconds: $rolloutSeconds, continuitySamples: $rolloutSamples, - minimumReadyProxyPods: $minimumReadyPods, - minimumReadyServiceEndpoints: $minimumReadyEndpoints}, + minimumReadyProxyPods: $rolloutMinimumReadyPods, + minimumReadyServiceEndpoints: $rolloutMinimumReadyEndpoints}, + rollbackExercise: {triggerAnnotation: $rollbackToken, restoredInitialRuntimeImageId: true, + fromRuntimeImageIds: $candidateRuntimeImageIds, + toRuntimeImageIds: $restoredRuntimeImageIds, + candidatePodUids: $candidatePodUids, restoredPodUids: $restoredPodUids, + retainedCandidatePodUids: 0, retainedInitialPodUids: 0, + rollbackSeconds: $rollbackSeconds, continuitySamples: $rollbackSamples, + minimumReadyProxyPods: $rollbackMinimumReadyPods, + minimumReadyServiceEndpoints: $rollbackMinimumReadyEndpoints}, workerExercise: {planned: {drainedAndStopped: $drainedWorker, recoverySeconds: $recoverySeconds}, abrupt: {stoppedWithoutDrain: $abruptWorker, remediation: "verified-down out-of-service:NoExecute taint plus forced API deletion", @@ -785,4 +991,4 @@ jq -n \ recoverySeconds: $abruptRecoverySeconds}}}' \ > "$output_dir/summary.json" -printf 'Kubernetes two-zone live rollout, planned-loss, and abrupt-loss proof passed; evidence: %s\n' "$output_dir" +printf 'Kubernetes two-zone content-distinct rollout, baseline rollback, planned-loss, and abrupt-loss proof passed; evidence: %s\n' "$output_dir" diff --git a/src/test/java/com/richmond423/loadbalancerpro/api/KubernetesLiveTopologyContractTest.java b/src/test/java/com/richmond423/loadbalancerpro/api/KubernetesLiveTopologyContractTest.java index 40b73a0c..14983c03 100644 --- a/src/test/java/com/richmond423/loadbalancerpro/api/KubernetesLiveTopologyContractTest.java +++ b/src/test/java/com/richmond423/loadbalancerpro/api/KubernetesLiveTopologyContractTest.java @@ -17,6 +17,7 @@ class KubernetesLiveTopologyContractTest { private static final Path DOCKERFILE = Path.of("Dockerfile"); private static final Path CLUSTER = Path.of("deploy/kubernetes/kind-cluster.yaml"); private static final Path WORKLOAD = Path.of("deploy/kubernetes/qualification.yaml"); + private static final Path CANDIDATE = Path.of("deploy/topology/RolloutCandidate.Dockerfile"); private static final Path PROFILE = Path.of("scripts/bench/kubernetes-topology-profile.example.json"); private static final Path RUNNER = Path.of("scripts/bench/proxy-kubernetes-topology.sh"); private static final Path CONTRACT = Path.of("scripts/bench/kubernetes-topology-contract-test.sh"); @@ -62,7 +63,7 @@ void clusterAndProfilePinTheDisposableTwoZoneBoundary() throws IOException { "kindest/node:v1.34.3@sha256:08497ee19eace7b4b5348db5c6a1591d7752b164530a36f855cb0f2bdcbadd48")); JsonNode profile = new ObjectMapper().readTree(read(PROFILE)); - assertEquals(3, profile.path("schemaVersion").asInt()); + assertEquals(4, profile.path("schemaVersion").asInt()); assertEquals("example", profile.path("review").path("status").asText()); assertEquals("v1.34.3", profile.path("cluster").path("kubectlVersion").asText()); assertEquals(2, profile.path("cluster").path("workers").asInt()); @@ -74,6 +75,10 @@ void clusterAndProfilePinTheDisposableTwoZoneBoundary() throws IOException { >= profile.path("objectives").path("maximumRolloutSeconds").asInt() + 5); assertTrue(profile.path("objectives").path("minimumRolloutSuccessRatio").asDouble() >= 0.95); assertTrue(profile.path("objectives").path("minimumPostRolloutSuccessRatio").asDouble() >= 0.95); + assertTrue(profile.path("workload").path("rollbackSeconds").asInt() + >= profile.path("objectives").path("maximumRollbackSeconds").asInt() + 5); + assertTrue(profile.path("objectives").path("minimumRollbackSuccessRatio").asDouble() >= 0.95); + assertTrue(profile.path("objectives").path("minimumPostRollbackSuccessRatio").asDouble() >= 0.95); assertTrue(profile.path("workload").path("abruptTransitionSeconds").asInt() >= profile.path("objectives").path("maximumAbruptEndpointWithdrawalSeconds").asInt() + 5); assertTrue(profile.path("objectives").path("minimumAbruptTransitionSuccessRatio").asDouble() >= 0.90); @@ -83,11 +88,14 @@ void clusterAndProfilePinTheDisposableTwoZoneBoundary() throws IOException { } @Test - void runnerExecutesLiveRollingReplacementPlannedAndAbruptWorkerLossChecks() throws IOException { + void runnerExecutesLiveImageTransitionRollbackPlannedAndAbruptWorkerLossChecks() throws IOException { String runner = read(RUNNER); + assertTrue(read(CANDIDATE).contains("metadata-only-local-candidate")); for (String behavior : List.of( "kind create cluster", "kind load docker-image", + "Kubernetes rollout candidate is not content-distinct from the baseline image", + "Kubernetes rollout candidate changed application layers", "Refusing to reuse or delete an existing kind cluster", "Live kube-proxy config is missing", "kube-proxy-config.yaml", @@ -96,17 +104,26 @@ void runnerExecutesLiveRollingReplacementPlannedAndAbruptWorkerLossChecks() thro "-keepalive=false", "${phase}-${pod}-metrics.txt", "loadbalancerpro.io/qualification-rollout", + "loadbalancerpro.io/qualification-release", + "--arg image \"$candidate_image\"", "kubectl rollout status deployment/loadbalancerpro", "rollout-continuity.csv", + "rollback-continuity.csv", ".metadata.deletionTimestamp == null", ".conditions.terminating != true", "Rolling replacement retained an initial proxy pod UID", - "Rolling replacement changed the immutable runtime image ID", + "Candidate rollout did not change the immutable runtime image ID", "post-rollout-distribution-delta.json", "bothReplacementProxyReplicasServed: true", + "Baseline rollback retained a prior proxy pod UID", + "Baseline rollback did not restore the initial immutable runtime image ID", + "post-rollback-distribution-delta.json", + "bothRestoredBaselineProxyReplicasServed: true", "priorPodUids: $priorPodUids", - "replacementPodUids: $replacementPodUids", - "sameRuntimeImageId: true", + "candidatePodUids: $candidatePodUids", + "restoredPodUids: $restoredPodUids", + "contentDistinctRuntimeImageId: true", + "restoredInitialRuntimeImageId: true", "kubectl drain", "docker stop", "ready Service endpoints while one worker is stopped", @@ -128,7 +145,7 @@ void runnerExecutesLiveRollingReplacementPlannedAndAbruptWorkerLossChecks() thro "bothProxyReplicasServed: true")) { assertTrue(runner.contains(behavior), "missing live Kubernetes proof behavior: " + behavior); } - assertTrue(read(CONTRACT).contains("rejected 30 unsafe profiles without creating a cluster")); + assertTrue(read(CONTRACT).contains("rejected 36 unsafe profiles without creating a cluster")); assertFalse(runner.contains("--insecure")); assertFalse(runner.contains("--validate=false")); } @@ -140,6 +157,7 @@ void ciPinsKindAndRunsAndUploadsTheLiveProof() throws IOException { assertTrue(ci.contains("ab60ca5f0fd60c1eb81b52909e67060e3ba0bd27e55a8ac147cbc2172ff14212")); assertTrue(ci.contains("bash scripts/bench/kubernetes-topology-contract-test.sh")); assertTrue(ci.contains("bash scripts/bench/proxy-kubernetes-topology.sh --mode smoke")); + assertTrue(ci.contains("LBP_KUBERNETES_CANDIDATE_SOURCE_IMAGE: loadbalancerpro:immutable-candidate-ci")); assertTrue(ci.contains("name: kubernetes-live-topology-smoke-")); }