Prove bounded Kubernetes API-key rotation and rollback

.github/workflows/ci.yml

@@ -12,7 +12,7 @@ jobs:
   build:
     name: Build, Test, Package, Smoke
     runs-on: ubuntu-latest
-    timeout-minutes: 35
+    timeout-minutes: 45
 
     steps:
       - name: Check out repository

README.md

@@ -15,9 +15,10 @@ The default posture is conservative: API-key authentication is selected, proxyin
   loaded rollout/rollback, plus a deployment-equivalent capacity staircase bound to the exact candidate and
   per-replica telemetry; CI-gated Compose and live two-zone Kubernetes proofs cover distribution, candidate abort,
   content-distinct image rollout and baseline rollback under load, immutable inbound-TLS identity rotation and rollback,
-  pod-identity turnover, endpoint continuity, replica loss, planned worker
+  bounded API-key overlap/commit/rollback, pod-identity turnover, endpoint continuity, replica loss, planned worker
   removal, operator-remediated abrupt worker loss, degraded service, and worker recovery.
-- API-key and OAuth2 resource-server modes with deny-by-default API classification.
+- API-key mode with a required primary key and at most one optional rotation key, plus OAuth2 resource-server mode,
+  with deny-by-default API classification.
 - Actuator health/readiness, optional Prometheus metrics, and optional OTLP metrics export with endpoint validation.
 - Capacity-aware, predictive, and evaluation-only allocation APIs in the separate source/Lab Tools runtime.
 - Deterministic request-level comparison, Decision Explorer/browser cockpit surfaces, Enterprise Lab scenarios, durable
@@ -46,9 +47,10 @@ and restores the prior digest. The Kubernetes adapter compiler supplies rollout,
 certificate-rotation, deployment-inspection, and capacity-sampling executables. A disposable two-worker/two-zone kind
   lane now deploys the restricted production image and proves live Service distribution, a metadata-only content-distinct
   candidate rollout and baseline rollback, then rotates between independently rooted certificates through versioned
-  immutable Secrets and restores the original identity. Both transitions require complete pod-UID turnover, unchanged
-  runtime image identity, endpoint continuity, and traffic through both replicas and backends before the lane exercises
-  worker drain/stop, degraded traffic, and
+  immutable Secrets and restores the original identity. It then rolls through immutable A-only, A+B, and B-only API-key
+  Secrets and reverses that sequence for rollback. Every credential transition preserves two ready endpoints, turns over
+  both pod UIDs, keeps the runtime image fixed, and proves the retired key is rejected before the lane exercises worker
+  drain/stop, degraded traffic, and
 operator-remediated no-drain worker loss and recovery. The next action remains to compile the adapters from the reviewed
 staging
 cluster identity, freeze the observed configuration/ingress hashes into the profiles, then run staging qualification
@@ -135,6 +137,7 @@ Important defaults in `application.properties`:
 | Property | Default | Effect |
 | --- | --- | --- |
 | `loadbalancerpro.auth.mode` | `api-key` | Protected API mode |
+| `loadbalancerpro.api.rotation-key` | empty | Optional second key accepted only during an operator-bounded rotation overlap |
 | `loadbalancerpro.proxy.enabled` | `false` | No forwarding until explicitly enabled |
 | `loadbalancerpro.lase.shadow.enabled` | `false` | No shadow evaluation by default |
 | `loadbalancerpro.api.max-request-bytes` | `16384` | Bounded API request bodies |
@@ -169,6 +172,11 @@ export LOADBALANCERPRO_API_KEY='supply-from-a-secret-manager'
 java -jar "$(bash scripts/resolve-executable-jar.sh)" --spring.profiles.active=prod
 ```
 
+API-key rotation is deliberately bounded to two credentials. Roll out a configuration containing primary A plus
+rotation key B, switch clients to B, then roll out B as the sole primary. Rollback reverses the sequence through the
+same A+B overlap. A rotation key cannot replace a missing primary, and the process does not dynamically reread mounted
+credential files.
+
 Do not commit API keys, OAuth tokens, AWS credentials, telemetry headers, private keys, or production targets. Terminate TLS at a trusted reverse proxy, ingress, managed load balancer, platform edge, or service mesh before shared-network exposure.
 
 OTLP metrics are opt-in. When enabled, the endpoint validator rejects blank or malformed URLs, embedded credentials, query strings, fragments, disallowed localhost, and obvious public hosts when private endpoints are required:

deploy/fixture/Dockerfile

@@ -11,4 +11,4 @@ RUN groupadd --gid 10001 fixture \
 COPY --from=build --chown=fixture:fixture /output/FixtureBackend.class /app/FixtureBackend.class
 USER fixture:fixture
 EXPOSE 8080
-ENTRYPOINT ["java", "--add-modules", "jdk.httpserver", "-cp", "/app", "FixtureBackend"]
+ENTRYPOINT ["java", "-Xmx32m", "-Xss256k", "--add-modules", "jdk.httpserver", "-cp", "/app", "FixtureBackend"]

deploy/fixture/FixtureBackend.java

@@ -6,23 +6,33 @@
 import java.net.URI;
 import java.nio.charset.StandardCharsets;
 import java.util.Arrays;
+import java.util.concurrent.ArrayBlockingQueue;
 import java.util.concurrent.CountDownLatch;
-import java.util.concurrent.ExecutorService;
-import java.util.concurrent.Executors;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
 
 public final class FixtureBackend {
     private static final int PORT = 8080;
     private static final int MAX_REQUEST_BYTES = 1_048_576;
     private static final int MAX_RESPONSE_BYTES = 1_048_576;
     private static final long MAX_DELAY_MILLIS = 10_000;
+    private static final int MAX_REQUEST_THREADS = 32;
+    private static final int MAX_PENDING_REQUESTS = 256;
 
     private FixtureBackend() {
     }
 
     public static void main(String[] args) throws Exception {
         String backendId = requiredEnvironment("FIXTURE_ID");
         HttpServer server = HttpServer.create(new InetSocketAddress("0.0.0.0", PORT), 0);
-        ExecutorService executor = Executors.newCachedThreadPool();
+        ThreadPoolExecutor executor = new ThreadPoolExecutor(
+                MAX_REQUEST_THREADS,
+                MAX_REQUEST_THREADS,
+                30,
+                TimeUnit.SECONDS,
+                new ArrayBlockingQueue<>(MAX_PENDING_REQUESTS),
+                new ThreadPoolExecutor.CallerRunsPolicy());
+        executor.allowCoreThreadTimeOut(true);
         CountDownLatch stopped = new CountDownLatch(1);
         server.createContext("/", exchange -> handle(exchange, backendId));
         server.setExecutor(executor);

deploy/kubernetes-proxy-prod.yaml

@@ -43,7 +43,7 @@ spec:
       serviceAccountName: loadbalancerpro
       automountServiceAccountToken: false
       enableServiceLinks: false
-      terminationGracePeriodSeconds: 40
+      terminationGracePeriodSeconds: 45
       topologySpreadConstraints:
         - maxSkew: 1
           minDomains: 2
@@ -132,7 +132,7 @@ spec:
           lifecycle:
             preStop:
               exec:
-                command: ["sh", "-c", "sleep 5"]
+                command: ["sh", "-c", "sleep 10"]
           securityContext:
             allowPrivilegeEscalation: false
             readOnlyRootFilesystem: true

deploy/kubernetes/qualification.yaml

@@ -26,9 +26,19 @@ data:
   LBP_UPSTREAM_1_URL: http://backend-b:8080
   LBP_TLS_HOSTNAME: lbp-kubernetes.local
   LBP_PROXY_STRATEGY: ROUND_ROBIN
+  LBP_HEALTH_CHECK_ENABLED: "false"
   LBP_HEALTH_CHECK_INTERVAL: 1s
   LBP_HEALTHY_THRESHOLD: "1"
   LBP_UNHEALTHY_THRESHOLD: "2"
+  LBP_COOLDOWN_ENABLED: "false"
+  LBP_RETRY_ENABLED: "true"
+  LBP_RETRY_MAX_ATTEMPTS: "3"
+  LBP_RETRY_BUDGET_PERCENT: "100"
+  LBP_RETRY_BACKOFF_BASE: 10ms
+  LBP_RETRY_BACKOFF_MAX: 50ms
+  LBP_RETRY_NON_IDEMPOTENT: "false"
+  LBP_RETRY_METHODS: GET,HEAD
+  LBP_RETRY_STATUSES: 502,503,504
 ---
 apiVersion: apps/v1
 kind: Deployment
@@ -217,7 +227,7 @@ spec:
       serviceAccountName: loadbalancerpro
       automountServiceAccountToken: false
       enableServiceLinks: false
-      terminationGracePeriodSeconds: 40
+      terminationGracePeriodSeconds: 45
       nodeSelector:
         loadbalancerpro.io/qualification-worker: "true"
       topologySpreadConstraints:
@@ -277,7 +287,7 @@ spec:
                   curl --fail --silent --show-error --cacert /run/tls/ca.pem
                   --header "X-API-Key: $(cat /run/secrets/loadbalancerpro.api.key)"
                   --resolve "${LBP_TLS_HOSTNAME}:8080:127.0.0.1"
-                  "https://${LBP_TLS_HOSTNAME}:8080/actuator/health" > /dev/null
+                  "https://${LBP_TLS_HOSTNAME}:8080/proxy/kubernetes/topology" > /dev/null
             periodSeconds: 3
             timeoutSeconds: 2
             failureThreshold: 3
@@ -297,7 +307,7 @@ spec:
           lifecycle:
             preStop:
               exec:
-                command: ["sh", "-c", "sleep 5"]
+                command: ["sh", "-c", "sleep 10"]
           securityContext:
             allowPrivilegeEscalation: false
             readOnlyRootFilesystem: true
@@ -322,10 +332,12 @@ spec:
       volumes:
         - name: api-key
           secret:
-            secretName: loadbalancerpro-api-key
+            secretName: loadbalancerpro-api-key-a
             items:
               - key: api-key
                 path: loadbalancerpro.api.key
+              - key: rotation-key
+                path: loadbalancerpro.api.rotation-key
         - name: server-tls
           secret:
             secretName: loadbalancerpro-server-tls-a

docs/DEPLOYMENT.md

@@ -59,7 +59,16 @@ curl --cacert "$LBP_TLS_DIRECTORY/ca.pem" --resolve lbp.local:18443:127.0.0.1 \
 docker compose -f deploy/docker-compose.proxy-prod.yml down
 ```
 
-The API key is mounted read-only as `/run/secrets/loadbalancerpro.api.key` and imported through Spring config trees. The temporary example uses read-only files inside a private `mktemp` parent so the image's non-root user can read the mounts; for a durable host path, grant read access only to the runtime UID/GID through the host's ownership or ACL mechanism. TLS, trust, client identity, and additional configuration directories are separate read-only mounts. Define backend custom trust or mTLS bundles only in the external configuration directory, for example:
+The required primary API key is mounted read-only as `/run/secrets/loadbalancerpro.api.key` and imported through Spring
+config trees. During a bounded rotation overlap only, a second key may be supplied as
+`/run/secrets/loadbalancerpro.api.rotation-key`; both authenticate, but the rotation key cannot replace a missing primary.
+Roll from A-only to A+B, switch clients to B, then roll to B-only. Rollback reverses the sequence. The process snapshots
+both values at startup and does not dynamically reread mounted credentials, so use versioned immutable Secrets and a
+zero-unavailable pod rollout instead of mutating an in-use Secret. The temporary example uses read-only files inside a
+private `mktemp` parent so the image's non-root user can read the mounts; for a durable host path, grant read access only
+to the runtime UID/GID through the host's ownership or ACL mechanism. TLS, trust, client identity, and additional
+configuration directories are separate read-only mounts. Define backend custom trust or mTLS bundles only in the
+external configuration directory, for example:
 
 ```properties
 spring.ssl.bundle.pem.backendtrust.truststore.certificate=file:/run/trust/ca.pem
@@ -91,19 +100,31 @@ Hostname verification remains mandatory; `tls.verify=false` is rejected. Server
 | `LBP_MAX_REQUEST_BYTES` | `loadbalancerpro.proxy.max-request-bytes` | `65536` |
 | `LBP_MAX_RESPONSE_BYTES` | `loadbalancerpro.proxy.max-response-bytes` | `0` (streaming/unbounded) |
 | `LBP_MAX_IN_FLIGHT` | `loadbalancerpro.proxy.limits.max-in-flight` | `100` |
+| `LBP_HEALTH_CHECK_ENABLED` | `loadbalancerpro.proxy.health-check.enabled` | `true` |
 | `LBP_HEALTH_CHECK_PATH` | `loadbalancerpro.proxy.health-check.path` | `/health` |
 | `LBP_HEALTH_CHECK_INTERVAL` | `loadbalancerpro.proxy.health-check.interval` | `5s` |
+| `LBP_COOLDOWN_ENABLED` | `loadbalancerpro.proxy.cooldown.enabled` | `true` |
 | `LBP_COOLDOWN_DURATION` | `loadbalancerpro.proxy.cooldown.duration` | `30s` |
 | `LBP_DRAIN_TIMEOUT` | `loadbalancerpro.proxy.reload.drain-timeout` | `30s` |
 | `LBP_SLOW_START_DURATION` | `loadbalancerpro.proxy.slow-start.duration` | `5s` |
+| `LBP_RETRY_ENABLED` | `loadbalancerpro.proxy.retry.enabled` | `false` |
+| `LBP_RETRY_MAX_ATTEMPTS` | `loadbalancerpro.proxy.retry.max-attempts` | `2` |
+| `LBP_RETRY_BUDGET_PERCENT` | `loadbalancerpro.proxy.retry.budget-percent` | `20` |
+| `LBP_RETRY_BACKOFF_BASE` | `loadbalancerpro.proxy.retry.backoff.base` | `50ms` |
+| `LBP_RETRY_BACKOFF_MAX` | `loadbalancerpro.proxy.retry.backoff.max` | `1s` |
+| `LBP_RETRY_NON_IDEMPOTENT` | `loadbalancerpro.proxy.retry.retry-non-idempotent` | `false` |
+| `LBP_RETRY_METHODS` | `loadbalancerpro.proxy.retry.methods` | `GET,HEAD` |
+| `LBP_RETRY_STATUSES` | `loadbalancerpro.proxy.retry.retry-statuses` | `502,503,504` |
 | `LBP_BACKEND_TRUST_BUNDLE` | `loadbalancerpro.proxy.backend-tls.truststore` | blank |
 | `LBP_UPSTREAM_0_CLIENT_CERT_BUNDLE` | `loadbalancerpro.proxy.upstreams[0].tls.client-cert` | blank |
 
 [`../deploy/kubernetes-proxy-prod.yaml`](../deploy/kubernetes-proxy-prod.yaml) is the canonical deployment base. It
 encodes two replicas, zero-unavailable rolling replacement, a two-domain zone-spread rule that permits a temporary
-surge pod, preferred host spreading, a one-replica disruption budget, startup/readiness/liveness probes, a five-second
-preStop delay, a 40-second termination window, a token-free service account, numeric non-root execution, and external
-Secret/ConfigMap mounts. Its image remains a deliberately non-resolving digest placeholder. The disposable
+surge pod, preferred host spreading, a one-replica disruption budget, startup/readiness/liveness probes, a ten-second
+preStop delay, a 45-second termination window, a token-free service account, numeric non-root execution, and external
+Secret/ConfigMap mounts. The drain delay exceeds the five-second qualification client timeout, while the termination
+window contains the application's 30-second graceful-shutdown bound. Its image remains a deliberately non-resolving
+digest placeholder. The disposable
 [`../scripts/bench/proxy-kubernetes-topology.sh`](../scripts/bench/proxy-kubernetes-topology.sh) lane applies the
 separate loopback qualification workload and proves a metadata-only content-distinct candidate rollout and baseline
 rollback under continuous traffic, complete pod-UID turnover in both directions, runtime-image identity transition and
@@ -112,7 +133,15 @@ continuous traffic windows. The TLS exercise uses independently generated roots,
 fingerprints, and single-CA positive/negative checks; it also requires fresh pod UIDs, unchanged runtime image identity,
 ready-endpoint continuity, and traffic through both replicas and backends in both directions. It proves application
 server TLS termination behind the loopback NodePort, not an ingress controller, external issuer, or trust-distribution
-system. The lane then proves two-zone Service distribution, planned worker removal, and
+system. The lane next proves bounded API-key rotation through immutable A-only, A+B, and B-only Secrets and reverses the
+sequence for rollback. Both keys are accepted only in the overlap phases; the retired key must return 401 after each
+commit, while zero-unavailable endpoint continuity, fresh pod UIDs, fixed runtime image identity, and traffic through
+both replicas and backends remain required. This is startup configuration rollout proof, not dynamic Secret reload or
+external secret-manager proof. Because each configured upstream is a Kubernetes Service rather than a pod, the local
+qualification lane uses EndpointSlice readiness as the pod-health authority and disables process-local active health
+checks and cooldown; up to three bounded attempts remain enabled for `GET`/`HEAD`, trying both Services before cycling
+after stale pooled connections have been discarded. The lane then proves two-zone
+Service distribution, planned worker removal, and
 operator-remediated no-drain worker loss and recovery. The abrupt-loss exercise forcibly stops the kind worker,
 confirms its container is down, applies the out-of-service `NoExecute` taint, and force-removes the three exact stateless
 qualification pods from the API. The disposable cluster also pins immediate EndpointSlice-triggered iptables updates

docs/LOAD_BALANCER_BUILD_OUT.md

@@ -165,15 +165,23 @@ an independently rooted immutable candidate Secret and back under continuous tra
 served-leaf fingerprints prove both identity transitions; fresh pods, unchanged runtime image identity, two-zone
 continuity, and positive traffic deltas on both replicas and backends are required in both directions. This exercises
 application server TLS behind the loopback NodePort, not an ingress controller, external issuer, or client
-trust-distribution system. One worker is then
+trust-distribution system. The restored-certificate deployment then rotates credentials through immutable A-only, A+B,
+and B-only API-key Secrets and rolls back through the same bounded overlap. Each of the four zero-unavailable rollouts
+must preserve two ready endpoints and both-zone placement, replace both pod UIDs without changing the runtime image,
+serve traffic through both replicas/backends, and enforce the expected positive and negative key boundary. This proves
+startup credential rollout mechanics, not dynamic Secret reload or an external secret manager. One worker is then
 drained and stopped under load, degraded traffic must continue through the remaining replica, and the stopped worker
 and second replica must recover inside the bound, and both recovered replicas and backends must serve new traffic. It
 then forcibly stops that recovered worker without a drain, confirms the container is down, and applies the documented
 out-of-service `NoExecute` remediation. It force-removes the three exact stateless workload pods from the API, bounds
 endpoint withdrawal, proves degraded traffic, rejects the failed pod identity after recovery, and requires both
 recovered replicas and backends to serve new traffic. The disposable cluster pins iptables-mode kube-proxy
 to immediate EndpointSlice-triggered updates and a one-second cleanup sync; deployment environments must review the
-equivalent Service/ingress failure-detection and reconciliation behavior. The candidate has a distinct local image
+equivalent Service/ingress failure-detection and reconciliation behavior. Its abrupt transition and degraded phases
+bound stale conntrack impact at 90% and 95% success with 5.5-second p99 ceilings; recovered traffic must return to the
+normal 99.9% success and 1.5-second p99 objectives. The ten-second endpoint drain exceeds the five-second qualification
+client timeout, and the 45-second termination grace contains the 30-second application shutdown bound. The candidate
+has a distinct local image
 content ID but preserves the baseline application layers, so it proves Kubernetes transition and rollback mechanics
 rather than compatibility between application releases. The reviewed deployment ingress, deployment-equivalent
 resources, registry
@@ -200,7 +208,9 @@ turnover and a content-distinct runtime image transition, then proves another co
 the initial runtime image identity. It restores two-zone placement and requires positive post-transition traffic deltas
 on both candidate/restored replicas and both backends. It applies the same zero-unavailable and distribution checks to
 versioned immutable TLS Secret rotation and rollback while proving the served leaf fingerprint changes and returns and
-the runtime image identity remains fixed.
+the runtime image identity remains fixed. It applies those rollout, pod-turnover, endpoint-continuity, and distribution
+checks again to bounded API-key overlap, candidate commit, rollback overlap, and baseline commit, with 401 checks for the
+retired credential after each commit.
 
 Use an immutable image digest and begin with a small, explicitly approved traffic slice. During every step, compare
 client success/latency, upstream health, proxy p95/p99, in-flight work, retries, sheds, cooldown trips, CPU, memory, GC,