From 74b1f6758304bc5a3a85ff4888039e7309324ddf Mon Sep 17 00:00:00 2001 From: RicheyWorks <730richey730@gmail.com> Date: Mon, 20 Jul 2026 23:59:12 -0700 Subject: [PATCH 1/5] docs: architecture audit, deployability + lab build plans, strategy playground --- docs/AUDIT_2026-07-21.md | 194 +++++++++++ docs/AUDIT_LAB_SHADOW_2026-07-21.md | 154 ++++++++ docs/BUILD_PLAN_DEPLOYABLE.md | 158 +++++++++ docs/BUILD_PLAN_LAB_SHADOW.md | 147 ++++++++ docs/strategy-playground.html | 523 ++++++++++++++++++++++++++++ 5 files changed, 1176 insertions(+) create mode 100644 docs/AUDIT_2026-07-21.md create mode 100644 docs/AUDIT_LAB_SHADOW_2026-07-21.md create mode 100644 docs/BUILD_PLAN_DEPLOYABLE.md create mode 100644 docs/BUILD_PLAN_LAB_SHADOW.md create mode 100644 docs/strategy-playground.html diff --git a/docs/AUDIT_2026-07-21.md b/docs/AUDIT_2026-07-21.md new file mode 100644 index 00000000..a7c9d3e7 --- /dev/null +++ b/docs/AUDIT_2026-07-21.md @@ -0,0 +1,194 @@ +# LoadBalancerPro — Deep Audit & Core Feature Plan + +**Repo:** RicheyWorks/LoadBalancerPro · audited at commit `e800ba06` (post-v2.5.0, branch `codex/command-ledger-restart-reconciliation`) +**Date:** 2026-07-21 +**Scope:** Full sweep — architecture, correctness/concurrency, security, performance, and feature gaps — focused on the core load balancer. Findings verified against source with file:line references. + +--- + +## 1. Executive summary + +LoadBalancerPro is ~102.6k lines of main-source Java across 493 files, but **the real load balancer — the code that moves actual network traffic — is about 3.7k lines (~3.5%)**: the `api/proxy/` package plus the five routing strategies it consumes. The remaining ~96% is a numeric simulation core (`core/LoadBalancer`, `ServerMonitor`), a calculation-only allocation API, and a very large lab/evidence/replay/explorer apparatus (`lab/` alone is 41.6k lines) that never carries a request. + +The proxy that does exist is defensively written — hop-by-hop stripping, SSRF-resistant authority pinning, idempotency-aware retries, fail-closed reload auth, verified backend TLS — but it is **narrow, buffered, and partially decorative**: + +1. **It's off by default and not enabled in prod** (`application.properties:31`, no override in `application-prod.properties`). The shipped configuration ships no data plane. +2. **Three of the five routing strategies don't actually adapt.** Least-connections, weighted-least-load, and tail-latency-P2C read `inFlightRequestCount`, latency, and error-rate from *static config* that no traffic ever updates. Only round-robin variants and health filtering behave as advertised on live traffic. +3. **Full request/response buffering** caps bodies at 64KB, makes large upstream responses a heap bomb, and forecloses streaming, SSE, WebSocket, and gRPC. +4. **Every call to the flagship allocation API leaks a JVM shutdown hook** plus the whole balancer object graph. + +The fastest path to a credible core LB: fix the four correctness/resource bugs below, then build the top four features (live telemetry loop, streaming, TLS termination, background health + real timeouts) — most of which convert already-written, dormant code into live capability. + +--- + +## 2. Architecture reality check + +**Entry point:** single Spring Boot app, `api/LoadBalancerApiApplication.java`. If any lab/experiment CLI flag is present it runs a batch command and exits; otherwise it boots the API on :8080 (Dockerfile, prod profile). + +**Three disjoint worlds:** + +| World | Size | Carries traffic? | +|---|---|---| +| **Data plane** — `api/proxy/` (ReverseProxyController → ReverseProxyService → JDK HttpClient) + 5 strategies in `core/` | ~3.7k lines | Yes, when `loadbalancerpro.proxy.enabled=true` (default false) | +| **Simulation core** — `core/LoadBalancer`, `ServerRegistry`, `ServerMonitor`, `ConsistentHashRing`, `LoadDistributionEngine`, driven by `POST /api/allocate/*`, CLI, GUI | ~15k lines | No — pure calculation; `ServerMonitor` fabricates CPU/mem/disk with `ThreadLocalRandom` (`ServerMonitor.java:347-356`) | +| **Lab/evidence apparatus** — `lab/` (41.6k), DecisionExplorer/Replay/Evidence half of `api/` (~30k), `cli/`, `gui/`, HTML cockpits | ~80k lines | No | + +Key structural facts: + +- The proxy does **not** use `core/LoadBalancer` or `ServerRegistry` at all. Upstreams come from `ReverseProxyProperties` (static config, hot-reloadable), converted per-request into transient `ServerStateVector`s (`ReverseProxyService.java:492-524`). +- `ConsistentHashRing` is real, tested code — but it is **not** a registered routing strategy and is unreachable from the proxy. +- Health checking exists twice: real HTTP probes in the proxy (`probeUpstream`, `ReverseProxyService.java:622-642`) and random-walk simulation in `ServerMonitor` (never probes anything). +- The two strategy systems are unrelated: `RoutingStrategyId` (5 strategies, live) vs `LoadBalancer.Strategy` (ROUND_ROBIN/LEAST_LOADED, simulation only). + +--- + +## 3. Verified defects (prioritized) + +### P0 — fix before anything else + +**D1. Adaptive strategies run on frozen config telemetry — live adaptive routing is non-functional.** +`ReverseProxyService.toCandidate()` (`:504-525`) builds each `ServerStateVector` from `ReverseProxyProperties.Upstream` config fields. `inFlightRequestCount` (default 0), `averageLatencyMillis`, `p95/p99`, `recentErrorRate`, `queueDepth` are **never updated by traffic** — `setInFlightRequestCount` is only called when copying config (`:887`). With `WEIGHTED_LEAST_CONNECTIONS` or `TAIL_LATENCY_POWER_OF_TWO`, a backend drowning in slow requests keeps reporting its configured count, so it keeps receiving full traffic; the strategy degenerates to a static tie-break. *Three independent audit passes converged on this.* + +**D2. Per-request JVM shutdown-hook + object-graph leak on the allocation API.** +Every `POST /api/allocate/capacity-aware|predictive` → `AllocatorService.createLoadBalancer()` (`:199`) → `new LoadBalancer(...)` → `new ServerMonitor(...)` (`LoadBalancer.java:72`) → `Runtime.getRuntime().addShutdownHook(...)` (`ServerMonitor.java:92`). The hook is never removed (`stop()` returns early because the monitor was never started), and each hook thread pins the full balancer graph. Sustained traffic → unbounded heap growth → OOM. + +**D3. Unhealthy servers are permanently evicted (simulation core).** +`ServerHealthCoordinator.detectFailedServers()` (`:41-52`) treats one bad sample or a manual drain flag as terminal: `serverRegistry.remove(failed)` with no re-admission path (only CLOUD servers get replacements). Combined with `ServerMonitor`'s random-walk metrics, a long-running monitor can randomly walk healthy servers over the threshold and empty the pool irreversibly. + +**D4. Shared strategy singletons corrupt state across routes.** +`RoutingStrategyRegistry.defaultRegistry()` (`:15-20`) hands the *same* `WeightedRoundRobinRoutingStrategy` instance to every route; its `retainOnly()` (`:85-90`) deletes accumulators absent from the current call's candidates. With two routes on WRR, interleaved traffic resets smooth-WRR every request — a 3:1 weight config delivers ~100:0. Retries similarly purge the just-attempted upstream's accumulator. + +### P1 — high + +**D5. Full buffering of request and response bodies.** +Request: Spring materializes the entire body as `byte[]` and `forward()` clones it *before* the 64KB `maxRequestBytes` check (`ReverseProxyService.java:102-109`); the `RequestSizeLimitFilter` does not cover `/proxy` (`RequestSizeLimitFilter.java:60-64`). Response: `BodyHandlers.ofByteArray()` (`:390`) — no cap; a multi-GB upstream response is fully heap-materialized per concurrent request. + +**D6. Blocking health probes on the request path + probe thundering herd.** +`effectiveHealth` (`:565-608`) runs due probes synchronously on the Tomcat request thread for *all* route targets before routing — up to `timeout` (1s) × N added to a live request every interval. The `probeStates` check-then-act (`:590-599`) lets N concurrent threads all fire the probe and each count the failure toward the cooldown threshold, tripping the breaker prematurely. `GET /api/proxy/status` also fires probes and mutates cooldown counters. + +**D7. No connect timeout on the shared HttpClient.** +`ReverseProxyConfiguration` builds the client bare (verified: zero `connectTimeout` occurrences). A black-holed backend holds a request thread for the OS TCP timeout (minutes). Only a single total `request-timeout` (2s default) exists — no separate connect/read/idle, no per-route override. + +**D8. No `X-Forwarded-For/Proto/Host` injection; client-supplied forwarding headers passed through.** +The proxy adds no forwarding headers (verified: no `X-Forwarded` in `api/proxy/`), so backends can't see the real client — and a client can *spoof* `X-Forwarded-For`/`X-Forwarded-Host` straight through to backends that trust them. + +**D9. Unauthenticated data plane outside prod/cloud-sandbox profiles.** +In default `api-key` mode the security chain is `anyRequest().permitAll()` (`ApiSecurityConfiguration.java:63`); the only credential check (`ProdApiKeyFilter`) is `@Profile({"prod","cloud-sandbox"})`. Without those profiles, `/proxy/**`, `/api/allocate/**`, `/api/routing/**`, and `/api/proxy/status` (which discloses upstream URLs) are open. Reload is separately protected and fails closed. OAuth2 mode is correct. + +### P2 — medium + +- **D10. Load-shedding priority inversion** (`LoadSheddingPolicy.java:68-74`): with `criticalBypassEnabled=false`, HARD pressure sheds CRITICAL unconditionally while USER can be allowed. Latent (default config masks it). +- **D11. Failover redistribution double-count/overwrite** (`ServerHealthCoordinator.java:74-85`): `leastLoaded()` merges redistribution into current totals, then `putAllAllocations` overwrites totals with only the slice — tracked load silently vanishes. +- **D12. `consistentHashing` reads the ring without the server lock** (`LoadBalancer.java:183-209`): NPE race when the monitor removes the last server between the empty-check and `firstEntry()`. +- **D13. Weight 0 promoted to full default weight** (`WeightedRoundRobinRoutingStrategy.java:100-105`): an operator "drain" weight of 0 silently becomes 1.0; discontinuous (0→1.0, 0.01→0.1). +- **D14. `reload()` wipes probe/cooldown state** (`:262-263`): a config reload instantly restores traffic to upstreams in failure cooldown. +- **D15. Actuator `metrics`/`prometheus` unauthenticated in default profile** (`application.properties:1`); prod correctly narrows to `health,info`. +- **D16. Rate-limiter bucket map never evicted**; key is attacker-mintable when `trust-forwarded-for=true` (`ApiRateLimitFilter.java:47,123-135`). + +### P3 — low (recorded, fix opportunistically) + +`Server.updateHistory` counter overflow → negative index after 2^31 updates (`Server.java:288`, should use `Math.floorMod`); rollback snapshot overwritten before validation (`:263-269`); lost updates between concurrent single-metric setters (`:453-463`); dead non-thread-safe `ServerRegistry.loadQueue`; `ServerMonitor.start()` check-then-act double-start; JVM-wide `isCloudServer` system property overrides every constructor arg (`Server.java:124-126`); ConsistentHashRing fallback-hash add/remove asymmetry; interrupted forwards misclassified as retriable; per-request stream/set rebuilds on the hot path (`normalizedRetryMethods` etc.). + +### What's genuinely good (keep) + +Hop-by-hop stripping both directions (request-smuggling safe); SSRF not client-steerable — authority pinning, scheme restriction, control-char rejection, redirects disabled; backend TLS uses default verification (no insecure TrustManager anywhere); idempotency-aware retry defaults that never re-hit an attempted upstream; fail-closed reload with constant-time key compare and atomic config swap; pervasive input validation with explained no-candidate decisions instead of throws; `LongAdder` metrics; correct `Math.floorMod` round-robin; non-root Docker; no secrets in repo. `HARDENING_AUDIT.md` is honest about its gaps. + +--- + +## 4. Feature gap matrix (verified in code) + +| Capability | Status | Notes | +|---|---|---| +| HTTP/1.1 keep-alive | ✅ framework default | No tuning | +| HTTP/2 | ⚠️ outbound only (JDK client default) | No inbound h2 | +| WebSocket / gRPC / SSE / streaming | ❌ | Foreclosed by full buffering + upgrade headers stripped | +| TCP (L4) mode | ❌ | | +| TLS termination / SNI / cert reload | ❌ | No `server.ssl.*` anywhere | +| Backend TLS | ⚠️ | Verified by default (good); no custom truststore/mTLS | +| Routing algorithms | ⚠️ 5 wired, 3 decorative | D1 | +| Path-based routing | ✅ | Longest-prefix, per-route strategy | +| Host/header-based routing | ❌ | | +| Sticky sessions / hash affinity | ❌ live path | `ConsistentHashRing` exists, unwired | +| Active health checks | ⚠️ | Real probes but on-request-path, single-probe flip, no thresholds (D6) | +| Passive outlier detection | ✅ basic | Consecutive-failure cooldown | +| Slow-start / warmup | ❌ | Recovered upstream gets full share instantly | +| Retries | ✅ basic | No budget, no backoff, no per-try timeout | +| Circuit breaking | ⚠️ | Cooldown ≈ crude breaker | +| Timeouts | ⚠️ | Single total timeout; no connect timeout (D7) | +| Load shedding / concurrency limits | ❌ live path | `AdaptiveConcurrencyLimiter` + `LoadSheddingPolicy` built, tested, unwired | +| Graceful shutdown / draining | ❌ | No `server.shutdown=graceful`, no drain state | +| Hot config reload | ✅ | Validate-before-apply, atomic swap; but wipes cooldowns (D14) | +| Dynamic backend registration | ⚠️ | Full-config reload only | +| Service discovery (DNS/cloud) | ❌ live path | AWS SDK serves the autoscaling demo only | +| Prometheus metrics | ⚠️ | Actuator+Micrometer on classpath; zero proxy metrics reach the registry; prod disables prometheus | +| Access logs | ❌ | Successful proxied requests log nothing | +| Per-backend stats | ✅ basic | JSON counters, no latency histograms | +| Rate limiting | ✅ disabled by default | Covers `/proxy/**` | +| X-Forwarded-* handling | ❌ | Neither injected nor sanitized (D8) | +| Weighted canary / splitting | ⚠️ | Static weights only | +| Mirroring | ❌ | LASE "shadow" is simulation, not mirroring | +| Body size limits | ⚠️ | Enforced after full buffering (D5) | +| All-backends-down behavior | ✅ | Fast 503 JSON, no hang | + +--- + +## 5. Feature plan — what the core load balancer needs + +Ordered by value. Items 1, 5, and 6 largely *wire up code that already exists*. + +### Phase A — make the existing proxy honest (1–2 weeks of focused work) + +**1. Live telemetry feedback loop (fixes D1).** +Track per-upstream state in the proxy: increment/decrement an in-flight counter around `forwardOnce`, keep rolling latency (EWMA + p95/p99 ring buffer) and error-rate windows, and feed *those* into `ServerStateVector` instead of config constants. Spec: a `UpstreamRuntimeStats` (LongAdder in-flight, `synchronized` or striped ring for latency) held in the existing `ConcurrentHashMap` keyed by upstream id; `toCandidate()` merges config (weight, healthy) with runtime stats. This single change converts least-connections, weighted-least-load, and tail-latency-P2C from decorative to real — the largest capability-per-line win in the codebase. + +**2. Timeout correctness (fixes D7).** +`connectTimeout` on the HttpClient builder (config: `proxy.connect-timeout`, default ~1s); per-route `request-timeout` override; distinct health-probe timeout already exists. Add retry backoff (exponential, jittered) and a retry budget (retries ≤ N% of a rolling window) while touching this code. + +**3. Background health checking (fixes D6).** +Move probing to a scheduled executor (one task per upstream, jittered), with rise/fall thresholds (e.g., 2 consecutive successes to admit, 3 failures to eject) instead of single-probe flips. Request path reads a volatile health snapshot only. Status endpoint becomes read-only. Kills the probe latency tax, the thundering herd, and the status-endpoint side effects in one move. + +**4. Forwarding headers (fixes D8).** +Append-safe `X-Forwarded-For`, set `X-Forwarded-Proto`/`X-Forwarded-Host` (and RFC 7239 `Forwarded`), configurable trust policy for inbound values (strip by default, trust-from-CIDR optional). Optional per-route header add/remove/set rules. + +**5. Wire load shedding + concurrency limits into `forward()`.** +`AdaptiveConcurrencyLimiter` and `LoadSheddingPolicy` are built and unit-covered. Enforce a global and per-upstream in-flight cap in the proxy (503 + `Retry-After` beyond limit). Fix the CRITICAL/USER inversion (D10) while wiring it. + +**6. Consistent-hash + sticky sessions on the live path.** +Register a `CONSISTENT_HASH` strategy (key = client IP or configured header) backed by the existing `ConsistentHashRing`, and a cookie-affinity mode. Reuses tested code; fixes the ring's lock and fallback-hash edge cases (D12, P3) as part of the port. + +### Phase B — make it deployable (2–4 weeks) + +**7. Streaming data path (fixes D5).** +Replace `byte[]`-in/`byte[]`-out with streamed bodies and bounded buffers — either servlet async streaming with `InputStream`/`BodyPublishers.ofInputStream` + capped response streaming, or a reactive/Netty rewrite of the proxy layer if HTTP/2 inbound and WebSocket are wanted soon. Enforce request size from `Content-Length` pre-read and a hard streamed cap. This is the prerequisite for SSE, WebSocket, gRPC, and large payloads. + +**8. TLS termination + SNI + cert hot-reload.** +Spring Boot SSL bundles give termination and file-watch reload nearly for free (`server.ssl.bundle`); add SNI mapping and optional per-upstream truststore/mTLS for backend connections. + +**9. Graceful shutdown + draining (fixes D14 partially).** +`server.shutdown=graceful` with a drain window; per-upstream administrative `DRAINING` state (no new picks, in-flight completes) usable from reload; make reload *carry over* probe/cooldown/runtime state for unchanged upstreams instead of clearing it. + +**10. Real observability.** +Port `ReverseProxyMetrics` to Micrometer (counters + timers tagged upstream/route/status-class, latency histograms → `/actuator/prometheus`, enabled in prod behind auth), add a structured access log line per proxied request (client, route, upstream, status, duration, retries), and secure actuator in the default profile (D15). + +**11. Slow-start/warmup.** +Time-based weight ramp for upstreams leaving cooldown or newly added (e.g., linear 0→full over `warmup-seconds`), preventing recovered-backend re-collapse. + +### Phase C — grow the control plane (later) + +**12.** Incremental backend add/remove/drain API (not full-config reload), then DNS-based discovery with periodic re-resolution. +**13.** Host/header-based routing rules and percentage-based canary splitting (the weights machinery generalizes). +**14.** Decide the fate of the simulation core: either clearly fence `core/LoadBalancer`/`ServerMonitor` as demo-only (and fix D2, D3, D11 anyway since the allocation API is live), or retire the random-walk monitor in favor of the proxy's real telemetry. The lab/evidence apparatus (80k lines) should be quarantined behind a Maven profile or module so the core stays auditable. + +### Also fix immediately regardless of roadmap (small, surgical) + +- D2: remove the shutdown hook (register once per process, or none for request-scoped balancers). +- D3: re-admission path for recovered servers; drain ≠ delete. +- D4: per-route strategy instances (registry returns a factory, not a singleton). +- D9: make `ProdApiKeyFilter` profile-independent when a key is configured, or fail startup in api-key mode with no key. +- D13: treat weight 0 as "no traffic," never promote to default. + +--- + +## 6. Suggested sequencing for Codex + +Each Phase A item is an independent, reviewable PR touching mostly `api/proxy/` — they fit the existing PR cadence. Recommended order: **A2 (timeouts) → A3 (background health) → A1 (live telemetry) → A5 (shedding) → A4 (headers) → A6 (hashing)**, then the immediate-fix list interleaved. A1 before A5 because the concurrency limiter wants the in-flight counter A1 introduces. Phase B7 (streaming) is the one item worth a design discussion first, since it decides whether the proxy stays servlet-based or moves to Netty. diff --git a/docs/AUDIT_LAB_SHADOW_2026-07-21.md b/docs/AUDIT_LAB_SHADOW_2026-07-21.md new file mode 100644 index 00000000..07e32838 --- /dev/null +++ b/docs/AUDIT_LAB_SHADOW_2026-07-21.md @@ -0,0 +1,154 @@ +# LoadBalancerPro — Deep Audit: Lab, Shadow & Analysis Subsystems + +**Repo:** RicheyWorks/LoadBalancerPro · commit `e800ba06` (post-v2.5.0) +**Date:** 2026-07-21 +**Companion to:** `AUDIT_2026-07-21.md` (the live proxy + basic simulation core). This document covers everything that audit deliberately set aside: the LASE shadow/adaptive experiment cluster, the Enterprise Lab apparatus (`lab/`), the DecisionExplorer/Replay/Evidence API surface, and the CLI/GUI/demo/cockpit/docs surfaces. + +Defect IDs here are namespaced by area (LASE `L#`, Lab `C#`/`S#`, Explorer `E#`, Operator `O#`) so they don't collide with the proxy audit's `D#`. All findings verified against source; representative file:line references included. + +--- + +## 1. Executive summary + +If the first audit's headline was "the real load balancer is 3.5% of the codebase," this one explains the other ~96%. It divides cleanly: + +- **~40k lines (`lab/`)** — an offline crash-recovery / durable-evidence harness that arms a routing decision over three fake loopback backends (`http://127.0.0.1:1`), evaluates it, and records everything in hash-chained, fsync'd, OS-locked evidence files. The durability engineering is real and some of it is genuinely good; but roughly half is duplication and proof-runners-proving-proof-runners, and it ships integration-test tooling inside the production jar. +- **~24k lines (DecisionExplorer + RoutingDecisionReplay + Evidence services in `api/`)** — a analysis layer that takes one routing-comparison result and restates it through ~40 derived "evidence" objects. About 90% is derivational restatement; there is a service whose sole job is to report whether *other response objects* have null fields. +- **~10k lines (LASE shadow + Adaptive experiment + Cloud in `core/`)** — a "shadow evaluation" pipeline plus real AWS autoscaling integration. The plumbing is production-grade but it evaluates **fabricated telemetry** (latency/error rates synthesized from load score), so its output is currently hollow; the AWS side has several dangerous-when-enabled defects. + +**The single most important structural finding:** the same idea — "compare/explain routing strategies on synthetic traffic" — is implemented **at least five times** (LASE agreement tracking, AdaptiveRoutingExperiment fixtures, AdaptiveRoutingScenario matrix, `core/RoutingComparisonEngine`, and the DecisionExplorer/Replay stack). Consolidating these would delete an estimated **20,000+ lines with zero capability loss**, and — more valuably — the pieces worth keeping (factor-level explainability, counterfactual weight scenarios, shadow evaluation) become genuinely useful the moment they're fed **real proxy decisions** instead of synthetic inputs. That's the throughline of the build plan: *delete the ceremony, wire the survivors to live traffic.* + +**Security note that spans all three areas:** in the default profile the Spring security chain is `anyRequest().permitAll()` and the only credential filter is `@Profile("prod","cloud-sandbox")` — so on a default run, the destructive lab endpoints (`POST /api/lab/.../retention` can delete all evidence), the DoS-amplifying explorer endpoint, and the status surfaces are all unauthenticated on 0.0.0.0:8080. This is the same root cause as the proxy audit's D9 and is the top cross-cutting fix. + +--- + +## 2. LASE shadow / adaptive / cloud cluster (`core/`) + +**What it is.** After each load distribution, `LaseShadowAdvisor.observe()` synthesizes an evaluation input, runs five sub-evaluators (routing choice, concurrency AIMD, load shedding, autoscaling, failure scenario), records a bounded (100) synchronized event, and compares its pick to the actual allocation. Off by default everywhere (`loadbalancerpro.lase.shadow.enabled=false` in all three property files — verified). Reachable via `LoadBalancer` when the flag is set, or `GET /api/allocator/lase/shadow`. The Adaptive policy engine (`off|observe|shadow|recommend|active-experiment`) gates whether a recommendation may replace baseline — but **no caller ever applies a passing decision to traffic**; `AdaptiveTrafficDecisionOrchestrator` literally appends "decision record only; no traffic action performed." + +### Verified defects + +**L1 (High, feature-hollow). Shadow telemetry is fabricated, not measured.** `LaseShadowAdvisor.java:279-290` computes `averageLatency = 50.0 + loadScore`, `p95 = avg + 40 + queueDepth*0.5`, `errorRate = healthy ? min(0.10, loadScore/1000) : 0.30`. Every downstream signal (concurrency, shedding, autoscaling, failure) is thus a linear transform of `Server.getLoadScore()`. The advisor never receives real latency or error data, even on the live path. The entire evaluation output is decorative. + +**L2 (High, feature-broken). The AIMD concurrency limiter's feedback loop never closes.** `LaseEvaluationEngine.java:37-39` constructs a fresh `AdaptiveConcurrencyLimiter` per call and discards `nextLimit` (used only in report strings); `currentConcurrencyLimit` is re-derived from server capacities every time. The "adaptive" limit never walks — same transition reported every event, nothing enforces it. + +**L3 (High→Med, wrong analytics). Utilization unit mismatch.** `LaseShadowAdvisor.java:246-275` feeds *load units* (e.g. 300.0) where `AutoscalingSignal.utilization()` expects a 0–1 ratio (`inFlight/capacity`), and clamps the concurrency limit to `min(100, …)` regardless of real capacity. Result: 3 servers + load 100 → "utilization 33.3 exceeded threshold 0.85" → SCALE_UP always; nearly every load classifies as CAPACITY_SATURATION. + +**L4 (Med). Agreement rate is statistically meaningless.** `:131-135` compares the argmax of a full distribution against a `TailLatencyPowerOfTwo` pick that samples only 2 random candidates — expected agreement ≤ ~2/N even for a perfect strategy. The headline metric is uninterpretable. + +**Cloud (real AWS integration) — dangerous when enabled:** + +**L5 (High, live+autonomous). `predictCapacityWithAI` is nonsense with a NaN path.** `CloudManager.java:431-440`: `totalLoad / (totalLoad/n * 1.5)` = `n/1.5`, independent of load; `totalLoad==0` → `0/0=NaN` → `ceil(NaN)`→0; fetches `getCurrentCapacity()` (a live AWS call) and discards it. + +**L6 (High, live+autonomous). `preemptiveInstancePooling` is an unbounded capacity ratchet.** `:651-662` adds `preemptivePoolSize` every 300s forever with no scale-down and no demand check, until it hits `maxDesiredCapacity`. Pure cost burn. + +**L7 (High, live). Orphaned billable ASGs.** `CloudConfig.java:99-100` regenerates the ASG name with `UUID.randomUUID()` per process; `canDeleteCloudResources` requires the tag to equal the *current* name. A killed process's ASG (with running EC2 instances) can never be seen or deleted by any later run → indefinite billing. This is the standout cloud risk. + +**L8–L16 (Med/Low, live).** Metric-cache TTL mismatch causing uncached CloudWatch double-fetches (`:953-955`); resource/thread leak + accumulating shutdown hooks on failed cloud init (`:76-83`); live init holding the `LoadBalancer` write lock for up to 5 min (`LoadBalancer.java:102-120`); memory/disk metrics fabricated from CPU (`:410-416`); nested retry amplification (~9 attempts/metric); synchronous LASE evaluation on the request path; brittle gate evaluator hard-coded to `EXPECTED_TOTAL_DECISIONS=100`; a replay format with **no producer** (the export endpoint emits a different JSON shape than `--lase-replay` consumes). + +**Safe-by-default confirmed:** live AWS requires `liveMode=true` AND `allowLiveMutation=true` AND `operatorIntent=LOADBALANCERPRO_LIVE_MUTATION` AND account-ID allow-list AND region allow-list AND non-zero caps — all default off/empty. Credentials are static keys from props/env (no default-credential-chain), held as plain strings with public getters (mild exposure). Event logs and audit logs are bounded and synchronized (no unbounded per-request growth). Experiment/replay determinism holds (fixed clocks, seeded RNG). + +**Value verdict:** `AdaptiveTrafficDecisionOrchestrator` + `LoadDistributionPlanner.recommendTrafficShares` + `TrafficAllocation` guardrails are the **best code in this audit** — real evidence-windowing, score→share allocation with movement caps, feasibility-checked projection. It only ever sees fixed-clock fixtures; it could genuinely drive weighted allocation of live traffic. The decision functions (concurrency limiter, load-shedding policy, shadow autoscaler, failure classifier) are clean in isolation but none is actuated, and the failure/shedding/autoscaler triplicate the same threshold-pressure logic three times. + +--- + +## 3. Enterprise Lab apparatus (`lab/`, ~41.6k lines) + +**What it is.** A self-contained offline crash-recovery/evidence harness. "Experiments" arm an adaptive allocation decision over three fake backends, drive bounded HTTP GETs at literal loopback stubs (`127.0.0.1:1`, deliberately unroutable), evaluate/roll back, and record everything in hash-chained durable evidence. Moving parts: evidence ownership (OS `FileLock` + lease + inode/generation fencing, ~3.3k lines), an optional separate supervisor JVM over loopback TCP (~4k), per-side append-only hash-chained JSONL command ledgers, per-experiment journals with compaction/quarantine, an allocation transaction coordinator with restart reconciliation, and five "proof runners." Invoked via CLI early-exit flags and `/api/lab/**`; zero GUI usage. + +### Verified defects + +**C1 (Med-High). Cross-process torn reads are structurally guaranteed.** `EnterpriseLabApplicationCommandLedger.java:56` sets `MAX_WRITE_CHUNK_BYTES = 256` (verified) and deliberately splits each ~600–1000-byte event line into ≤256-byte `write()` calls, so a frame is visible half-written between chunks. The peer process reads the file **lock-free** (`SupervisorAllocationBridge.java:593-606`, `SupervisorService.java:985`) and `replayLocked` then throws `TRUNCATED_TAIL` — the *same* code used for genuine crash damage — which escalates to `failAllocationAdmission("SUPERVISOR_RECONCILIATION_FAILED")`. So allocation admission fails on a healthy system with a corruption-shaped reason. This is the exact "cross-process coordination" the recent Codex PRs claim to have solved, and it is not sound for concurrent reader/writer. No test exercises two live processes appending/reading one file. + +**C2 (Med). Supervisor lock is vulnerable to lock-file-deletion split-brain.** `EnterpriseLabSupervisorOwnership.java:90-98` checks only `lock.isValid()` and stats the path — it never compares the locked channel's inode to the file currently at the path. Delete `supervisor.lock` (e.g. a `target/` cleanup) → supervisor A's lock on the unlinked inode stays valid, supervisor B recreates and locks the new inode, both pass `requireHeld()`, both append → chain break → ledger permanently bricked with no repair path. The application side got the correct inode+creationTime identity check; the supervisor side didn't — the asymmetry itself is evidence of copy-paste divergence. + +**C3 (Med). Fail-stop at hard caps with no rotation and no recovery.** Ledgers and the allocation state store cap at 8MB/4096 events and then throw `EVENT_LIMIT_EXCEEDED` forever. Journals have compaction; **the ledgers and state store have none**. After ~500–1000 lifetime commands, every future experiment fails permanently; the only remedy is manually deleting files the system treats as tamper-evident. "Bounded by design" here means "designed to brick." + +**C5 (Med, ops). Restart within the lease window crash-loops the whole app.** `classifyPrior` (`:445-449`) refuses takeover on an unexpired lease even when the taker holds the exclusive OS lock (which on a single host proves the prior owner is dead); the controller then throws from bean creation and the app exits. `kill -9` + fast systemd restart → multi-restart outage of the entire API (not just the lab) for up to 30s. + +**C6 (Med-Low). Orphaned temp files brick listing endpoints.** A crash between write and `ATOMIC_MOVE` leaves a `.installing` temp; `compactedManifests()` then throws `VERIFICATION_FAILED` for *every* future call, and `replayedExperimentStates` throws on any non-VERIFIED journal — one stray file prevents the app from booting in allocation mode. No cleanup path exists. + +**C4/C7/C8 (Low).** Three full-file replays per append (O(n²) decode+re-encode+SHA per append); directory entries not fsync'd despite `FORCE_DATA_AND_METADATA` receipts (a durability lie the codebase spends thousands of lines claiming to preclude); static `PROCESS_MUTEXES` maps never evicted. + +**S1 (High, config-dependent). `/api/lab/**` unauthenticated in the default profile.** Same root cause as proxy-audit D9. Anyone on the network can arm/cancel experiments, `POST .../durable/retention {maximumTerminalJournals:0, dryRun:false}` (deletes all terminal journals), and hammer `GET .../durable` which re-verifies up to 256×16MB of journals per request — a cheap CPU/IO DoS. Prod profile fails closed correctly. + +**Path traversal: none** — experiment IDs are whitelisted then SHA-256-hashed into filenames, with `NOFOLLOW_LINKS` + `toRealPath` parent checks. This part is genuinely well done. Supervisor TCP is loopback-pinned with a 64-byte SecureRandom credential and constant-time compare — sound. + +**Engineering quality.** `SupervisorCommandLedger` is a ~90% verbatim copy of `ApplicationCommandLedger` (~1,100 duplicated lines); `AllocationStateStore` is a third copy of the same JSONL engine and `LocalJournal` a fourth variant — C2's inode-check asymmetry is the live cost of that copy-paste. The five proof runners are **7,250 lines of integration-test tooling shipped in the production jar** and reachable via production CLI flags. Of the 41.6k lines, ~18k is substance; ~23k is duplication (~3k), proof tooling in main (~7k), and layered evidence/receipt ceremony (~13k) whose failure mode is validating its own validators. Tests are bimodal: the `Enterprise*` family (real second-JVM forking, torn-tail injection, tamper rejection) is strong; the `LocalLab*` family (~96 files) is ~75–80% ceremony — 28 tests assert markdown files contain specific English sentences, 18 assert source strings don't contain certain words. + +--- + +## 4. DecisionExplorer / Replay / Evidence API (`api/`, ~24k lines) + +**What it is.** Every service here is a pure reshaper of one `RoutingComparisonResponse` (which runs `core/RoutingComparisonEngine` over client-supplied synthetic `ServerStateVector`s). For each result it chains ~20 derived "evidence" objects (snapshot → trace → capsule → readiness-checklist → source-map → field-inventory → null-safety-summary → … → closure-checklist), and DecisionExplorer adds a parallel confidence→diagnostics→tradeoff→shadow→counterfactual chain. Stateless per request (good) — except the shared strategy registry carries cross-request state (see E3). + +### Verified defects + +**E1 (Critical DoS). Uncapped candidate list × ~O(n²) payload amplification.** `RoutingComparisonService.java:158` validates only non-empty — no upper bound, no `@Size`/`@Max` (verified). A ~150-byte server entry means ~100 candidates fit in the 16KB body cap. Measured against `target/classes`: n=10 → 8.2MB payload; n=50 → 141MB; **n=100 → 533MB per strategy**, and with 5 default strategies that's multi-GB heap in one request. Amplification ≈ 16KB in → >500MB out (~32,000×). Unauthenticated in the default profile, rate-limiting off by default → trivial remote OOM. CI doesn't catch it (the perf test uses 3-server inputs). + +**E2 (High, wrong analytics). Explainability fuses two unrelated scoring models.** `RoutingComparisonService.java:687-739` always computes factor contributions with `core.ServerScoreCalculator` (the tail-latency model: P95 weight 0.45, P99 0.35…) *regardless of strategy*, then pairs them with `explanation.scores()` — which for `WEIGHTED_LEAST_LOAD` is a different formula, for WRR is *effective weights*, and for ROUND_ROBIN is empty. So for 4 of 5 strategies the reported factor contributions don't sum to the reported score, and "the factor that separated the candidates" was never part of the strategy's selection math. The core value proposition — "explain this routing decision" — is wrong for most strategies. + +**E3 (Med). Cross-request state → nondeterministic `/compare` + unstable fingerprints.** The singleton service holds stateful WRR `currentWeights` and RR `cursor` (and a seeded `Random` for tie-breaks). Two byte-identical `POST /compare` requests can return different `chosenServerId`, so the SHA-256 "deterministic fingerprint" differs for identical input — directly contradicting the reproducibility contract, and leaking prior-request routing history. + +**E4 (Low-Med). Fingerprint delimiter injection.** Builders join `candidateIds` with `,` / fields with `\n` after only trimming; a serverId like `a,b` collides with two ids `a`,`b`. Fingerprints are advisory, so low exploitability, but it breaks the uniqueness claim. + +**N1 — "Null-safety theater" (the audit's explicit question: confirmed).** `RoutingDecisionReplayEvidenceNullSafetySummaryService` (864 lines) exists solely to emit metadata about whether *other response objects the same request just built* have non-null fields — nullness that is statically knowable. It even asserts its own boundary strings contain the phrase "not production certification." Companions `FieldInventoryService` (1007 lines, 201 `field()` calls), `BoundarySummaryService`, `StatusRollupService`, and five `Lane*SummaryService`s / four `Reviewer*Service`s are the same metadata-about-metadata pattern. It's HTTP-reachable, computes nothing, and materially inflates the E1 payload. + +**Duplication & value.** Five overlapping "explain/compare a decision" stacks (§1). The Replay/ReplayEvidence chain (~11k lines) and the DecisionExplorer chain (~12k lines) are ~90% derivational restatement of the ~1.9k lines of *actual* factor data in `RoutingComparisonService`. Realistically **~18k–20k lines are deletable/consolidatable** into a ~2–3k-line explainability module; the only non-derivational pieces are dominant/delta factor analysis and counterfactual ±10% weight scenarios (<2k lines combined). Both would have real value as an **"explain this actual proxied request"** endpoint if fed the real `RoutingDecision` the proxy made (with each strategy's own score model, fixing E2) — the proxy makes real decisions today that are captured into none of this machinery. + +--- + +## 5. Operator surfaces (CLI / GUI / demo / cockpits / docs / build) + +**O0 (High, structural). The shipped jar cannot reach the flagship CLI.** `pom.xml:268` sets the main class to `LoadBalancerApiApplication`, whose `main` dispatches only EnterpriseLab/Lase commands and whose `shouldStartApi()` ignores `RemediationReportCli.isRequested`. `RemediationReportCli` is wired only into `cli/LoadBalancerCLI.main`, which nothing references. So `java -jar …jar --remediation-report --input … --output …` (the exact form used ~40× in `docs/REMEDIATION_REPORT_CLI.md`) silently boots a web server and writes nothing. All 31KB of that doc is non-runnable as written. + +**O1 (High). CLI "Scale Cloud" can scale down on an aborted prompt.** `LoadBalancerCLI.java:757` guards only `adjust == Integer.MIN_VALUE`, but `promptForInt` returns **-1** on abort/max-attempts, and -1 is a valid adjustment in range → three fat-fingered inputs execute `scaleServersAsync(current-1)`, an unrequested cloud scale-down (a real AWS mutation in live mode). + +**O2 (Critical for the feature, but the feature is dead). JavaFX GUI action buttons are runtime-broken.** All five commands run via `runAsync` on the common pool and then call `dialog.showAndWait()` off the FX thread → `IllegalStateException: Not on FX application thread`, surfaced as "Command failed." Add/Fail/Balance/InitCloud/ScaleCloud never work. Plus off-thread table mutation races, an unbounded alerts list, a no-op `GuiConfig.Builder` (16 setters that `return this` and ignore their value), and `javafx-controls` at **compile scope** so it's baked into the headless server jar. + +**O3 (High). Dead undo persistence + unfiltered deserialization.** `Command` isn't `Serializable`, so `UndoManager` truncates `undo_history.ser` then throws `NotSerializableException` (swallowed) every exit — history never survives restart. Worse, load path is `ObjectInputStream.readObject()` on a CWD-relative file with no `ObjectInputFilter` → a planted gadget-chain file in the launch directory is code execution at startup (mitigated only by the write path being dead). + +**O4 (Med). XSS in two cockpit pages.** `enterprise-lab.html` (5 `innerHTML` sinks — verified) interpolates server JSON (`scenario.displayName`, `event.rollbackReason`, `policy.warning`, guardrail reasons) into template literals without escaping; `enterprise-lab-reviewer.html:608` does the same with reviewer-summary path fields. The other 11 pages correctly use `textContent`/`escapeHtml` — the two vulnerable ones are precisely those missing the helper. + +**O5 (Med/Low). Overwrite & tooling inconsistencies.** `RemediationReportCli` outputs (`--output`, `--manifest`, redaction sidecars, bundle zips) clobber without a `--force` prompt, while `EvidencePolicyExampleService` correctly checks-and-forces — inconsistent in a tool selling evidence integrity. Jar-selection drifts three ways across `operator-distribution-smoke.sh` (lexical sort → picks 2.5.0 over 2.10.0), the `.ps1` (mtime), and Docker/CI (`ls -t`); `local-artifact-verify` hardcodes `2.5.0`. CLI idle-timeout is inverted (only checked between prompts, but prompts block on `nextLine()`), menu items 11/12 are unimplemented stubs, and a `checkMonitorStatus` sleeps up to ~31s on the input thread. + +**Well-built (keep as-is):** `demo/ProxyDemoFixtureLauncher` (loopback-enforced, clean lifecycle — the best operator surface in the repo), the Dockerfile (digest-pinned, non-root, healthcheck), all bash scripts (`set -euo pipefail`, loopback-only), the Postman collection, and the zip-slip guard in bundle verification. + +**Docs.** ~75–80% agent-generated ceremony: ~113 files matching PLAN/CHECKLIST/ROLLUP/LANE/CLOSURE patterns (a 260KB `REVIEWER_TRUST_MAP.md`, a 97KB `DECISION_VECTOR.md`), plus stale evidence (`RELEASE_ARTIFACT_EVIDENCE.md` still at v1.9.0, `PERFORMANCE_BASELINE.md` with no measured numbers). ~30 files are genuinely operator-useful (OPERATIONS_GUIDE, RUNBOOK, DEPLOYMENT_HARDENING, CONTAINER_*, API_CONTRACTS). + +--- + +## 6. Cross-cutting themes + +1. **Fabricated inputs everywhere.** LASE synthesizes latency from load score (L1); the simulation `ServerMonitor` random-walks metrics (proxy audit); cloud memory/disk are `cpu*1.2`/`cpu*0.8` (L15); explorer runs on client-supplied synthetic candidates (E-cluster); lab experiments hit `127.0.0.1:1`. **Nothing in ~96% of the codebase observes a real request.** The highest-value move across all four areas is the same: capture the real `RoutingDecision` + per-upstream latency/error the proxy already produces, and feed *that* into the shadow evaluator, the explainability layer, and the traffic-share orchestrator. +2. **Five parallel compare/explain stacks** (§1) — consolidate to one. +3. **Ceremony that validates itself** — null-safety-summary services, evidence-of-evidence proof runners, tests that assert prose. ~30k+ lines deletable across lab + explorer with no capability loss. +4. **Default-open security** — one `permitAll` + one `@Profile`-gated filter leaves destructive and DoS-amplifying endpoints unauthenticated by default. One fix (proxy-audit D9 / lab S1) closes it everywhere. +5. **Real durability engineering worth keeping** — the ownership lease/fencing, the journal store, and the transaction coordinator are sound designs; they're just wrapped in duplication and pointed at a simulator. + +--- + +## 7. Consolidated verdict table + +| Subsystem | Lines (approx) | Verdict | One-line rationale | +|---|---|---|---| +| `AdaptiveTrafficDecisionOrchestrator` + `recommendTrafficShares` + TrafficAllocation guardrails | ~3k | **KEEP & WIRE LIVE** | Best code in the audit; could drive real weighted allocation | +| LASE shadow advisor + event log + endpoint | ~4k | **KEEP shell, FIX inputs (L1–L4)** | Production-grade plumbing evaluating fabricated telemetry | +| AIMD limiter / shedding / autoscaler / failure classifier | ~2k | **CONSOLIDATE (3 dup pressure classifiers) + ACTUATE** | Clean functions, none wired to anything | +| Cloud* (AWS) | ~3k | **KEEP guardrails, FIX L5–L12 before any live use** | Good gating, broken math + orphan-ASG cost risk | +| Evidence ownership (lease/fence/paths) | ~3.3k | **KEEP (fix C5)** | Genuinely sound single-host design | +| Command ledgers ×2 + AllocationStateStore + LocalJournal | ~5k → ~1.5k | **CONSOLIDATE to one JSONL engine, then fix C1/C3** | Four copies of one store | +| Journal directory / replay / verifier | ~3k | **KEEP (fix C6)** | Core durable evidence; ID-hashing is correct | +| Allocation transaction coordinator + reconcilers | ~4k | **KEEP** | The actual crash-window state machine; well tested | +| Supervisor server/service/client/protocol | ~4k | **KEEP if external-supervisor is a requirement, else QUARANTINE** | Sound, but exists to prove process separation of a simulator | +| 5 proof runners + reports + exporters | ~7.2k | **MOVE to src/test or a tool module** | Integration tests shipped in the prod jar | +| DecisionExplorer + Replay + ReplayEvidence chains | ~23k → ~2-3k | **CONSOLIDATE to one explainability module** | ~90% derivational restatement | +| Null-safety/field-inventory/lane/reviewer "evidence" services | ~6k | **DELETE** | Metadata about metadata; computes nothing | +| `RoutingComparisonEngine` + dominant/delta factor analysis | ~1k | **KEEP & WIRE LIVE** | The one real explainability capability | +| Interactive `LoadBalancerCLI` + UndoManager | ~2k | **QUARANTINE/DELETE** | Synthetic state, dead undo, deser risk, dangerous sentinel | +| Evidence/report CLI tooling (`RemediationReportCli` +svcs) | ~4k | **KEEP (fix O0 wiring, O5 overwrite)** | Real file work, reuses api service | +| JavaFX GUI | ~2k | **DELETE (or fix O2/O3 if UI is wanted)** | All action buttons runtime-broken; drags CVE surface into server jar | +| `ProxyDemoFixtureLauncher`, Dockerfile, scripts, Postman | — | **KEEP** | Best-built surfaces | +| docs/ ceremony (~113 files) | — | **ARCHIVE ~80%** | Agent-generated plans/rollups; keep ~30 operator docs | + +Top-priority fixes across this whole surface, in order: **S1/D9 default-deny → E1 explorer OOM cap → O1 CLI scale-down sentinel → O4 cockpit XSS → C1 ledger torn reads → L7 orphan-ASG guard → O0 CLI wiring → C2/C3/C5/C6 lab durability.** diff --git a/docs/BUILD_PLAN_DEPLOYABLE.md b/docs/BUILD_PLAN_DEPLOYABLE.md new file mode 100644 index 00000000..b8eb7f9c --- /dev/null +++ b/docs/BUILD_PLAN_DEPLOYABLE.md @@ -0,0 +1,158 @@ +# LoadBalancerPro — Build Plan: Path to Deployable + +**Companion to:** `AUDIT_2026-07-21.md` (defect IDs D1–D16 and phase items referenced below come from that document) +**Goal:** take the reverse proxy from "defensively-written demo, off by default" to a load balancer you can put in front of real traffic. +**Shape:** 5 milestones, ~18 PRs, each PR independently reviewable and shippable. Written to be executed by Codex PR-by-PR; every PR lists scope, touched files, config surface, and acceptance criteria. + +Existing config baseline (verified in `application-proxy-demo-*.properties` / `ReverseProxyProperties`): `loadbalancerpro.proxy.{enabled,strategy,request-timeout,max-request-bytes}`, nested `health-check.*`, `retry.*`, `cooldown.*`, `routes.*`, `upstreams[n].*`. All new keys below extend this namespace — no breaking renames in Milestones 1–3. + +--- + +## Milestone 0 — Stop the bleeding (bug fixes, no new features) + +Small surgical PRs. All are pure fixes to verified defects; land before feature work so later PRs build on sound ground. + +### PR-0.1 Remove per-request shutdown-hook leak (D2) +- **Change:** `ServerMonitor` constructor (`core/ServerMonitor.java:92`) must not register a JVM shutdown hook. Move hook registration into `start()` (register once, keep the `Thread` reference) and deregister in `stop()` via `removeShutdownHook`, guarding `IllegalStateException` during actual shutdown. `AllocatorService`-created balancers never call `start()`, so they get no hook at all. +- **Files:** `core/ServerMonitor.java`, test in `core/ServerMonitorTest`. +- **Accept:** loop 10k `POST /api/allocate/capacity-aware` in a test → `ApplicationShutdownHooks` size stable (assert via reflection or a counter hook); heap stable under `-Xmx64m`. + +### PR-0.2 Health eviction → drain + re-admission (D3) +- **Change:** `ServerHealthCoordinator.detectFailedServers` stops deleting servers. Introduce `ServerDegradationState` transitions (the enum already exists): HEALTHY → DEGRADED (threshold breach) → EVICTED only after N consecutive bad cycles; recovered servers (M consecutive good cycles) return to rotation. Manual `setHealthy(false)` = DRAINING, never eviction. Registry removal only via explicit admin call. +- **Files:** `core/ServerHealthCoordinator.java`, `core/LoadBalancer.java:146-163`, `core/Server.java`. +- **Accept:** unit test — one bad sample does not remove; N bad samples evicts; recovery re-admits; drained server is skipped by allocation but present in registry. + +### PR-0.3 Per-route strategy instances (D4) +- **Change:** `RoutingStrategyRegistry` returns a **factory** per `RoutingStrategyId`; `ReverseProxyRoutePlanner` instantiates one strategy per route at config build/reload time and stores it on the route object. Stateful strategies (WRR cursor/accumulators, RR cursor) are therefore route-scoped. On reload, carry the old instance over when the route's strategy id and upstream set are unchanged (preserves smooth-WRR state). +- **Files:** `core/RoutingStrategyRegistry.java`, `api/proxy/ReverseProxyRoutePlanner.java`, `api/proxy/ReverseProxyService.java`. +- **Accept:** integration test with two WRR routes (weights 3:1) interleaved → each route's observed split within 5% of 3:1 over 1k requests. + +### PR-0.4 Weight-0 = drain (D13) + retry classification fix (interrupted ≠ retriable) +- **Change:** `effectiveWeight`: weight 0 → excluded from candidates (equivalent to `healthy=false` for selection) in WRR/WLC; validation rejects negative weights, allows 0 with documented drain semantics. `forwardOnce` marks `InterruptedException` non-retriable. +- **Files:** `core/WeightedRoundRobinRoutingStrategy.java:100-105`, `core/WeightedLeastConnectionsRoutingStrategy.java:80-85`, `api/proxy/ReverseProxyService.java:405-413`. +- **Accept:** weight-0 upstream receives zero requests; weight 0.01 gets ~1% vs weight 1.0 peer. + +### PR-0.5 Auth fail-closed in api-key mode (D9) + actuator lockdown (D15) +- **Change:** `ProdApiKeyFilter` drops `@Profile` gating → active whenever auth-mode=api-key; **startup fails** if api-key mode with empty key unless `loadbalancerpro.api.auth-mode=none` is explicitly set (new explicit "I know it's open" mode for local dev). Default `application.properties` exposure narrowed to `health,info`; `metrics,prometheus` moved behind auth (see PR-3.2). +- **Files:** `api/config/ProdApiKeyFilter.java`, `api/config/ApiSecurityConfiguration.java`, `application.properties`. +- **Accept:** default profile + no key → app refuses to start with clear message; auth-mode=none logs a prominent warning; prod behavior unchanged. + +### PR-0.6 Simulation-core correctness batch (D10, D11, D12 + P3 items) +- Load-shedding priority ordering fixed (CRITICAL never shed before USER at same pressure); redistribution merge bug (`ServerHealthCoordinator.java:74-85`) — use `leastLoaded()`'s merged result, drop `putAllAllocations` overwrite; `consistentHashing` takes the server read-lock and handles empty-ring between checks; `Math.floorMod` in `Server.updateHistory`; validate-before-snapshot in `updateMetrics`; synchronize single-metric setters properly; delete dead `loadQueue`; remove `isCloudServer` global property override. +- **Accept:** existing test suite green + new unit tests per fix. + +**Milestone 0 exit:** all P0/P1 defects in the audit closed except D1 (telemetry — Milestone 1) and D5 (streaming — Milestone 2). + +--- + +## Milestone 1 — An honest adaptive proxy (Phase A) + +### PR-1.1 Timeout correctness (D7) +- **Config:** `proxy.connect-timeout` (default `1s`) applied via `HttpClient.newBuilder().connectTimeout(...)`; per-route `routes..request-timeout` overriding the global; existing `health-check.timeout` unchanged. +- **Files:** `api/proxy/ReverseProxyConfiguration.java`, `ReverseProxyProperties.Route`, `ReverseProxyService`. +- **Accept:** upstream that accepts-then-blackholes → request fails in ≈connect/request timeout, not minutes; per-route override observed. + +### PR-1.2 Upstream runtime stats (foundation for D1) +- **New class:** `api/proxy/UpstreamRuntimeStats` — per-upstream: `LongAdder inFlight` (incremented before `httpClient.send`, decremented in `finally`), rolling latency (fixed 256-slot ring of recent millis + EWMA; compute p50/p95/p99 on snapshot), error-rate window (sliding 30s success/failure counts), last-updated timestamp. Held in `ConcurrentHashMap` in `ReverseProxyService`; survives reload for unchanged upstream ids (fixes half of D14). +- **Accept:** concurrent load test → inFlight returns to 0 after quiesce (no drift, including on exceptions); stats visible in status endpoint. + +### PR-1.3 Live telemetry → routing (closes D1) +- **Change:** `toCandidate()` builds `ServerStateVector` from `UpstreamRuntimeStats` (in-flight, avg/p95/p99 latency, error rate, queueDepth = inFlight) merged with config (weight, admin healthy). Config telemetry fields on `Upstream` become **seed/fallback values** (deprecation note in javadoc, kept for compatibility one release). +- **Accept:** integration test with one artificially slow upstream under WEIGHTED_LEAST_CONNECTIONS → slow upstream's share drops materially (assert < 30% of requests); same test under TAIL_LATENCY_POWER_OF_TWO shows tail improvement vs ROUND_ROBIN. + +### PR-1.4 Background health checking (D6) +- **Change:** dedicated `ScheduledExecutorService` (daemon, name-prefixed threads), one jittered task per upstream per interval; **rise/fall thresholds**: `health-check.healthy-threshold` (default 2), `health-check.unhealthy-threshold` (default 3). Request path reads a volatile `HealthSnapshot`; `statusSnapshot` becomes read-only (no probes, no cooldown mutation). Cooldown failure-counting driven only by real forwarding failures + prober results (single-count). +- **Files:** new `api/proxy/UpstreamHealthProber.java`; `ReverseProxyService` sheds `probeDue`/inline probing. +- **Accept:** zero probe I/O on request threads (assert via instrumentation); flapping upstream (alternating probe results) does not flap state; status GET has no side effects. + +### PR-1.5 Forwarding headers (D8) +- **Config:** `proxy.forwarded.mode=strip-and-set|append|off` (default `strip-and-set`), `proxy.forwarded.trusted-proxies=` (inbound values honored only from these), `routes..headers.{add,set,remove}` map for static rewrite rules. +- **Change:** inject `X-Forwarded-For` (append-safe), `X-Forwarded-Proto`, `X-Forwarded-Host`, RFC 7239 `Forwarded`; strip inbound spoofables unless from trusted CIDR. +- **Accept:** backend fixture asserts correct XFF chain for direct and chained-proxy cases; spoofed inbound XFF from untrusted source not forwarded. + +### PR-1.6 Live load shedding + concurrency limits (wires dormant core code) +- **Config:** `proxy.limits.max-in-flight` (global), `upstreams[n].max-in-flight`, `proxy.shedding.enabled` + pressure thresholds mapping to existing `LoadSheddingConfig`. +- **Change:** `forward()` consults limits using PR-1.2 counters → 503 + `Retry-After` when exceeded; `AdaptiveConcurrencyLimiter` optional mode (`proxy.limits.adaptive=true`) adjusting the global cap from latency feedback. +- **Accept:** saturate a 2-upstream setup beyond cap → excess gets fast 503, upstreams never see > cap concurrent; CRITICAL priority requests (header-mapped) shed last. + +### PR-1.7 Consistent-hash strategy + cookie affinity +- **Config:** `routes..strategy=CONSISTENT_HASH`, `routes..hash-on=client-ip|header:`; `routes..affinity.cookie-name` (cookie mode independent of strategy). +- **Change:** register `ConsistentHashRingStrategy` implementing `RoutingStrategy` over the existing ring (port fixes ring locking per PR-0.6); affinity cookie (HMAC of upstream id, key from config) checked before strategy, fallback to strategy when target unhealthy. +- **Accept:** same key always → same healthy upstream; upstream removal remaps only ~1/N keys; cookie pin survives across requests and fails over cleanly. + +### PR-1.8 Retry budget + backoff, slow-start +- **Config:** `retry.budget-percent` (default 20), `retry.backoff={base,max}` (exponential, jittered); `proxy.slow-start.duration` (default 0=off) — linear weight ramp for upstreams newly added or exiting cooldown; cooldown expiry no longer resets failure memory to zero (keep half). +- **Accept:** brownout test — retries capped at budget; recovering upstream's traffic ramps rather than steps. + +**Milestone 1 exit:** all five strategies honest on live traffic; health, timeouts, shedding, affinity real. This is the "credible single-node L7 LB (buffered)" checkpoint. + +---## Milestone 2 — Deployable data path + +### PR-2.1 Streaming request path (D5, part 1) +- **Change:** `ReverseProxyController` stops taking `@RequestBody byte[]`; reads `HttpServletRequest.getInputStream()` and forwards via `BodyPublishers.ofInputStream`. Pre-check `Content-Length` against `max-request-bytes` before reading; for chunked inbound, enforce cap with a counting bounded stream (abort with 413 mid-stream). Remove the redundant `clone()`. +- **Accept:** 1GB request rejected instantly on Content-Length; chunked over-limit aborted at the cap with bounded memory (`-Xmx128m` test); normal POSTs byte-identical at backend. + +### PR-2.2 Streaming response path (D5, part 2) +- **Change:** `BodyHandlers.ofInputStream()` → stream to `HttpServletResponse` output with a fixed copy buffer; new `proxy.max-response-bytes` (default 0=unlimited, streamed so memory-safe either way); flush strategy compatible with SSE (`text/event-stream` passes through incrementally — do not buffer-and-forward). +- **Note:** retry semantics change — a response can only be retried before first byte is written to the client; encode that rule explicitly in `forward()`. +- **Accept:** 2GB response proxied under `-Xmx128m`; SSE fixture streams events with < 100ms added latency per event; retries still work for connect-phase failures. + +### PR-2.3 TLS termination + SNI + hot cert reload +- **Config:** standard Spring `server.ssl.bundle` (SSL bundles give file-watch reload); document keystore/PEM setup in `docs/DEPLOYMENT.md`; optional second connector for cleartext health traffic if needed. +- **Backend TLS:** `proxy.backend-tls.truststore` (custom CA), `upstreams[n].tls.{verify (default true), client-cert}` for mTLS to backends. Never expose a "verify=false" without a loud startup warning. +- **Accept:** TLS termination e2e test with self-signed bundle; cert file swap picked up without restart; backend mTLS fixture handshake verified. + +### PR-2.4 Graceful shutdown + draining reload (D14 complete) +- **Change:** `server.shutdown=graceful` + `spring.lifecycle.timeout-per-shutdown-phase` (default 30s) in all profiles; prober/executors implement `SmartLifecycle` stop; reload diff engine — unchanged upstreams keep runtime stats/health/cooldown, removed upstreams enter DRAINING (no new picks, config retained until in-flight drains or timeout), added upstreams start in slow-start. +- **Accept:** SIGTERM under load → zero failed in-flight requests, exit within window; reload that drops an upstream mid-load → no 5xx from that transition. + +### PR-2.5 Deployment packaging: proxy on by default in a real prod story +- **Change:** new `application-proxy-prod.properties` profile: proxy enabled, api-key auth enforced, actuator behind auth, health-check + cooldown + limits enabled with sane defaults; Dockerfile gains `HEALTHCHECK` against LB health (not just API health), documented env-var config surface (`LBP_UPSTREAM_0_URL` style via Spring relaxed binding); `docs/DEPLOYMENT.md` with docker-compose example (LB + 2 backends), K8s manifest sketch (readiness = `/api/health`, preStop drain sleep), and config reference table generated from `ReverseProxyProperties`. +- **Accept:** `docker compose up` from the example proxies traffic with TLS, auth, health checks, metrics — no code edits required. + +**Milestone 2 exit:** streaming, TLS, graceful lifecycle, and a documented turnkey deployment. This is the "you can actually put it in front of something" checkpoint. + +--- + +## Milestone 3 — Operable + +### PR-3.1 Micrometer metrics +- Port `ReverseProxyMetrics` to `MeterRegistry`: `lbp.proxy.requests` (counter; tags: route, upstream, status_class, outcome), `lbp.proxy.latency` (timer w/ histogram buckets; route, upstream), `lbp.proxy.inflight` (gauge per upstream), `lbp.proxy.retries`, `lbp.proxy.sheds`, `lbp.proxy.health` (gauge 0/1), `lbp.proxy.cooldown.trips`. Keep the JSON status endpoint reading from the same source. +- **Accept:** `/actuator/prometheus` (behind auth, enabled in proxy-prod profile) exposes all series; Grafana-ready; cardinality bounded by config (no per-client tags). + +### PR-3.2 Access log +- Structured per-request line (JSON or combined-log configurable): timestamp, client, method, path, route, upstream, status, bytes in/out, duration, retries, shed/cooldown flags. Async appender; `proxy.access-log.{enabled,format,path}`. Sampling knob for high QPS. +- **Accept:** every proxied request (success and failure) produces exactly one line; overhead < 5% at saturation benchmark. + +### PR-3.3 Admin API v1 (incremental config) +- `POST /api/proxy/upstreams` (add), `DELETE /api/proxy/upstreams/{id}` (→ DRAINING then remove), `PATCH /api/proxy/upstreams/{id}` (weight/healthy/drain), `GET /api/proxy/config` (redacted effective config + generation). Same auth as reload; every mutation audit-logged with generation bump. Full-config reload remains for bulk changes. +- **Accept:** add/drain/remove cycle under load with zero dropped requests; concurrent mutations serialized (or 409 on generation conflict). + +### PR-3.4 Benchmark + soak harness (CI-gated) +- `scripts/bench/` — wrk/vegeta scenario set (steady, spike, slow-backend, backend-kill, reload-under-load, drain-under-load) against the compose stack; nightly soak (1h) asserting: no heap growth trend, inFlight returns to zero, p99 within budget, zero 5xx during drain/reload scenarios. This is the regression net that keeps Milestones 0–2 fixed. + +--- + +## Milestone 4 — Growth (post-deployable, design-doc first) + +- **Host/header-based routing rules** and percentage canary splitting (generalize route matching; precedence: host > path-prefix length; `routes..match.{host,header.}`; `split` groups with percentage weights). +- **DNS service discovery:** `upstreams[n].discovery=dns::`, periodic re-resolution with per-IP health, respecting TTL floor. +- **HTTP/2 inbound + WebSocket passthrough:** decision point — Tomcat h2 + servlet upgrade handling vs migrating the proxy layer to Netty/reactive. Write `docs/adr/ADR-streaming-stack.md` first (the Milestone 2 servlet-streaming work is compatible with either, but WebSocket forces the choice). +- **Simulation core quarantine:** move `lab/`, DecisionExplorer/Replay/Evidence services, GUI, demo CLIs behind a Maven profile/module (`-P lab`) so the deployable artifact is the ~15% that's real. Retire `ServerMonitor`'s random-walk or fence it behind `demo` profile. This roughly halves the audit surface and jar size, and stops sim code (with its own bug tail) from shipping in prod builds. + +--- + +## Sequencing & dependency graph + +``` +M0: 0.1 0.2 0.3 0.4 0.5 0.6 (parallel-safe, independent) +M1: 1.1 → 1.2 → 1.3 → 1.6 (1.2 is the spine: stats feed 1.3, 1.6, 3.1) + 1.4 (independent after 0.x) + 1.5 (independent) + 1.7 (after 0.3, 0.6) + 1.8 (after 1.2) +M2: 2.1 → 2.2 → 2.3 → 2.5 (2.4 after 1.4; 2.5 last, integrates all) +M3: 3.1 (after 1.2), 3.2, 3.3 (after 2.4), 3.4 (after 2.5) +``` + +Suggested PR cadence for Codex: land M0 as a batch of 6 small PRs first (each < ~300 lines diff), then M1 in the arrow order above. Every PR: unit tests + one integration test against the `ProxyDemoFixtureLauncher` backends (extend the fixture with `slow`, `blackhole`, `flaky`, and `sse` modes in PR-1.1 — several acceptance tests above need them). Definition of done for "deployable" = Milestone 2 exit + PR-3.4 soak green. diff --git a/docs/BUILD_PLAN_LAB_SHADOW.md b/docs/BUILD_PLAN_LAB_SHADOW.md new file mode 100644 index 00000000..de08ef23 --- /dev/null +++ b/docs/BUILD_PLAN_LAB_SHADOW.md @@ -0,0 +1,147 @@ +# LoadBalancerPro — Build Plan: Lab, Shadow & Analysis Subsystems + +**Companion to:** `AUDIT_LAB_SHADOW_2026-07-21.md` (defect IDs L#/C#/S#/E#/O# below come from that document) and `BUILD_PLAN_DEPLOYABLE.md` (the proxy roadmap; its milestones are referenced where they interlock). + +**Framing.** The proxy build plan makes the load balancer *real*. This plan decides the fate of the other ~96% of the codebase. It has three distinct goals, and every PR serves exactly one: + +- **SECURE & STABILIZE** — close the defects that are dangerous *today* (unauthenticated destructive endpoints, a remote-OOM endpoint, a CLI that can scale down AWS by accident, cockpit XSS, ledger torn reads, orphan-ASG cost risk). +- **CONSOLIDATE & QUARANTINE** — collapse the five duplicate compare/explain stacks and the four duplicate JSONL stores into one each, and move test tooling out of the production jar. Target: delete ~30k lines with zero capability loss. +- **WIRE TO LIVE TRAFFIC** — the payoff. Feed the real `RoutingDecision` + per-upstream telemetry the proxy produces (once `BUILD_PLAN_DEPLOYABLE.md` Milestone 1 lands) into the survivors: shadow evaluation, explainability, and the traffic-share orchestrator. This is what turns dormant machinery into product. + +Sequencing rule: **Milestone L0 (security) can and should land immediately**, in parallel with the proxy's Milestone 0. Consolidation (L1–L2) should precede wiring (L3) so we wire *one* clean component, not five. Live-wiring (L3) depends on proxy Milestone 1 (live telemetry, PR-1.2/1.3) existing. + +--- + +## Milestone L0 — Secure & stabilize (land now, parallel to proxy M0) + +### PR-L0.1 Default-deny security posture (closes S1; shared with proxy D9) +- **Change:** this is the *same* fix as proxy `PR-0.5` — do it once and both plans depend on it. `ProdApiKeyFilter` no longer `@Profile`-gated; active whenever auth-mode=api-key; app fails startup in api-key mode with no key unless `auth-mode=none` is explicit. Add explicit matchers so `/api/enterprise-lab/**`, `/api/evidence-training/**`, `/api/remediation`, `/api/scenarios/replay` require `allocationRole` (they currently fall through to `authenticated()`), and destructive lab endpoints (`/durable/retention`, `/durable/*/compact`) require an admin role. +- **Files:** `api/config/ApiSecurityConfiguration.java`, `api/config/ProdApiKeyFilter.java`, `application.properties`. +- **Accept:** default profile + no key → refuses to start; `POST /api/lab/.../durable/retention` returns 401/403 without credentials; role matrix test covers every `/api/**` prefix. + +### PR-L0.2 Cap explorer input size (closes E1 — critical remote OOM) +- **Change:** `@Size(max=…)` on the server/candidate list in `RoutingServerStateInput` (default cap e.g. 32; configurable `loadbalancerpro.api.max-candidates`), and a hard cap on strategies-per-request. Reject over-cap with 400 *before* building any payload. Add a response-size guard on the decision-explorer path. +- **Files:** `api/RoutingController.java`, `api/RoutingComparisonService.java` (`:158`), the input DTO. +- **Accept:** the audit's PoC (n=100, 5 strategies) returns 400, not multi-GB heap; new CI test asserts the cap at the large-N boundary (the current perf test uses n=3 and misses this). + +### PR-L0.3 CLI scale-down sentinel + prompt-abort safety (closes O1) +- **Change:** `promptForInt` returns a dedicated sentinel (or `OptionalInt.empty()`) on abort, distinct from any valid value; `LoadBalancerCLI.java:757` treats abort as "cancel, no action." Audit every caller of `promptForInt` for the same -1-in-range trap. +- **Accept:** aborting the Scale Cloud prompt performs zero mutation; unit test for abort → no `scaleServersAsync` call. + +### PR-L0.4 Cockpit XSS (closes O4) +- **Change:** add the shared `escapeHtml` helper (already present in two other pages) to `enterprise-lab.html` and `enterprise-lab-reviewer.html`; escape every server-derived value at the 5 + 1 `innerHTML` sinks, or switch to `textContent`/`createElement`. +- **Accept:** a guardrail-reason / event-id containing `` renders inert; manual check of both pages against a malicious fixture. + +### PR-L0.5 Orphan-ASG guard (closes L7 — cloud cost risk) +- **Change:** ASG name must be **deterministic and recoverable** — derive from `resourceNamePrefix + environment` (stable across restarts), or persist the generated name to the trusted evidence dir and reload it on startup. `canDeleteCloudResources` matches on the stable name + ownership tag. Add a startup reconciliation that lists ASGs by ownership tag and adopts/cleans prior-run groups. +- **Files:** `core/CloudConfig.java:99-100`, `core/CloudManager.java` (describe/delete/ownership). +- **Accept:** restart re-discovers the prior run's ASG; simulated killed-process ASG is adoptable, not orphaned. (Test with mocked AWS clients — `liveMode=false` path.) + +### PR-L0.6 Ledger torn-read fix (closes C1 — the "cross-process coordination" bug) +- **Change:** two parts. (1) Make each event frame a **single atomic write** (remove the 256-byte chunk splitting in `EnterpriseLabApplicationCommandLedger.java:56-772` and the supervisor twin) so a reader never sees a half-frame; if a single `write()` can still short-write, hold a shared file-region read lock for cross-process reads. (2) In `replayLocked`, distinguish a *transient* tail (last line incomplete, no newline) from genuine corruption — retry-on-transient with a short backoff before classifying `TRUNCATED_TAIL`, and never escalate a transient to `SUPERVISOR_RECONCILIATION_FAILED`. +- **Files:** `lab/EnterpriseLabApplicationCommandLedger.java`, `lab/EnterpriseLabSupervisorCommandLedger.java`, `lab/EnterpriseLabSupervisorAllocationBridge.java`, `lab/EnterpriseLabSupervisorService.java`. +- **Accept:** new test — two live JVMs, one appending continuously while the other replays in a tight loop for N seconds → zero false `TRUNCATED_TAIL`/`CONCURRENT_CHANGE`; genuine mid-frame corruption still detected. + +**L0 exit:** nothing in the lab/shadow/analysis surface is remotely dangerous in the default profile, the cloud integration can't orphan billable resources, and the ledger no longer fails healthy systems. + +--- + +## Milestone L1 — Consolidate the analysis layer (delete ~20k lines) + +Do this *before* live-wiring so we wire one clean explainability module, not five stacks. + +### PR-L1.1 Delete the metadata-about-metadata services (N1) +- **Change:** remove `RoutingDecisionReplayEvidenceNullSafetySummaryService`, `FieldInventoryService`, `BoundarySummaryService`, `StatusRollupService`, the five `Lane*SummaryService`s, and the four `Reviewer*Service`s, plus their DTOs and the ~30 `*DocumentationTest`s that assert prose. Strip their fields from the response objects. +- **Accept:** build green; the DecisionExplorer payload shrinks by an order of magnitude; no remaining service references the deleted ones. (~6k lines out.) + +### PR-L1.2 Collapse Replay + DecisionExplorer into one explainability module +- **Change:** define one `RoutingExplanation` result carrying the *non-derivational* content only: per-candidate factor contributions (dominant + delta analysis) and counterfactual ±weight scenarios. Delete the ~20-object derivational chain (snapshot→trace→capsule→…→closure) and the parallel confidence/diagnostics/tradeoff/shadow restatements — keep a single confidence score and a single tradeoff summary if they carry real signal. `/api/routing/decision-explorer` returns the new compact shape. +- **Files:** new `api/explain/RoutingExplanationService.java`; retire `DecisionExplorer*` (80 files) and `RoutingDecisionReplay*` (54 files) down to the survivors. +- **Accept:** the counterfactual weight-scenario test and dominant/delta factor tests still pass against the new module; a golden-payload test documents the new (much smaller) contract. (~12–14k lines out.) + +### PR-L1.3 Fix explainability correctness (E2) + determinism (E3, E4) +- **Change:** compute factor contributions using **each strategy's own score model**, not always `ServerScoreCalculator` — the explanation must reflect what the strategy actually optimized (weights for WRR, load formula for least-load, empty/positional for RR). Make `/compare` deterministic: either use per-request fresh strategy instances (aligns with proxy PR-0.3) or exclude cross-request cursor state from the fingerprint; seed tie-break RNG deterministically from the request. Fix fingerprint delimiter injection (length-prefix or hash-per-field instead of `join`). +- **Accept:** for a WRR decision the reported factors reconcile with the strategy's selection; two identical `/compare` requests return identical `chosenServerId` and fingerprint; `a,b` vs `a`+`b` no longer collide. + +### PR-L1.4 Collapse the duplicate compare/experiment surfaces +- **Change:** route `RoutingComparisonService`, `AdaptiveRoutingExperimentService`, and `AdaptiveRoutingScenarioRunner` through the single `core/RoutingComparisonEngine`. Keep one experiment/fixture entry point; delete `AdaptiveRoutingStrategyComparisonMatrixBuilder` and the self-referential `GateEvaluator` (L14) or reduce it to a real assertion. Merge the three pressure classifiers (`FailureScenarioRunner`, `ShadowAutoscaler`, `LoadSheddingPolicy` predicates) into one. +- **Accept:** one comparison code path; experiment/scenario endpoints produce equivalent output via the shared engine; pressure-classification tests consolidated. + +**L1 exit:** one explainability module, one comparison engine, one pressure classifier — correct and deterministic. Estimated ~20k lines removed. + +--- + +## Milestone L2 — Consolidate & quarantine the lab (`lab/`) + +### PR-L2.1 One JSONL store engine (fixes C2 asymmetry structurally) +- **Change:** extract a single `ChainedJsonlStore` (bounded, hash-chained, OS-locked, inode+creationTime identity, atomic single-frame append from PR-L0.6) and re-express `ApplicationCommandLedger`, `SupervisorCommandLedger`, `AllocationStateStore`, and `LocalJournal` on top of it. The supervisor lock automatically inherits the correct inode identity check (C2 fixed by construction). +- **Accept:** all four stores' existing tests pass against the shared engine; a lock-file-deletion split-brain test (delete lock file, second locker) is refused. (~2.5–3k lines out.) + +### PR-L2.2 Rotation & recovery for ledgers/state store (C3, C6) +- **Change:** add compaction/rotation to the shared engine (ledgers and state store, not just journals) — archive-and-truncate at a soft threshold, keeping the hash chain across a rotation boundary. Add `.installing`/temp-file cleanup on startup (C6) so an orphaned temp doesn't brick listing endpoints. Provide an operator repair command for a bricked chain. +- **Accept:** 10k-command soak never hits `EVENT_LIMIT_EXCEEDED`; an injected orphan `.installing` file is cleaned on boot and endpoints stay healthy. + +### PR-L2.3 Takeover honors the OS lock (C5) +- **Change:** at takeover, if the taker holds the exclusive OS `FileLock`, treat an unexpired lease as stale (the lock proves the prior single-host owner is dead) instead of refusing and crashing the app. Keep lease semantics for the genuinely-multi-host case behind an explicit flag. +- **Accept:** `kill -9` + immediate restart → clean single takeover, no crash-loop; multi-writer exclusion still holds. + +### PR-L2.4 Move proof runners out of the production jar +- **Change:** relocate the five proof runners + reports + exporters (~7.2k lines) to `src/test` or a separate `lab-tools` Maven module; remove their CLI early-exit dispatch from `LoadBalancerApiApplication` (or gate behind a `-tools` classifier artifact). Drop `javafx-controls` to `provided`/`runtime` scope so it leaves the server jar. +- **Accept:** production jar no longer contains proof-runner or JavaFX classes; the proofs still run in CI from their new home. + +### PR-L2.5 Durability honesty (C7) + logging (diagnosability) +- **Change:** fsync parent directories on create/rename/delete so `FORCE_DATA_AND_METADATA` receipts don't outlive their directory entries; add real logging to the lab storage classes (currently zero) so a corruption is diagnosable without a debugger; evict `PROCESS_MUTEXES` entries (C8). +- **Accept:** a crash-consistency test (fsync fault injection) no longer produces a receipt for a lost entry; log output present on every failure path. + +**L2 exit:** the durability engineering worth keeping is deduplicated, rotatable, recoverable, honest, and diagnosable; test tooling is out of the shipped artifact. + +--- + +## Milestone L3 — Wire the survivors to live traffic (the payoff) + +**Depends on `BUILD_PLAN_DEPLOYABLE.md` Milestone 1 (PR-1.2 runtime stats, PR-1.3 live telemetry).** Until the proxy produces real per-upstream latency/error/in-flight data and captures its real `RoutingDecision`, these components have nothing real to consume. + +### PR-L3.1 Capture real proxy decisions +- **Change:** in `ReverseProxyService.forward`, after a routing decision + response, emit a `LiveRoutingDecisionRecord` (chosen upstream, candidate states from PR-1.2 runtime stats, actual latency/status) into a bounded ring buffer / event stream. +- **Accept:** `GET /api/proxy/decisions/recent` returns real decisions from live traffic; bounded memory. + +### PR-L3.2 Feed real telemetry into LASE shadow (fixes L1–L4) +- **Change:** replace the fabricated telemetry in `LaseShadowAdvisor.java:279-290` with the real per-upstream latency/error/in-flight from PR-1.2. Fix the unit mismatch (L3 — pass ratios, not load units), close the AIMD feedback loop by persisting limiter state across evaluations (L2), and compute agreement against the *same* candidate set the proxy actually chose from (L4). Run shadow evaluation async off the request path (L13). +- **Accept:** shadow recommendations track real backend degradation; agreement rate is interpretable; no per-request latency added. + +### PR-L3.3 "Explain this actual request" endpoint +- **Change:** point the consolidated `RoutingExplanationService` (L1.2) at `LiveRoutingDecisionRecord`s instead of client-supplied synthetic candidates. `GET /api/proxy/decisions/{id}/explain` returns dominant-factor + counterfactual analysis for a real decision, using the correct per-strategy score model (L1.3). +- **Accept:** explaining a real proxied request shows why *that* upstream was chosen and what weight/telemetry shift would have changed it. + +### PR-L3.4 Actuate the traffic-share orchestrator (optional, gated) +- **Change:** the `AdaptiveTrafficDecisionOrchestrator` (the best code in the audit) currently only records decisions. Behind a default-off `loadbalancerpro.lase.policy.mode=active-experiment` flag with the existing guardrails, let it adjust per-upstream weights on the live proxy within movement caps — a real, guardrailed adaptive-routing capability. Wire `ShadowAutoscaler` recommendations to the (now-safe, L5–L7-fixed) `CloudManager.scaleServersAsync` similarly gated. +- **Accept:** with the flag on in a test harness, sustained backend degradation shifts weight away from the bad upstream within guardrail limits; flag off = today's record-only behavior exactly. + +**L3 exit:** shadow evaluation, explainability, and adaptive allocation all operate on *real* traffic — the machinery that was 96% dormant becomes a genuine adaptive-routing + observability product. + +--- + +## Milestone L4 — Operator surface cleanup + +- **PR-L4.1** Fix or retire the CLI (O0 jar wiring so `--remediation-report` works from the shipped jar, or correct the docs to `java -cp`; consistent `--force` overwrite policy O5; fix inverted idle-timeout; remove dead undo persistence + add `ObjectInputFilter` for O3, or delete the interactive menu entirely and keep only the evidence tooling). +- **PR-L4.2** Delete or fix the JavaFX GUI (O2/O3). Recommendation: delete — all action buttons are runtime-broken and it drags a CVE surface into the server jar; the cockpit HTML pages already cover the UI need. +- **PR-L4.3** Unify jar-selection across `operator-distribution-smoke.sh` / `.ps1` / Docker / CI (O5) and de-hardcode the version. +- **PR-L4.4** Consolidate the five ~600–1080-line evidence-viewer cockpit pages into one parameterized page + a shared `lib.js`; archive ~80% of `docs/` (the PLAN/ROLLUP/LANE/CLOSURE ceremony) into a `docs/archive/`, keep the ~30 operator docs, refresh the stale evidence stubs (v1.9.0, empty performance baseline). + +--- + +## Sequencing & dependency graph + +``` +L0 (security/stabilize): L0.1 L0.2 L0.3 L0.4 L0.5 L0.6 ← land NOW, parallel to proxy M0; independent +L1 (consolidate analysis): L1.1 → L1.2 → L1.3 → L1.4 ← before L3 +L2 (consolidate lab): L2.1 → L2.2 → L2.3 → L2.4 → L2.5 (L2.1 needs L0.6's atomic-append) +L3 (wire live): needs proxy PR-1.2/1.3 → L3.1 → L3.2, L3.3 (need L1.2/L1.3) → L3.4 (needs L0.5, L2 cloud fixes) +L4 (operator): L4.1 L4.2 L4.3 L4.4 ← independent, any time after L0 +``` + +**Recommended order for Codex:** land all of L0 first (six small, high-value security/stability PRs — several overlap the proxy plan's M0, so coordinate `L0.1`=`proxy PR-0.5`). Then L1 to shrink the analysis surface before wiring, L2 in parallel to dedup the lab. Hold L3 until the proxy's live-telemetry PRs exist — that's the dependency that unlocks the entire "wire to real traffic" payoff. L4 is cleanup, fit it in opportunistically. + +**Definition of done for this plan:** L0 complete (nothing dangerous by default) + L1/L2 complete (~30k lines removed, one of each core abstraction) + at least L3.1–L3.3 (shadow + explainability operating on real proxy decisions). L3.4 and L4 are stretch. + +**Estimated net effect:** roughly **-30k lines** (deletions/consolidation) and **+~3k lines** (live-wiring), leaving a codebase where the proxy is real (first plan), the analysis layer explains real decisions, and the lab is a lean, correct durability harness rather than 41k lines of self-validating ceremony. diff --git a/docs/strategy-playground.html b/docs/strategy-playground.html new file mode 100644 index 00000000..0cdf83cc --- /dev/null +++ b/docs/strategy-playground.html @@ -0,0 +1,523 @@ + + + + + +Load Balancer Strategy Playground — RicheyWorks/LoadBalancerPro + + + +
+
+

Load Balancer Strategy Playground

+ a faithful, live model of all five routing strategies + RicheyWorks/LoadBalancerPro · core/*RoutingStrategy +
+

+ +
+
+
req/s 320
+
speed 3×
+ + +
+ +
+ Scenario + + + + + + + + +
+ +
+
+

Backends — live routing share & observed telemetry

+
+
+
+

Fleet metrics

+
+
Throughput
0 req/s
+
In-flight
0
+
p50 / p95 latency
0 ms
+
p99 latency
0 ms
+
Error rate
0%
+
Fairness (Gini)
0
+
+ +
+ throughput + p95 latency + error % +
+ +
+
+ + +
+
+
+

All five strategies run independent live simulations against the same backend config and request rate. The highlighted card is the current best by fleet p99 (lower is better). Edit the backends above and watch which strategy copes — this is the head-to-head the repo's RoutingComparisonEngine does on synthetic inputs, but here on live queueing telemetry.

+
+ +
+ How this maps to the code. Each strategy is reimplemented from the actual Java in + core/: RoundRobinRoutingStrategy (floorMod cursor), + WeightedRoundRobinRoutingStrategy (smooth WRR, nginx-style), + WeightedLeastConnectionsRoutingStrategy (in-flight ÷ weight), + TailLatencyPowerOfTwoStrategy (sample 2, pick lower p95/p99 score), + ConsistentHashRing (ceiling entry on a vnode ring). + The key difference from the shipped proxy: here the strategies see live in-flight counts and observed p95/p99 that rise with load — + exactly the real telemetry the audit found the production path never feeds them (defect D1). This is what the adaptive strategies do once that loop is closed. + Keys: space pause · 1–5 switch strategy · 6 compare all. +
+
+ + + + From 3a7b5daa66c19a6d9efc7833b20afe4e12f4f836 Mon Sep 17 00:00:00 2001 From: RicheyWorks <730richey730@gmail.com> Date: Tue, 21 Jul 2026 00:01:04 -0700 Subject: [PATCH 2/5] docs: link audit, build plans, and strategy playground from README --- README.md | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/README.md b/README.md index fb3f0583..7f70300b 100644 --- a/README.md +++ b/README.md @@ -26,6 +26,19 @@ The Enterprise Lab Cockpit provides controlled lab evidence, local reproducibili - It is not replay execution, evidence/report generation, storage/export proof unless a specific implemented lane and verification result says so. - It is not permission to add CI/Maven wiring, Docker/Compose behavior, runtime behavior, endpoints, secrets, external/cloud/tenant targets, or production-looking defaults outside an explicitly scoped change. +## Independent Architecture Audit, Build Plans & Strategy Playground + +An independent, code-level architecture review of the repository lives under `docs/`. It is a candid engineering assessment for reviewers and agents — a map of what is real data-plane/control-plane code versus simulation, lab, and evidence apparatus, with prioritized findings and file-level references. It is an audit and planning surface only: it does not itself change behavior, and it does not add production-readiness, certification, live-cloud, real-tenant, benchmark, throughput/p95/p99, or replay/export/storage claims. + +- Live proxy + core simulation audit: [`docs/AUDIT_2026-07-21.md`](docs/AUDIT_2026-07-21.md). +- Lab, LASE shadow, Decision Explorer & operator-surface audit: [`docs/AUDIT_LAB_SHADOW_2026-07-21.md`](docs/AUDIT_LAB_SHADOW_2026-07-21.md). +- Deployability build plan (proxy → real load balancer): [`docs/BUILD_PLAN_DEPLOYABLE.md`](docs/BUILD_PLAN_DEPLOYABLE.md). +- Lab / shadow / analysis build plan (secure, consolidate, wire to live traffic): [`docs/BUILD_PLAN_LAB_SHADOW.md`](docs/BUILD_PLAN_LAB_SHADOW.md). + +The build plans are PR-sized, sequenced roadmaps bounded by the same trust contract as this README: they describe proposed, separately-scoped changes and do not themselves authorize production, CI/Maven, Dockerfile, endpoint, secret, or cloud/tenant changes outside an explicitly scoped PR. + +An interactive [`docs/strategy-playground.html`](docs/strategy-playground.html) accompanies the audit — a single self-contained page (no build, no network) that faithfully reimplements all five routing strategies (`RoundRobin`, `WeightedRoundRobin`, `WeightedLeastConnections`, `TailLatencyPowerOfTwo`, `ConsistentHashRing`) in JavaScript and runs them against a live queueing model with configurable backend weights, latency, error rate, and health. It is a reviewer/educational simulation only: it is browser-side, uses no production code path, carries no real traffic, and is not throughput/p95/p99 evidence, production proof, or live-cloud/real-tenant validation. + ## Current Local-Lab Status - The local-lab Compose path is optional/manual/local-lab-only and remains bounded by the Compose readiness gate and app-service/runbook guardrails. From f227391b2999786a6452d29b99e49b4a71083c8b Mon Sep 17 00:00:00 2001 From: RicheyWorks <730richey730@gmail.com> Date: Tue, 21 Jul 2026 00:47:56 -0700 Subject: [PATCH 3/5] docs: note proposed (not-wired) CSRBT ecosystem integration in README; commit proposal ledger --- README.md | 8 ++ .../CSRBT_ECOSYSTEM_INTEGRATION_PROPOSAL.md | 100 ++++++++++++++++++ 2 files changed, 108 insertions(+) create mode 100644 docs/agent/CSRBT_ECOSYSTEM_INTEGRATION_PROPOSAL.md diff --git a/README.md b/README.md index 7f70300b..ac41535c 100644 --- a/README.md +++ b/README.md @@ -39,6 +39,14 @@ The build plans are PR-sized, sequenced roadmaps bounded by the same trust contr An interactive [`docs/strategy-playground.html`](docs/strategy-playground.html) accompanies the audit — a single self-contained page (no build, no network) that faithfully reimplements all five routing strategies (`RoundRobin`, `WeightedRoundRobin`, `WeightedLeastConnections`, `TailLatencyPowerOfTwo`, `ConsistentHashRing`) in JavaScript and runs them against a live queueing model with configurable backend weights, latency, error rate, and health. It is a reviewer/educational simulation only: it is browser-side, uses no production code path, carries no real traffic, and is not throughput/p95/p99 evidence, production proof, or live-cloud/real-tenant validation. +## Proposed CSRBT Ecosystem Integration (planning only — not wired) + +A WARN-classified proposal to integrate the RicheyWorks CSRBT ecosystem — an adaptive ordered index with exact O(log n) order statistics ([CSRBT](https://github.com/RicheyWorks/CSRBT)) and the SmokeHouse log-structured record store — into LoadBalancerPro in lab mode lives at [`docs/agent/CSRBT_ECOSYSTEM_INTEGRATION_PROPOSAL.md`](docs/agent/CSRBT_ECOSYSTEM_INTEGRATION_PROPOSAL.md). It is a planning ledger only. **Current wiring state: none** — there is no Maven dependency, no source wiring, and no endpoint; nothing has been merged. + +The proposal scopes five separately-approved, off-by-default lab-mode lanes: an allocation-evidence store (SmokeHouse, embedded), an exact tail-latency percentile scoring substrate (CSRBT order statistics, giving exact `p95`/`p99` instead of estimates), reviewer-facing decision-history views, evidence retention/archival, and anti-thrash strategy-promotion gates. It is bounded by precondition P0: no dependency lane may merge until the ecosystem artifacts are published (they currently install via `publishToMavenLocal` only) or a reviewer-approved local-lab resolution posture is documented with the same explicitness as the Compose readiness gate. The dependency direction is one-way — LoadBalancerPro would consume the libraries; it does not join the ecosystem's build. + +This proposal adds no production capability, no supported dependency, and no runtime behavior, and it does not relax this README's trust contract. No lane is complete until its own scoped PR is merged and main checks are green. + ## Current Local-Lab Status - The local-lab Compose path is optional/manual/local-lab-only and remains bounded by the Compose readiness gate and app-service/runbook guardrails. diff --git a/docs/agent/CSRBT_ECOSYSTEM_INTEGRATION_PROPOSAL.md b/docs/agent/CSRBT_ECOSYSTEM_INTEGRATION_PROPOSAL.md new file mode 100644 index 00000000..fc3705ed --- /dev/null +++ b/docs/agent/CSRBT_ECOSYSTEM_INTEGRATION_PROPOSAL.md @@ -0,0 +1,100 @@ +# CSRBT Ecosystem Integration Proposal (WARN-classified planning surface) + +This document is a WARN-classified planning surface only. It is a proposal ledger for +scoped, lab-mode-only integration of RicheyWorks CSRBT-ecosystem libraries into +LoadBalancerPro. It is not implementation permission, not a merged capability, not +production behavior, and not evidence. No lane below is complete until its own PR is +merged under the campaign rules and main checks are green. + +This proposal does not relax the README trust contract. It does not authorize Maven config +changes, CI/workflow changes, Dockerfile changes, Compose behavior changes, runtime +behavior changes, endpoint changes, secrets, external/cloud/tenant targets, or +production-looking defaults outside an explicitly scoped, separately approved PR per lane. + +## What is being proposed, at claim level + +The CSRBT ecosystem is a twelve-engine, zero-runtime-dependency, Java 17 library family by +the same author ([map](https://github.com/RicheyWorks/SuperBeefSort/blob/main/docs/ECOSYSTEM.md)): +an adaptive ordered index with exact O(log n) order statistics +([CSRBT](https://github.com/RicheyWorks/CSRBT)), a log-structured record store whose +append-only CRC'd log is the only truth +([SmokeHouse](https://github.com/RicheyWorks/SmokeHouse)), and derived engines (views, +time travel, crash-atomic batches, cold archives) that are all rebuildable caches of that +log. Every engine ships seeded oracle tests and composed integration tests +([WholeHog](https://github.com/RicheyWorks/WholeHog)). + +The claim-level fit: LoadBalancerPro's trust posture is built on controlled lab evidence, +local reproducibility, and explicit proof boundaries. SmokeHouse's design doctrine — the +log is the only truth; every index is a rebuildable cache; a crash can never lose evidence +that was durably written — is the same posture expressed as a storage engine. CSRBT's +order statistics give exact (not estimated) sliding-window percentiles, which is the +substrate the LASE Core Expansion ledger's tail-latency-aware scoring goals need. + +## Dependency posture (a named precondition, not a lane) + +The ecosystem artifacts (`io.github.richeyworks:*:0.1.0`) currently install via +`publishToMavenLocal` only. **Precondition P0:** no lane that adds a Maven dependency may +merge until either (a) the artifacts are published to Maven Central (the ecosystem's own +open Phase 9 item), or (b) a reviewer-approved local-lab-only resolution posture is +documented in the lane's PR with the same explicitness as the Compose readiness gate. +Local-lab docs must state that mavenLocal resolution is manual, local, and not CI-proof. +The dependency direction is one-way: LoadBalancerPro consumes the libraries; it does not +join the ecosystem's composite build, and the ecosystem takes no dependency on +LoadBalancerPro. + +## Proposed lanes (each PR-sized, each separately scoped and approved) + +### Lane E1 — Lab-mode allocation-evidence store (SmokeHouse, embedded) + +Append every allocation decision (request descriptor, candidate readouts, decision vector, +chosen target, evaluation metrics) as a record in an embedded SmokeHouse store, behind a +lab-only configuration flag that is off by default. Deliverables: the evidence-record +codec, the append path in the allocation facade's lab seam (no endpoint changes), seeded +oracle tests (TreeMap reference, in the ecosystem's house style and this repo's +tested-invariant style), and reopen/replay tests proving the evidence trail survives a +crash by construction. Claim boundary: this is lab evidence capture, not +replay/evidence/report/storage/export proof in the README's sense until a PR says so. + +### Lane E2 — Exact tail-latency scoring substrate (CSRBT order statistics) + +A lab-mode per-backend sliding-window latency tracker backed by CSRBT's windowed ordered +set: `percentileKey(95)`/`percentileKey(99)` are exact order-statistics walks, not sketch +estimates. Exposed to the load-distribution planner/evaluator as an optional scoring input +behind configuration, off by default. Deliverables: the tracker, property tests comparing +against brute-force percentile computation on the same window, and a scenario-evidence lab +run in the existing local-lab manner. Claim boundary: this enables tail-latency-aware +scoring experiments in lab mode; it is not throughput/p95/p99 production evidence. + +### Lane E3 — Reviewer-facing decision-history views (Renderer over E1) + +Materialized counts and rankings (decisions per backend, per strategy, per outcome class) +folded live off the E1 store's tail, for reviewer/operator surfaces. Depends on E1. +Deliverables: view definitions, fold-vs-brute-force oracle tests, and wiring into an +existing reviewer surface only if that surface's own scope allows it in the same PR. + +### Lane E4 — Evidence retention and archival (DryAge + Jerky over E1) — ledger-only + +Preserve evidence-store generations at scenario boundaries; cure them into CRC-verified +archives for the evidence directory. Recorded here for completeness; not proposed for +implementation until E1 has merged and a reviewer names a retention requirement. + +### Lane E5 — Anti-thrash strategy-promotion gates (MorphPolicy pattern) — ledger-only + +CSRBT's promotion gates (cooldown, minimum improvement, stability wins) are the shape the +adaptive routing policy would need to avoid strategy thrash under regime shifts. Recorded +as a design pointer for the LASE ledger; any implementation is its own scoped campaign. + +## Verification expectations + +Each lane follows VERIFICATION_PROTOCOL.md: focused checks while editing, full local +verification before merge, current-head remote checks before merge, post-merge main checks +before the lane counts. Each lane's PR states what was actually verified and nothing more. +Failures land in FAILURE_LOG.md. + +## What this proposal does not claim + +It does not prove production readiness, production certification, live-cloud validation, +real-tenant validation, runtime enforcement, load/stress/benchmarking, throughput/p95/p99 +production evidence, replay/evidence/report/storage/export proof, or broader automation. +It does not make the CSRBT ecosystem a supported production dependency of LoadBalancerPro; +it proposes lab-mode library use behind flags that default off, one bounded PR at a time. From 29a8aa3b5526139d1d93ea1fdc6614c66f289539 Mon Sep 17 00:00:00 2001 From: RicheyWorks <730richey730@gmail.com> Date: Tue, 21 Jul 2026 00:50:20 -0700 Subject: [PATCH 4/5] docs: add status badges and tagline to README header --- README.md | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/README.md b/README.md index ac41535c..14f73c78 100644 --- a/README.md +++ b/README.md @@ -1,5 +1,15 @@ # LoadBalancerPro +[![CI](https://github.com/RicheyWorks/LoadBalancerPro/actions/workflows/ci.yml/badge.svg)](https://github.com/RicheyWorks/LoadBalancerPro/actions/workflows/ci.yml) +[![CodeQL](https://github.com/RicheyWorks/LoadBalancerPro/actions/workflows/codeql.yml/badge.svg)](https://github.com/RicheyWorks/LoadBalancerPro/actions/workflows/codeql.yml) +[![license: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE) +[![Java 17](https://img.shields.io/badge/Java-17-orange.svg)](https://openjdk.org/projects/jdk/17/) +[![Spring Boot 3.5](https://img.shields.io/badge/Spring%20Boot-3.5-6DB33F.svg)](https://spring.io/projects/spring-boot) +![version 2.5.0](https://img.shields.io/badge/version-2.5.0-informational.svg) +![scope: local-lab](https://img.shields.io/badge/scope-local--lab-e8a33d.svg) + +> A Java 17 / Spring Boot adaptive-routing lab: a config-driven reverse-proxy data plane, a calculation-only allocation core with pluggable routing strategies, and a controlled, evidence-first lab apparatus — bounded by an explicit trust contract (see below). + ## Enterprise Lab Cockpit LoadBalancerPro is an Enterprise Lab Cockpit for controlled pre-production routing validation. It is not a demo. From 43ac9ea654a7320b983a00e59f96a1774ffd4af0 Mon Sep 17 00:00:00 2001 From: RicheyWorks <730richey730@gmail.com> Date: Tue, 21 Jul 2026 01:57:54 -0700 Subject: [PATCH 5/5] docs: add CSRBT lane E2 integration ADR (exact tail-latency routing) + README link --- README.md | 2 + .../ADR_E2_EXACT_TAIL_LATENCY_ROUTING.md | 108 ++++++++++++++++++ 2 files changed, 110 insertions(+) create mode 100644 docs/agent/ADR_E2_EXACT_TAIL_LATENCY_ROUTING.md diff --git a/README.md b/README.md index 14f73c78..524d6450 100644 --- a/README.md +++ b/README.md @@ -55,6 +55,8 @@ A WARN-classified proposal to integrate the RicheyWorks CSRBT ecosystem — an a The proposal scopes five separately-approved, off-by-default lab-mode lanes: an allocation-evidence store (SmokeHouse, embedded), an exact tail-latency percentile scoring substrate (CSRBT order statistics, giving exact `p95`/`p99` instead of estimates), reviewer-facing decision-history views, evidence retention/archival, and anti-thrash strategy-promotion gates. It is bounded by precondition P0: no dependency lane may merge until the ecosystem artifacts are published (they currently install via `publishToMavenLocal` only) or a reviewer-approved local-lab resolution posture is documented with the same explicitness as the Compose readiness gate. The dependency direction is one-way — LoadBalancerPro would consume the libraries; it does not join the ecosystem's build. +Lane E2 (exact tail-latency percentile scoring) is elaborated as an implementable, PR-by-PR design record in [`docs/agent/ADR_E2_EXACT_TAIL_LATENCY_ROUTING.md`](docs/agent/ADR_E2_EXACT_TAIL_LATENCY_ROUTING.md) — including a self-contained SPI seam that lets the feature merge and pass CI with no external dependency, keeping precondition P0 satisfied until the ecosystem publishes. + This proposal adds no production capability, no supported dependency, and no runtime behavior, and it does not relax this README's trust contract. No lane is complete until its own scoped PR is merged and main checks are green. ## Current Local-Lab Status diff --git a/docs/agent/ADR_E2_EXACT_TAIL_LATENCY_ROUTING.md b/docs/agent/ADR_E2_EXACT_TAIL_LATENCY_ROUTING.md new file mode 100644 index 00000000..f88300aa --- /dev/null +++ b/docs/agent/ADR_E2_EXACT_TAIL_LATENCY_ROUTING.md @@ -0,0 +1,108 @@ +# ADR — Exact Tail-Latency Routing via Order Statistics (CSRBT lane E2, with E1/E5 appendices) + +**Status:** Proposed (WARN-classified planning surface — elaborates `docs/agent/CSRBT_ECOSYSTEM_INTEGRATION_PROPOSAL.md`) +**Date:** 2026-07-21 +**Scope:** LoadBalancerPro. Lab-mode, off by default. No endpoint, CI, Dockerfile, or production-default changes. One-way dependency (LoadBalancerPro consumes; the CSRBT ecosystem takes nothing back). +**Trust note:** This is a design record, not implementation permission. No lane is complete until its own scoped PR is merged with green main checks. This ADR does not relax the README trust contract; it adds no production capability, throughput/p95/p99 production evidence, or supported dependency. + +--- + +## 1. Context — the defect this closes + +The independent audit (`docs/AUDIT_2026-07-21.md`, defect **D1**) established that LoadBalancerPro's adaptive routing strategies do not adapt on the live path. `ReverseProxyService.toCandidate()` builds each `ServerStateVector` from **static configuration** fields on `ReverseProxyProperties.Upstream` — `inFlightRequestCount` (default 0), `averageLatencyMillis`, `p95LatencyMillis`, `p99LatencyMillis`, `recentErrorRate`, `queueDepth`. Nothing on the request path ever updates them (`setInFlightRequestCount` is called only when copying config). Consequently `WEIGHTED_LEAST_CONNECTIONS` and `TAIL_LATENCY_POWER_OF_TWO` — the flagship strategies — decide on frozen numbers, and three of five strategies are decorative in the shipped proxy. + +The `BUILD_PLAN_DEPLOYABLE.md` roadmap opens (PR-1.2/1.3) with "close the live-telemetry loop": track real per-upstream in-flight counts and rolling latency, and feed them into `ServerStateVector`. The open question that roadmap left is *how to compute the percentiles*. Naïve options are a full sort per read (O(n log n) on the hot path) or a bounded histogram / t-digest sketch (approximate, and a new dependency of its own). + +CSRBT resolves this precisely. Its headline capability is **exact O(log n) order statistics over a subtree-size augmentation** — `select(rank)`, `rank(key)`, and derived `percentileKey(p)`. A per-backend sliding window backed by an order-statistics tree yields **exact** p95/p99 in O(log n) per query and O(log n) per sample, no sort, no sketch error. This is lane **E2** of the ecosystem proposal, and it is the cleanest fit in either codebase: one project's headline feature closes the other's headline defect. + +## 2. Decision + +Introduce a **lab-mode, off-by-default per-upstream latency window** exposing exact rolling p50/p95/p99, and let it (when enabled) supply the telemetry fields of `ServerStateVector` in place of the frozen config values. Do it behind a narrow **SPI seam** so the merge is not gated on the CSRBT dependency (see §4, which resolves proposal precondition P0). + +### 2.1 The seam: `RankedLatencyWindow` + +Define a small interface in LoadBalancerPro (new package `…loadbalancerpro.telemetry`): + +```java +/** Bounded, thread-safe rolling window of latency samples with exact order statistics. */ +public interface RankedLatencyWindow { + void record(long latencyNanos); // O(log n): insert sample, evict oldest if over capacity + long percentileNanos(double p); // O(log n): exact p in [0,100]; 0 if empty + long p50(); long p95(); long p99(); + int size(); + void clear(); +} +``` + +Two implementations, chosen at wiring time by config: + +1. **`SelfContainedRankedLatencyWindow` (default, zero new dependency).** A ~200-line size-augmented balanced BST (AVL or WB — the audit-verified rotation+size-maintenance logic can be lifted directly) storing composite keys `(latencyNanos, monotonicSeq)` so duplicate latencies coexist, plus a FIFO ring of the live keys for O(1) eviction of the oldest sample. `percentileNanos(p)` = `select(ceil(p/100 · n))`. This ships in-repo, needs no CSRBT artifact, and **lets every E2 PR merge and pass CI with no external dependency** — directly satisfying precondition P0 without waiting on Maven Central. +2. **`CsrbtRankedLatencyWindow` (adapter, lab-only).** Delegates to CSRBT's windowed ordered set / `RankedSet.percentileKey`. Available only when the `io.github.richeyworks:csrbt:0.1.0` artifact resolves (mavenLocal in lab, or Maven Central once the ecosystem's Phase 9 publishes). Selected by config; never the CI/default path until P0's option (a) or (b) is formally met. + +Rationale: the interface is the contract (mirroring CSRBT's own "the contract is the JSON, deliberately" stance in its visualizer). The self-contained default makes E2 shippable and CI-safe today; the CSRBT adapter is a drop-in upgrade that carries the ecosystem's tested engine when the dependency posture allows. This is the load-bearing design decision of the ADR — it dissolves the P0 blocker instead of waiting on it. + +### 2.2 Wiring into the proxy hot path + +- **Sample capture.** In `ReverseProxyService.forwardOnce(...)`, in the `finally`/completion block that already knows the attempt's outcome, record `window(upstreamId).record(elapsedNanos)`. Windows live in a `ConcurrentHashMap` on the service, created per configured upstream and **carried across reload for unchanged upstream ids** (this also fixes half of audit F-D14, where reload wipes runtime state). Pair with the in-flight `LongAdder` from build-plan PR-1.2 so `queueDepth`/`inFlightRequestCount` are live too. +- **Consumption.** In `toCandidate()`, when `loadbalancerpro.proxy.telemetry.exact-percentiles.enabled=true`, populate `ServerStateVector`'s `averageLatencyMillis`/`p95`/`p99`/`queueDepth` from the window + in-flight counters; when `false`, behave exactly as today (config values). The config fields become documented seed/fallback values, not the live source. +- **No endpoint change.** `GET /api/proxy/status` may *optionally* surface the live percentiles in its existing JSON (additive, no new route). No new controller, no schema break. + +### 2.3 Calculation-core path (optional, later) + +The allocation core already models `ServerObservationWindow`; E2's `RankedLatencyWindow` can back it too, giving `LoadDistributionPlanner`/`Evaluator` exact percentiles for the `/api/allocate/*` scoring experiments. Deferred to a follow-up PR; the proxy path is the primary deliverable. + +## 3. Flag & configuration surface + +All new keys default to the inert value; the shipped and `prod` profiles are unchanged. + +| Key | Default | Effect | +|---|---|---| +| `loadbalancerpro.proxy.telemetry.exact-percentiles.enabled` | `false` | Master switch. `false` = today's config-driven behavior, bit-for-bit. | +| `loadbalancerpro.proxy.telemetry.window-size` | `256` | Samples retained per upstream (bounded memory: O(upstreams × window-size)). | +| `loadbalancerpro.proxy.telemetry.impl` | `self-contained` | `self-contained` (default, no dep) or `csrbt` (lab-only, requires the artifact). | +| `loadbalancerpro.proxy.telemetry.min-samples` | `20` | Below this, fall back to config/seed percentiles (avoids cold-start noise; pairs with slow-start, build-plan PR-1.8). | + +## 4. Dependency posture (satisfies precondition P0) + +The proposal's P0 forbids merging a Maven dependency until the ecosystem publishes to Maven Central *or* a reviewer-approved local-lab resolution posture is documented. This ADR's SPI design means **the E2 feature does not require the dependency to merge at all**: the `self-contained` implementation is the CI/default path and pulls nothing external. The `csrbt` implementation is an *optional lab adapter*, added in its own PR that (a) declares the dependency `provided`/optional so it is never on the default resolution or CI classpath, and (b) documents mavenLocal resolution as manual, local, and not CI-proof — the same explicitness the Compose readiness gate uses. Dependency direction stays one-way. This keeps the whole lane green and honest before Maven Central exists. + +## 5. Test strategy + +Following `docs/agent/VERIFICATION_PROTOCOL.md` — focused checks while editing, full local verification before merge, current-head + post-merge main checks. + +1. **Percentile correctness (property test).** For random sample streams and window sizes, assert `RankedLatencyWindow.percentileNanos(p)` equals a brute-force percentile over the *same* live window (sort the current contents, index by the same `ceil` convention) for p ∈ {50, 90, 95, 99, 100} across thousands of seeded iterations. Run against **both** implementations so the CSRBT adapter and the self-contained tree are held to the identical oracle. +2. **Windowing / eviction.** Assert size never exceeds capacity, that eviction is FIFO (oldest sample leaves), and that percentiles track a shifting distribution (e.g. a step change from 40ms to 200ms is reflected within `window-size` samples). +3. **Order-statistics invariants.** For the self-contained tree, reuse the audit's approach: randomized mixed insert/evict against a sorted-list oracle with structural + subtree-size validation after every op (the audit's companion visualizer already demonstrates this passes for the lifted rotation logic). +4. **Concurrency.** Hammer one window from N threads (`record` on Tomcat request threads, `percentile*` on the routing thread) and assert no exception, no lost update beyond the window bound, and monotonic-ish readings — the interface must be thread-safe (the self-contained impl guards with a lock or a stamped lock; the CSRBT adapter inherits the engine's `StampedLock`). +5. **Flag inertness.** With the flag `false`, a golden test asserts routing decisions and `ServerStateVector` contents are identical to the pre-E2 build for a fixed scenario — proving zero behavior change when off. +6. **Scenario-evidence lab run.** In the existing local-lab manner, a one-slow-backend scenario under `TAIL_LATENCY_POWER_OF_TWO` with the flag on shows traffic shifting away from the slow upstream (the behavior the audit proved is *absent* today), captured as lab evidence — explicitly **not** throughput/p95/p99 production evidence. + +## 6. Consequences + +**Positive.** Closes audit D1 on the proxy path; makes least-connections and tail-latency-P2C genuinely adaptive; adds exact (not sketch) percentiles at O(log n); ships with no new dependency; leaves a clean upgrade seam to the CSRBT engine; partially fixes reload state loss (F-D14). Converts existing-but-decorative strategy code into live capability — the highest capability-per-line change identified in the build plan. + +**Negative / risks.** Memory grows O(upstreams × window-size) (bounded, configurable). A second order-statistics implementation now exists in-repo (the self-contained tree) — acceptable, and by design swappable for CSRBT. Hot-path cost rises from O(1) config read to O(log window) per candidate per request — negligible at window≤256, but the `min-samples` and flag gates keep it opt-in. The CSRBT adapter's correctness depends on the artifact; the shared oracle test (§5.1) guards against drift. + +**Neutral.** No endpoint, CI, Docker, or default-profile change. Prod behavior identical unless an operator sets the flag. + +## 7. Sequencing (PR-by-PR, each independently mergeable and green) + +- **PR-E2.1** — `RankedLatencyWindow` interface + `SelfContainedRankedLatencyWindow` + tests §5.1–5.4. Pure library, no wiring. Merges with zero dependency. +- **PR-E2.2** — Capture: per-upstream windows in `ReverseProxyService`, sample on `forwardOnce`, in-flight counters (build-plan PR-1.2), reload-carryover. Still no consumption; flag absent from decisions. +- **PR-E2.3** — Consume: `toCandidate()` reads the window behind `exact-percentiles.enabled`; golden flag-inertness test §5.5; scenario-evidence run §5.6. **This is the PR that closes D1.** +- **PR-E2.4** — Optional additive status surfacing of live percentiles in existing `/api/proxy/status` JSON. +- **PR-E2.5** — Lab-only `CsrbtRankedLatencyWindow` adapter behind `impl=csrbt`, dependency declared optional/provided with the documented mavenLocal posture (P0 option b). Shared oracle test runs both impls. + +--- + +## Appendix A — Lane E1 (crash-safe allocation-evidence log, SmokeHouse) + +The lab-side audit (`docs/AUDIT_LAB_SHADOW_2026-07-21.md`) found the evidence ledger has structurally-possible torn cross-process reads (C1) and non-atomic, fsync-less persistence that can silently destroy the prior snapshot (F-P1). SmokeHouse's doctrine — *the append-only CRC'd log is the only truth; every index is a rebuildable cache; a durably-written record cannot be lost to a crash* — is that exact posture as a storage engine. E1 proposes appending each allocation decision (request descriptor, candidate readouts, decision vector, chosen target, metrics) to an embedded SmokeHouse store behind an off-by-default lab flag, with seeded oracle tests (TreeMap reference) and reopen/replay-after-crash tests. Same SPI discipline applies: define an `AllocationEvidenceSink` interface with a self-contained append-only-file default and a SmokeHouse-backed adapter, so E1 merges without the dependency and upgrades cleanly. Claim boundary: lab evidence capture, not replay/evidence/export proof in the README's sense. + +## Appendix B — Lane E5 (anti-thrash promotion gates, MorphPolicy) + +CSRBT's `control.MorphPolicy` — verified sound in the audit — gates strategy changes with **cooldown**, a **minimum-improvement margin** (a candidate must be ≥20% cheaper to win, making A→B→A oscillation impossible at a fixed workload), and **stability wins** (N consecutive agreeing evaluations before acting). This is precisely the hysteresis the LoadBalancer's adaptive weight/strategy selection needs to avoid thrash under regime shifts, which the build plan flagged as missing (alongside slow-start, PR-1.8). E5 is a *design-pattern* transfer, not a code dependency: port the gate shape (three parameters, a `MorphHistory` cooldown counter) into the routing policy that acts on E2's now-live telemetry. Ledger-only until E2 lands and a reviewer names the requirement. + +## Appendix C — Why order statistics, not a sketch + +A t-digest / DDSketch gives approximate percentiles in O(1) space and is the usual production choice at massive scale. E2 chooses exact order statistics because (a) per-backend windows are small (≤256) so O(log n) is trivially cheap and exactness is free; (b) the project's entire trust posture is "controlled, reproducible, exact — not estimated," and CSRBT's exact percentiles are that posture expressed as a data structure; (c) it reuses an audited, tested engine from the same author rather than adding an approximate third-party dependency. If windows ever need to be large (thousands+), a sketch-backed `RankedLatencyWindow` implementation can be added behind the same interface without touching the proxy.