Skip to content

feat(k8s): add opt-in multi-replica Claude control plane - #651

Closed
liubing0806 wants to merge 476 commits into
dofastted:mainfrom
liubing0806:codex/kubernetes-control-plane
Closed

liubing0806 wants to merge 476 commits into
dofastted:mainfrom
liubing0806:codex/kubernetes-control-plane

Conversation

@liubing0806

@liubing0806 liubing0806 commented Oct 21, 2026 •

Copy link
Copy Markdown

Add an explicit kubernetes-ha mode so multiple control-plane Pods can concurrently serve ordinary Claude Docker Messages requests using shared PostgreSQL state and a pinned TLS node protocol. The existing kubernetes deployment remains single-instance. Slot payloads reuse the existing prebuilt kernel and CLI packages.

This is a scoped implementation of #648 and remains Draft. It depends on the PostgreSQL work in #649, which is still open; that dependency is included in this branch's diff.

Behavior

  • PostgreSQL stores authentication/session state, routing/sticky state, FIFO turns, key/account/VM admissions, rate limits, durable jobs and idempotent request settlements. Each replica handles requests independently.
  • Node allocations, initial credential publication and launch require private grants, approved policies and original receipts. Refresh tokens stay in the node's unmounted canonical directory; guests receive access-token-only projections.
  • A lost controller does not release an unknown execution by timeout. Recovery requires loss of the original owner lock and the original node/command/execution identity's durable completion proof. Missing usage or pricing evidence retains the reservation.
  • Recovery scanning, background ownership, maintenance and shutdown share the same persisted authority. HTTP/SSE drain accepted work; idle WebSockets close with 1001 during draining.
  • deploy/k8s-ha provides two replicas, required node anti-affinity, Pod-local runtime volumes, restricted security settings, readiness/draining, PDB and an optional HPA. The offline bootstrap CLI verifies an already migrated business schema before private initialization or node actions.

Supported scope

Ordinary Claude Docker: Anthropic Messages and Claude Chat/Completions conversion over HTTP/SSE. GPT/Codex, API backends, Search, native/KVM, terminals and legacy management/import/restore paths are rejected in HA mode. Claude Responses retains its existing 400 protocol_not_allowed; successful WebSocket model inference is not supported by this implementation.

Validation

  • Final code ff4e43486fa98af21aee366f8c3779202e81e1d0: fixed-archive CLI/database checks 25/25, 779-file format check and changed-file syntax checks passed. Final independent read-only Codex session found no new demonstrable P0/P1/P2 in the final delta. Review reports and earlier fixes are committed.
  • 48aa2a28: 50/50 targeted checks and 9/9 real two-node Kubernetes smoke scenarios passed. Running rootfs layers and eight runtime source files were compared with the frozen source.
  • f53e005f: 73/73 targeted checks; 2→3→2 scaling and rolling update with 45/45 HTTP/SSE requests returning 200; HPA 2→3 with a test-only CPU target of 1% and 31/31 requests returning 200; CRI exit 137 followed by recovery from the original node receipt and exactly one settlement. WebSocket upgrade/error/drain/bounded reconnect was checked, including a transient reconnect error.
  • These cluster tests used actual PostgreSQL, Kubernetes and Go TLS node processes with a synthetic kernel/provider and fake Engine. A separate actual prebuilt ELF test verified AT projection reload without restarting the kernel/CLI. Real model inference, real provider RT refresh, existing production nodes and Ingress runtime behavior remain unverified.
  • Full-suite checkpoint 4de403f9: unit 4339 passed / 24 failed / 10 skipped; E2E 117 passed / 17 failed / 1 skipped. Seven unit and all 17 E2E failures match the prior verified baseline. The additional 17 unit failures were missing test database initialization; the runner fix was separately rechecked 18/18, not represented as a full-suite rerun. Go race: 299 passed / 1 auxiliary skip; subsequent changes do not modify worker sources.

See the verification report for exact SHAs, image identities, review sessions, failure summaries and validation boundaries. No production deployment was performed.

Refs #648

@liubing0806 liubing0806 changed the title feat(k8s): add single-controller deployment foundation feat(k8s): add opt-in multi-replica Claude control plane Oct 22, 2026
cedar-cli and others added 8 commits October 22, 2026 17:29
* feat(protocol): official Claude Code passthrough for cli-hop

Co-authored-by: ted <kuhlmangarry24@gmail.com>

* feat(codex): add official Codex Responses passthrough

---------

Co-authored-by: cedar <yf>
Co-authored-by: ted <kuhlmangarry24@gmail.com>
Co-authored-by: ted <kuhlmangarry24@gmail.com>
* fix: rebuild outbound session on official passthrough paths

Official Claude Code and Codex content passthrough (#674) incorrectly
forced session mode passthrough, reusing inbound session IDs. Restore
attempt/VM rebuild semantics for outbound session while keeping body
content passthrough and slot identity rebind.

Co-authored-by: ted <kuhlmangarry24@gmail.com>

* fix: rebuild Codex prompt_cache_key on official passthrough

Force rebuild session mode for official CC/Codex passthrough paths so
prompt_cache_key follows outbound session, matching non-passthrough hop
semantics while keeping instructions/input/tools content intact.

Co-authored-by: ted <kuhlmangarry24@gmail.com>

* chore: biome format assert.match in handle-codex test

Co-authored-by: ted <kuhlmangarry24@gmail.com>

---------

Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: ted <kuhlmangarry24@gmail.com>
* origin/main:
  release: cut v1.3.213
  fix(213): 官方透传仍 rebuild outbound session,内容继续完全透传 (#676)
Merge the official annotated tag v1.4.011, peeled to
3ca3e01, into the local branch at
2adb081. Preserve both histories.

Keep PostgreSQL asynchronous persistence, HA fencing and unknown-execution
reservations, account isolation, and the independent encrypted Claude
subscription credential path. Adapt upstream image usage using new
PostgreSQL migration 023; historical migrations 001-022 are unchanged.
Combine scheduling-state, session-slot, model and panel updates with local
shared-state semantics and asynchronous test fixtures.

Known subscription PUT/delete race and missing HA subscription route and
egress integration remain intentionally unchanged. No push or deployment.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants