feat(sre): register privileged authority and isolate Kubernetes credentials

[Pal Lakatos-Toth (pallakatos)]

Current control-plane compatibility repair

Current head: 4e012ec7f9cece155c19a8f272f976a7ae2ff469. The bootstrap failure was not a rejected controller Pod: Kubernetes v1.31.0's controller-manager crashed in CEL OpenAPI type checking when the new policy inspected KarsSREAction.spec.action.params represented by boolean additionalProperties: true.

The repair preserves type: object and arbitrary nested JSON using field-local x-kubernetes-preserve-unknown-fields: true, consistently in Rust schema generation and Helm. No admission policy, approval requirement or namespace restriction is removed. Hosted A/B evidence recovered controller-manager and real Deployment/ReplicaSet/Pod creation, with all nine ordinary/private admission and JSON round-trip cases matching: https://github.com/Azure/kars/actions/runs/34262068112/job/102182569705 . That diagnostic job intentionally retained the original baseline failure; it was not reported green.

The repaired shipped-schema probe now executes those cases too. Parent corrected the diagnostic test imports for the actual full-harness discovery command; all 35 Python cases, Helm lint and Rust syntax/format checks pass locally. Shared Cargo was not taken from another owner: Rust/build/Clippy/schema-drift and full fatal migration qualification remain required from the new hosted run, https://github.com/Azure/kars/actions/runs/34265939336 .

This PR remains draft, outside the three completed author-waived foundation landings. No application readiness, customer-cluster change, main promotion or full SRE privacy closure is claimed yet. Older bootstrap hypotheses and failures below are historical and superseded by this evidence.

---

Current integration candidate and staged acceptance

Current head: ee24037189eff7beb96e1ba7d1961f5569e93ee8, now directly based on assembled foundations 365483a1 in kars-bridge. The ancestry refresh carried only predecessor audit records; all SRE repairs remain intact.

The preceding full Kind run confirmed real authority-stage preview/apply succeeds after the Helm 3/4 discovery fix. It then exposed a harness sequencing conflict: Helm 4's default watcher waited for SRE inference-policy acknowledgements before the required explicit enrollment phase could run.

Repair 7f7a4ac4 changes only prepared-legacy test setup to Helm 4's built-in-resource (--wait=legacy) readiness strategy, preserving Helm 3's existing wait. The immediately following real SRE migration remains mandatory/fatal before unrelated tests. No production readiness, policy, admission check or migration requirement is relaxed.

Fresh full qualification: https://github.com/Azure/kars/actions/runs/34252594680 . This PR remains draft and technically unqualified until that lifecycle completes. It is not covered by the author waiver used for #547-#549. Earlier failed-run details below are historical.

---

Current lifecycle follow-up: Helm 3/4 release discovery

Current head: 86162297a2ee6f6044c9487b3a978a8c2fe47a46. The corrected CRD was accepted in the full Kind run, which then exposed a separate real CLI compatibility defect: Helm 4.2.4 rejects the removed helm list --all flag during authority staging.

The shared install/stage discovery helper preserves Helm 3 behavior. It retries without --all only after that exact flag error and a confirmed Helm 4 version, whose default includes all release statuses. Authorization, malformed inventory, namespace/context, ownership and other error paths stay fail-closed; an error never becomes an absent release or permission to adopt. No dependencies or Rust implementation changed.

Targeted command/helper/authority coverage and the actual installed Helm flag parser passed locally. Full hosted lifecycle qualification is running at https://github.com/Azure/kars/actions/runs/34247989763 . This PR remains draft and outside the three author-waived foundation landings. Earlier schema repair and failure evidence below are retained as history.

---

Current schema repair and remaining qualification

Current head: 447312d1afb0a77ccbd2d144d74a4a293a6ceedd. The real Kubernetes v1.31 API rejected the SRE registration CRD with HTTP 422 because both custom namespace CEL accesses were undefined. The repair uses __namespace__ in CEL while preserving the JSON field names, namespace equality and singleton-name constraints.

Actual shipped-schema gate now passes: https://github.com/Azure/kars/actions/runs/34241285988/job/102112345148 . The prior controlled API comparison demonstrated corrected CRD/canonical acceptance (201) and the intended foreign-namespace/noncanonical rejections (422): https://github.com/Azure/kars/actions/runs/34239442081/job/102106020705 .

Full SRE migration/lifecycle qualification remains pending, and this PR remains draft. This is not closure of the downstream credential-privacy blocker and is not covered by the author's signature waiver for #547-#549. No source constraints, required gates or customer deployment safeguards were weakened. Earlier preparation/failed-run details below are historical.

---

Publication order

This is the security prerequisite between #549 and #550. It targets the qualified credential-source branch; the governed-service layer must incorporate this prerequisite and its privacy-epoch guard before its remaining HIGH finding can close. Do not merge into the intermediate feature branch. After preceding slices land, retarget to protected kars-bridge, refresh exact-head qualification and obtain genuine approval. Final promotion to main remains separate.

Why this prerequisite is required

The existing SRE agent holds a Kubernetes credential with cluster-wide Secret access. Merely mounting the new governed-service control token only in the router does not protect it from that API credential. Safe replacement requires operator-authorized UID enrollment, reviewed retirement of legacy grants, and a real compatible diagnostic path—not a permission drop that leaves SRE broken.

Scope

• Add cluster-scoped KarsSRERegistration/canonical, pinned to exact controller/release, source Sandbox and runtime Namespace identities. Registrar permission is unbound by default; occupancy, names, labels and Helm-looking metadata do not delegate privilege.
• Stage only unprivileged sources; atomically create genuinely new sources or require explicit reviewed UIDs for existing sources, consumers and legacy grants. Retire only approved legacy subjects with UID/resourceVersion fencing, preserve unrelated resources, and require live GET/LIST/WATCH denial before private issuance.
• Keep the Pod Azure identity/federated subject. Give the diagnostic proxy a separate, renewable, router-private Kubernetes identity. Old Hermes clients retain their standard token/CA/namespace file interface through loopback HTTPS and an opaque non-Kubernetes agent credential.
• Preserve bounded diagnostic reads, logs, metrics and Pending-only proposals. Secret key names remain visible; values, annotations/labels and copied credential material do not. Token requests, exec/proxy, unsafe paths/media and arbitrary writes are blocked.
• Protect reserved sources, identities, private material and workload/connect paths with retained admission policies. Ban legacy ServiceAccount-token Secret aliases for the private identity, detect prestaged/current/prior-UID aliases, quarantine owned authority and reject watch-only legacy grants. Unknown Secrets and replacement identities are not adopted or deleted.
• Add CLI stage/preview/enroll/migrate/retire and installation, upgrade, push, rollback and removal safeguards. Preserve unrelated core behavior and pinned images; refuse insecure migration or rollback before mutation.
• Add real Kind acceptance phases for legacy fixtures, delegated and tenant callers, token-controller/watch bypass attempts, migration ordering, unchanged Hermes TLS behavior and retirement/fresh re-enrollment.

Documentation: docs/how-to/sre-authority.md. Shared privacy logic is covered by audit, no-stub, crypto and runtime-affecting Kind guards. Dependencies add standard TLS support only; unrelated lockfile resolution changes were removed.

Initial CLI compatibility correction

The first hosted CLI run caught four external-mesh rejection regressions: SRE preflight ran before an already-invalid external mesh target was rejected. Repair 05b3a77c restores that no-probe rejection order while keeping authority preflight before artifact resolution and every deployment write. The existing assertions remain intact, and a valid-core/preflight-failure regression proves no writes occur. New exact-head qualification is required.

Confirmed request-boundary evidence and approved alert triage

The initial full run found registration display/status conventions missing; these are now implemented with real standard conditions and shared transition-time helpers. The existing 17-criterion conformance suite passes. Secret-type mutations now require the specific Kubernetes immutable-field 422 cause, while schema-valid CREATE/annotation probes still require the intended VAP 403.

For CodeQL alerts 780/781/782/783, independent source/SARIF review confirmed that the reported flows originate in Axum server-injected State rather than HTTP input. Eight hostile header/query/body scenarios exercised fixed-file rereads, renewal and inventory; operations stayed on the selected backend. Flagged production blobs and authentication were not rewritten. All 50 focused SRE tests and strict combined Clippy passed.

The user explicitly approved false-positive dispositions for ONLY those four alerts, each with its own evidence comment. CodeQL queries and source coverage remain enabled. This assumes trusted controller/kubelet configuration and private-volume integrity; it is not a declaration that privileged configuration tampering is harmless, and it is not audit sign-off or merge/deployment approval. The separate CLI fixture alert was addressed in code by exact resource-argument matching, not dismissed. Real Kubernetes migration qualification and genuine author/reviewer signatures remain required.

Isolated TLS regression setup correction

The full 941d6c2b run passed 2,572 of 2,573 cases; the new boundary case failed before executing its inputs because isolated nextest execution had no Rustls CryptoProvider selected. The longer step included test-binary compilation, not a hanging boundary request. Commit 99787682 initializes the existing AWS-LC provider within that test. It now passes alone, without relying on another test initializing process-global state. Production proxy code and the reviewed CodeQL input boundaries are unchanged. The real Kubernetes migration gate has not yet run successfully and remains required.

Current held acceptance issue

The 99787682 run clears Rust and CodeQL, but Kind stops during legacy-fixture preparation before the authority/migration proof. The sanitized failure identifies a kubectl command after the old source/consumer were created; bounded diagnosis is in progress. The migration gate is not considered passed. This later draft remains outside the foundation integration queue and must not block assembling already-qualified earlier slices.

Evidence and remaining gates

Current candidate: 997876825aa54cded225b2ef4a82050b2e5bc8c8, based on #549 head 8b206065.

• Local repair qualification: 46 focused Rust tests, 116 affected CLI/Helm tests, typecheck and strict combined controller/router Clippy passed. Earlier unchanged-Hermes TLS/proposal compatibility cases are retained.
• Parent re-enrollment fix replaces the complete reviewed specification with JSON Patch after UID/resourceVersion tests, preventing stale retired consumer data from surviving. Targeted regressions and the real persisted-spec assertion remain strict.
• Ten pure acceptance-harness tests, Bash syntax, shared security-guard regressions and committed-diff gates passed. These are not live Kubernetes proof.
• Whole source review and both additional HIGH alias/watch repairs have bounded source-level closure. Final harness/re-enrollment review and hosted API/CEL/authorization/migration acceptance remain pending.
• Capability audit: docs/security-audits/2026-09-08-sre-authority-prerequisite.md. Genuine author and independent reviewer sign-offs are pending; no signatures or human approvals have been fabricated.

This draft does not yet close #550 until actual API qualification and integration are complete. It is not full Bridge publication, arbitrary-credential integration, a task runtime or a shared budget broker. No customer/H100 deployment, Azure mutation, public image release, integration merge or main merge is performed by preparing it.

Latest acceptance-fixture correction (3c9523e8)

The previous hosted Kind run stopped during fixture preparation: the legacy control consumer omitted required spec.inferenceRef. This test-only correction supplies that field in both the consumer fixture and the reserved-source admission probe, with regression coverage against the actual Helm schema. Command failures now identify harness source coordinates, exit status and an allowlisted error category without printing argv, credentials or response bodies.

Thirteen pure harness regressions pass. Fresh hosted CI/Kind is pending; this does not claim completed real-API migration qualification or closure of the downstream governed-services privacy blocker. No production behavior, permissions, dependencies or audit requirements changed. The PR remains draft and outside the qualified foundation assembly queue.

[github-actions]

Dependency Review

✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.

OpenSSF Scorecard

Package	Version	Score	Details
cargo/rcgen	0.13.2	Unknown	Unknown
cargo/yasna	0.5.2	🟢 4.1	Details Check Score Reason Maintained ⚠️ 0 0 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 0 Dangerous-Workflow 🟢 10 no dangerous workflow patterns detected Binary-Artifacts 🟢 10 no binaries found in the repo Code-Review 🟢 6 Found 7/11 approved changesets -- score normalized to 6 Packaging ⚠️ -1 packaging workflow not detected Token-Permissions ⚠️ 0 detected GitHub workflow tokens with excessive permissions Pinned-Dependencies ⚠️ 0 dependency not pinned by hash detected -- score normalized to 0 CII-Best-Practices ⚠️ 0 no effort to earn an OpenSSF best practices badge detected Security-Policy ⚠️ 0 security policy file not detected Fuzzing 🟢 10 project is fuzzed Signed-Releases ⚠️ -1 no releases found Branch-Protection ⚠️ 0 branch protection not enabled on development/release branches License 🟢 10 license file detected SAST ⚠️ 0 SAST tool is not run on all commits -- score normalized to 0
npm/js-yaml	4.3.2	🟢 6	Details Check Score Reason Code-Review ⚠️ 0 Found 1/30 approved changesets -- score normalized to 0 Maintained 🟢 10 30 commit(s) and 28 issue activity found in the last 90 days -- score normalized to 10 Packaging ⚠️ -1 packaging workflow not detected Token-Permissions 🟢 10 GitHub workflow tokens follow principle of least privilege Dangerous-Workflow 🟢 10 no dangerous workflow patterns detected Binary-Artifacts 🟢 10 no binaries found in the repo Pinned-Dependencies 🟢 3 dependency not pinned by hash detected -- score normalized to 3 CII-Best-Practices ⚠️ 0 no effort to earn an OpenSSF best practices badge detected License 🟢 10 license file detected Fuzzing 🟢 10 project is fuzzed Branch-Protection ⚠️ 0 branch protection not enabled on development/release branches Signed-Releases ⚠️ -1 no releases found SAST ⚠️ 0 SAST tool is not run on all commits -- score normalized to 0 Security-Policy 🟢 4 security policy file detected
npm/js-yaml	4.3.2	🟢 6	Details Check Score Reason Code-Review ⚠️ 0 Found 1/30 approved changesets -- score normalized to 0 Maintained 🟢 10 30 commit(s) and 28 issue activity found in the last 90 days -- score normalized to 10 Packaging ⚠️ -1 packaging workflow not detected Token-Permissions 🟢 10 GitHub workflow tokens follow principle of least privilege Dangerous-Workflow 🟢 10 no dangerous workflow patterns detected Binary-Artifacts 🟢 10 no binaries found in the repo Pinned-Dependencies 🟢 3 dependency not pinned by hash detected -- score normalized to 3 CII-Best-Practices ⚠️ 0 no effort to earn an OpenSSF best practices badge detected License 🟢 10 license file detected Fuzzing 🟢 10 project is fuzzed Branch-Protection ⚠️ 0 branch protection not enabled on development/release branches Signed-Releases ⚠️ -1 no releases found SAST ⚠️ 0 SAST tool is not run on all commits -- score normalized to 0 Security-Policy 🟢 4 security policy file detected
npm/js-yaml	4.3.2	🟢 6	Details Check Score Reason Code-Review ⚠️ 0 Found 1/30 approved changesets -- score normalized to 0 Maintained 🟢 10 30 commit(s) and 28 issue activity found in the last 90 days -- score normalized to 10 Packaging ⚠️ -1 packaging workflow not detected Token-Permissions 🟢 10 GitHub workflow tokens follow principle of least privilege Dangerous-Workflow 🟢 10 no dangerous workflow patterns detected Binary-Artifacts 🟢 10 no binaries found in the repo Pinned-Dependencies 🟢 3 dependency not pinned by hash detected -- score normalized to 3 CII-Best-Practices ⚠️ 0 no effort to earn an OpenSSF best practices badge detected License 🟢 10 license file detected Fuzzing 🟢 10 project is fuzzed Branch-Protection ⚠️ 0 branch protection not enabled on development/release branches Signed-Releases ⚠️ -1 no releases found SAST ⚠️ 0 SAST tool is not run on all commits -- score normalized to 0 Security-Policy 🟢 4 security policy file detected

Scanned Files

• Cargo.lock
• mesh-plugin/package-lock.json
• runtimes/openclaw/package-lock.json
• tools/headlamp-plugin/package-lock.json