feat(budget): enforce durable governed-inference reservations - #553
Draft
Pal Lakatos-Toth (pallakatos) wants to merge 12 commits into
Draft
feat(budget): enforce durable governed-inference reservations#553Pal Lakatos-Toth (pallakatos) wants to merge 12 commits into
Pal Lakatos-Toth (pallakatos) wants to merge 12 commits into
Conversation
Local integration checkpoint only. Rust and real Kind qualification remain pending; no publication or enforcement approval is implied. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: eb3654cd-f1e0-445a-8734-430800af1903
Forward exact 7dc7281. Preserve SRE projection and privacy lifecycle logic; use existing workspace TLS package versions and add only controller dependency edges. Budget Rust/Kind qualification remains pending; no Cargo invocation or publication. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: eb3654cd-f1e0-445a-8734-430800af1903
…umers Capture stable same-workspace UID/RV ancestry, canonical readiness/attenuation/full authorization, and live Team ownership without coupling identity resolution to budget state. Budget callers verify their persisted pins separately. Add focused API/race/owner tests; Rust execution remains pending the coordinated Cargo lease. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: eb3654cd-f1e0-445a-8734-430800af1903
Merge exact 068ae16 without changing Pending authorization semantics. Move the pending budget audit to the existing required security-audit path; no human signature or gate waiver is fabricated. Budget Rust and live Kind qualification remain pending the Cargo allocation. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: eb3654cd-f1e0-445a-8734-430800af1903
Record actual post-068ae160 static and cached-test results without claiming budget Rust, Kind or human signoff completion. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: eb3654cd-f1e0-445a-8734-430800af1903
Repair the five bounded source-review findings: module scopes, supported Team launches, non-destructive admission waits, final Anthropic usage evidence, and legacy planning selection. Preserve Task UIDs and existing funded executions while denying new admissions; retain explicit pause/policy/UID revocation. Add full Team/Task API interleaving and actual stream-settlement regressions. Rust tests are authored but unrun pending the parent's Cargo grant. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: eb3654cd-f1e0-445a-8734-430800af1903
Local-only qualification checkpoint on aa34c3f. Retain all previously uncommitted parent repairs: existing workspace hex and schemars dependency edges, kube 3 typed replace_status, ledger borrow/move, concurrent test root lifetime, required legacy inferenceRef fixture, and real text/event-stream MIME for all three wiremock SSE fixtures. The original HEAD was not compile/test qualified; these repairs are part of this checkpoint. Resolve strict Clippy without new allow/expect attributes, feature flags, or new dependencies. Rename Amounts.add to checked_add with identical arithmetic. Keep OutputField operator wire names via explicit serde renames and add exact serde/JsonSchema compatibility coverage. Move unchanged request normalization, shape validation and final route/egress classification into a shared router-only dispatch module, included by the controller only in tests. Keep the public VerifiedTaskNode.generation field for credential composition and explicitly verify that generation in the second live inventory, with regression coverage. Actual Helm/Rust drift tests revealed three pre-existing candidate failures: Task, Team and Profile budget scope nullability/descriptions, Task's untyped status budget binding, Team account reference schema, and CEL rule ordering. Align these templates to the existing generated Rust contract; retain the same CEL rule strings, lifetime opt-in/retention and legacy launch prohibition. No fallback prices, raw-token authority, all-compute scope, or feature expansion. All Cargo commands below used the existing guard, cwd this worktree, and only /Users/pallakatos/Private/Repos/kars/target. The guard sets CARGO_INCREMENTAL=0, CARGO_BUILD_JOBS=2, CARGO_NET_OFFLINE=true and the existing CARGO_TARGET_DIR; it enforces an 8.5 GiB free-space floor. Both packages kept default features. Exact invocation prefix: python3 /Users/pallakatos/.copilot/session-state/a7f32227-c055-4df4-81ee-ce27d7f2d74d/files/run-cargo-guard.py --cwd /Users/pallakatos/.copilot/session-state/a7f32227-c055-4df4-81ee-ce27d7f2d74d/kars-pr11-budgets -- Passing guarded commands: 1. cargo test --quiet --offline --locked -p kars-controller -p kars-inference-router --lib --bins -- budget task_identity 75 controller + 57 router = 132 passed, zero failures/ignored. Includes all 16 incomplete Anthropic cases, positive final SSE settlement, complete funded-Team interleavings, new generation and wire-schema tests. 2. cargo test --quiet --offline --locked -p kars-controller -p kars-inference-router --lib --bins -- helm_drift::tests::helm_ kars_task::tests:: kars_task::authorization_tests:: kars_team_reconciler::tests:: kars_team_reconciler::persistence_tests:: routes::model_routing routes::chat_completions routes::anthropic_messages routes::inference routes::spawn_policy forward_proxy:: failover:: blocklist:: 68 controller + 83 router = 151 passed, including all 15 Helm drift tests. Counts describe executions; some selectors overlap command 1. 3. cargo test --quiet --offline --locked -p kars-controller -p kars-inference-router --test proxy_fake_upstream --test foundry_route_guard --test failover_walk --test multi_provider_guardrails --test anthropic_buffered_guardrail --test chat_output_guardrail_nonjson 24 real loopback HTTP tests passed: proxy 3, Foundry 9, failover 3, multi-provider 5, Anthropic 2, chat non-JSON 2; no external services. 4. cargo clippy --quiet --offline --locked -p kars-controller -p kars-inference-router --all-targets -- -D warnings Passed final all-target check, zero warnings and no new waivers. 5. cargo fmt --package kars-controller --package kars-inference-router 6. cargo fmt --package kars-controller --package kars-inference-router --check Passed. Only affected files required formatting. The compatibility/HTTP test invocations set TMPDIR=$PWD/.qualification-fixtures inside this worktree; that empty scratch directory has been removed. Minimum observed free space across qualification: 10.03 GiB; floor never hit. Passing existing Node/static commands, cwd this worktree unless noted: node cli/node_modules/vitest/vitest.mjs run --root cli --configLoader runner --no-cache src/commands/budget.test.ts 7 passed. Existing symlinked cache only; no dependency install. node cli/node_modules/vitest/vitest.mjs run --root deploy/helm/kars/tests --globals --no-cache src/inference-budget.test.ts src/local-inference.test.ts 18 passed (12 budget + 6 local-inference), rerun after template repair. cd cli && npm run typecheck Passed, no emit. git diff --check git diff --cached --check bash ci/check-copyright-headers.sh 736 source files passed with the new shared module staged. BASE_REF=068ae160 bash ci/check-loc.sh Passed against the reviewed parent. Additional source comparison confirmed normalize, shape validation, Quote accounting, mediated egress and operation classification unchanged modulo whitespace/relocation; Cargo.lock adds only hex and schemars 1.2.1 package edges. Limits: this is source/compile/strict-lint/test qualification, not production acceptance. Real budget/SRE Kind, human review of these repairs, combined credential/budget authority composition, and an independently fully-qualified router image digest remain pending. Privacy epoch remains an additional fence, not Pod UID/custom-audience TokenReview authentication. No public push, merge, main/customer change, Azure, Docker, image build/push or private Bridge actions. cli/node_modules remains untracked and is not part of this checkpoint. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: eb3654cd-f1e0-445a-8734-430800af1903
Dependency Review✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.Scanned FilesNone |
…andidate Carry the shared js-yaml lock repair and current SRE retirement/fixture work without replacing budget accounting or caller authority. Real combined Kubernetes qualification and human audit gates remain required. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: eb3654cd-f1e0-445a-8734-430800af1903
Hosted nextest runs each case independently; the pending-leaf fixture constructed a Kubernetes client before selecting the existing AWS-LC provider. Initialize it in the shared test setup so the real authority assertions execute without depending on another test process. No production authority or TLS policy changes. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: eb3654cd-f1e0-445a-8734-430800af1903
Preserve the existing budget ledger, accounting and caller authentication; forward only the prerequisite resource-specific policy correction and its evidence. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: eb3654cd-f1e0-445a-8734-430800af1903
Bounded local source checkpoint on 2b2e0b8. Addresses the actual #553 hosted CNCF failures c3_conditions_array_present and c12_state_printer_column without changing conformance checks or scope. Add observational status.phase, observedGeneration, and standard Kubernetes Ready/LedgerValid conditions to both Rust and Helm. The Phase printer points to .status.phase; the existing .status.ledger.phase and ledger wire remain unchanged. Conditions use the existing project helper and standard Condition and jiff-backed Time types, list-map keys by type, and stable transition times. Populate status after UID-anchor creation, during ledger initialization, after sealing, with every changed ledger transaction, and in periodic recovery/backfill. Derive bootstrap, active, temporarily reserved, exhausted, revoked, capacity-blocked, frozen, closing/retired, invalid-ledger and API- unavailable observations from actual account state. No blanket Ready signal, router-health assertion, fallback pricing, private errors, or spend authority. Reporting reloads a fresh account and uses bounded UID/RV whole-status PUT. It preserves the exact ledger, including corrupt or missing data; it cannot initialize/reset money. Initialization also uses complete status PUT. Lost acknowledgements still fail; unchanged ledger mutations retain their original no-write/idempotent return. A failed reporting read/write is not successful publication; an unreachable API necessarily retains its last observation. No funded Task deletion, authority reset, or changed account limit/schema. Add 11 Rust regressions covering real store transitions and fault injection: - reserved headroom versus durable exhaustion, preserving funded dispatch; - expiry/headroom recovery and retained replay fences; - revoked/retired authority and accepted-work liability; - provider breach plus forged observational Ready not authorizing spending; - live recovery API failure, stable Unknown, and subsequent root retirement; - corrupt/missing ledger reporting without repair or reinitialization; - failed reads/writes, UID replacement, conflicts and lost reporting acks; - legacy observation backfill and stable timestamp across generations; - Helm/generated-Rust reporting schema and printer-column equality; - report CAS conflict with concurrent reservation, preserving new money; - lost bootstrap ledger-status acknowledgement without reset before sealing. Extend the existing Helm tests for new reporting fields in enabled/default shapes. Update the directly related operator documentation. FAST GATES ACTUALLY RUN AND PASSED, cwd this worktree: rustfmt --check --edition 2024 controller/src/inference_budget/store.rs Includes nested store/status test modules; syntax/format only, not compilation. rustfmt --check --edition 2024 --config skip_children=true controller/src/inference_budget/account.rs controller/src/inference_budget/status.rs controller/src/inference_budget/status_tests.rs controller/src/inference_budget/store.rs controller/src/inference_budget/store_tests.rs controller/src/inference_budget/recovery.rs controller/src/inference_budget/mod.rs controller/src/kars_team_reconciler/budget_interleaving_tests.rs node cli/node_modules/vitest/vitest.mjs run --root deploy/helm/kars/tests --globals --no-cache src/inference-budget.test.ts src/local-inference.test.ts 20 passed: 14 budget and 6 local-inference. node cli/node_modules/vitest/vitest.mjs run --root cli --configLoader runner --no-cache src/commands/budget.test.ts 7 passed. cd cli && npm run typecheck Passed, no emit. git diff --check git diff --cached --check bash ci/check-copyright-headers.sh 739 tracked source files passed with new files staged. BASE_REF=2b2e0b8018b5059b05467ac76d3aa6115575be78 bash ci/check-loc.sh Passed; every touched Rust module is <=800 lines; no new whitelist/waiver. Parsed-YAML comparison proved the prior financial spec, immutable-grant CEL and status.ledger schema exactly unchanged. No Cargo manifest or lock edits. NOT YET RUST/CNCF QUALIFIED: no Cargo grant was held or used for this checkpoint. The root target remains exclusively leased to the credential RPC owner. After explicit parent grant, run the existing guarded offline/locked commands with existing root target, CARGO_INCREMENTAL=0 and CARGO_BUILD_JOBS=2: cargo test --quiet --offline --locked -p kars-controller -p kars-inference-router --lib --bins -- budget task_identity helm_drift cargo test --quiet --offline --locked -p kars-controller -p kars-inference-router -p kars-cncf-conformance --test criteria The latter is the actual 17-case CNCF criteria/report runner, not a substitute. cargo clippy --quiet --offline --locked -p kars-controller -p kars-inference-router --all-targets -- -D warnings cargo fmt --package kars-controller --package kars-inference-router --check No public push, parent review claim, merge, customer/main/private Bridge edit, Azure/cloud action, Docker/image action, install or new target. Pending: actual Rust compile/tests, generated schema drift, CNCF17, strict Clippy, parent review, and independent existing SRE Kind/cross-credential production qualification. cli/node_modules remains an unstaged existing symlink. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: eb3654cd-f1e0-445a-8734-430800af1903
Actual guarded Rust qualification of abbfc7d found and closed four bounded issues rather than treating source checks as compilation evidence: - Whole-status acknowledgement comparison now compares exact serialized wire values: standard Kubernetes Time serializes at second precision, so comparing unpublished subsecond clock values falsely reported committed status corrupt. Ledger counters and all other fields remain exact in this comparison. - KarsBudgetAccount::crd() hit kube's structural-schema panic on the existing tagged MaximumPrice union. Its generated ledger schema now uses the same bounded structural envelope already deployed by Helm, with existing contract version and capacity constants. The actual typed Ledger and all runtime financial/authority validation are unchanged. The schema regression now checks the ledger envelope too; only documentation, unordered required sets and equivalent JSON-Schema numeric minima are canonicalized for comparison. - Move the concurrent-reservation and bootstrap-lost-ack tests to module scope. Both are now genuinely registered/executed, not nested inactive functions. - Explicitly install the controller's AWS-LC Rustls provider in the legacy Pod fixture before creating its Client. Verify this fixture in isolation too. PASSING ACTUAL CARGO COMMANDS, cwd kars-pr11-budgets: All invoked through: python3 /Users/pallakatos/.copilot/session-state/a7f32227-c055-4df4-81ee-ce27d7f2d74d/files/run-cargo-guard.py --cwd /Users/pallakatos/.copilot/session-state/a7f32227-c055-4df4-81ee-ce27d7f2d74d/kars-pr11-budgets -- The guard used only /Users/pallakatos/Private/Repos/kars/target, default core package features, CARGO_INCREMENTAL=0, CARGO_BUILD_JOBS=2, CARGO_NET_OFFLINE=true, and the existing 8.5 GiB free-space floor. cargo test --quiet --offline --locked -p kars-controller -p kars-inference-router --lib --bins -- budget task_identity helm_drift 116 controller + 57 router = 173 passed; zero failed/ignored. Includes all 11 new status regressions, bootstrap/active/reserved/exhausted/revoked/frozen/ corrupt/unavailable transitions, report CAS races and lost acks, timestamp stability, exact generated/Helm reporting+ledger schema checks, and existing budget/interleaving/identity/Helm gates. cargo test --quiet --offline --locked -p kars-controller -p kars-inference-router -p kars-cncf-conformance --test criteria 17/17 passed, including c3_conditions_array_present, c12_state_printer_column, full_report_is_all_pass and report stability. cargo test --quiet --offline --locked -p kars-controller -p kars-inference-router --lib --bins -- no_task_owner_no_reference_is_a_byte_safe_legacy_noop_without_api_access 1 controller fixture passed in isolation, without another test installing TLS. cargo clippy --quiet --offline --locked -p kars-controller -p kars-inference-router --all-targets -- -D warnings Passed with no new lint waivers. cargo fmt --package kars-controller --package kars-inference-router --check Passed. FAST GATES: node cli/node_modules/vitest/vitest.mjs run --root deploy/helm/kars/tests --globals --no-cache src/inference-budget.test.ts src/local-inference.test.ts Re-run: 20 passed (14 budget + 6 local-inference). CLI budget 7 and CLI typecheck passed at abbfc7d; no CLI source changed since. git diff --check; git diff --cached --check bash ci/check-copyright-headers.sh 739 source files passed. BASE_REF=2b2e0b8018b5059b05467ac76d3aa6115575be78 bash ci/check-loc.sh Passed; all changed Rust modules remain <=800 lines; no whitelist changes. No Cargo manifest or lock changes, no dependency installs or alternate targets. Minimum observed free space during this lease: 9.84 GiB; guard floor not hit. This qualifies the local budget-account status/CNCF repair, not production rollout. Parent review, hosted CI, real SRE/Kind and cross-credential composition remain independent gates; no claim of a qualified production router digest. No public push/merge, main/customer/private Bridge edits, Azure, Docker or image operations. Existing cli/node_modules symlink remains untracked. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: eb3654cd-f1e0-445a-8734-430800af1903
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Current shared dependency and prerequisite refresh
Current head:
c11c55a8552ab5150be0be6b1d6122afad8f3e9c. Forwarded the same reviewed SRE retirement/preflight updates and verified js-yaml 4.3.2 lock repair; no independent dependency re-resolution or gate waiver. The original runtime audit failure is addressed by the patched dependency, with fresh exact-head hosted qualification required.The SRE prerequisite still needs the resource-specific built-in-controller/private-ReplicaSet admission correction and complete migration acceptance. This PR remains draft and must not bypass that dependency or merge into an intermediate feature branch. Historical head references below are retained as evidence, not current readiness claims. No main/customer deployment or private Bridge publication is performed.
Publication position and scope
This is the already-approved governed-inference budget candidate, composed on #550 and its SRE prerequisite #551. It is a parallel bounded layer to managed MCP and credential integration; do not merge into the intermediate feature base. Final combined qualification and direct retargeting to
kars-bridgeare required before landing. Bridge remains private and core remains independently usable.The new explicit scope is governed inference only: durable token and operator-configured maximum-price caps. It does not claim compute/tool/storage/network charges, tax/FX or all-in invoice accuracy. Omitted/unbounded standalone behavior stays unchanged; legacy positive budgets remain planning-only rather than being silently upgraded.
Implementation
Evidence and required gates
Current head
dd888762c79490d5aff11b4744f98bbea7c5e0b1. Independent automated review findings were repaired and bounded re-review found no significant issues. Actual local Rust execution passed 132 budget/identity cases, 151 compatibility/schema/dispatch cases and 24 HTTP cases (some selectors overlap). Strict paired Clippy, formatting, CLI typecheck/seven cases, eighteen Helm cases, LOC and source-policy gates passed. The final commit records exact commands and limits; no new lint or security waivers were introduced.DRAFT: real hosted budget/SRE Kind, combined credential/MCP integration, qualified router-image enforcement and genuine human audit sign-offs remain required. No existing human waiver applies to this capability. Local source and HTTP evidence is not full runtime acceptance, financial certification or merge approval. No customer/H100 deployment, image release, private Bridge source exposure or main promotion is performed.
Audit:
docs/security-audits/2026-09-08-governed-inference-budgets.md.