Repository navigation
Make the system headroom of unified-memory cache sizing configurable - #323
Open
javieratomic wants to merge 1 commit into
Open
javieratomic wants to merge 1 commit into
javieratomic wants to merge 1 commit into
Conversation
On a unified-memory GPU, Helios sizes the KV page pool once at startup from the memory that is free after the model loads. Before giving the rest to the pool it sets aside a 64 MiB driver margin, 2 GiB for the server's later host allocations, any configured reserve, and a twentieth of total memory for the operating system and other processes. The twentieth is a constant (`HOST_SYSTEM_SHARE` in `selene/src/offload.rs`) and no setting lowers it: `--memory-limit=-SIZE` and `[memory] reserve` only subtract more, and an occupied limit caps what the server occupies without counting the twentieth. On a host that does nothing but serve, the twentieth is much larger than what the rest of the system uses, and memory is what bounds the pool. On three hosts with 121.6 GiB of unified memory each (DGX Spark class), serving DeepSeek V4.1 Flash at tensor parallelism 3 with one rank per host, the twentieth is 6.08 GiB per host, while with the server stopped resident anonymous memory is 0.4 to 0.6 GiB. The right size depends on what else the host runs, which the server cannot know, so it is made a host-level setting whose default stays a twentieth. `flare run` and `flare orbit run` accept `--system-headroom SIZE`, and the configuration file accepts `[memory] system_headroom = "SIZE"`. Precedence is the flag, then the key, then a twentieth of memory. `0` leaves no headroom, and any other value below 1 MiB is rejected, so a fraction such as `0.05` is not read as bytes. The shared settings that application clients load apply the key as well, and Ignition's configuration validator accepts and checks the key without using it. Before this change the integrated-GPU cache budget (`budget` in `helios/src/runtime_core/memory.rs`) and the host budget for prefix-cache intake and pool staging (`host_pool_budget` in `selene/src/offload.rs`) used the constant directly. Both now call `flare_selene::offload::system_headroom`, which returns the configured size or the twentieth. Cache sizing on a discrete GPU keeps no system headroom and ignores the setting. On an integrated GPU, a configured headroom at least as large as the memory free after the driver margin, any reserve and the host allowance is refused by name, before the model loads under every form of memory limit and again when caches are sized; when nothing is free, the existing capacity error is reported. `flare run --hosts` forwards the value rank zero resolved, from its flag or its key, to every rank, as it does for `[memory] reserve`. `flare orbit run` forwards only the flag, and each node otherwise applies its own key. When the setting is given, the load display's cache plan shows it on an integrated GPU or when the host prefix cache is enabled. docs/cli-design.md describes the flag, the key, when to use a reserve instead, which value each rank uses and the risk of a small value; docs/model-preparation.md states the qualified configuration, and docs/TODO.txt lists the configurations not run. With the setting absent, every budget is computed as before. Measured result, with this change on `6b68b9b4`, on the three hosts above, which have 16 GiB of swap each: lowering the headroom raises KV capacity by the memory returned. The pool holds 8.17 M KV tokens with the default, 10.68 M with `4G` (+31 %), 13.06 M with `2G` (+60 %) and 14.36 M with `1G` (+76 %). With one start per value, decode throughput shows no difference larger than the spread between repetitions, and no out-of-memory event occurs in the load tested. Startup is slower: the server is ready in 41 s with the default and in 44 to 107 s over the five starts with a smaller headroom. The server uses more swap as the headroom shrinks: when serving begins, the three rank processes together hold 1.9 GiB of their own memory in swap with the default and 4.9 to 5.5 GiB at `2G` and `1G`. This is a capacity change for hosts where memory bounds the pool; it does nothing where an occupied limit, the KV ceiling or the prepared extent bounds it. Evidence: each value is one fresh server start that differs only in the headroom option, given to every rank, with the page cache dropped on all hosts before the start. The core serves 4 concurrent requests, the prefix cache is on disk only, and the binary is built with `--features roce`. Rank zero's cache plan reports budgets of 6.86, 8.93, 10.91 and 11.98 GiB (default, `4G`, `2G`, `1G`), which grow by 2.07, 4.05 and 5.12 GiB for the 2.08, 4.08 and 5.08 GiB returned, and every start reports `limited by memory`. A start with `[memory] system_headroom = "2G"` in a configuration file on every host and no flag holds 13.28 M KV tokens. With `--system-headroom 200G` every rank stops after GPU initialization, before loading the model, with the refusal that names the headroom. Each start runs about eight minutes of load: prose chat completions on one and four streams, four concurrent code completions, four concurrent cold prompts of 120,000 tokens, cold prompts of 20,000 and 100,000 tokens, and a streamed chat completion during a cold 120,000-token prefill. Minimum `MemAvailable` on the lowest host is 9.1, 7.1, 5.6 and 4.3 GiB, the four cold prompts take 175.6, 173.9, 175.1 and 169.3 s, and no `NV_ERR_NO_MEMORY` line or OOM kill occurs. One server at `2G` ran for 60 minutes through 10 rounds of a similar load: 179 requests completed and none failed, `MemAvailable` never fell under 5.2 GiB on any host, and swap-out over the hour was 930 MiB across the three hosts. Validation and limitations: unit tests cover the budget under every limit form with the default, a configured size, zero and an oversized value; the refusal and its exemption when nothing is free; parsing in the Rust flag, the Rust configuration loader and the Python validator; precedence; forwarding to peers with and without their own keys; the host prefix-cache intake budget; and the display entry. The selene, selene-manager, helios (without a GPU), flare and flare-cli suites and tests/cli/test_config.py with tests/cli/test_commands.py were run. Thirteen tests fail, and the same thirteen fail with the same messages on a tree whose Rust sources equal `6b68b9b4`: seven helios template tests that need `FLARE_TEST_TEMPLATE_DIR`, three DeepSeek frontend-settings tests, two `orbit::agent` tests and selene-manager's `record::tests::persisted_header_matches_the_descriptor_layout`. Two selene-manager tests fail intermittently on both trees under a CPU quota. No unit test loads a core, so the call sites that pass the setting into the check before loading, cache sizing and prefix-cache preparation are exercised only by the served runs. Not run: a discrete GPU; Windows and the desktop client's settings path; `flare run --hosts` and `flare orbit run` (the measured ranks were started one per host with the flag on each); a headroom of `0`; a host without swap, or with less than 16 GiB of it; hosts whose other processes use significant memory; and more than one start per value under the full load, except `2G`, which was started three times. Design-Signed-Off-By: Claude Fable 5.2; design-review=ed22 Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Signed-off-by: Javier López <59578422+javieratomic@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
On a unified-memory GPU, Helios sizes the KV page pool once at startup from the memory that is free after the model loads. Before giving the rest to the pool it sets aside a 64 MiB driver margin, 2 GiB for the server's later host allocations, any configured reserve, and a twentieth of total memory for the operating system and other processes. The twentieth is a constant (
HOST_SYSTEM_SHAREinselene/src/offload.rs), and no setting lowers it:--memory-limit=-SIZEand[memory] reserveonly subtract more, and an occupied limit (--memory-limit SIZEor a fraction) caps what the server occupies without counting the twentieth.On a host that does nothing but serve, the twentieth is much larger than what the rest of the system uses while the server is serving, and memory is what bounds the pool. Measured case: three hosts with 121.6 GiB of unified memory each (DGX Spark class), one rank per host, serving DeepSeek V4.1 Flash at tensor parallelism 3. The twentieth is 6.08 GiB per host. With the server stopped and the page cache dropped,
MemAvailableis 115.8 to 116.4 GiB, resident anonymous memory is 0.4 to 0.6 GiB, and 0.8 to 1.2 GiB of each host's 16 GiB of swap is in use. With the default, the load display reports6.86 GiB cache budgetandlimited by memory, the pool holds 8.17 M KV tokens, and under the load described belowMemAvailablenever falls under 9.1 GiB on any host. Each GiB returned to the pool holds about 1.2 M more KV tokens for this model, so the constant withholds several million tokens of KV capacity that these hosts held allocated through the load tested.The right size depends on what else the host runs, which the server cannot know. A host-level setting is therefore the place for it: the default stays a twentieth, and an operator who knows the host can state a smaller value.
What changed
flare runandflare orbit runaccept--system-headroom SIZE, and the configuration file accepts[memory] system_headroom = "SIZE". Precedence is the flag, then the key, then a twentieth of memory. Sizes use the existing binary suffixes.0leaves no headroom; any other value below 1 MiB is rejected, so a fraction such as0.05is not read as bytes. The shared settings that application clients load apply the key as well.flare_selene::offload::system_headroomreturns the configured size or the twentieth. Both places that used the constant call it: the integrated-GPU cache budget (budgetinhelios/src/runtime_core/memory.rs) and the host budget for prefix-cache intake and pool staging (host_pool_budgetinselene/src/offload.rs). Cache sizing on a discrete GPU keeps no system headroom and ignores the setting.flare run --hostsforwards the value rank zero resolved, from its flag or its key, to every rank, as it does for[memory] reserve; a peer's own key applies only when rank zero resolves none.flare orbit runforwards only the flag: the controller's key is not applied, and each node otherwise applies its own.… · system headroom 2GiB), on an integrated GPU or when the host prefix cache is enabled.docs/cli-design.mddescribes the flag, the key, when to use a reserve instead, which value each rank uses, and the risk of a small value: pools are fixed at startup, so later demand from the system or from the server's own host allocations beyond the 2 GiB allowance can be met only by reclaim, swap or the out-of-memory killer, which can stall or kill the server.docs/model-preparation.mdstates the qualified configuration, including that the hosts had swap, anddocs/TODO.txtlists the configurations not run.With the setting absent, every budget is computed as before.
Claimed result
On the measured hosts, lowering the headroom raises KV capacity by the memory returned. With one start per value, decode throughput shows no difference larger than the spread between repetitions. Cold prefill is timed once per start: the four-prompt and beside-the-stream times differ by less than the round-to-round spread of the sustained run at
2G, and the 100,000-token time differs by 2.0 s. No out-of-memory event occurs in the load tested. Startup is slower: the server is ready in 41 s with the default and in 44 to 107 s over the five starts with a smaller headroom (44, 53 and 107 s for the three at2G). The hosts have 16 GiB of swap each, and the server uses more of it as the headroom shrinks: when serving begins, the three rank processes together hold 1.9 GiB of their own memory in swap with the default and 4.9 to 5.5 GiB at2Gand1G.--system-headroomMemAvailableunder load (lowest host)4G2G1GThis is a capacity change for hosts where memory bounds the pool. It does nothing where an occupied limit, the KV ceiling or the prepared extent bounds it. The numbers are for this pull request alone, on a build of this branch with no other change.
Evidence
Conditions. Three hosts as above, connected over RoCE, one rank per host; DeepSeek V4.1 Flash core at tensor parallelism 3 for 4 concurrent requests, prepared from 6b68b9b plus the preparation-only change of #122 (compile headroom 4 GiB); 16 GiB of swap per host; disk Engram tables with the default 1 GiB row cache;
--scheduler-policy throughput; prefix cache on disk only. The binary is this change on 6b68b9b built with--features roce; the branch has since been rebased onto 5fd4070, which adds only a Windows ARM64 build fix (#315), and checked to compile. Before each server start the page cache is dropped on all hosts. Every row of the table is one fresh server start that differs only in the headroom option, given to every rank.Sizing. The load display shows the value reaching the sizing path. Rank zero's lines per start:
Every start reports
limited by memory. The budget grows by 2.07, 4.05 and 5.12 GiB for the 2.08, 4.08 and 5.08 GiB returned, within 0.04 GiB. The last line is a start with[memory] system_headroom = "2G"in a configuration file on every host and no flag: 13.28 M KV tokens, against 13.06 M with the flag; rank zero's budget is 0.18 GiB larger and its loaded memory 0.16 GiB smaller. With--system-headroom 200Gevery rank stops after GPU initialization, before loading the model. Rank zero reportssystem headroom is 214748364800 bytes, but only 121987493888 bytes are free after the driver margin, any reserve and the host allowance; the peers report the same message with their own free byte counts.Load per start. Prose chat completions with one stream and with four streams (three repetitions each); four concurrent cold prompts of 120,000 random tokens each with 256 generated tokens; cold natural-text prompts of 20,000 tokens (6.2 to 7.1 s) and 100,000 tokens; a streamed chat completion during a cold 120,000-token prefill; four concurrent code completions (two repetitions). Each start lasts about eight minutes.
MemAvailableis sampled every 2 s on every host; swap-out is the change inpswpoutbetween the server becoming ready and the end of the load, so it excludes startup;NV_ERR_NO_MEMORYcounts kernel log lines and OOM kills the change inoom_kill, on all hosts. The configuration-key start ran only the prose completions.--system-headroomNV_ERR_NO_MEMORY/ OOM kills4G2G1GThe cold 120,000-token prefill beside the stream took 43.9, 44.5, 44.6 and 48.6 s (default,
4G,2G,1G), and the stream's longest stall was 2.5, 2.5, 3.7 and 2.9 s.Startup. In the table's starts, swap use and startup time are larger with a smaller headroom, though
2Gexceeds1Gin both (values for default,4G,2G,1G). The rank processes'VmSwap, summed over the three hosts, was 1.9, 3.4, 5.5 and 5.4 GiB when serving began and 1.9, 2.9, 3.7 and 3.9 GiB at the end of the load. The server was ready 41, 47, 107 and 60 s after launch, and the slowest rank's cache allocation took 11.6, 15.5, 74.8 and 29.0 s. During the load the three hosts together took 81 k, 216 k, 489 k and 489 k major page faults. The two other starts at2Gwere ready in 44 s (configuration key) and 53 s (sustained run), with slowest allocations of 13.7 and 21.0 s and 4.9 and 5.5 GiB ofVmSwapwhen serving began; the 107 s and 74.8 s of the table's2Gstart did not recur.Sustained load at
2G. One server (13.08 M KV tokens) ran for 60 minutes through 10 rounds. Each round ran four concurrent cold 120,000-token prompts, the two natural-text prompts and a 78,072-token chat prompt (all three cold in the first round and restored from the prefix cache afterwards), prose and code completions on four streams, and a streamed chat completion during a cold 120,000-token prefill. Rounds 1, 4, 7 and 10 added a cold 250,000-token prompt and its repeat. Result:MemAvailablenever fell under 5.2 GiB on any host, and was 5.5 to 6.2 GiB at the end of every round, with no trend.NV_ERR_NO_MEMORYline and no OOM kill. Swap-out over the hour was 930 MiB across the three hosts, 716 MiB of it on rank zero's host.2Grow above: seven rounds took 181.2 to 195.8 s and three took 171.3 to 175.5 s.VmSwap, summed over the three hosts, was 5.5 GiB when serving began, 3.9 GiB after the first round and 4.6 GiB from the sixth on.Tests. Unit tests cover the budget under every limit form with the default, a configured size, zero and an oversized value; the refusal and its exemption when nothing is free; parsing in the Rust flag, the Rust configuration loader and the Python validator; precedence; forwarding to peers with and without their own keys; the host prefix-cache intake budget; and the display entry. Suites run on this branch: selene, selene-manager, helios (without a GPU), flare, flare-cli, and
tests/cli/test_config.pywithtests/cli/test_commands.py. No unit test loads a core, so the call sites that pass the setting into the check before loading, cache sizing and prefix-cache preparation are exercised only by the served runs. Thirteen tests fail, and the same thirteen fail with the same messages without this change (on a tree whose Rust sources equal 6b68b9b): seven helios template tests that needFLARE_TEST_TEMPLATE_DIR, three DeepSeek frontend-settings tests, twoorbit::agenttests and selene-manager'srecord::tests::persisted_header_matches_the_descriptor_layout. Two selene-manager tests (a_freed_window_credit_is_granted_without_waiting_for_the_clock,d7_a_corrupt_put_and_a_cancelled_put_over_the_channel) fail intermittently on both trees under a CPU quota.Not run.
flare run --hostsandflare orbit run, including a persistent group; the measured ranks were started one per host with the flag on each.0.2Gwas started three times (flag, configuration key, sustained run); the default,4Gand1Gonce each.--memory-limit.🤖 Generated with Claude Code