Skip to content

Make the system headroom of unified-memory cache sizing configurable - #323

Open
javieratomic wants to merge 1 commit into
NVIDIA:mainfrom
javieratomic:system-headroom
Open

javieratomic wants to merge 1 commit into
NVIDIA:mainfrom
javieratomic:system-headroom

Conversation

@javieratomic

Copy link
Copy Markdown
Contributor

Why

On a unified-memory GPU, Helios sizes the KV page pool once at startup from the memory that is free after the model loads. Before giving the rest to the pool it sets aside a 64 MiB driver margin, 2 GiB for the server's later host allocations, any configured reserve, and a twentieth of total memory for the operating system and other processes. The twentieth is a constant (HOST_SYSTEM_SHARE in selene/src/offload.rs), and no setting lowers it: --memory-limit=-SIZE and [memory] reserve only subtract more, and an occupied limit (--memory-limit SIZE or a fraction) caps what the server occupies without counting the twentieth.

On a host that does nothing but serve, the twentieth is much larger than what the rest of the system uses while the server is serving, and memory is what bounds the pool. Measured case: three hosts with 121.6 GiB of unified memory each (DGX Spark class), one rank per host, serving DeepSeek V4.1 Flash at tensor parallelism 3. The twentieth is 6.08 GiB per host. With the server stopped and the page cache dropped, MemAvailable is 115.8 to 116.4 GiB, resident anonymous memory is 0.4 to 0.6 GiB, and 0.8 to 1.2 GiB of each host's 16 GiB of swap is in use. With the default, the load display reports 6.86 GiB cache budget and limited by memory, the pool holds 8.17 M KV tokens, and under the load described below MemAvailable never falls under 9.1 GiB on any host. Each GiB returned to the pool holds about 1.2 M more KV tokens for this model, so the constant withholds several million tokens of KV capacity that these hosts held allocated through the load tested.

The right size depends on what else the host runs, which the server cannot know. A host-level setting is therefore the place for it: the default stays a twentieth, and an operator who knows the host can state a smaller value.

What changed

  • Setting. flare run and flare orbit run accept --system-headroom SIZE, and the configuration file accepts [memory] system_headroom = "SIZE". Precedence is the flag, then the key, then a twentieth of memory. Sizes use the existing binary suffixes. 0 leaves no headroom; any other value below 1 MiB is rejected, so a fraction such as 0.05 is not read as bytes. The shared settings that application clients load apply the key as well.
  • Sizing. flare_selene::offload::system_headroom returns the configured size or the twentieth. Both places that used the constant call it: the integrated-GPU cache budget (budget in helios/src/runtime_core/memory.rs) and the host budget for prefix-cache intake and pool staging (host_pool_budget in selene/src/offload.rs). Cache sizing on a discrete GPU keeps no system headroom and ignores the setting.
  • Refusal. On an integrated GPU, a configured headroom at least as large as the memory free after the driver margin, any reserve and the host allowance is refused by name. The check runs before the model loads under every form of memory limit, and again when caches are sized. When nothing is free, the headroom is not the cause and the existing capacity error is reported.
  • Ranks. flare run --hosts forwards the value rank zero resolved, from its flag or its key, to every rank, as it does for [memory] reserve; a peer's own key applies only when rank zero resolves none. flare orbit run forwards only the flag: the controller's key is not applied, and each node otherwise applies its own.
  • Display. When the setting is given, the load display's cache plan shows it (… · system headroom 2GiB), on an integrated GPU or when the host prefix cache is enabled.
  • Validation. Ignition's configuration validator accepts and checks the key; Ignition does not use it.
  • Documentation. docs/cli-design.md describes the flag, the key, when to use a reserve instead, which value each rank uses, and the risk of a small value: pools are fixed at startup, so later demand from the system or from the server's own host allocations beyond the 2 GiB allowance can be met only by reclaim, swap or the out-of-memory killer, which can stall or kill the server. docs/model-preparation.md states the qualified configuration, including that the hosts had swap, and docs/TODO.txt lists the configurations not run.

With the setting absent, every budget is computed as before.

Claimed result

On the measured hosts, lowering the headroom raises KV capacity by the memory returned. With one start per value, decode throughput shows no difference larger than the spread between repetitions. Cold prefill is timed once per start: the four-prompt and beside-the-stream times differ by less than the round-to-round spread of the sustained run at 2G, and the 100,000-token time differs by 2.0 s. No out-of-memory event occurs in the load tested. Startup is slower: the server is ready in 41 s with the default and in 44 to 107 s over the five starts with a smaller headroom (44, 53 and 107 s for the three at 2G). The hosts have 16 GiB of swap each, and the server uses more of it as the headroom shrinks: when serving begins, the three rank processes together hold 1.9 GiB of their own memory in swap with the default and 4.9 to 5.5 GiB at 2G and 1G.

--system-headroom KV tokens change minimum MemAvailable under load (lowest host)
default (6.08 GiB) 8.17 M — 9.1 GiB
4G 10.68 M +31 % 7.1 GiB
2G 13.06 M +60 % 5.6 GiB
1G 14.36 M +76 % 4.3 GiB

This is a capacity change for hosts where memory bounds the pool. It does nothing where an occupied limit, the KV ceiling or the prepared extent bounds it. The numbers are for this pull request alone, on a build of this branch with no other change.

Evidence

Conditions. Three hosts as above, connected over RoCE, one rank per host; DeepSeek V4.1 Flash core at tensor parallelism 3 for 4 concurrent requests, prepared from 6b68b9b plus the preparation-only change of #122 (compile headroom 4 GiB); 16 GiB of swap per host; disk Engram tables with the default 1 GiB row cache; --scheduler-policy throughput; prefix cache on disk only. The binary is this change on 6b68b9b built with --features roce; the branch has since been rebased onto 5fd4070, which adds only a Windows ARM64 build fix (#315), and checked to compile. Before each server start the page cache is dropped on all hosts. Every row of the table is one fresh server start that differs only in the headroom option, given to every rank.

Sizing. The load display shows the value reaching the sizing path. Rank zero's lines per start:

default  Plan cache.  per GPU: 99.71 GiB loaded · 6.86 GiB cache budget
4G       Plan cache.  per GPU: 99.80 GiB loaded · 8.93 GiB cache budget · system headroom 4GiB
2G       Plan cache.  per GPU: 99.89 GiB loaded · 10.91 GiB cache budget · system headroom 2GiB
1G       Plan cache.  per GPU: 99.86 GiB loaded · 11.98 GiB cache budget · system headroom 1GiB
key 2G   Plan cache.  per GPU: 99.73 GiB loaded · 11.09 GiB cache budget · system headroom 2GiB

Every start reports limited by memory. The budget grows by 2.07, 4.05 and 5.12 GiB for the 2.08, 4.08 and 5.08 GiB returned, within 0.04 GiB. The last line is a start with [memory] system_headroom = "2G" in a configuration file on every host and no flag: 13.28 M KV tokens, against 13.06 M with the flag; rank zero's budget is 0.18 GiB larger and its loaded memory 0.16 GiB smaller. With --system-headroom 200G every rank stops after GPU initialization, before loading the model. Rank zero reports system headroom is 214748364800 bytes, but only 121987493888 bytes are free after the driver margin, any reserve and the host allowance; the peers report the same message with their own free byte counts.

Load per start. Prose chat completions with one stream and with four streams (three repetitions each); four concurrent cold prompts of 120,000 random tokens each with 256 generated tokens; cold natural-text prompts of 20,000 tokens (6.2 to 7.1 s) and 100,000 tokens; a streamed chat completion during a cold 120,000-token prefill; four concurrent code completions (two repetitions). Each start lasts about eight minutes. MemAvailable is sampled every 2 s on every host; swap-out is the change in pswpout between the server becoming ready and the end of the load, so it excludes startup; NV_ERR_NO_MEMORY counts kernel log lines and OOM kills the change in oom_kill, on all hosts. The configuration-key start ran only the prose completions.

--system-headroom prose, 1 stream (tok/s) prose, 4 streams (tok/s) code, 4 streams, after the load (tok/s) 4 × 120k cold (s) 100k natural cold (s) swap-out during the run, all hosts (MiB) NV_ERR_NO_MEMORY / OOM kills
default 48.6 95.7 169.0 175.6 30.9 91 0 / 0
4G 46.5 96.9 168.4 173.9 32.8 307 0 / 0
2G 48.1 96.9 169.1 175.1 32.9 192 0 / 0
1G 47.5 95.9 169.0 169.3 30.9 381 0 / 0

The cold 120,000-token prefill beside the stream took 43.9, 44.5, 44.6 and 48.6 s (default, 4G, 2G, 1G), and the stream's longest stall was 2.5, 2.5, 3.7 and 2.9 s.

Startup. In the table's starts, swap use and startup time are larger with a smaller headroom, though 2G exceeds 1G in both (values for default, 4G, 2G, 1G). The rank processes' VmSwap, summed over the three hosts, was 1.9, 3.4, 5.5 and 5.4 GiB when serving began and 1.9, 2.9, 3.7 and 3.9 GiB at the end of the load. The server was ready 41, 47, 107 and 60 s after launch, and the slowest rank's cache allocation took 11.6, 15.5, 74.8 and 29.0 s. During the load the three hosts together took 81 k, 216 k, 489 k and 489 k major page faults. The two other starts at 2G were ready in 44 s (configuration key) and 53 s (sustained run), with slowest allocations of 13.7 and 21.0 s and 4.9 and 5.5 GiB of VmSwap when serving began; the 107 s and 74.8 s of the table's 2G start did not recur.

Sustained load at 2G. One server (13.08 M KV tokens) ran for 60 minutes through 10 rounds. Each round ran four concurrent cold 120,000-token prompts, the two natural-text prompts and a 78,072-token chat prompt (all three cold in the first round and restored from the prefix cache afterwards), prose and code completions on four streams, and a streamed chat completion during a cold 120,000-token prefill. Rounds 1, 4, 7 and 10 added a cold 250,000-token prompt and its repeat. Result:

  • 179 requests completed, none failed or cancelled, with 7.2 M prompt tokens prefilled.
  • MemAvailable never fell under 5.2 GiB on any host, and was 5.5 to 6.2 GiB at the end of every round, with no trend.
  • No NV_ERR_NO_MEMORY line and no OOM kill. Swap-out over the hour was 930 MiB across the three hosts, 716 MiB of it on rank zero's host.
  • The four concurrent prompts took 171.3 to 195.8 s per round (mean 184.1 s), against 175.1 s in the 2G row above: seven rounds took 181.2 to 195.8 s and three took 171.3 to 175.5 s.
  • The prefill beside the stream took 43.0 to 49.8 s per round, and the stream's longest stall was 1.8 to 3.8 s.
  • The ranks' VmSwap, summed over the three hosts, was 5.5 GiB when serving began, 3.9 GiB after the first round and 4.6 GiB from the sixth on.
  • The 250,000-token prompts took 87.1 to 94.2 s cold, and each repeat restored all 250,000 tokens with the same first-token log-probability.

Tests. Unit tests cover the budget under every limit form with the default, a configured size, zero and an oversized value; the refusal and its exemption when nothing is free; parsing in the Rust flag, the Rust configuration loader and the Python validator; precedence; forwarding to peers with and without their own keys; the host prefix-cache intake budget; and the display entry. Suites run on this branch: selene, selene-manager, helios (without a GPU), flare, flare-cli, and tests/cli/test_config.py with tests/cli/test_commands.py. No unit test loads a core, so the call sites that pass the setting into the check before loading, cache sizing and prefix-cache preparation are exercised only by the served runs. Thirteen tests fail, and the same thirteen fail with the same messages without this change (on a tree whose Rust sources equal 6b68b9b): seven helios template tests that need FLARE_TEST_TEMPLATE_DIR, three DeepSeek frontend-settings tests, two orbit::agent tests and selene-manager's record::tests::persisted_header_matches_the_descriptor_layout. Two selene-manager tests (a_freed_window_credit_is_granted_without_waiting_for_the_clock, d7_a_corrupt_put_and_a_cancelled_put_over_the_channel) fail intermittently on both trees under a CPU quota.

Not run.

  • A discrete GPU, where the setting changes only the host prefix-cache budgets and the display entry.
  • Windows and the desktop client's settings path.
  • flare run --hosts and flare orbit run, including a persistent group; the measured ranks were started one per host with the flag on each.
  • A headroom of 0.
  • Hosts whose other processes use significant memory; with the server stopped, resident anonymous memory on the measured hosts was 0.4 to 0.6 GiB. The load generator ran on rank zero's host.
  • A sustained run with the default headroom, so the spread of the four-prompt time across rounds has no baseline.
  • A host without swap, or with less than 16 GiB of it.
  • More than one start per headroom value under the full load. 2G was started three times (flag, configuration key, sustained run); the default, 4G and 1G once each.
  • A pool filled to capacity. The per-start load left about 1.1 M tokens in the pool at every headroom, and the sustained run ended its rounds with 1.5 M rising to 8.4 M of 13.08 M.
  • The refusal before loading combined with a --memory-limit.

🤖 Generated with Claude Code

On a unified-memory GPU, Helios sizes the KV page pool once at startup
from the memory that is free after the model loads. Before giving the
rest to the pool it sets aside a 64 MiB driver margin, 2 GiB for the
server's later host allocations, any configured reserve, and a twentieth
of total memory for the operating system and other processes. The
twentieth is a constant (`HOST_SYSTEM_SHARE` in `selene/src/offload.rs`)
and no setting lowers it: `--memory-limit=-SIZE` and `[memory] reserve`
only subtract more, and an occupied limit caps what the server occupies
without counting the twentieth. On a host that does nothing but serve,
the twentieth is much larger than what the rest of the system uses, and
memory is what bounds the pool. On three hosts with 121.6 GiB of unified
memory each (DGX Spark class), serving DeepSeek V4.1 Flash at tensor
parallelism 3 with one rank per host, the twentieth is 6.08 GiB per
host, while with the server stopped resident anonymous memory is 0.4 to
0.6 GiB. The right size depends on what else the host runs, which the
server cannot know, so it is made a host-level setting whose default
stays a twentieth.

`flare run` and `flare orbit run` accept `--system-headroom SIZE`, and
the configuration file accepts `[memory] system_headroom = "SIZE"`.
Precedence is the flag, then the key, then a twentieth of memory. `0`
leaves no headroom, and any other value below 1 MiB is rejected, so a
fraction such as `0.05` is not read as bytes. The shared settings that
application clients load apply the key as well, and Ignition's
configuration validator accepts and checks the key without using it.
Before this change the integrated-GPU cache budget (`budget` in
`helios/src/runtime_core/memory.rs`) and the host budget for
prefix-cache intake and pool staging (`host_pool_budget` in
`selene/src/offload.rs`) used the constant directly. Both now call
`flare_selene::offload::system_headroom`, which returns the configured
size or the twentieth. Cache sizing on a discrete GPU keeps no system
headroom and ignores the setting. On an integrated GPU, a configured
headroom at least as large as the memory free after the driver margin,
any reserve and the host allowance is refused by name, before the model
loads under every form of memory limit and again when caches are sized;
when nothing is free, the existing capacity error is reported. `flare
run --hosts` forwards the value rank zero resolved, from its flag or its
key, to every rank, as it does for `[memory] reserve`. `flare orbit run`
forwards only the flag, and each node otherwise applies its own key.
When the setting is given, the load display's cache plan shows it on an
integrated GPU or when the host prefix cache is enabled.
docs/cli-design.md describes the flag, the key, when to use a reserve
instead, which value each rank uses and the risk of a small value;
docs/model-preparation.md states the qualified configuration, and
docs/TODO.txt lists the configurations not run. With the setting absent,
every budget is computed as before.

Measured result, with this change on `6b68b9b4`, on the three hosts
above, which have 16 GiB of swap each: lowering the headroom raises KV
capacity by the memory returned. The pool holds 8.17 M KV tokens with
the default, 10.68 M with `4G` (+31 %), 13.06 M with `2G` (+60 %) and
14.36 M with `1G` (+76 %). With one start per value, decode throughput
shows no difference larger than the spread between repetitions, and no
out-of-memory event occurs in the load tested. Startup is slower: the
server is ready in 41 s with the default and in 44 to 107 s over the
five starts with a smaller headroom. The server uses more swap as the
headroom shrinks: when serving begins, the three rank processes together
hold 1.9 GiB of their own memory in swap with the default and 4.9 to 5.5
GiB at `2G` and `1G`. This is a capacity change for hosts where memory
bounds the pool; it does nothing where an occupied limit, the KV ceiling
or the prepared extent bounds it.

Evidence: each value is one fresh server start that differs only in the
headroom option, given to every rank, with the page cache dropped on all
hosts before the start. The core serves 4 concurrent requests, the
prefix cache is on disk only, and the binary is built with `--features
roce`. Rank zero's cache plan reports budgets of 6.86, 8.93, 10.91 and
11.98 GiB (default, `4G`, `2G`, `1G`), which grow by 2.07, 4.05 and 5.12
GiB for the 2.08, 4.08 and 5.08 GiB returned, and every start reports
`limited by memory`. A start with `[memory] system_headroom = "2G"` in a
configuration file on every host and no flag holds 13.28 M KV tokens.
With `--system-headroom 200G` every rank stops after GPU initialization,
before loading the model, with the refusal that names the headroom. Each
start runs about eight minutes of load: prose chat completions on one
and four streams, four concurrent code completions, four concurrent cold
prompts of 120,000 tokens, cold prompts of 20,000 and 100,000 tokens,
and a streamed chat completion during a cold 120,000-token prefill.
Minimum `MemAvailable` on the lowest host is 9.1, 7.1, 5.6 and 4.3 GiB,
the four cold prompts take 175.6, 173.9, 175.1 and 169.3 s, and no
`NV_ERR_NO_MEMORY` line or OOM kill occurs. One server at `2G` ran for
60 minutes through 10 rounds of a similar load: 179 requests completed
and none failed, `MemAvailable` never fell under 5.2 GiB on any host,
and swap-out over the hour was 930 MiB across the three hosts.

Validation and limitations: unit tests cover the budget under every
limit form with the default, a configured size, zero and an oversized
value; the refusal and its exemption when nothing is free; parsing in
the Rust flag, the Rust configuration loader and the Python validator;
precedence; forwarding to peers with and without their own keys; the
host prefix-cache intake budget; and the display entry. The selene,
selene-manager, helios (without a GPU), flare and flare-cli suites and
tests/cli/test_config.py with tests/cli/test_commands.py were run.
Thirteen tests fail, and the same thirteen fail with the same messages
on a tree whose Rust sources equal `6b68b9b4`: seven helios template
tests that need `FLARE_TEST_TEMPLATE_DIR`, three DeepSeek
frontend-settings tests, two `orbit::agent` tests and selene-manager's
`record::tests::persisted_header_matches_the_descriptor_layout`. Two
selene-manager tests fail intermittently on both trees under a CPU
quota. No unit test loads a core, so the call sites that pass the
setting into the check before loading, cache sizing and prefix-cache
preparation are exercised only by the served runs. Not run: a discrete
GPU; Windows and the desktop client's settings path; `flare run --hosts`
and `flare orbit run` (the measured ranks were started one per host with
the flag on each); a headroom of `0`; a host without swap, or with less
than 16 GiB of it; hosts whose other processes use significant memory;
and more than one start per value under the full load, except `2G`,
which was started three times.

Design-Signed-Off-By: Claude Fable 5.2; design-review=ed22
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Signed-off-by: Javier López <59578422+javieratomic@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant