Repository navigation
Prepare and serve Inkling 2 Small NVFP4 at TP4 on four DGX Sparks - #304
Open
original-el8 wants to merge 3 commits into
Open
original-el8 wants to merge 3 commits into
original-el8 wants to merge 3 commits into
Conversation
The Inkling 2 Small NVFP4 recipe prepared only on four GPUs of one host: it rejected a multi-host rendezvous, built its collectives on CUDA peer memory, and admitted compute capability 12.0 alone. Each rank holds a quarter of the 344 GB checkpoint, which fits a DGX Spark (GB10, 121 GiB unified memory), so four Sparks joined by RoCE can serve the model once the recipe places one rank per host. The recipe follows the multi-host shape the DeepSeek V4 recipe uses (51864a2, #240): host placement and the ring or chain choice travel in PreparationConfig, and one RoCEnante runtime replaces the per-role peer-memory channels when the rank group spans hosts. RoCEnante reductions carry BF16 only, so cross-host ranks round each partial sum to BF16 before reducing; the single-host path keeps its FP32 reductions, because that precision is available there and its prepared graphs must not change. What changed, in ignition/recipes/inkling2. Before, recipe.prepare raised when given `ranks` and always used process-local GPUs 0 through 3. Now, with `ranks`, each rank uses GPU 0 of its own host, and hosts, roce_ring and roce_chain flow from the plan into PreparationConfig, which requires four distinct (host, GPU) placements and validates the routing shape with ignition.planning.host_shape_error. bootstrap._collective_runtimes allocates the channels: when the coordinator spans hosts, one RoCEnante runtime carries the hidden reductions and the vocabulary, sampling and log-probability gathers, sized from the decode rows, the drafter's block of every request, the prefill chunk and each rank's vocabulary partition; otherwise it creates the same all-reduce and all-gather peer-memory runtimes with the same arguments as before. The shared runtime is closed once. decoder.reduction_dtype selects BF16 for a RoCE runtime and FP32 otherwise, and the embedding, attention and expert partial sums are cast to it before each reduction. prepare_rank admits compute capability (12, 1) beside (12, 0); the exact check existed because the recipe had been tested on RTX PRO 6000 alone. On an integrated GPU it releases the checkpoint's page cache after loading weights, since unified memory counts those pages against the GPU. The request limit stays 8 and the declared defaults are unchanged. docs/model-preparation.md gives the four-host export, join and serve commands and the qualified configuration, docs/roce-ring.md lists the recipe among multi-host preparations, and docs/TODO.txt records the unrun comparisons. The result is that `ignition export --ranks 4` with three `ignition join` peers prepares this recipe on four hosts, and `flare run` serves the core across them. Single-host preparation builds the same channels and FP32 reductions as before. Evidence, on four DGX Sparks (GB10, compute capability 12.1, one GPU each), RoCE over ConnectX-7 with mesh routing, the checkpoint on rank zero only and peers reading their tensor ranges from it (80.27 GiB each, about 2.3 Gbit/s). Before this change the recipe rejected `ranks`. After it, a reduced core (--no-mtp --language-model-only --max-concurrency 1 --prefill-chunk-size 512 --autotune-effort off --no-compile, no memory reserve) exported 8 graphs in 18 min 43 s and served across the four hosts: 82.91 GiB loaded per rank and a 24.5 GiB cache with 6,414,336 KV tokens. A chat completion, a streamed completion (90 events), two requests against the one slot, and a 7,150-token lookup prefilled in fourteen 512-token chunks returned the expected answers, and SIGTERM shutdown completed in 3.5 s. The RoCE loss counters (out_of_sequence, packet_seq_err, local_ack_timeout_err, rnr_nak_retry_err, summed over sixteen ports) did not change during export or serving. tests/recipes/test_inkling2_hosts.py (14 passed, CPU only) covers placement and routing validation, the plan fitting four GB10 hosts, dispatch of the plan to the rank group, the RoCE runtime's capacities, the unchanged peer-memory channel arguments, and the reduction element type. With tests/cli/test_inkling2.py, tests/cli/test_planning.py and tests/recipes/test_declarations.py: 249 passed, 9 failed, 14 skipped; the same three files on the parent commit give 235 passed, 14 skipped and the same 9 failures (Qwen declaration assertions, a missing Kimi checkpoint directory, DeepSeek V4.1 and Muse Spark collective-policy cases, and the collective ladder round trip). Limitations. An export of the declared eight-request configuration with drafting, vision, tuning and compilation was started on the four Sparks and stopped at graph construction after about seven minutes; it has not completed there. Not run: a numerical comparison of BF16 cross-host reductions with single-host FP32 output, the server tests of tests/server/test_inkling2.py on Sparks, ring and chain routing, and single-host GPU preparation after this change. Design-Signed-Off-By: flare-02; design-review=ed22
The Inkling 2 Small NVFP4 recipe rejected more than eight concurrent requests: PreparationConfig bounded the request capacity to [1, 8] and the recipe declared max_concurrency=8. Four DGX Sparks hold enough memory beside the weights for a 32-request core with 8192-token prefill chunks, and nothing in the recipe depends on eight. Request slots, batch capacities, pool rows, DSpark scratch, vision slots, gather rows, prefill scratch and the page count already derive from the configured request count, so raising the bound is sufficient and no capacity needed a separate change. What changed. Before, bootstrap.PreparationConfig raised for max_active above 8 and planning rejected --max-concurrency above 8 for this recipe. Now the bound is [1, 32] and the recipe declares max_concurrency=32. The default concurrency (8), default prefill chunk (4096) and default verification capacity (64 rows, 64-graph budget) are unchanged, so an export without flags prepares the same catalog as before. tests/server/test_inkling2.py no longer pins eight requests and 4096-token chunks: it accepts a service with at least eight requests and 4096- or 8192-token chunks and scales its chunk-crossing prompts with the chunk size the service reports. tests/cli/test_inkling2.py asserts the declared maximum, and tests/recipes/test_inkling2_hosts.py adds the 32-request bound, round shapes, RoCE capacities and planning fit. docs/model-preparation.md documents the 32-request, 8192-token DGX Spark configuration in place of the eight-request one, and docs/TODO.txt replaces the pending eight-request Spark export with the planner estimate described below. The result is that the recipe prepares and serves up to 32 concurrent requests. On four DGX Sparks the 32-request, 8192-token configuration with seven drafts and vision exports, loads with 7.6 GiB or more available per host, and answers 32 concurrent requests correctly. These are functional results; no throughput was measured. Evidence, on four DGX Sparks (GB10, 121 GiB unified memory each, one rank per host, RoCE over ConnectX-7 with mesh routing). Before this change the configuration could not be requested. After it, export with --max-concurrency 32 --image-concurrency 4 --prefill-chunk-size 8192 --mtp-draft-tokens 7 --speculative-token-capacity 128 --speculative-graph-budget 64 --autotune-effort default and a 4 GiB memory reserve took 1 h 25 min 30 s and produced 152 graphs in a 351 GB core (350,544,415,417 bytes): kernel tuning 675 s, compilation 1,893 s, compiled warmup 586 s, capture 485 s, export 1,078 s. The lowest MemAvailable during export was 11.5, 12.6, 12.5 and 13.1 GiB on ranks 0 through 3, and the graph pool is 2,840 MiB per rank. Verification capacity 128 with graph budget 64 is the value the DeepSeek V4.1 recipe declares for 32 requests and seven drafts. Serving loaded 90.06 to 90.24 GiB per rank. With the exported 4 GiB reserve the cache was 12.4 GiB with 2,392,064 KV tokens and 78 shared snapshot slots, leaving 12.0 to 12.5 GiB MemAvailable. With --memory-limit=-0G it was 16.5 GiB with 3,252,224 KV tokens and 106 snapshot slots, leaving 7.76 to 8.26 GiB after loading and 7.64 GiB at the lowest sample during 32 concurrent requests. On that service, tests/server/test_inkling2.py -k default-effort passed 7 cases: eight concurrent requests, streamed reasoning, prefix reuse and branching across a chunk, a tool call, a structured response, images across a chunk, and four images with four concurrent image requests. 16 and 32 concurrent exact-repeat requests were all correct, and a streamed completion succeeded. An 18,832-token lookup was answered correctly in four of four runs with a 2048-token output limit and in one of two runs with a 512-token limit; the other run spent all 512 tokens in reasoning. Its cause was not investigated, and docs/TODO.txt already records an intermittent long-prefix output-limit failure for this recipe. The RoCE loss counters did not change and earlyoom, the hosts' out-of-memory daemon, killed no process. CPU tests: tests/recipes/test_inkling2_hosts.py 18 passed; with tests/cli/test_inkling2.py, tests/cli/test_planning.py and tests/recipes/test_declarations.py, 253 passed, 9 failed, 14 skipped, the 9 failures being the ones the same files produce on main (Qwen declaration assertions, a missing Kimi checkpoint directory, DeepSeek V4.1 and Muse Spark collective-policy cases, and the collective ladder round trip). Limitations. The planner estimated 22.9 GiB of KV pages per rank where serving allocated 12.4 GiB at the same reserve; docs/TODO.txt records the correction of the declared rank costs. Not run: the 32-request configuration on RTX PRO 6000, a 16-request core, the eight-request configuration on DGX Sparks, the effort-zero server cases, and benchmarks. Design-Signed-Off-By: flare-02; design-review=ed22
This was referenced Nov 15, 2026
When `flare run` starts a multi-host core, rank zero reads every peer's rank
core through the page cache and sends it over TCP. On a host whose GPU
allocates from system memory, such as a DGX Spark (GB10), those cached pages
are not free memory for CUDA, so the pages of several rank cores can keep rank
zero from creating its CUDA context. Helios already releases the page cache of
a rank's own core files before it creates the context
(Files::release_page_cache in helios/src/runtime_core/model.rs), and a
receiving rank syncs the files it writes; the files rank zero reads on behalf
of other ranks were never released. Releasing them where the distribution
source stops closes that gap with the same eos operation and needs no change
to loading or to the transfer protocol.
Before this change the distribution source in flare-cli/src/distribution.rs
returned when its serving threads had joined and left the pages it had read in
the cache. Now release_served_pages calls flare_eos::release_page_cache
(POSIX_FADV_DONTNEED on Linux, a no-op on Windows) on every file of ranks one
and above when the source thread finishes, whether distribution succeeded or
failed. Rank zero's own files are left to the existing release in Helios.
flare-cli gains a path dependency on the in-tree eos crate, which the lock
file records.
The result is that a rank-zero host returns the memory used to read peer
cores before GPU initialization, without a manual page-cache drop.
Evidence, on four DGX Sparks (121 GiB unified memory each, one rank per host)
serving a 351 GB four-rank Inkling 2 Small NVFP4 core. Before: after rank
zero distributed three rank cores, its host showed 114 GiB of page cache and
2 GiB free, and rank zero failed at "Initialize GPU" with CudaError {
operation: "cuCtxCreate_v2", code: 2, name: "CUDA_ERROR_OUT_OF_MEMORY" }. The
failure was intermittent: the first launch of a smaller core went through the
same distribution without it. After, with rebuilt binaries on all four hosts,
the peers' rank cores removed and no manual cache drop: three rank cores were
sent in 70.2 s, /proc/meminfo on rank zero at the end of distribution showed
MemAvailable 115.9 GiB and Cached 0.12 GiB, and GPU initialization succeeded.
The Linux test
source_releases_the_page_cache_of_peer_core_files_when_it_finishes serves a
4 MiB on-disk file to one peer and checks with mincore that the file has
resident pages after the transfer and none after the source finishes.
cargo test --release --manifest-path flare-cli/Cargo.toml distribution: 30
passed, 0 failed, 2 ignored (they need external fixtures). Not run: other
models, two- and three-rank cores, and hosts with discrete GPUs, where the
release changes only the page cache.
Design-Signed-Off-By: flare-02; design-review=ed22
This was referenced Nov 15, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Inkling 2 Small NVFP4 could not be prepared or served at tensor parallelism 4 with one rank per host. This change makes it export, load and serve on four DGX Sparks at up to 32 concurrent requests with 8192-token prefill chunks. It has three commits, each described below with its own reasoning, change, result and evidence, in commit order.
1. Prepare Inkling 2 Small NVFP4 across four hosts for DGX Spark TP4
The Inkling 2 Small NVFP4 recipe prepared only on four GPUs of one host: it rejected a multi-host rendezvous, built its collectives on CUDA peer memory, and admitted compute capability 12.0 alone. Each rank holds a quarter of the 344 GB checkpoint, which fits a DGX Spark (GB10, 121 GiB unified memory), so four Sparks joined by RoCE can serve the model once the recipe places one rank per host. The recipe follows the multi-host shape the DeepSeek V4 recipe uses (51864a2, #240): host placement and the ring or chain choice travel in PreparationConfig, and one RoCEnante runtime replaces the per-role peer-memory channels when the rank group spans hosts. RoCEnante reductions carry BF16 only, so cross-host ranks round each partial sum to BF16 before reducing; the single-host path keeps its FP32 reductions, because that precision is available there and its prepared graphs must not change.
What changed, in ignition/recipes/inkling2. Before, recipe.prepare raised when given
ranksand always used process-local GPUs 0 through 3. Now, withranks, each rank uses GPU 0 of its own host, and hosts, roce_ring and roce_chain flow from the plan into PreparationConfig, which requires four distinct (host, GPU) placements and validates the routing shape with ignition.planning.host_shape_error. bootstrap._collective_runtimes allocates the channels: when the coordinator spans hosts, one RoCEnante runtime carries the hidden reductions and the vocabulary, sampling and log-probability gathers, sized from the decode rows, the drafter's block of every request, the prefill chunk and each rank's vocabulary partition; otherwise it creates the same all-reduce and all-gather peer-memory runtimes with the same arguments as before. The shared runtime is closed once. decoder.reduction_dtype selects BF16 for a RoCE runtime and FP32 otherwise, and the embedding, attention and expert partial sums are cast to it before each reduction. prepare_rank admits compute capability (12, 1) beside (12, 0); the exact check existed because the recipe had been tested on RTX PRO 6000 alone. On an integrated GPU it releases the checkpoint's page cache after loading weights, since unified memory counts those pages against the GPU. The request limit stays 8 and the declared defaults are unchanged. docs/model-preparation.md gives the four-host export, join and serve commands and the qualified configuration, docs/roce-ring.md lists the recipe among multi-host preparations, and docs/TODO.txt records the unrun comparisons.The result is that
ignition export --ranks 4with threeignition joinpeers prepares this recipe on four hosts, andflare runserves the core across them. Single-host preparation builds the same channels and FP32 reductions as before.Evidence, on four DGX Sparks (GB10, compute capability 12.1, one GPU each), RoCE over ConnectX-7 with mesh routing, the checkpoint on rank zero only and peers reading their tensor ranges from it (80.27 GiB each, about 2.3 Gbit/s). Before this change the recipe rejected
ranks. After it, a reduced core (--no-mtp --language-model-only --max-concurrency 1 --prefill-chunk-size 512 --autotune-effort off --no-compile, no memory reserve) exported 8 graphs in 18 min 43 s and served across the four hosts: 82.91 GiB loaded per rank and a 24.5 GiB cache with 6,414,336 KV tokens. A chat completion, a streamed completion (90 events), two requests against the one slot, and a 7,150-token lookup prefilled in fourteen 512-token chunks returned the expected answers, and SIGTERM shutdown completed in 3.5 s. The RoCE loss counters (out_of_sequence, packet_seq_err, local_ack_timeout_err, rnr_nak_retry_err, summed over sixteen ports) did not change during export or serving. tests/recipes/test_inkling2_hosts.py (14 passed, CPU only) covers placement and routing validation, the plan fitting four GB10 hosts, dispatch of the plan to the rank group, the RoCE runtime's capacities, the unchanged peer-memory channel arguments, and the reduction element type. With tests/cli/test_inkling2.py, tests/cli/test_planning.py and tests/recipes/test_declarations.py: 249 passed, 9 failed, 14 skipped; the same three files onmaingive 235 passed, 14 skipped and the same 9 failures (Qwen declaration assertions, a missing Kimi checkpoint directory, DeepSeek V4.1 and Muse Spark collective-policy cases, and the collective ladder round trip).Limitations. An export of the declared eight-request configuration with drafting, vision, tuning and compilation was started on the four Sparks and stopped at graph construction after about seven minutes; it has not completed there. Not run: a numerical comparison of BF16 cross-host reductions with single-host FP32 output, the server tests of tests/server/test_inkling2.py on Sparks, ring and chain routing, and single-host GPU preparation after this change.
2. Raise the Inkling 2 Small NVFP4 request limit to 32
The Inkling 2 Small NVFP4 recipe rejected more than eight concurrent requests: PreparationConfig bounded the request capacity to [1, 8] and the recipe declared max_concurrency=8. Four DGX Sparks hold enough memory beside the weights for a 32-request core with 8192-token prefill chunks, and nothing in the recipe depends on eight. Request slots, batch capacities, pool rows, DSpark scratch, vision slots, gather rows, prefill scratch and the page count already derive from the configured request count, so raising the bound is sufficient and no capacity needed a separate change.
What changed. Before, bootstrap.PreparationConfig raised for max_active above 8 and planning rejected --max-concurrency above 8 for this recipe. Now the bound is [1, 32] and the recipe declares max_concurrency=32. The default concurrency (8), default prefill chunk (4096) and default verification capacity (64 rows, 64-graph budget) are unchanged, so an export without flags prepares the same catalog as before. tests/server/test_inkling2.py no longer pins eight requests and 4096-token chunks: it accepts a service with at least eight requests and 4096- or 8192-token chunks and scales its chunk-crossing prompts with the chunk size the service reports. tests/cli/test_inkling2.py asserts the declared maximum, and tests/recipes/test_inkling2_hosts.py adds the 32-request bound, round shapes, RoCE capacities and planning fit. docs/model-preparation.md documents the 32-request, 8192-token DGX Spark configuration in place of the eight-request one, and docs/TODO.txt replaces the pending eight-request Spark export with the planner estimate described below.
The result is that the recipe prepares and serves up to 32 concurrent requests. On four DGX Sparks the 32-request, 8192-token configuration with seven drafts and vision exports, loads with 7.6 GiB or more available per host, and answers 32 concurrent requests correctly. These are functional results; no throughput was measured.
Evidence, on four DGX Sparks (GB10, 121 GiB unified memory each, one rank per host, RoCE over ConnectX-7 with mesh routing). Before this change the configuration could not be requested. After it, export with --max-concurrency 32 --image-concurrency 4 --prefill-chunk-size 8192 --mtp-draft-tokens 7 --speculative-token-capacity 128 --speculative-graph-budget 64 --autotune-effort default and a 4 GiB memory reserve took 1 h 25 min 30 s and produced 152 graphs in a 351 GB core (350,544,415,417 bytes): kernel tuning 675 s, compilation 1,893 s, compiled warmup 586 s, capture 485 s, export 1,078 s. The lowest MemAvailable during export was 11.5, 12.6, 12.5 and 13.1 GiB on ranks 0 through 3, and the graph pool is 2,840 MiB per rank. Verification capacity 128 with graph budget 64 is the value the DeepSeek V4.1 recipe declares for 32 requests and seven drafts. Serving loaded 90.06 to 90.24 GiB per rank. With the exported 4 GiB reserve the cache was 12.4 GiB with 2,392,064 KV tokens and 78 shared snapshot slots, leaving 12.0 to 12.5 GiB MemAvailable. With --memory-limit=-0G it was 16.5 GiB with 3,252,224 KV tokens and 106 snapshot slots, leaving 7.76 to 8.26 GiB after loading and 7.64 GiB at the lowest sample during 32 concurrent requests. On that service, tests/server/test_inkling2.py -k default-effort passed 7 cases: eight concurrent requests, streamed reasoning, prefix reuse and branching across a chunk, a tool call, a structured response, images across a chunk, and four images with four concurrent image requests. 16 and 32 concurrent exact-repeat requests were all correct, and a streamed completion succeeded. An 18,832-token lookup was answered correctly in four of four runs with a 2048-token output limit and in one of two runs with a 512-token limit; the other run spent all 512 tokens in reasoning. Its cause was not investigated, and docs/TODO.txt already records an intermittent long-prefix output-limit failure for this recipe. The RoCE loss counters did not change and earlyoom, the hosts' out-of-memory daemon, killed no process. CPU tests: tests/recipes/test_inkling2_hosts.py 18 passed; with tests/cli/test_inkling2.py, tests/cli/test_planning.py and tests/recipes/test_declarations.py, 253 passed, 9 failed, 14 skipped, the 9 failures being the ones the same files produce on main (Qwen declaration assertions, a missing Kimi checkpoint directory, DeepSeek V4.1 and Muse Spark collective-policy cases, and the collective ladder round trip).
Limitations. The planner estimated 22.9 GiB of KV pages per rank where serving allocated 12.4 GiB at the same reserve; docs/TODO.txt records the correction of the declared rank costs. Not run: the 32-request configuration on RTX PRO 6000, a 16-request core, the eight-request configuration on DGX Sparks, the effort-zero server cases, and benchmarks.
3. Release the page cache of rank cores sent to peer ranks
This commit is shared runtime behaviour in
flare runfor every multi-host core; it is here because a cold launch of the 32-request Inkling 2 core on DGX Sparks fails GPU initialization on rank zero without it.When
flare runstarts a multi-host core, rank zero reads every peer's rank core through the page cache and sends it over TCP. On a host whose GPU allocates from system memory, such as a DGX Spark (GB10), those cached pages are not free memory for CUDA, so the pages of several rank cores can keep rank zero from creating its CUDA context. Helios already releases the page cache of a rank's own core files before it creates the context (Files::release_page_cache in helios/src/runtime_core/model.rs), and a receiving rank syncs the files it writes; the files rank zero reads on behalf of other ranks were never released. Releasing them where the distribution source stops closes that gap with the same eos operation and needs no change to loading or to the transfer protocol.Before this change the distribution source in flare-cli/src/distribution.rs returned when its serving threads had joined and left the pages it had read in the cache. Now release_served_pages calls flare_eos::release_page_cache (POSIX_FADV_DONTNEED on Linux, a no-op on Windows) on every file of ranks one and above when the source thread finishes, whether distribution succeeded or failed. Rank zero's own files are left to the existing release in Helios. flare-cli gains a path dependency on the in-tree eos crate, which the lock file records.
The result is that a rank-zero host returns the memory used to read peer cores before GPU initialization, without a manual page-cache drop.
Evidence, on four DGX Sparks (121 GiB unified memory each, one rank per host) serving a 351 GB four-rank Inkling 2 Small NVFP4 core. Before: after rank zero distributed three rank cores, its host showed 114 GiB of page cache and 2 GiB free, and rank zero failed at "Initialize GPU" with CudaError { operation: "cuCtxCreate_v2", code: 2, name: "CUDA_ERROR_OUT_OF_MEMORY" }. The failure was intermittent: the first launch of a smaller core went through the same distribution without it. After, with rebuilt binaries on all four hosts, the peers' rank cores removed and no manual cache drop: three rank cores were sent in 70.2 s, /proc/meminfo on rank zero at the end of distribution showed MemAvailable 115.9 GiB and Cached 0.12 GiB, and GPU initialization succeeded. The Linux test source_releases_the_page_cache_of_peer_core_files_when_it_finishes serves a 4 MiB on-disk file to one peer and checks with mincore that the file has resident pages after the transfer and none after the source finishes. cargo test --release --manifest-path flare-cli/Cargo.toml distribution: 30 passed, 0 failed, 2 ignored (they need external fixtures). Not run: other models, two- and three-rank cores, and hosts with discrete GPUs, where the release changes only the page cache.