Skip to content

Prepare and serve Inkling 2 Small NVFP4 at TP4 on four DGX Sparks - #304

Open
original-el8 wants to merge 3 commits into
mainfrom
inkling2/dgx-spark-tp4
Open

original-el8 wants to merge 3 commits into
mainfrom
inkling2/dgx-spark-tp4

Conversation

@original-el8

@original-el8 original-el8 commented Nov 15, 2026 •

Copy link
Copy Markdown
Collaborator

Inkling 2 Small NVFP4 could not be prepared or served at tensor parallelism 4 with one rank per host. This change makes it export, load and serve on four DGX Sparks at up to 32 concurrent requests with 8192-token prefill chunks. It has three commits, each described below with its own reasoning, change, result and evidence, in commit order.

1. Prepare Inkling 2 Small NVFP4 across four hosts for DGX Spark TP4

The Inkling 2 Small NVFP4 recipe prepared only on four GPUs of one host: it rejected a multi-host rendezvous, built its collectives on CUDA peer memory, and admitted compute capability 12.0 alone. Each rank holds a quarter of the 344 GB checkpoint, which fits a DGX Spark (GB10, 121 GiB unified memory), so four Sparks joined by RoCE can serve the model once the recipe places one rank per host. The recipe follows the multi-host shape the DeepSeek V4 recipe uses (51864a2, #240): host placement and the ring or chain choice travel in PreparationConfig, and one RoCEnante runtime replaces the per-role peer-memory channels when the rank group spans hosts. RoCEnante reductions carry BF16 only, so cross-host ranks round each partial sum to BF16 before reducing; the single-host path keeps its FP32 reductions, because that precision is available there and its prepared graphs must not change.

What changed, in ignition/recipes/inkling2. Before, recipe.prepare raised when given ranks and always used process-local GPUs 0 through 3. Now, with ranks, each rank uses GPU 0 of its own host, and hosts, roce_ring and roce_chain flow from the plan into PreparationConfig, which requires four distinct (host, GPU) placements and validates the routing shape with ignition.planning.host_shape_error. bootstrap._collective_runtimes allocates the channels: when the coordinator spans hosts, one RoCEnante runtime carries the hidden reductions and the vocabulary, sampling and log-probability gathers, sized from the decode rows, the drafter's block of every request, the prefill chunk and each rank's vocabulary partition; otherwise it creates the same all-reduce and all-gather peer-memory runtimes with the same arguments as before. The shared runtime is closed once. decoder.reduction_dtype selects BF16 for a RoCE runtime and FP32 otherwise, and the embedding, attention and expert partial sums are cast to it before each reduction. prepare_rank admits compute capability (12, 1) beside (12, 0); the exact check existed because the recipe had been tested on RTX PRO 6000 alone. On an integrated GPU it releases the checkpoint's page cache after loading weights, since unified memory counts those pages against the GPU. The request limit stays 8 and the declared defaults are unchanged. docs/model-preparation.md gives the four-host export, join and serve commands and the qualified configuration, docs/roce-ring.md lists the recipe among multi-host preparations, and docs/TODO.txt records the unrun comparisons.

The result is that ignition export --ranks 4 with three ignition join peers prepares this recipe on four hosts, and flare run serves the core across them. Single-host preparation builds the same channels and FP32 reductions as before.

Evidence, on four DGX Sparks (GB10, compute capability 12.1, one GPU each), RoCE over ConnectX-7 with mesh routing, the checkpoint on rank zero only and peers reading their tensor ranges from it (80.27 GiB each, about 2.3 Gbit/s). Before this change the recipe rejected ranks. After it, a reduced core (--no-mtp --language-model-only --max-concurrency 1 --prefill-chunk-size 512 --autotune-effort off --no-compile, no memory reserve) exported 8 graphs in 18 min 43 s and served across the four hosts: 82.91 GiB loaded per rank and a 24.5 GiB cache with 6,414,336 KV tokens. A chat completion, a streamed completion (90 events), two requests against the one slot, and a 7,150-token lookup prefilled in fourteen 512-token chunks returned the expected answers, and SIGTERM shutdown completed in 3.5 s. The RoCE loss counters (out_of_sequence, packet_seq_err, local_ack_timeout_err, rnr_nak_retry_err, summed over sixteen ports) did not change during export or serving. tests/recipes/test_inkling2_hosts.py (14 passed, CPU only) covers placement and routing validation, the plan fitting four GB10 hosts, dispatch of the plan to the rank group, the RoCE runtime's capacities, the unchanged peer-memory channel arguments, and the reduction element type. With tests/cli/test_inkling2.py, tests/cli/test_planning.py and tests/recipes/test_declarations.py: 249 passed, 9 failed, 14 skipped; the same three files on main give 235 passed, 14 skipped and the same 9 failures (Qwen declaration assertions, a missing Kimi checkpoint directory, DeepSeek V4.1 and Muse Spark collective-policy cases, and the collective ladder round trip).

Limitations. An export of the declared eight-request configuration with drafting, vision, tuning and compilation was started on the four Sparks and stopped at graph construction after about seven minutes; it has not completed there. Not run: a numerical comparison of BF16 cross-host reductions with single-host FP32 output, the server tests of tests/server/test_inkling2.py on Sparks, ring and chain routing, and single-host GPU preparation after this change.

2. Raise the Inkling 2 Small NVFP4 request limit to 32

The Inkling 2 Small NVFP4 recipe rejected more than eight concurrent requests: PreparationConfig bounded the request capacity to [1, 8] and the recipe declared max_concurrency=8. Four DGX Sparks hold enough memory beside the weights for a 32-request core with 8192-token prefill chunks, and nothing in the recipe depends on eight. Request slots, batch capacities, pool rows, DSpark scratch, vision slots, gather rows, prefill scratch and the page count already derive from the configured request count, so raising the bound is sufficient and no capacity needed a separate change.

What changed. Before, bootstrap.PreparationConfig raised for max_active above 8 and planning rejected --max-concurrency above 8 for this recipe. Now the bound is [1, 32] and the recipe declares max_concurrency=32. The default concurrency (8), default prefill chunk (4096) and default verification capacity (64 rows, 64-graph budget) are unchanged, so an export without flags prepares the same catalog as before. tests/server/test_inkling2.py no longer pins eight requests and 4096-token chunks: it accepts a service with at least eight requests and 4096- or 8192-token chunks and scales its chunk-crossing prompts with the chunk size the service reports. tests/cli/test_inkling2.py asserts the declared maximum, and tests/recipes/test_inkling2_hosts.py adds the 32-request bound, round shapes, RoCE capacities and planning fit. docs/model-preparation.md documents the 32-request, 8192-token DGX Spark configuration in place of the eight-request one, and docs/TODO.txt replaces the pending eight-request Spark export with the planner estimate described below.

The result is that the recipe prepares and serves up to 32 concurrent requests. On four DGX Sparks the 32-request, 8192-token configuration with seven drafts and vision exports, loads with 7.6 GiB or more available per host, and answers 32 concurrent requests correctly. These are functional results; no throughput was measured.

Evidence, on four DGX Sparks (GB10, 121 GiB unified memory each, one rank per host, RoCE over ConnectX-7 with mesh routing). Before this change the configuration could not be requested. After it, export with --max-concurrency 32 --image-concurrency 4 --prefill-chunk-size 8192 --mtp-draft-tokens 7 --speculative-token-capacity 128 --speculative-graph-budget 64 --autotune-effort default and a 4 GiB memory reserve took 1 h 25 min 30 s and produced 152 graphs in a 351 GB core (350,544,415,417 bytes): kernel tuning 675 s, compilation 1,893 s, compiled warmup 586 s, capture 485 s, export 1,078 s. The lowest MemAvailable during export was 11.5, 12.6, 12.5 and 13.1 GiB on ranks 0 through 3, and the graph pool is 2,840 MiB per rank. Verification capacity 128 with graph budget 64 is the value the DeepSeek V4.1 recipe declares for 32 requests and seven drafts. Serving loaded 90.06 to 90.24 GiB per rank. With the exported 4 GiB reserve the cache was 12.4 GiB with 2,392,064 KV tokens and 78 shared snapshot slots, leaving 12.0 to 12.5 GiB MemAvailable. With --memory-limit=-0G it was 16.5 GiB with 3,252,224 KV tokens and 106 snapshot slots, leaving 7.76 to 8.26 GiB after loading and 7.64 GiB at the lowest sample during 32 concurrent requests. On that service, tests/server/test_inkling2.py -k default-effort passed 7 cases: eight concurrent requests, streamed reasoning, prefix reuse and branching across a chunk, a tool call, a structured response, images across a chunk, and four images with four concurrent image requests. 16 and 32 concurrent exact-repeat requests were all correct, and a streamed completion succeeded. An 18,832-token lookup was answered correctly in four of four runs with a 2048-token output limit and in one of two runs with a 512-token limit; the other run spent all 512 tokens in reasoning. Its cause was not investigated, and docs/TODO.txt already records an intermittent long-prefix output-limit failure for this recipe. The RoCE loss counters did not change and earlyoom, the hosts' out-of-memory daemon, killed no process. CPU tests: tests/recipes/test_inkling2_hosts.py 18 passed; with tests/cli/test_inkling2.py, tests/cli/test_planning.py and tests/recipes/test_declarations.py, 253 passed, 9 failed, 14 skipped, the 9 failures being the ones the same files produce on main (Qwen declaration assertions, a missing Kimi checkpoint directory, DeepSeek V4.1 and Muse Spark collective-policy cases, and the collective ladder round trip).

Limitations. The planner estimated 22.9 GiB of KV pages per rank where serving allocated 12.4 GiB at the same reserve; docs/TODO.txt records the correction of the declared rank costs. Not run: the 32-request configuration on RTX PRO 6000, a 16-request core, the eight-request configuration on DGX Sparks, the effort-zero server cases, and benchmarks.

3. Release the page cache of rank cores sent to peer ranks

This commit is shared runtime behaviour in flare run for every multi-host core; it is here because a cold launch of the 32-request Inkling 2 core on DGX Sparks fails GPU initialization on rank zero without it.

When flare run starts a multi-host core, rank zero reads every peer's rank core through the page cache and sends it over TCP. On a host whose GPU allocates from system memory, such as a DGX Spark (GB10), those cached pages are not free memory for CUDA, so the pages of several rank cores can keep rank zero from creating its CUDA context. Helios already releases the page cache of a rank's own core files before it creates the context (Files::release_page_cache in helios/src/runtime_core/model.rs), and a receiving rank syncs the files it writes; the files rank zero reads on behalf of other ranks were never released. Releasing them where the distribution source stops closes that gap with the same eos operation and needs no change to loading or to the transfer protocol.

Before this change the distribution source in flare-cli/src/distribution.rs returned when its serving threads had joined and left the pages it had read in the cache. Now release_served_pages calls flare_eos::release_page_cache (POSIX_FADV_DONTNEED on Linux, a no-op on Windows) on every file of ranks one and above when the source thread finishes, whether distribution succeeded or failed. Rank zero's own files are left to the existing release in Helios. flare-cli gains a path dependency on the in-tree eos crate, which the lock file records.

The result is that a rank-zero host returns the memory used to read peer cores before GPU initialization, without a manual page-cache drop.

Evidence, on four DGX Sparks (121 GiB unified memory each, one rank per host) serving a 351 GB four-rank Inkling 2 Small NVFP4 core. Before: after rank zero distributed three rank cores, its host showed 114 GiB of page cache and 2 GiB free, and rank zero failed at "Initialize GPU" with CudaError { operation: "cuCtxCreate_v2", code: 2, name: "CUDA_ERROR_OUT_OF_MEMORY" }. The failure was intermittent: the first launch of a smaller core went through the same distribution without it. After, with rebuilt binaries on all four hosts, the peers' rank cores removed and no manual cache drop: three rank cores were sent in 70.2 s, /proc/meminfo on rank zero at the end of distribution showed MemAvailable 115.9 GiB and Cached 0.12 GiB, and GPU initialization succeeded. The Linux test source_releases_the_page_cache_of_peer_core_files_when_it_finishes serves a 4 MiB on-disk file to one peer and checks with mincore that the file has resident pages after the transfer and none after the source finishes. cargo test --release --manifest-path flare-cli/Cargo.toml distribution: 30 passed, 0 failed, 2 ignored (they need external fixtures). Not run: other models, two- and three-rank cores, and hosts with discrete GPUs, where the release changes only the page cache.

The Inkling 2 Small NVFP4 recipe prepared only on four GPUs of one host: it
rejected a multi-host rendezvous, built its collectives on CUDA peer memory,
and admitted compute capability 12.0 alone. Each rank holds a quarter of the
344 GB checkpoint, which fits a DGX Spark (GB10, 121 GiB unified memory), so
four Sparks joined by RoCE can serve the model once the recipe places one rank
per host. The recipe follows the multi-host shape the DeepSeek V4 recipe uses
(51864a2, #240): host placement and the ring or chain choice travel in
PreparationConfig, and one RoCEnante runtime replaces the per-role peer-memory
channels when the rank group spans hosts. RoCEnante reductions carry BF16
only, so cross-host ranks round each partial sum to BF16 before reducing; the
single-host path keeps its FP32 reductions, because that precision is
available there and its prepared graphs must not change.

What changed, in ignition/recipes/inkling2. Before, recipe.prepare raised when
given `ranks` and always used process-local GPUs 0 through 3. Now, with
`ranks`, each rank uses GPU 0 of its own host, and hosts, roce_ring and
roce_chain flow from the plan into PreparationConfig, which requires four
distinct (host, GPU) placements and validates the routing shape with
ignition.planning.host_shape_error. bootstrap._collective_runtimes allocates
the channels: when the coordinator spans hosts, one RoCEnante runtime carries
the hidden reductions and the vocabulary, sampling and log-probability
gathers, sized from the decode rows, the drafter's block of every request, the
prefill chunk and each rank's vocabulary partition; otherwise it creates the
same all-reduce and all-gather peer-memory runtimes with the same arguments as
before. The shared runtime is closed once. decoder.reduction_dtype selects
BF16 for a RoCE runtime and FP32 otherwise, and the embedding, attention and
expert partial sums are cast to it before each reduction. prepare_rank admits
compute capability (12, 1) beside (12, 0); the exact check existed because the
recipe had been tested on RTX PRO 6000 alone. On an integrated GPU it releases
the checkpoint's page cache after loading weights, since unified memory counts
those pages against the GPU. The request limit stays 8 and the declared
defaults are unchanged. docs/model-preparation.md gives the four-host export,
join and serve commands and the qualified configuration, docs/roce-ring.md
lists the recipe among multi-host preparations, and docs/TODO.txt records the
unrun comparisons.

The result is that `ignition export --ranks 4` with three `ignition join`
peers prepares this recipe on four hosts, and `flare run` serves the core
across them. Single-host preparation builds the same channels and FP32
reductions as before.

Evidence, on four DGX Sparks (GB10, compute capability 12.1, one GPU each),
RoCE over ConnectX-7 with mesh routing, the checkpoint on rank zero only and
peers reading their tensor ranges from it (80.27 GiB each, about 2.3 Gbit/s).
Before this change the recipe rejected `ranks`. After it, a reduced core
(--no-mtp --language-model-only --max-concurrency 1 --prefill-chunk-size 512
--autotune-effort off --no-compile, no memory reserve) exported 8 graphs in
18 min 43 s and served across the four hosts: 82.91 GiB loaded per rank and a
24.5 GiB cache with 6,414,336 KV tokens. A chat completion, a streamed
completion (90 events), two requests against the one slot, and a 7,150-token
lookup prefilled in fourteen 512-token chunks returned the expected answers,
and SIGTERM shutdown completed in 3.5 s. The RoCE loss counters
(out_of_sequence, packet_seq_err, local_ack_timeout_err, rnr_nak_retry_err,
summed over sixteen ports) did not change during export or serving.
tests/recipes/test_inkling2_hosts.py (14 passed, CPU only) covers placement
and routing validation, the plan fitting four GB10 hosts, dispatch of the plan
to the rank group, the RoCE runtime's capacities, the unchanged peer-memory
channel arguments, and the reduction element type. With
tests/cli/test_inkling2.py, tests/cli/test_planning.py and
tests/recipes/test_declarations.py: 249 passed, 9 failed, 14 skipped; the
same three files on the parent commit give 235 passed, 14 skipped and the
same 9 failures (Qwen declaration assertions, a missing Kimi checkpoint
directory, DeepSeek V4.1 and Muse Spark collective-policy cases, and the
collective ladder round trip).

Limitations. An export of the declared eight-request configuration with
drafting, vision, tuning and compilation was started on the four Sparks and
stopped at graph construction after about seven minutes; it has not completed
there. Not run: a numerical comparison of BF16 cross-host reductions with
single-host FP32 output, the server tests of tests/server/test_inkling2.py on
Sparks, ring and chain routing, and single-host GPU preparation after this
change.

Design-Signed-Off-By: flare-02; design-review=ed22
The Inkling 2 Small NVFP4 recipe rejected more than eight concurrent requests:
PreparationConfig bounded the request capacity to [1, 8] and the recipe
declared max_concurrency=8. Four DGX Sparks hold enough memory beside the
weights for a 32-request core with 8192-token prefill chunks, and nothing in
the recipe depends on eight. Request slots, batch capacities, pool rows,
DSpark scratch, vision slots, gather rows, prefill scratch and the page count
already derive from the configured request count, so raising the bound is
sufficient and no capacity needed a separate change.

What changed. Before, bootstrap.PreparationConfig raised for max_active above
8 and planning rejected --max-concurrency above 8 for this recipe. Now the
bound is [1, 32] and the recipe declares max_concurrency=32. The default
concurrency (8), default prefill chunk (4096) and default verification
capacity (64 rows, 64-graph budget) are unchanged, so an export without flags
prepares the same catalog as before. tests/server/test_inkling2.py no longer
pins eight requests and 4096-token chunks: it accepts a service with at least
eight requests and 4096- or 8192-token chunks and scales its chunk-crossing
prompts with the chunk size the service reports. tests/cli/test_inkling2.py
asserts the declared maximum, and tests/recipes/test_inkling2_hosts.py adds
the 32-request bound, round shapes, RoCE capacities and planning fit.
docs/model-preparation.md documents the 32-request, 8192-token DGX Spark
configuration in place of the eight-request one, and docs/TODO.txt replaces
the pending eight-request Spark export with the planner estimate described
below.

The result is that the recipe prepares and serves up to 32 concurrent
requests. On four DGX Sparks the 32-request, 8192-token configuration with
seven drafts and vision exports, loads with 7.6 GiB or more available per
host, and answers 32 concurrent requests correctly. These are functional
results; no throughput was measured.

Evidence, on four DGX Sparks (GB10, 121 GiB unified memory each, one rank per
host, RoCE over ConnectX-7 with mesh routing). Before this change the
configuration could not be requested. After it, export with --max-concurrency
32 --image-concurrency 4 --prefill-chunk-size 8192 --mtp-draft-tokens 7
--speculative-token-capacity 128 --speculative-graph-budget 64
--autotune-effort default and a 4 GiB memory reserve took 1 h 25 min 30 s and
produced 152 graphs in a 351 GB core (350,544,415,417 bytes): kernel tuning
675 s, compilation 1,893 s, compiled warmup 586 s, capture 485 s, export
1,078 s. The lowest MemAvailable during export was 11.5, 12.6, 12.5 and
13.1 GiB on ranks 0 through 3, and the graph pool is 2,840 MiB per rank.
Verification capacity 128 with graph budget 64 is the value the DeepSeek V4.1
recipe declares for 32 requests and seven drafts. Serving loaded 90.06 to
90.24 GiB per rank. With the exported 4 GiB reserve the cache was 12.4 GiB
with 2,392,064 KV tokens and 78 shared snapshot slots, leaving 12.0 to
12.5 GiB MemAvailable. With --memory-limit=-0G it was 16.5 GiB with 3,252,224
KV tokens and 106 snapshot slots, leaving 7.76 to 8.26 GiB after loading and
7.64 GiB at the lowest sample during 32 concurrent requests. On that service,
tests/server/test_inkling2.py -k default-effort passed 7 cases: eight
concurrent requests, streamed reasoning, prefix reuse and branching across a
chunk, a tool call, a structured response, images across a chunk, and four
images with four concurrent image requests. 16 and 32 concurrent exact-repeat
requests were all correct, and a streamed completion succeeded. An
18,832-token lookup was answered correctly in four of four runs with a
2048-token output limit and in one of two runs with a 512-token limit; the
other run spent all 512 tokens in reasoning. Its cause was not investigated,
and docs/TODO.txt already records an intermittent long-prefix output-limit
failure for this recipe. The RoCE loss counters did not change and earlyoom,
the hosts' out-of-memory daemon, killed no process. CPU tests:
tests/recipes/test_inkling2_hosts.py 18 passed; with tests/cli/test_inkling2.py, tests/cli/test_planning.py and
tests/recipes/test_declarations.py, 253 passed, 9 failed, 14 skipped, the 9
failures being the ones the same files produce on main (Qwen declaration
assertions, a missing Kimi checkpoint directory, DeepSeek V4.1 and Muse Spark
collective-policy cases, and the collective ladder round trip).

Limitations. The planner estimated 22.9 GiB of KV pages per rank where
serving allocated 12.4 GiB at the same reserve; docs/TODO.txt records the
correction of the declared rank costs. Not run: the 32-request configuration
on RTX PRO 6000, a 16-request core, the eight-request configuration on DGX
Sparks, the effort-zero server cases, and benchmarks.

Design-Signed-Off-By: flare-02; design-review=ed22
When `flare run` starts a multi-host core, rank zero reads every peer's rank
core through the page cache and sends it over TCP. On a host whose GPU
allocates from system memory, such as a DGX Spark (GB10), those cached pages
are not free memory for CUDA, so the pages of several rank cores can keep rank
zero from creating its CUDA context. Helios already releases the page cache of
a rank's own core files before it creates the context
(Files::release_page_cache in helios/src/runtime_core/model.rs), and a
receiving rank syncs the files it writes; the files rank zero reads on behalf
of other ranks were never released. Releasing them where the distribution
source stops closes that gap with the same eos operation and needs no change
to loading or to the transfer protocol.

Before this change the distribution source in flare-cli/src/distribution.rs
returned when its serving threads had joined and left the pages it had read in
the cache. Now release_served_pages calls flare_eos::release_page_cache
(POSIX_FADV_DONTNEED on Linux, a no-op on Windows) on every file of ranks one
and above when the source thread finishes, whether distribution succeeded or
failed. Rank zero's own files are left to the existing release in Helios.
flare-cli gains a path dependency on the in-tree eos crate, which the lock
file records.

The result is that a rank-zero host returns the memory used to read peer
cores before GPU initialization, without a manual page-cache drop.

Evidence, on four DGX Sparks (121 GiB unified memory each, one rank per host)
serving a 351 GB four-rank Inkling 2 Small NVFP4 core. Before: after rank
zero distributed three rank cores, its host showed 114 GiB of page cache and
2 GiB free, and rank zero failed at "Initialize GPU" with CudaError {
operation: "cuCtxCreate_v2", code: 2, name: "CUDA_ERROR_OUT_OF_MEMORY" }. The
failure was intermittent: the first launch of a smaller core went through the
same distribution without it. After, with rebuilt binaries on all four hosts,
the peers' rank cores removed and no manual cache drop: three rank cores were
sent in 70.2 s, /proc/meminfo on rank zero at the end of distribution showed
MemAvailable 115.9 GiB and Cached 0.12 GiB, and GPU initialization succeeded.
The Linux test
source_releases_the_page_cache_of_peer_core_files_when_it_finishes serves a
4 MiB on-disk file to one peer and checks with mincore that the file has
resident pages after the transfer and none after the source finishes.
cargo test --release --manifest-path flare-cli/Cargo.toml distribution: 30
passed, 0 failed, 2 ignored (they need external fixtures). Not run: other
models, two- and three-rank cores, and hosts with discrete GPUs, where the
release changes only the page cache.

Design-Signed-Off-By: flare-02; design-review=ed22
@original-el8 original-el8 changed the title Prepare Inkling 2 Small NVFP4 across four hosts for DGX Spark TP4 Prepare and serve Inkling 2 Small NVFP4 at TP4 on four DGX Sparks Nov 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant