Skip to content

Raise the Inkling 2 Small NVFP4 request limit to 32 - #305

Merged
original-el8 merged 1 commit into
inkling2/dgx-spark-tp4from
inkling2/request-limit-32
Nov 15, 2026
Merged

original-el8 merged 1 commit into
inkling2/dgx-spark-tp4from
inkling2/request-limit-32

Conversation

@original-el8

Copy link
Copy Markdown
Collaborator

Stacked on inkling2/dgx-spark-tp4 ("Prepare Inkling 2 Small NVFP4 across four hosts for DGX Spark TP4"), which adds multi-host preparation of this recipe; review and merge that change first. Branch: inkling2/request-limit-32. The diff against inkling2/dgx-spark-tp4 is this change alone.

The Inkling 2 Small NVFP4 recipe rejected more than eight concurrent requests: PreparationConfig bounded the request capacity to [1, 8] and the recipe declared max_concurrency=8. Four DGX Sparks hold enough memory beside the weights for a 32-request core with 8192-token prefill chunks, and nothing in the recipe depends on eight. Request slots, batch capacities, pool rows, DSpark scratch, vision slots, gather rows, prefill scratch and the page count already derive from the configured request count, so raising the bound is sufficient and no capacity needed a separate change.

What changed. Before, bootstrap.PreparationConfig raised for max_active above 8 and planning rejected --max-concurrency above 8 for this recipe. Now the bound is [1, 32] and the recipe declares max_concurrency=32. The default concurrency (8), default prefill chunk (4096) and default verification capacity (64 rows, 64-graph budget) are unchanged, so an export without flags prepares the same catalog as before. tests/server/test_inkling2.py no longer pins eight requests and 4096-token chunks: it accepts a service with at least eight requests and 4096- or 8192-token chunks and scales its chunk-crossing prompts with the chunk size the service reports. tests/cli/test_inkling2.py asserts the declared maximum, and tests/recipes/test_inkling2_hosts.py adds the 32-request bound, round shapes, RoCE capacities and planning fit. docs/model-preparation.md documents the 32-request, 8192-token DGX Spark configuration in place of the eight-request one, and docs/TODO.txt replaces the pending eight-request Spark export with the planner estimate described below.

The result is that the recipe prepares and serves up to 32 concurrent requests. On four DGX Sparks the 32-request, 8192-token configuration with seven drafts and vision exports, loads with 7.6 GiB or more available per host, and answers 32 concurrent requests correctly. These are functional results; no throughput was measured.

Evidence, on four DGX Sparks (GB10, 121 GiB unified memory each, one rank per host, RoCE over ConnectX-7 with mesh routing). Before this change the configuration could not be requested. After it, export with --max-concurrency 32 --image-concurrency 4 --prefill-chunk-size 8192 --mtp-draft-tokens 7 --speculative-token-capacity 128 --speculative-graph-budget 64 --autotune-effort default and a 4 GiB memory reserve took 1 h 25 min 30 s and produced 152 graphs in a 351 GB core (350,544,415,417 bytes): kernel tuning 675 s, compilation 1,893 s, compiled warmup 586 s, capture 485 s, export 1,078 s. The lowest MemAvailable during export was 11.5, 12.6, 12.5 and 13.1 GiB on ranks 0 through 3, and the graph pool is 2,840 MiB per rank. Verification capacity 128 with graph budget 64 is the value the DeepSeek V4.1 recipe declares for 32 requests and seven drafts. Serving loaded 90.06 to 90.24 GiB per rank. With the exported 4 GiB reserve the cache was 12.4 GiB with 2,392,064 KV tokens and 78 shared snapshot slots, leaving 12.0 to 12.5 GiB MemAvailable. With --memory-limit=-0G it was 16.5 GiB with 3,252,224 KV tokens and 106 snapshot slots, leaving 7.76 to 8.26 GiB after loading and 7.64 GiB at the lowest sample during 32 concurrent requests. On that service, tests/server/test_inkling2.py -k default-effort passed 7 cases: eight concurrent requests, streamed reasoning, prefix reuse and branching across a chunk, a tool call, a structured response, images across a chunk, and four images with four concurrent image requests. 16 and 32 concurrent exact-repeat requests were all correct, and a streamed completion succeeded. An 18,832-token lookup was answered correctly in four of four runs with a 2048-token output limit and in one of two runs with a 512-token limit; the other run spent all 512 tokens in reasoning. Its cause was not investigated, and docs/TODO.txt already records an intermittent long-prefix output-limit failure for this recipe. The RoCE loss counters did not change and earlyoom, the hosts' out-of-memory daemon, killed no process. CPU tests: tests/recipes/test_inkling2_hosts.py 18 passed; with tests/cli/test_inkling2.py, tests/cli/test_planning.py and tests/recipes/test_declarations.py, 253 passed, 9 failed, 14 skipped, the 9 failures being the ones the same files produce on main (Qwen declaration assertions, a missing Kimi checkpoint directory, DeepSeek V4.1 and Muse Spark collective-policy cases, and the collective ladder round trip).

Limitations. The planner estimated 22.9 GiB of KV pages per rank where serving allocated 12.4 GiB at the same reserve; docs/TODO.txt records the correction of the declared rank costs. Not run: the 32-request configuration on RTX PRO 6000, a 16-request core, the eight-request configuration on DGX Sparks, the effort-zero server cases, and benchmarks.

The Inkling 2 Small NVFP4 recipe rejected more than eight concurrent requests:
PreparationConfig bounded the request capacity to [1, 8] and the recipe
declared max_concurrency=8. Four DGX Sparks hold enough memory beside the
weights for a 32-request core with 8192-token prefill chunks, and nothing in
the recipe depends on eight. Request slots, batch capacities, pool rows,
DSpark scratch, vision slots, gather rows, prefill scratch and the page count
already derive from the configured request count, so raising the bound is
sufficient and no capacity needed a separate change.

What changed. Before, bootstrap.PreparationConfig raised for max_active above
8 and planning rejected --max-concurrency above 8 for this recipe. Now the
bound is [1, 32] and the recipe declares max_concurrency=32. The default
concurrency (8), default prefill chunk (4096) and default verification
capacity (64 rows, 64-graph budget) are unchanged, so an export without flags
prepares the same catalog as before. tests/server/test_inkling2.py no longer
pins eight requests and 4096-token chunks: it accepts a service with at least
eight requests and 4096- or 8192-token chunks and scales its chunk-crossing
prompts with the chunk size the service reports. tests/cli/test_inkling2.py
asserts the declared maximum, and tests/recipes/test_inkling2_hosts.py adds
the 32-request bound, round shapes, RoCE capacities and planning fit.
docs/model-preparation.md documents the 32-request, 8192-token DGX Spark
configuration in place of the eight-request one, and docs/TODO.txt replaces
the pending eight-request Spark export with the planner estimate described
below.

The result is that the recipe prepares and serves up to 32 concurrent
requests. On four DGX Sparks the 32-request, 8192-token configuration with
seven drafts and vision exports, loads with 7.6 GiB or more available per
host, and answers 32 concurrent requests correctly. These are functional
results; no throughput was measured.

Evidence, on four DGX Sparks (GB10, 121 GiB unified memory each, one rank per
host, RoCE over ConnectX-7 with mesh routing). Before this change the
configuration could not be requested. After it, export with --max-concurrency
32 --image-concurrency 4 --prefill-chunk-size 8192 --mtp-draft-tokens 7
--speculative-token-capacity 128 --speculative-graph-budget 64
--autotune-effort default and a 4 GiB memory reserve took 1 h 25 min 30 s and
produced 152 graphs in a 351 GB core (350,544,415,417 bytes): kernel tuning
675 s, compilation 1,893 s, compiled warmup 586 s, capture 485 s, export
1,078 s. The lowest MemAvailable during export was 11.5, 12.6, 12.5 and
13.1 GiB on ranks 0 through 3, and the graph pool is 2,840 MiB per rank.
Verification capacity 128 with graph budget 64 is the value the DeepSeek V4.1
recipe declares for 32 requests and seven drafts. Serving loaded 90.06 to
90.24 GiB per rank. With the exported 4 GiB reserve the cache was 12.4 GiB
with 2,392,064 KV tokens and 78 shared snapshot slots, leaving 12.0 to
12.5 GiB MemAvailable. With --memory-limit=-0G it was 16.5 GiB with 3,252,224
KV tokens and 106 snapshot slots, leaving 7.76 to 8.26 GiB after loading and
7.64 GiB at the lowest sample during 32 concurrent requests. On that service,
tests/server/test_inkling2.py -k default-effort passed 7 cases: eight
concurrent requests, streamed reasoning, prefix reuse and branching across a
chunk, a tool call, a structured response, images across a chunk, and four
images with four concurrent image requests. 16 and 32 concurrent exact-repeat
requests were all correct, and a streamed completion succeeded. An
18,832-token lookup was answered correctly in four of four runs with a
2048-token output limit and in one of two runs with a 512-token limit; the
other run spent all 512 tokens in reasoning. Its cause was not investigated,
and docs/TODO.txt already records an intermittent long-prefix output-limit
failure for this recipe. The RoCE loss counters did not change and earlyoom,
the hosts' out-of-memory daemon, killed no process. CPU tests:
tests/recipes/test_inkling2_hosts.py 18 passed; with tests/cli/test_inkling2.py, tests/cli/test_planning.py and
tests/recipes/test_declarations.py, 253 passed, 9 failed, 14 skipped, the 9
failures being the ones the same files produce on main (Qwen declaration
assertions, a missing Kimi checkpoint directory, DeepSeek V4.1 and Muse Spark
collective-policy cases, and the collective ladder round trip).

Limitations. The planner estimated 22.9 GiB of KV pages per rank where
serving allocated 12.4 GiB at the same reserve; docs/TODO.txt records the
correction of the declared rank costs. Not run: the 32-request configuration
on RTX PRO 6000, a 16-request core, the eight-request configuration on DGX
Sparks, the effort-zero server cases, and benchmarks.

Design-Signed-Off-By: flare-02; design-review=ed22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant