Skip to content

Name the native Flash Next checkpoint in its recipe #705

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ashhart merged 1 commit into from
Oct 13, 2026
Merged

Name the native Flash Next checkpoint in its recipe #705

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 16 additions & 6 deletions docs/recipes/qwen3.8-flash-next.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,13 +1,22 @@
# Qwen3.8 Flash Next

The `qwen4_exp` family has Gated DeltaNet, sparse attention, MoE, hyper-connections and hashed n-gram
embeddings. The supported checkpoint uses MLX affine 4-bit weights in groups of 32 and includes an MTP head.
embeddings. The native engine's supported Metal checkpoint uses MLX affine 6-bit weights in groups of 32
and includes an MTP head. The native engine refuses the 4-bit/group-32 checkpoint at load.

```bash
tensorfold pull TensorFold/Qwen3.8-Flash-Next-MLX-4bit-MTP
tensorfold serve TensorFold/Qwen3.8-Flash-Next-MLX-4bit-MTP --name bench
tensorfold pull TensorFold/Qwen3.8-Flash-Next-MLX-6bit-MTP
tensorfold serve TensorFold/Qwen3.8-Flash-Next-MLX-6bit-MTP --name bench --temperature 0
```

For native serving, see the [Metal runbook](../../RUNBOOK.md#flash-next-and-paired-metal-serving).
Native Flash Next has no CUDA backend yet; its port is tracked in
[#472](https://github.com/ashhart/TensorFold/issues/472).

The sections below describe the Python 0.6.6 engine, available on
[`python-0.6`](https://github.com/ashhart/TensorFold/tree/python-0.6).
The MLX section uses `TensorFold/Qwen3.8-Flash-Next-MLX-4bit-MTP`; CUDA checkpoints are listed below.

On MLX, a supported conversion without the head runs without MTP drafting. On CUDA, pass `--no-drafts`
for such a conversion; the default positive draft depth otherwise refuses the missing head.

Expand All @@ -16,8 +25,8 @@ for such a conversion; the default positive draft depth otherwise refuses the mi
N-gram tables stay in host file mappings when the checkpoint's model files exceed 75% of the GPU's
recommended working set. `TF_NGRAM_HOST=1` forces this mode; `TF_NGRAM_HOST=0` keeps the tables in MLX.
Startup admission uses the same choice as the loader and subtracts the mapped weights, scales and biases
from the checkpoint's size. For the named 4-bit checkpoint, about 29.8 GiB of its 105.4 GiB is mapped,
leaving a conservative 75.6 GiB resident-weight estimate. The default command selects host mode on an
from the checkpoint's size. For the 4-bit checkpoint, about 29.8 GiB of its 105.4 GiB is mapped,
leaving a conservative 75.6 GiB resident-weight estimate. With that checkpoint, the Python engine selects host mode on an
M4 Max with 128 GiB, whose process budget is 89.6 GiB, including the 3 GiB process reserve.

Mapped pages still use RAM while cached. The loader prefetches them after its initial forwards; macOS
Expand Down Expand Up @@ -61,7 +70,8 @@ to one stream, while successful checks allow the lane engine to combine requests

On CUDA Flash Next serves NVFP4 and EXL3 checkpoints, and the MLX 4-bit checkpoint as the portable option: the same
files a Mac serves, and the only format two ranks and `--ple-on-ssd` read. `tensorfold serve` loads the checkpoint
you name; it picks none by itself. Use the [container setup](../../RUNBOOK.md#nvidia-gpus) for any of them. Prompts
you name; it picks none by itself. Use the [Python CUDA setup](cuda.md#python-06-cuda-families)
for the container and pinned 0.6.6 installation. Prompts
take bf16 activations by default; what that costs against the FP8 prompt path (`--prefill-fp8`) depends on the
format ([prompt precision](cuda.md#prompt-precision)):

Expand Down