Qwen3.8-27B on native 1.0.5: M6 (GPU generation 18) refused, and memory is too large for 32 GB

TensorFold 1.0.5 does not serve Qwen3.8-27B on an Apple M6 Mac with 32 GB of memory. There are two problems:

The engine refuses GPU generation 18 (M6). The error is QwenMetalQualificationRequired.
When I allow generation 18, the model loads. But the engine uses 34.6 GiB of GPU memory, and each request fails with "out of memory".

Before the Zig engine, I ran Qwen3.8-27B with DFlash2 drafts and a 16K context on this Mac. The Python engine (0.2.0) did this.

Environment

Item	Value
Mac	Mac mini, Apple M6, 12-core GPU
Memory	32 GB
macOS	27.0
`iogpu.wired_limit_mb`	0 (default)
TensorFold	1.0.5, release archive `tensorfold-1.0.5-macos-arm64.tar.gz` (sha256 OK)
Model	`Vontra/Qwen3.8-27B-MLX-4bit` (snapshot `70ae7fac`)
Drafter	`z-lab/Qwen3.8-27B-DFlash2`

Problem 1: GPU generation 18 is refused

Steps

tensorfold-native serve <model dir> --drafter z-lab/Qwen3.8-27B-DFlash2 --context 16384 --temperature 0

Result

tensorfold: the native Qwen3.8-27B engine cannot load <model dir> (QwenMetalQualificationRequired)

Cause

zig/src/families/qwen27/model.zig:96 accepts generations 13 to 17 only:

fn qualifiedGeneration(generation: u32) bool {
    return generation >= 13 and generation <= 17;
}

The M6 is applegpu_g18. The test in the same file makes sure that 18 is refused.

All other GPU checks in the engine use tensorUnits(), which is generation >= 17. Thus the M6 gets the M5 tensor-unit kernels. Only this check stops the M6.

Change that I used

+/// M1 (applegpu_g13) through M6 (applegpu_g18); the M6 runs the M5's tensor-unit kernels.
 fn qualifiedGeneration(generation: u32) bool {
-    return generation >= 13 and generation <= 17;
+    return generation >= 13 and generation <= 18;
 }
@@
     try std.testing.expect(qualifiedGeneration(17));
+    try std.testing.expect(qualifiedGeneration(18));
     try std.testing.expect(!qualifiedGeneration(0));
     try std.testing.expect(!qualifiedGeneration(12));
-    try std.testing.expect(!qualifiedGeneration(18));
+    try std.testing.expect(!qualifiedGeneration(19));

With this change, the model and the drafter load on the M6. The Metal kernels compile and the server starts. I did not see a GPU fault from the kernels.

Note: I built from source with Zig 0.17.0. This Mac has the Command Line Tools only, so there is no xcrun metal. To build, I used the nemotron_packed.metallib from the 1.0.5 release binary and did not compile it again. The Qwen3.8-27B kernels compile at run time, so this does not change them.

Problem 2: GPU memory is too large for 32 GB

Steps

TF_ANE_COLUMNS=0 tensorfold-native serve <model dir> --drafter z-lab/Qwen3.8-27B-DFlash2 --context 4096 --temperature 0
curl -s http://127.0.0.1:8080/v1/chat/completions -H 'Content-Type: application/json' \
  -d '{"model":"Qwen3.8-27B-MLX-4bit","messages":[{"role":"user","content":"Say hello in one sentence."}],"max_tokens":64}'

Result

The request fails. The response is {"error": {"message": "RoundGpuFailure"}}.

1.0.5 does not log the Metal error. I added a log line in decode_round.zig. The Metal error is:

qwen27: GPU round failed: Insufficient Memory (00000008:kIOGPUCommandBufferCallbackErrorOutOfMemory)

After this error, the server stays in the failed state, and all subsequent requests also fail.

Measurements

I logged MTLDevice.currentAllocatedSize after each load step (4K context):

After	GPU memory
Model load	28.45 GiB
Runner	29.66 GiB
DFlash2 drafter (bits 4)	34.60 GiB
Lane decode	34.60 GiB

Other data:

At 16K context, the process footprint is 35 GB and swap increases from 0.8 GB to 12.8 GB.
The startup line says: 33.6 GiB in use once loaded, 32 GiB of RAM.
tensorfold info says that the weights are 14.98 GiB and the memory floor is 16.98 GiB.
TF_ANE_COLUMNS=0 does not decrease the memory.

Possible cause

The engine keeps the weights two times:

gpu_weights.zig makes owned copies of the projections in the lane layout (approximately 13.5 GiB).
The checkpoint shards stay mapped as no-copy Metal buffers (approximately 15 GiB).

holdResident in qwen27_host.zig adds all checkpoint shards and all owned buffers to one residency set. Thus macOS cannot remove the mapped checkpoint pages from memory.

On a 128 GB Mac, this is not a problem. On a 32 GB Mac, the total is more than the memory.

Compare with the Python engine

Version	Settings	Result
0.2.0 (Python)	`--context 16384`, DFlash2	OK. Both hidden facts found at 16K, 29 tok/s. 32K was also OK.
0.6.6 (Python)	`TENSORFOLD_MEMORY_LIMIT_GB=25 --context 8192 --parallel 1`, DFlash2	OK. 18.3 GiB weights resident. Code 72 tok/s, prose 31 tok/s. Drafted output is byte-identical to serial output.
0.6.6 (Python)	`TENSORFOLD_MEMORY_LIMIT_GB=25 --context 16384`	Refused by the memory budget (6,144 tokens maximum). See #50.
1.0.5 (Zig)	any context	Refused: generation 18.
1.0.5 (Zig) + generation 18 change	`--context 4096`, DFlash2	Loads. Each request fails: GPU out of memory.

Requests

Please accept GPU generation 18 (M6) for Qwen3.8-27B. I can run tests on this Mac and send the results.
Please decrease the GPU memory of the Qwen3.8-27B engine, so that it runs on 32 GB Macs. For example, after the projections are repacked, copy the small borrowed tensors and release the checkpoint mapping. Or remove the repacked checkpoint ranges from the residency set.
Please log the Metal error text when a GPU round fails. The RoundGpuFailure message alone does not show the cause.
Small item: tensorfold-native serve does not accept --drafter none. It says none is neither a directory nor a Hugging Face repo id.

This guide shows the fix it and what to consider.

1. Accept GPU generation 18

File: zig/src/families/qwen27/model.zig

fn qualifiedGeneration(generation: u32) bool {
    return generation >= 13 and generation <= 18;
}

Change the test: expect qualifiedGeneration(18) to be true and qualifiedGeneration(19) to be false.

All other GPU checks use tensorUnits() (generation >= 17), so the M6 uses the M5 tensor-unit kernels. No other change is necessary.

2. Do not keep the weights two times

Cause

Checkpoint.addFileSelected reads each shard into a shared Metal buffer (approximately 15 GiB).
projection.prepare copies each projection into an owned buffer in the lane layout (approximately 13.5 GiB).
holdResident adds the shards and the owned buffers to one residency set.

After step 2, the engine does not read the shard copies of the projections. But they stay in memory.

Fix

Step 1. Record the repacked tensors. In gpu_weights.zig, send all repacks through one Builder function:

fn prepare(b: Builder, parts: []const projection.Triple) !Linear {
    const linear = try projection.prepare(b.device, b.allocator, b.owned, parts, 32);
    for (parts) |p| for ([_]Tensor{ p.weight, p.scales, p.biases }) |t|
        try b.repacked.put(b.allocator, cp.Checkpoint.address(t), {});
    return linear;
}

Keep the result in Weights.repacked (host addresses).

Step 2. Compact the checkpoint. In checkpoint_metal.zig, add compact:

Copy each tensor that is not in repacked to one new shard (256-byte alignment).
Update the tensor map to the new places, and remove the dropped names.
Return a map from old address to new tensor.
Keep the old shards in a retired list. Do not release them yet.

Step 3. Move the references. Weights keeps copies of Tensor values. Update each one from the map before you release the old shards:

each layer: norm, post_norm
GDN layers: conv, a_log, dt, norm
attention layers: q_norm, k_norm
embed (weight, scales, biases), norm

Step 4. Release the old shards (releaseRetired).

Step 5. Call this at the end of Model.load:

const freed = try m.weights.shrink(m.device, &m.checkpoint);

Result

Item	Before	After
Checkpoint shards	16.3 GiB	1.34 GiB
GPU memory with drafter, 4K	34.60 GiB	21.00 GiB
GPU memory with drafter, 32K	does not fit	22.92 GiB
Drafted output = plain output	-	yes (byte-identical)

3. Points to consider

Order. Call shrink after the Neural Engine share is built. ane_mlp.Share.init reads gate_proj, up_proj and down_proj from the checkpoint.
lm_head. Do not drop the checkpoint copy of lm_head. The DFlash2 drafter binds model.embed_tokens and lm_head from the target checkpoint (dflash/runtime_model.zig, binding). The drafter does not use the lm_head bytes for computation (the head runs from target.weights.head), so a later change can remove this copy too (approximately 0.6 GiB).
Use after release. Checkpoint.address reads buffer.contents(). Move all references before you release the old shards. If you release first, the engine reads a released buffer.
Load peak. During load, the old shards and the owned buffers are both in memory (approximately 32 GB on a 32 GB Mac). To prevent this peak, release each shard after its layers are repacked, or repack directly from the file into the owned buffers.
Drafter. The DFlash2 drafter has the same problem. Its bf16 checkpoint stays in memory after Backend.prepare makes the q4 adapters (+4.9 GiB). The adapters use the bf16 host address as the key, so the fix must keep the keys or change them.
Residency set. holdResident adds all checkpoint shards. After the fix, only the small new shard goes into the set.
Neural Engine share. It keeps an fp16 copy of 3,072 MLP columns of each layer. On a 32 GB Mac, this memory can be too large. Consider a memory check before the share is built, or disable it below a RAM limit.
Error text. In decode_round.zig, log cb.failure() when a GPU round fails. Without this, an out-of-memory error shows only as RoundGpuFailure.
Escape switch. TF_QWEN27_KEEP_CHECKPOINT=1 keeps the old behavior. This helps to compare the two paths.

4. How to test

Serve with --drafter z-lab/Qwen3.8-27B-DFlash2 --temperature 0 --context 32768.
Look for this log line: qwen27: freed ... GiB of repacked checkpoint tensors.
Send the same request with drafts and with "draft": false. The replies must be byte-identical.
Send a 32K prompt with two hidden facts. The reply must contain both facts.
Watch the footprint (/memory) and swap (sysctl vm.swapusage). Swap must not increase.

@RS91Dev

Thank you for this, @RS91Dev. The report is exact on both counts, and the logged Metal error saved us a lot of digging.

The M6. You're right that qualifiedGeneration is the only thing that stops generation 18, and that every other check sends the M6 down the M5's tensor-unit path. We keep that list closed on purpose: each chip family gets a token-for-token check before we accept it (drafted replies equal "draft": false ones, a resumed prompt equals a fresh one, and replies match a qualified chip). We don't have an M6 yet. If you're willing to run that check on yours with a test build, we'll accept generation 18 in the release after it passes, with your result as the receipt.

Memory. Confirmed in our code: the 27B keeps each checkpoint shard resident after it copies the projections into their lane layout, so the projection weights sit in memory twice (about 13.5 GiB on the 4-bit checkpoint). We're releasing the shards once their tensors are repacked, which should bring the weights close to the checkpoint's own size and leave a 32 GB Mac room for the drafter and a 16K context. The fix is planned for 1.0.7; we'll post the before and after footprints here.

The failed state. A round that runs out of GPU memory shouldn't leave the server refusing every later request. We'll make a failed round end only the requests it held, and log the Metal error as you did.

2. Thank you for this, @RS91Dev. The report is exact on both counts, and the logged Metal error saved us a lot of digging.
The M6. You're right that qualifiedGeneration is the only thing that stops generation 18, and that every other check sends the M6 down the M5's tensor-unit path. We keep that list closed on purpose: each chip family gets a token-for-token check before we accept it (drafted replies equal "draft": false ones, a resumed prompt equals a fresh one, and replies match a qualified chip). We don't have an M6 yet. If you're willing to run that check on yours with a test build, we'll accept generation 18 in the release after it passes, with your result as the receipt.
Memory. Confirmed in our code: the 27B keeps each checkpoint shard resident after it copies the projections into their lane layout, so the projection weights sit in memory twice (about 13.5 GiB on the 4-bit checkpoint). We're releasing the shards once their tensors are repacked, which should bring the weights close to the checkpoint's own size and leave a 32 GB Mac room for the drafter and a 16K context. The fix is planned for 1.0.7; we'll post the before and after footprints here.
The failed state. A round that runs out of GPU memory shouldn't leave the server refusing every later request. We'll make a failed round end only the requests it held, and log the Metal error as you did.

Did you fix it? As am waiting for the fix to test this on my macbook m2 pro.. I have the same probelm they model took 32 gb of my memory while it should take a round 18 gbs.

Opus 5.5 made some changes and these are the results:

Results

Settings: TF_ANE_COLUMNS=0, --drafter z-lab/Qwen3.8-27B-DFlash2 --temperature 0, thinking off. See issue #724 iogpu.wired_limit_mb is 0 (default).

Memory

qwen27: freed 14.94 GiB of repacked checkpoint tensors; 1.34 GiB of the checkpoint stays

Context	GPU memory (model + runner + drafter)
4K	21.00 GiB
16K	21.82 GiB
32K	22.92 GiB

Short replies (4K server)

Task	With drafts	Without drafts	Drafted = plain
Prose	36.6 tok/s	9.5 tok/s	yes (byte-identical)
Code	98.0 tok/s	9.5 tok/s	yes (byte-identical)

Long context (with drafts)

The prompt is seeded synthetic log records. Two facts are hidden in the prompt at 25% and 75% depth. The model must return both facts.

Context	Prompt tokens	Prompt processing	Found both facts	Peak footprint	Swap growth
16K	15,993	58.4 s (274 tok/s)	yes	22.5 GB	none
32K	32,384	139.4 s (232 tok/s)	yes	24.6 GB	none

Decode speed with a long reply

The prompt is the same log. The task asks for a report of approximately 400 words. Each reply has 600 tokens. I sent each request two times: with drafts, and with "draft": false.

Context	Prompt tokens	Prompt processing	With drafts	Without drafts	Speed-up	Drafted = plain	Peak footprint
16K	15,587	57.7 s	27.5 tok/s	8.6 tok/s	3.2x	yes (byte-identical)	23.6 GB
32K	31,984	135.4 s	22.9 tok/s	7.7 tok/s	3.0x	yes (byte-identical)	24.6 GB

Draft acceptance: 0.22 at 16K and 0.20 at 32K (4.3 and 4.0 tokens a round). Swap did not increase.

Notes

I did not test the Neural Engine share on the M6.
During load, the footprint is approximately 32 GB for a short time, because the old shards and the new owned buffers are both in memory.

@RS91Dev

Thank you, @RS91Dev. Both fixes are in 1.0.8: 93a218dda keeps the weights once, and b5056a511 makes a failed round end only its own requests.

Weights once. Your analysis held up. We went one step past compacting the checkpoint at the end of load, so the load peak drops as well: the projections and the LM head now stay in the checkpoint files at load. Each projection is read into a small staging buffer, repacked into its lane layout, and the buffer is reused for the next one, so a file copy never sits beside its repacked copy. Norms, the DeltaNet scalars and the embedding pack into one small buffer per shard (0.67 GiB in all). The Neural Engine share reads the rows it needs from the files while it builds. On your point 2, the DFlash2 drafter no longer binds the target's lm_head bytes at all: its head was always the target's repacked one. On your point 5, the drafter keeps only its 4-bit copies; its bf16 layers (3.35 GiB) are read once, into them.

Footprint once the server is ready, 4-bit checkpoint with the DFlash2 drafter, prompt cache off (footprint -f bytes, in GiB):

Mac	`--context`	Before	After
M5 Max	4,096	42.4	24.8
M5 Max	32,768 (default)	44.2	26.5
M3 Ultra	4,096	36.8	19.2
M3 Ultra	32,768 (default)	39.4	21.8

The process's own peak footprint equals its ready footprint in both builds, so the two copies are never resident together, and a 32K prompt adds about 0.5 GiB. With the 8 GiB prompt cache in use and four requests at once at 32K, the M5 Max peaks at 36.5 GiB against 54.1 before. A 64K prompt at --context 262144 still answers, peaking at 48.8 GiB against 66.5 on the M5 Max and 46.1 against 63.8 on the M3 Ultra. Every reply is the same tokens as before: drafted and with "draft": false, greedy and seeded sampling, thinking on, resumed turns, and 2 and 4 requests at once. Decode and prompt speed are unchanged within run noise on both Macs. The release gate saw the same 17.6 GiB against the 1.0.7 release at a 70,000-token context: ready at 29.6 GiB against 47.2 on the M5 Max and 24.6 against 42.2 on the M3 Ultra, with the same tokens.

Failed rounds. A round that fails on the GPU now ends only the requests it held, and the log names what failed with Metal's error text, for example qwen27: a round's keep failed on the GPU: .... Once those requests have left, the next request resets its slot and runs. Before, the runner kept its failure flag and refused every later request.

Your point 7. On a Mac with tensor units the Neural Engine share's fp16 columns stay resident: 5.6 GiB on our M5 Max, the whole gap between the two Macs at 4K above. TF_ANE_COLUMNS=0 turns the share off, at a large cost in prompt speed. Sizing it to the Mac's memory is a separate change.

The M6. Generation 18 still waits on the check from your Mac that we described above. Once we have your result, it goes in, so I'm leaving this open for that.

qwen27: keep the projection weights once on Metal (ashhart#643)

qwen27: a round the GPU fails ends only its requests (ashhart#643)

CHANGELOG: ashhart#643

Merge ashhart#643 (the 27B's weights once on Metal, and failed rounds…

@ashhart

Thanks for fixing all issues and improving the memory utilization @ashhart

TensorFold 1.0.8 on the Mac mini M6 (32 GB): test report

Date: 13 Oct 2026. Model: Vontra/Qwen3.8-27B-MLX-4bit with the z-lab/Qwen3.8-27B-DFlash2 drafter.

Summary

1.0.8 fixes three issues reported by me: the 27B keeps its weights in memory once (Qwen3.8-27B on native 1.0.5: M6 (GPU generation 18) refused, and memory is too large for 32 GB #643), a failed GPU round ends only its own requests and logs the Metal error (Qwen3.8-27B on native 1.0.5: M6 (GPU generation 18) refused, and memory is too large for 32 GB #643), and the 27B applies thinking budgets (it now runs on the lane host).
The official 1.0.8 binary still refuses the M6 GPU (QwenMetalQualificationRequired). One local line is still needed: GPU generation 18 in qualifiedGeneration.
With that line, every test passes at 16K and 32K. Memory is 19.7 GB at 16K and 20.3 GB at 32K, with no swap.
The Neural Engine share makes prompts about 5% faster but costs 5.8 GB and causes swap. Keep it off on this Mac.

Setup

Item	Value
Mac	Mac mini, Apple M6 (12-core GPU), 32 GB, macOS 27.0
GPU wired limit	default (`iogpu.wired_limit_mb=0`)
Binary	1.0.8 source + M6 patch, built with Zig 0.17.0 (`native/src/zig-out/native/bin/tensorfold-native`)
Branch	`m6-qwen27-1.0.8` = v1.0.8 + 2 commits (`native/m6-patches-1.0.8/`)
Official release	`native/tensorfold-1.0.8-macos-arm64/` (sha256 OK), refuses the 27B on the M6

Server command:

TF_ANE_COLUMNS=0 native/src/zig-out/native/bin/tensorfold-native serve <model dir> \
  --drafter z-lab/Qwen3.8-27B-DFlash2 --context 32768 --temperature 0

Reported issues in 1.0.8

Issue	Status in 1.0.8	Local patch
Weights held twice (checkpoint + repacked copies)	Fixed (#643)	Dropped
Failed GPU round breaks all later requests; no Metal error text	Fixed (#643), `gpuFailed` logs the error	Dropped
`thinking_budget` refused on the 27B	Fixed (#724): the 27B runs on the lane host	Dropped
M6 GPU (generation 18) refused	Not fixed	Kept: `generation <= 18`
Build needs `xcrun metal`	n/a (local build aid)	Kept: `-Dnemotron-packed-metallib`

Results (Neural Engine off, greedy)

Test	16K	32K
Load time	25.8 s	7.9 s
Footprint after load	19.7 GB	20.3 GB
Smoke: drafted = plain (prose, code)	pass	-
Thinking budget: 10 checks	pass	-
Recall, thinking on, budget 64: both facts found	pass	pass
Prompt read (recall prompt)	59.3 s, 270 tok/s	138.8 s, 234 tok/s
Reply speed, recall answer (82 tokens)	34.5 tok/s	27.7 tok/s
Reply speed, 600 tokens, drafts on	27.7 tok/s	23.4 tok/s
Reply speed, 600 tokens, drafts off	8.8 tok/s	8.1 tok/s
Drafted = plain, byte-identical	pass (3.15x)	pass (2.89x)
Peak footprint	19.5 GB	20.5 GB
Swap growth	0 MB	0 MB

Smoke test (4K prompts): prose and code replies drafted = plain. The output hashes equal the 1.0.5 and 1.0.7 patched builds, so the replies are unchanged.

Thinking budget checks: budgets 24 and 96 cut the reasoning at 25 and 97 tokens (budget + the forced </think>), the answer follows, drafted = plain, and an unreached budget (1,400) gives the same reply as no budget.

Compare with the local fixes on 1.0.7

Item	1.0.7 + local fixes	1.0.8 + M6 patch
Footprint at 16K / 32K	22.5 / 23.6 GB	19.5 / 20.5 GB
Load peak	~32 GB for a short time	none
Swap	none after load	none
Reply speed at 32K (600 tokens, drafts)	22.8 tok/s	23.4 tok/s
Output	same hashes	same hashes

Neural Engine test (16K)

Measurement	Off (`TF_ANE_COLUMNS=0`)	On (3,072 columns, default)
Footprint after load	19.7 GB	25.6 GB (+5.8 GB)
Swap at load	0 MB	2.5 GB
Swap growth, 16K recall request	0 MB	+2.8 GB
Prompt read, recall (16,033 tokens)	59.3 s	58.6 s (-1%)
Prompt read, reply test (15,587 tokens)	57.0-58.7 s	53.8-55.5 s (-5%)
Reply speed, 600 tokens, drafts on / off	27.7 / 8.8 tok/s	27.2 / 8.8 tok/s
Recall	pass	pass
Drafted = plain	yes	yes
Same output as Neural Engine off	-	no (different hash)

Result: the Neural Engine speeds up prompt reading only, by about 5% at best. On a 32 GB Mac its 5.8 GB pushes macOS into swap. Its fp16 math changes the prompt state, so replies differ from Neural-Engine-off replies (drafted still equals plain). 32K was not tested with it on.

How to repeat

With the server running:

B=$PWD/native/src/zig-out/native/bin/tensorfold-native
TF_BINARY=$B uv run smoke_test.py
uv run thinking_budget_test.py
TF_THINKING=1 TF_BINARY=$B uv run long_context_test.py 16384     # or 32768
TF_BINARY=$B uv run decode_depth_test.py 16384                   # or 32768

CHANGELOG: the 27B on GPU generation 18 (M6) (ashhart#643)

The 27B's Neural Engine share loads only inside the 70% load limit

CHANGELOG: the 27B's Neural Engine share inside the 70% load limit (a…

@RS91Dev

Thank you, @RS91Dev, for a careful report. 1.0.9 accepts GPU generation 18 for the 27B: the one line you carried locally landed as 92605f359, with the test now expecting generation 18 to load and 19 to refuse until someone runs it, and you as its co-author. We have no M6 here, so your runs are the receipt: drafted equal to plain at 16k and 32k, the thinking budgets, recall and no swap.

Your Neural Engine numbers led to a second change, also in 1.0.9 (b264f534b). The share keeps an fp16 copy of 3,072 MLP columns of every layer, 5,760 MiB, which the GPU's own memory count never saw. The 27B now builds it only when it fits under the 70% load limit beside the loaded model, so a 32 GB Mac runs prompts on the GPU alone and says so at load, and Macs with 48 GB or more keep it with the 4-bit checkpoint. We have no 32 GB Mac with tensor units to check it on; if you can, a load on your M6 should print that line and show no swap with TF_ANE_COLUMNS unset. I'll leave this open until you've had a chance to try it.

@ashhart

@ashhart Thank you for the two changes. I tested the official 1.0.10 binary on the Mac mini M6 (32 GB).

GPU generation 18: works. The official binary loads the 27B on the M6. The smoke test passes (drafted = plain, prose and code). I removed my local patch.

Neural Engine fit check: the share still loads on 32 GB. With TF_ANE_COLUMNS unset:

The load does not print the "prompts run on the GPU alone" line, at 16K or 32K.
The footprint after load is 25.7 GB (16K) and 26.3 GB (32K). With the share off it is 19.7 and 20.3 GB.
Swap goes from 0 MB to 5.3 GB.

Cause. I added one log line to shareFits (qwen27_served.zig) and built 1.0.10:

Context	Footprint at the check	+ share	+ margin	70% cap	Result
16K	14,684 MiB	5,760 MiB	2,048 MiB	22,937 MiB	fits, 445 MiB left
32K	14,694 MiB	5,760 MiB	2,048 MiB	22,937 MiB	fits, 435 MiB left

Model.load calls shareFits before the DFlash2 drafter and the stream buffers load. These add about 4 GiB later. The server's own load line then says 24.0 GiB (16K) and 24.5 GiB (32K) "in use once loaded", which is more than the 22.4 GiB cap.

Possible fix. Make the check count the drafter and the streams. Two ways:

Decide the share after attach, when the drafter and the slots are loaded. Use the same "in use once loaded" value that the prompt cache line uses.
Or give shareFits an estimate of the bytes that load after Model.load (drafter + slotCost for the streams).

I can test a build with the fix on the M6. Until then I use TF_ANE_COLUMNS=0.

@RS91Dev

Thanks again, @RS91Dev. The fit-check fix is on the 1.1.2 line with your credit. The share now waits for the drafter and stream buffers before its check, its price includes a measured 32 MiB of private overhead, and its full size is reserved in the prompt-cache budget.

On an M5 Max with the memory limit set to that of a 32 GB Mac, a 256-column share is declined with a line saying why, and memory and tokens match the GPU-only run. With the full memory, 256 and 512 columns load after the stream slots are in place. With the share left unset, replies and memory match the build without the change.

The same checks found that the fp16 share changes the greedy reply within 64 tokens and gives no prefill gain on that Mac. So in 1.1.2 it stays opt-in: TF_ANE_COLUMNS=N turns it on, and only on GPUs with tensor units. We'll keep this open until 1.1.2 ships.

Posted by Ash's AI Assistant Synthena

qwen27: keep the projection weights once on Metal (ashhart#643)

qwen27: a round the GPU fails ends only its requests (ashhart#643)

CHANGELOG: ashhart#643

Merge ashhart#643 (the 27B's weights once on Metal, and failed rounds…

CHANGELOG: the 27B on GPU generation 18 (M6) (ashhart#643)

The 27B's Neural Engine share loads only inside the 70% load limit

CHANGELOG: the 27B's Neural Engine share inside the 70% load limit (a…

glm: prepare bounded host context and stream fit selection

qwen27: decide the Neural Engine share after the drafter and residenc…

Changelog: the contributed changes carried into 1.1.2, with their cre…

Thank you for the report. Version 1.1.2 accounts for the optional 27B Neural Engine share after the drafter and residency costs, including its private overhead and prompt-cache reservation. Please retest the original low-memory case and share sanitized startup and peak-memory results.

- Synthema, Ash's AI Assistant.

Thank you. I retested the original low-memory case with the official 1.1.2 binary (no local patch) on the Mac mini M6, 32 GB, macOS 27.0. Model: Vontra/Qwen3.8-27B-MLX-4bit with the z-lab/Qwen3.8-27B-DFlash2 drafter, --temperature 0.

Result: fixed. With TF_ANE_COLUMNS=3072 the share is refused at 16K and 32K. Peak memory stays below the 70% limit (22.4 GiB) during the full test suite.

Startup (sanitized), 16K, TF_ANE_COLUMNS=3072:

info: qwen27: 19.3 GiB kept wired between requests, inside 70% of 32 GiB of RAM
info: qwen27: the Neural Engine's prompt share (5760 MiB of fp16 weights) would take this Mac past its load limit (864 MiB left after its drafter, streams and images); prompts run on the GPU alone
info: qwen27: Neural Engine prompt share requested_columns=3072 loaded_columns=0 program_weight_bytes=0 loaded=false exact_output_qualified=false
info: prompt cache: 0.8 GiB from 0.8 GiB free under the 70% cap (19.6 GiB in use once loaded, 32 GiB of RAM, 2 GiB kept for prompt buffers)
info: Qwen3.8-27B: 1 streams of 16384 tokens, DFlash2 drafts for one of them
[tensorfold] prompt chunks of up to 512 tokens, cut at replies 256+ tokens apart
[tensorfold] serving Qwen3.8-27B-MLX-4bit at http://127.0.0.1:8080/v1 (sampling: greedy; drafts: on; context: 16384; loaded in 8.6s)

Startup (sanitized), 32K, TF_ANE_COLUMNS=3072:

info: qwen27: 20.3 GiB kept wired between requests, inside 70% of 32 GiB of RAM
info: qwen27: the Neural Engine's prompt share (5760 MiB of fp16 weights) would take this Mac past its load limit (0 MiB left after its drafter, streams and images); prompts run on the GPU alone
info: qwen27: Neural Engine prompt share requested_columns=3072 loaded_columns=0 program_weight_bytes=0 loaded=false exact_output_qualified=false
info: prompt cache: 0.0 GiB from 0.0 GiB free under the 70% cap (20.6 GiB in use once loaded, 32 GiB of RAM, 2 GiB kept for prompt buffers)
info: Qwen3.8-27B: 1 streams of 32768 tokens, DFlash2 drafts for one of them

Load matrix (physical_footprint from /memory, 5 s after "serving"):

Context	`TF_ANE_COLUMNS`	Share	Room at the check	Footprint	Peak	Prompt cache
16K	unset	off	-	19.59 GiB	19.62 GiB	0.8 GiB
16K	3072	refused	864 MiB	19.56 GiB	19.56 GiB	0.8 GiB
16K	256	loaded (480 MiB)	fits	20.06 GiB	20.42 GiB	0.3 GiB
32K	unset	off	-	20.56 GiB	20.56 GiB	0.0 GiB
32K	3072	refused	0 MiB	20.56 GiB	20.56 GiB	0.0 GiB
32K	256	refused	0 MiB	20.56 GiB	20.56 GiB	0.0 GiB

Peak memory under load, TF_ANE_COLUMNS=3072 (smoke, thinking budget, a recall prompt of about 16K / 32K tokens, and a 600-token reply drafted and plain):

	16K	32K
Peak footprint (`lifetime_peak_physical_footprint_bytes`)	20.72 GiB	20.76 GiB
Swap, start to end of suite	+139 MB	-84 MB
All tests pass, drafted = plain (byte-identical)	yes	yes
600-token reply, drafts on / off	29.1 / 8.9 tok/s	21.0 / 8.2 tok/s

Observations (not a request):

With the share off, the footprint after load is about 1.4 GB higher than 1.0.10 (19.59 vs 18.2-18.4 GiB at 16K). 1.1.2 serves 1 stream instead of 2 and reads prompts in 512-token chunks instead of 128. At 32K the prompt cache budget is 0 GiB.
The first cold load after idle (16K, share off) increased swap from 0 to 1.2 GB. The later loads did not increase swap.
Output hashes differ from 1.0.10 at the same prompts (drafted still equals plain), which I assume comes from the new prompt chunk and tile changes.

You can close the issue for the 32 GB case.

Thanks for confirming on 22 October that the 32 GB M6 case now works: Qwen3.8-27B with DFlash2 passes your 16K and 32K matrix. That path shipped in 1.1.2, and 1.1.3 keeps it. Closing; if a later build's footprint grows on your Mac again, please open a new issue with the build and the startup line.

- Synthema, Ash's AI Assistant.

Qwen3.8-27B on native 1.0.5: M6 (GPU generation 18) refused, and memory is too large for 32 GB #643

Description

Environment

Problem 1: GPU generation 18 is refused

Steps

Result

Cause

Change that I used

Problem 2: GPU memory is too large for 32 GB

Steps

Result

Measurements

Possible cause

Compare with the Python engine

Requests

Activity

RS91Dev commented on Oct 11, 2026

1. Accept GPU generation 18

2. Do not keep the weights two times

Cause

Fix

Result

3. Points to consider

4. How to test

ashhart commented on Oct 11, 2026

BryarGh commented on Oct 11, 2026

RS91Dev commented on Oct 12, 2026

Results

Memory

Short replies (4K server)

Long context (with drafts)

Decode speed with a long reply

Notes

ashhart commented on Oct 12, 2026

RS91Dev commented on Oct 13, 2026

TensorFold 1.0.8 on the Mac mini M6 (32 GB): test report

Summary

Setup

Reported issues in 1.0.8

Results (Neural Engine off, greedy)

Compare with the local fixes on 1.0.7

Neural Engine test (16K)

How to repeat

ashhart commented on Oct 13, 2026

RS91Dev commented on Oct 14, 2026

5 remaining items

ashhart commented on Oct 19, 2026

ashhart commented on Oct 21, 2026

RS91Dev commented on Oct 22, 2026

ashhart commented on Oct 23, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions