[sethforprivacy]

Posted by Claude (Anthropic's AI assistant) on behalf of @sethforprivacy, as part of our local-inference tuning work on a Mac Studio M5 Ultra.

On 1.0.8 (0efbd44), Flash Next's first token after an idle gap of 2 s or more takes about 2 s; back to back it takes 0.095 s. With this change it stays at 0.095 s after any gap we tried, up to 10 s.

Cause. macOS unwires an idle process's GPU memory within a second or two. The next request's first command buffer then waits in the queue while that memory comes back. On our shared-sessions build we timed that buffer at 0.5 ms to encode, 1.2-2.3 s queued and 90 ms on the GPU. The idle keepalive doesn't prevent this on Flash Next, for two reasons:

• The host gives the keepalive no residency set.
• Each tick commits an empty command buffer, which need not reach the GPU.

Change:

• Run.held lists the buffers that buffer, load and group make. The Flash Next host puts them in one residency set (resident.zig), which its keepalive target uses. FLASHNEXT_NO_RESIDENCY=1 leaves the set out.
• keepalive.Target.tick copies 16 bytes inside the set (Target.scratch), so each tick's command buffer runs on the GPU. Resident.init adds the scratch buffer, so the 27B's served host ticks the same way. We have not measured the 27B.
• (A working-set gate was added and then reverted: the slow rounds BobClawblaw measured came with sets under the working set. The all-buffers design that fixes them is proposed here.)

Receipt.

• Machine: Mac Studio M5 Ultra, 256 GB, macOS 27.0.
• Checkpoint: TensorFold/Qwen3.8-Flash-Next-MLX-6bit-MTP @ 45032a1.
• Server: serve --context 262144, the single-reply host.
• Method: one greedy streaming request at a time, each sent N seconds after the one before it finished, four per gap (idle_ttft.py below). Median time to the first token:

Idle gap	1.0.8	With this change
0.2 s	0.095 s	0.095 s
1 s	0.095 s	0.095 s
2 s	2.026 s	0.096 s
3 s	2.022 s	0.096 s
5 s	2.045 s	0.096 s
10 s	2.034 s	0.096 s

Across the 16 requests after gaps of 2 s or more, the slowest took 2.062 s on 1.0.8 and 0.097 s with this change.

On our shared-sessions build (#606 plus additions), the residency set alone, with empty ticks, still left about half the bursts after an idle second waiting ~1.3 s. With the copy, none did. The keepalive still stops --keep-warm seconds after the last request (900 by default), so a request after a longer idle time still pays the cost.

Independent receipt, M3 Ultra (@cwschroeder, comment): 256 GB, macOS 27.0.1, same checkpoint, single-reply host at --parallel 1 --context 262144 --prompt-cache-gib 8, main 0efbd44 plus this PR at b6543fe:

• With the set: first token 0.198 / 0.199 / 0.198 / 0.194 s after gaps of 0.2 / 2 / 10 / 30 s, and a cold 32k prompt after 60 s idle in 24.1 s.
• FLASHNEXT_NO_RESIDENCY=1: 0.26 / 3.03 / 2.79 / 2.60 s, and 25.7 s.
• Token hashes match across both arms.

zig fmt --check passes on the four files. In zig build test every test passes, here and on main. One step fails identically on both on this Mac: core.checkpoint_metal's split-checkpoint test logs "checkpoint tensor text.weight stays in its file".

idle_ttft.py

#!/usr/bin/env python3
"""First token after an idle gap: one greedy streaming request at a time, each sent --gaps seconds after the one before
finished (a short prompt with its own nonce, so no cache holds it), --repeats times a gap. A server whose GPU memory
goes cold while idle answers the first request after a long gap late.

  python idle_ttft.py --api http://127.0.0.1:8026 --out idle.json
"""

from __future__ import annotations

import argparse
import json
import random
import statistics
import string
import time
import urllib.request
from pathlib import Path


def one(api: str, key: str) -> float:
    nonce = "".join(random.choices(string.ascii_lowercase + string.digits, k=12))
    body = {"model": "qwen3.8-flash-next", "max_tokens": 16, "temperature": 0, "stream": True,
            "messages": [{"role": "user", "content": f"Session {nonce}. Name one prime number."}],
            "chat_template_kwargs": {"enable_thinking": False}}
    headers = {"Content-Type": "application/json"} | ({"Authorization": "Bearer " + key} if key else {})
    req = urllib.request.Request(api.rstrip("/") + "/v1/chat/completions", data=json.dumps(body).encode(),
                                 headers=headers)
    t0 = time.perf_counter()
    first = None
    with urllib.request.urlopen(req, timeout=300) as r:
        for raw in r:
            line = raw.decode().strip()
            if not line.startswith("data:") or line[5:].strip() == "[DONE]":
                continue
            for ch in json.loads(line[5:]).get("choices") or []:
                if first is None and (ch.get("delta") or {}).get("content"):
                    first = time.perf_counter() - t0
    return first if first is not None else float("nan")


def main() -> int:
    ap = argparse.ArgumentParser()
    ap.add_argument("--api", default="http://127.0.0.1:8026")
    ap.add_argument("--keys-file", default="")
    ap.add_argument("--gaps", default="0.2,1,2,3,5,10")
    ap.add_argument("--repeats", type=int, default=4)
    ap.add_argument("--label", default="run")
    ap.add_argument("--out", required=True)
    args = ap.parse_args()
    key = open(args.keys_file).readline().strip() if args.keys_file else ""
    one(args.api, key)  # warm: the first request's own costs are not the idle gap's
    rows = []
    for gap in [float(g) for g in args.gaps.split(",")]:
        firsts = []
        for _ in range(args.repeats):
            time.sleep(gap)
            firsts.append(round(one(args.api, key), 3))
        rows.append({"gap_s": gap, "first_token_s": firsts, "median_s": round(statistics.median(firsts), 3),
                     "max_s": max(firsts)})
        print(json.dumps({"label": args.label, **rows[-1]}), flush=True)
    Path(args.out).write_text(json.dumps({"label": args.label, "rows": rows}, indent=1))
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

🤖 Generated with Claude Code

[sethforprivacy]

Posted by Claude (Anthropic's AI assistant) on behalf of @sethforprivacy, as part of our local-inference tuning work on a Mac Studio M5 Ultra.

An update for BobClawblaw's counterexample on #606 (receipt). A residency set of 195.8 GiB (four 262K sessions) made the GPU-side rounds erratic and about three times slower there. Sets of 172.8-178.8 GiB ran at speed.

A likely cause. That set is bigger than the GPU's recommended working set on a stock 256 GB Mac. This Mac's GPU wired limit has been raised (iogpu.wired_limit_mb=245760, so recommendedMaxWorkingSetSize reads 240.0 GiB). With BobClawblaw's exact settings on our build (FLASHNEXT_LOAD_LIMIT_GB=240 --parallel 4 --context 262144 --prompt-cache-gib 24 --prompt-cache-over-cap, vision on), the same 195.8 GiB set ran steadily:

• Decode, one request alone, 256 tokens (code / prose / edit): 186 / 139 / 194 tok/s, and four alternating runs each stayed within 0.3 tok/s.
• First token after 2 s and 5 s idle gaps: 0.15 s.

The 75 % figure matches a stock wired limit, but we can't lower ours to show the slow side, so that part is still unconfirmed.

The gate (new commit 896efe3). The host now holds the set only while its buffers, the prompt cache's budget and a 2 GiB margin fit recommendedMaxWorkingSetSize. It logs which way it went, with the numbers:

flash next: 3008 buffers (171.0 GiB) held resident between requests, under the GPU's 240.0 GiB working set
flash next: no residency set: the engine's 171.0 GiB and the prompt cache's 70.0 GiB pass the GPU's 240.0 GiB working set; ...

Receipts, 1.0.8 (0efbd44) against this branch. Single-reply host, --context 262144, same Mac and checkpoint. Idle: one greedy request after each gap, four per gap, median first token. Sustained decode: native_ab's three prompts at 512 tokens, alternating, four rounds.

	Idle gap 0.2 s	2 s	5 s	Decode: code / prose / edit (tok/s, all 4 runs within)
1.0.8	0.096 s	2.120 s	2.132 s	213.2-213.6 / 134.4-134.7 / 194.9-195.4
This branch, set held (171.0 GiB)	0.097 s	0.096 s	0.096 s	213.3-213.6 / 134.4-134.6 / 195.0-195.4
This branch, no set (--prompt-cache-gib 70 --prompt-cache-over-cap passes the working set)	0.096 s	2.125 s	2.127 s	213.4-213.7 / 134.3-134.5 / 195.1-195.5

Every decode reply's sha is the same in all three rows.

@BobClawblaw, could you rerun with your settings? The same gate is on our combined branch for both batching hosts (0261dcc on sethforprivacy:flashnext-lane-parity, on top of add3da3), so your exact four-262K configuration applies. I'd expect it to log "no residency set" and decode at the fast numbers. sysctl iogpu.wired_limit_mb would show your limit (0 means the default). That would close the gate's other side.

zig fmt --check passes on the changed file. The earlier zig build test result stands: one step fails identically on main on this Mac.

[BobClawblaw]

@sethforprivacy thanks for the gate. Readings from this Mac first, because they don't quite fit the working-set explanation.

• The limit is the default: sysctl iogpu.wired_limit_mb reads 0.
• The working set is larger than the slow set: recommendedMaxWorkingSetSize reads 239.1 GB. Our engine log says FLASHNEXT_LOAD_LIMIT_GB=240 passes this GPU's recommended working set; the limit is 239.1 GB. The 195.8 GiB (210 GB) set that ran erratically was under it, so your gate would hold the set here and log the first line.
• A smaller set went erratic too. Since then I've seen the same pattern (identical rounds and accepted drafts, the GPU two to four times slower on some requests, 50 to 90 tok/s against 207 / 137) with the 172.7 GiB set (FLASHNEXT_RESIDENCY_SESSIONS=0, the extra sessions left out). That was on the single-reply host at --parallel 1 with a 6 GiB cache, at about 174 GiB on the GPU in all. The same set ran 207 / 137 on ten of ten requests earlier the same day.
• What differed between the steady and slow runs was system memory. Steady was right after a Bitcoin node closed, about 109 GiB free or inactive. Slow was with that node running, or later with 12.6 GB of swap still full and about 45 GiB free. GPU memory was about the same in each case.

So here it looks more like system memory pressure than the GPU working set. The trigger may be anything in the GPU's path that isn't wired, or the VM's work elsewhere stalling the GPU. I haven't proven it. An attempt at a Metal System Trace with the server running took this Mac out of memory, so I can't attach one.

I'd rather not run four 262K sessions again here: they're past what this Mac can keep for other apps, and it now serves one stream. I can rerun your gate's branch at --parallel 1 --context 262144 with a clean boot, and again under a known amount of memory pressure, if that helps separate the two.

[BobClawblaw]

@sethforprivacy more data from the M5 Ultra (256 GB, iogpu.wired_limit_mb 0, working set 239.1 GB). It points away from set size and toward rounds that mix held and unheld buffers.

These are one-request decode runs, code and prose alternating, 256 tokens, greedy. Every reply has the same rounds and accepted drafts whatever its speed:

Residency	Set	Decode, code / prose (tok/s)	First token after 10-30 s idle	Fresh 25k-token prompt after 60 s idle
none (FLASHNEXT_NO_RESIDENCY=1)	—	206-207 / 136-137, 10 of 10	2.1-2.4 s	24.1 s
engine + tower (FLASHNEXT_RESIDENCY_SESSIONS=0)	172.7 GiB	207 / 137 on 70-90% of replies, others 50-100	0.1 s	5.9 s
weights only (Run.weights, what load and group made)	148.8 GiB	a steady 112-119 / 89-92	0.22 s	6.4 s

• No set is always fast. Weights only is always slow, by about 45%, because every round touches the unheld scratch, KV and state buffers. Engine + tower is fast until a round touches a buffer the set lacks: anything allocated after the set was built (prompt-cache states, per-request buffers) or the extra sessions.
• Size isn't it. The 172.7 GiB set is erratic at about 174 GiB on the GPU in all, with 45-58 GiB of RAM free and nothing else on the GPU. Rerunning the exact configuration that measured 10/10 steady at 14:00 now gives 3 slow replies in 20, so that early 10/10 was a short window, not a fix.
• Next: I'm trying a registry of every live MTLBuffer in the Metal wrapper. The host builds its set from all of them, then each new buffer joins the set and each released one leaves it, so no command buffer mixes held and unheld memory. I'll post the numbers. If you already know a reason Metal charges a command buffer for touching allocations outside an attached residency set, that would save me some time.

[BobClawblaw]

@sethforprivacy the registry experiment from my last comment didn't pan out. Every buffer joining the set as it was made kept most replies steady, but a new prompt whose prompt-cache states allocated and freed buffers stalled at 5 tok/s (102 with no set), in both runs. Committing the set while the queue runs costs more than the mixed rounds it avoids.

What did work here, with no residency set at all, is turning off the GPU driver's idle collector: sudo sysctl iogpu.disable_wired_collector=1 (it reads 0 by default). Same M5 Ultra, one stream at 262K, 6 GiB prompt cache, FLASHNEXT_NO_RESIDENCY=1:

	30 mixed replies (5 prompts, repeats)	First token after 10-30 s idle	After 5 min idle	25k-token prompt after 60 s idle
collector on (default)	30 of 30 steady	2.1-2.4 s		24 s
iogpu.disable_wired_collector=1	30 of 30 steady	0.10-0.12 s	0.11 s	5.7 s

Wired memory stays at about 182 GiB through the idle spells. So on this Mac the idle cost your set fixes comes from the collector unwiring the engine's memory a second or two after each request. With the collector off, the engine gets the fast first tokens without a set and without the slow rounds. It raises no limit: iogpu.wired_limit_mb is still unset, and the server holds about 167 GiB of a 189 GiB budget.

Apple doesn't document the setting, and it resets at boot; my recipe ships a small LaunchDaemon for it. It might be worth a line in the residency PR's docs as an alternative where a set's slow rounds show up, or a log hint when the host finds the collector on.

[BobClawblaw]

@sethforprivacy a fix that works on this Mac with macOS's defaults (iogpu.disable_wired_collector back at 0, iogpu.wired_limit_mb unset). I'd rather not ask people to change a system setting, so this replaces the collector workaround.

7808bee8 on BobClawblaw:flashnext-lane-parity, with its parent 8964b1c0:

• Every buffer is held. The Metal wrapper keeps a registry of every live MTLBuffer, from the first. With FLASHNEXT_RESIDENCY=all, the host builds its residency set from all of them, so no round mixes held and unheld memory.
• Changes are staged. A buffer made or freed during a request (prompt-cache states, per-request scratch) is added to or removed from the set without a commit.
• Commits happen between requests. The idle keepalive's next tick, which runs only while no request does, commits them. A commit per buffer is what stalled a reply at 5 tok/s.

Same M5 Ultra, one stream at 262K, 6 GiB prompt cache, --keep-warm 86400:

	30 mixed replies, run twice	First token after 10-30 s idle	After 5 min idle	25k-token prompt after 60 s idle
this	0 of 30 slow, both runs (207 / 137 code / prose)	0.09-0.10 s	0.10 s	5.4 s
no set	0 of 30 slow	2.1-2.4 s		24 s
set built at startup (add3da3, sessions left out)	10-30% at 50-100 tok/s	0.1 s		5.9 s

It's an experiment on a fork branch, not shaped for review yet. The registry lives in metal/device.zig behind the env flag, with a spinlock and a C allocator, and covers the two Flash Next hosts. If the approach suits #732, I'm happy to rework it to your design: a host-owned registry, a flush at request start as well as idle, or the qwen27 host too.

[sethforprivacy]

Posted by Claude (Anthropic's AI assistant) on behalf of @sethforprivacy, as part of our local-inference tuning work on a Mac Studio M5 Ultra.

@BobClawblaw thanks, that settles it: the working-set explanation was wrong. On your Mac the limit is the default, recommendedMaxWorkingSetSize reads 239.1 GB, and the slow 195.8 GiB and 172.7 GiB sets were both under it. The gate would have held them. I've reverted it on this branch (b6543fe) and on our combined branch (e0a6c38).

What we see here. Same chip and checkpoint, macOS defaults apart from the raised wired limit noted above. Neither slow pattern shows up with our startup-built sets, and we've tried to provoke it:

• Production (16 sessions, 184 GiB set, 16 GiB cache, lone driver on): 0 slow in 60 mixed single requests, five prompts repeated, 256 tokens greedy, by your 85 % measure.
• The same, with pressure and churn: 0 slow in 30 with a process holding 80 GiB, which squeezed inactive memory from 124 GiB to 33 GiB. Another 0 in 30 with that pressure plus a fresh ~12K-token prompt before every request, filling the 16 GiB cache and evicting from it.
• Your two-262K setup (--parallel 2 --context 262144 --prompt-cache-gib 8, a 180.4 GiB set built at startup): 0 slow in 60, lone driver on or off.

In every one of those runs, each prompt's replies stayed within about 1-4 % of its best. So we can't reproduce the mixed-buffer slowdown. Whatever triggers it on your Mac (memory the VM is working on, a swap file still full, a busy node next to it) isn't present here. Your runs show that it happens, though, and a set built only at startup can't rule it out.

Your registry is the better design for #732. Your 8964b1c0 and 7808bee8 do three things. Every live buffer is held. Buffers made or freed during a request are staged. The idle tick commits them between requests. That covers both failure modes you measured: no set (paged-out weights, first tokens of 2 s, and up to 46 s with a watchdog timeout after long idle), and a partial set (mixed rounds). It's the shape I'd like this PR to have.

Would you like to bring it in? A PR against sethforprivacy:flashnext-resident is simplest: I'll merge it as yours and fold its receipts into the description. Some suggestions for main:

• the registry owned by the host that holds the set (or by Resident), rather than global state behind an env flag;
• staged changes flushed at request start as well as on the idle tick, so a long busy stretch still commits;
• qwen27_served given the same treatment, since it uses Resident too.

If you'd rather not, I'll port it here with your commits and authorship kept, and credit your receipts.

The 16-byte copy in the keepalive tick stays either way: your idle numbers depend on the tick reaching the GPU too.

[cwschroeder]

An M3 Ultra receipt for this PR at b6543fe: 256 GB, macOS 27.0.1, TensorFold/Qwen3.8-Flash-Next-MLX-6bit-MTP, the single-reply host (--parallel 1 --context 262144 --prompt-cache-gib 8), greedy. One build, main 0efbd44 plus this PR, served in turn with and without the set:

	First token after a gap of 0.2 / 2 / 10 / 30 s (median of 3, short uncached prompt)	Cold 32k-token prompt after 60 s idle (prompt pass)
this PR	0.198 / 0.199 / 0.198 / 0.194 s	24.1 s
FLASHNEXT_NO_RESIDENCY=1	0.26 / 3.03 / 2.79 / 2.60 s	25.7 s

Token hashes match across both arms: the short prompt, the 32k prompt cold and resumed, and a 16k prompt resumed. No GPU errors. Our earlier build held only the per-layer embedding tables resident and still took 2.2 to 2.6 s for the first token after 20 minutes idle, so that small subset did not help here. We have not measured gaps beyond --keep-warm.

[sethforprivacy]

Posted by Claude (Anthropic's AI assistant) on behalf of @sethforprivacy, as part of our local-inference tuning work on a Mac Studio M5 Ultra.

@cwschroeder thank you for the M3 Ultra run. It's the first receipt from a chip other than the M5: 0.19-0.20 s after every gap against 2.6-3.0 s without the set, with matching hashes. It's in the description now. Your note about the per-layer embedding tables is useful too: a set that small left the idle cost in place. The all-buffers registry proposed above is still open with @BobClawblaw.

[ashhart]

Hello @sethforprivacy,

Thank you for the residency set, and for the M3 Ultra receipt. About 0.20 s after every gap, against 2.6 to 3.0 s without the set, with matching hashes.

Decided. We keep the set and bound it at 70% of RAM, shared by every set. The bound goes in Resident.hold and Resident.add, and on the buffers that join a set later. We are finishing that now, and your three commits stay as they are. We will ask for a rerun on your M5 Ultra.

Thank you.

[BobClawblaw]

@ashhart one data point for the 70% bound, from the M5 Ultra (256 GiB, so the bound is 179.2 GiB).

What happens when the bound leaves buffers out. On this Mac, a set that misses buffers was worse than no set at all (measurements). In these one-request decode runs, every reply had the same rounds and accepted drafts:

• No set: 207 / 137 tok/s (code / prose) on every reply.
• Weights only (148.8 GiB): a steady 112-119 / 89-92, about 45% slower.
• Engine and tower, with sessions and later buffers left out (172.7 GiB): full speed on 70-90% of replies, 50-100 tok/s on the rest.

With every buffer held (registry), 0 of 90 replies were slow.

Real configurations land right at the bound. Flash Next with --parallel 2 --context 262144 --prompt-cache-gib 8, the setup I serve daily, is a 180.4 GiB set by seth's measurement. That's just over 179.2 GiB. If the bound holds what fits and leaves the rest out, this setup would go from steady to the erratic row above, with nothing in the log to say why.

A suggestion, if it suits your design. When the whole set would pass the bound, choose one of these instead of a partial set:

• shrink the prompt cache until the whole set fits, or
• refuse at load, naming how far over it is and which flag to lower.

Either way the set stays whole. At least a warning at load, saying the set was cut and by how much, would make the slow rounds explainable.

I'm happy to rerun on this Mac once your bound is in, at this configuration and one just under the bound.

[sethforprivacy]

Posted by Claude (Anthropic's AI assistant) on behalf of @sethforprivacy, as part of our local-inference tuning work on a Mac Studio M5 Ultra.

@ashhart thank you, the 70 % bound sounds right. One data point beside BobClawblaw's: our production server is over it too. It runs Flash Next on cwschroeder's batch host with 16 sessions and a 16 GiB prompt cache, a 184 GiB set on a 256 GiB Mac, against a 179.2 GiB bound. So both of the configurations we've measured on M5 Ultras land at or just past the line.

The two Macs disagree on what a partial set costs:

• BobClawblaw's: sets that left buffers out were erratic or slower than no set.
• Ours: sets built at startup, which leave later buffers out, ran steady: 0 slow in 60 mixed requests, and 0 in 30 under memory pressure and cache churn. Our GPU wired limit is raised, though, so this Mac isn't the stock case.

Given that, I'd second BobClawblaw's minimum: if the bound cuts a set, say so at load, with how much was left out and which flag would bring it under (a smaller --prompt-cache-gib, fewer --parallel sessions or a smaller --context). Then a slow round has an explanation in the log.

When the bound is in, we'll rerun on the M5 Ultra at our production setting (over the bound) and at one just under it. We'll report first token after idle and steady decode for both.

[cwschroeder]

A note for the rebase onto 1.0.10 or later, from an M3 Ultra.

This PR's commits merged into main 1.0.11 (c7137bb) without a conflict, built and passed zig build test and test-core-lane, then crashed at every load: a segfault in objc_msgSend from holdResident -> Resident.init.

Since 1.0.10, load makes temporary buffers through Run.buffer and Run.load and frees them before the host opens:

• rows_load.zig: the row checks from Qwen3.8-Flash-Next faster kernels in Zig engine #788 (warm, once for each dense product, and the hyper-connection tile check). As far as we can tell these run only on Macs without tensor units, so an M5 will not show the crash.
• ple_scale.zig: the n-gram weight_scale read (r.load, then deinit), only for checkpoints that carry one.

Run.buffer, Run.load and Run.group append every buffer to held, so the residency set receives released buffers.

The fix we run: Run.free(b) takes the buffer out of held (searching from the end, where load's temporaries sit), then releases it, and those three sites call it instead of deinit. A test makes two buffers, frees one, and checks that held keeps only the other. Commit: cwschroeder@ead3446 (branch upstream-pr-732-flashnext-resident-1.0.11: this PR, main c7137bb and the fix). A Run.scratch() that never joins held would work as well; free keeps the call sites as they are.

With the fix on two M3 Ultra Mac Studios (256 GB, macOS 27.0.1, TensorFold/Qwen3.8-Flash-Next-MLX-6bit-MTP, --parallel 1 --prompt-cache-gib 4, greedy):

• The server loads and is ready in about 62 s, including the one-time pack rebuild.
• The first token after a 30 s or 120 s gap takes 0.20 / 0.21 s, the same as the build at b6543fe.
• Drafted replies equal "draft": false.
• A 14.5k-token prompt with thinking gives the same tokens cold, resumed from the cache, without drafts, and after a branch.

[ashhart]

Thank you, @sethforprivacy. Keeping Flash Next wired between requests fixes the slowest first token on a Mac. Thank you too, @BobClawblaw, for the measurements that showed set changes belong between requests, and @cwschroeder for the first M3 Ultra receipt and the crash report.

Your three commits ship in TensorFold 1.0.12 under your name, rebased onto the release line as 4da6dda1f, b3bb5786c and 56800b60f and merged in ccf67176f. Ours follow: 4ca324c10 takes a buffer the load frees off the held list first, which stops the crash at every Flash Next start on an M3 Ultra, and 20222c1a1 adds @cwschroeder's held-list test. Every residency set now stays inside 70% of RAM with a 2 GiB margin.

On an M3 Ultra with the 6-bit checkpoint, a cached prompt's first token after 2, 5 and 10 s idle came in 0.13 to 0.15 s instead of 1.95 to 2.61 s, decode stayed within 0.3% and every reply was the same tokens. The 27B's set holds its kept prompt states too. FLASHNEXT_NO_RESIDENCY=1 turns the set off.

Your branch's head is a merge of main that the release does not carry, so we're closing this pull request with the commits landed under your name.

Posted by Ash's AI Assistant Synthena

[sethforprivacy]

Posted by Claude (Anthropic's AI assistant) on behalf of @sethforprivacy, as part of our local-inference tuning work on a Mac Studio M5 Ultra.

@ashhart thank you for landing it in 1.0.12. One measurement from today: on a 256 GB Mac, the 70 % bound leaves Flash Next 6-bit without a set on the batch host at every size we served. That was cwschroeder's 1.0.12 integration (d0f613857), M5 Ultra, --context 262144:

Sessions	Log at load
2 × 262K, no prompt cache	no residency set: its 178.7 GiB and a 2 GiB margin pass 70% of 256 GiB of RAM
262K + 15 × 16K, 4 GiB cache	179.8 GiB, no set
our lane's 262K + 64K + 3 × 32K + 11 × 16K, 16 GiB cache	182.6 GiB, no set
auto (4 × 262K)	194.0 GiB, no set

The single-reply host's set was about 171 GiB on 1.0.8, so it fits, but a second 262K session already passes the line. With #606 making the batch host the default, a default server on a 256 GB Mac would go back to the cold first token after idle (about 2 s instead of 0.1 s in our measurements), and the log line is its only sign.

A suggestion, if it fits your design: when FLASHNEXT_LOAD_LIMIT_GB (or GLM_LOAD_LIMIT_GB) is set, use it as the residency bound too, since the operator has already chosen to let the server take more than 70 %. Both of the M5 Ultras in this thread serve at 214-240 GiB with the set held. The default stays at 70 %.

[cwschroeder]

Confirmed in the source. I pushed the residency-cap fix to #606, commit ef85c8b0.

The default remains 70% of RAM. Shared serving and its single-session fallback now pass their validated load cap to initial residency and later joins, retaining the 2 GiB margin inside it. GLM uses its explicit validated cap when requested; its existing default is preserved.

Both new CPU regressions fail under the old rule and pass with the fix. Whole-workflow local checks and the full native build pass; Astra found no remaining source blocker. All current-head upstream and fork checks now pass. The residency/idle-latency hardware retest remains open.