Skip to content

5090+256G内存无法运行红帽的GLM5.3FlashNVFP4 #397

Open
Open
5090+256G内存无法运行红帽的GLM5.3FlashNVFP4#397
Labels
Desktopproblem related to FreeToken DesktopbugSomething isn't workinglinux

Description

@Zanry
Zanry
opened on Sep 5, 2026

Before you start

  • I have read the FAQ and my problem is not answered there.
  • I have read the Roadmap and this is not already planned there.
  • I have searched existing issues and found no duplicate.
  • I have restarted the Desktop app to pick up the latest update and the problem still happens.

What happened

5090+256G内存无法运行红帽的GLM5.3FlashNVFP4，每次加载到77%就挂掉，调整了几个参数也是一样。

Desktop app version

v0.2.0-beta.17

OS

Ubuntu

OS details

anduinOS（基于26.04）

GPU and driver

RTX 5090, driver 595.84

CPU and system RAM

Ryzen 7 9850X3D

Checkpoint

RedHatAI/GLM-5.3-Flash-NVFP4

Model settings

No response

Engine log

[02:53:29] stdout/INFO  [2026-09-06|02:53:29|core|rank=0] INFO     GLM-5.3 resident quant: attn=none dense=none lm_head=none (FREETOKEN_GLM5_ATTN_FP8/FREETOKEN_GLM5_MLP_FP8; an FTW conversion records these choices implicitly -- serve with the same flags)
[02:53:29] stderr/INFO  Converting dense weights: 0it [00:00, ?it/s]
[02:53:29] stderr/INFO  Converting dense weights: 19it [00:00, 92.74it/s]
[02:53:29] stderr/INFO  Converting dense weights: 36it [00:00, 115.74it/s]
[02:53:29] stderr/INFO  Converting dense weights: 48it [00:00, 98.64it/s] 
[02:53:29] stderr/INFO  Converting dense weights: 85it [00:00, 129.51it/s]
[02:53:30] stderr/INFO  Converting dense weights: 106it [00:00, 138.95it/s]
[02:53:30] stderr/INFO  Converting dense weights: 127it [00:01, 150.97it/s]
[02:53:30] stderr/INFO  Converting dense weights: 152it [00:01, 160.90it/s]
[02:53:30] stderr/INFO  Converting dense weights: 175it [00:01, 177.93it/s]
[02:53:30] stderr/INFO  Converting dense weights: 196it [00:01, 181.45it/s]
[02:53:30] stderr/INFO  Converting dense weights: 217it [00:01, 182.78it/s]
[02:53:30] stderr/INFO  Converting dense weights: 242it [00:01, 184.43it/s]
[02:53:30] stderr/INFO  Converting dense weights: 265it [00:01, 189.06it/s]
[02:53:31] stderr/INFO  Converting dense weights: 286it [00:01, 185.27it/s]
[02:53:31] stderr/INFO  Converting dense weights: 307it [00:01, 178.43it/s]
[02:53:31] stderr/INFO  Converting dense weights: 332it [00:02, 186.52it/s]
[02:53:31] stderr/INFO  Converting dense weights: 355it [00:02, 190.55it/s]
[02:53:31] stderr/INFO  Converting dense weights: 376it [00:02, 185.99it/s]
[02:53:31] stderr/INFO  Converting dense weights: 397it [00:02, 184.36it/s]
[02:53:31] stderr/INFO  Converting dense weights: 445it [00:02, 199.65it/s]
[02:53:31] stderr/INFO  Converting dense weights: 466it [00:02, 186.72it/s]
[02:53:32] stderr/INFO  Converting dense weights: 487it [00:02, 179.83it/s]
[02:53:32] stderr/INFO  Converting dense weights: 512it [00:03, 181.93it/s]
[02:53:32] stderr/INFO  Converting dense weights: 535it [00:03, 193.01it/s]
[02:53:32] stderr/INFO  Converting dense weights: 556it [00:03, 183.23it/s]
[02:53:32] stderr/INFO  Converting dense weights: 577it [00:03, 177.30it/s]
[02:53:32] stderr/INFO  Converting dense weights: 602it [00:03, 179.24it/s]
[02:53:32] stderr/INFO  Converting dense weights: 629it [00:03, 196.03it/s]
[02:53:32] stderr/INFO  Converting dense weights: 650it [00:03, 196.56it/s]
[02:53:33] stderr/INFO  Converting dense weights: 670it [00:03, 189.36it/s]
[02:53:33] stderr/INFO  Converting dense weights: 692it [00:04, 181.70it/s]
[02:53:33] stderr/INFO  Converting dense weights: 715it [00:04, 193.89it/s]
[02:53:33] stderr/INFO  Converting dense weights: 736it [00:04, 187.97it/s]
[02:53:33] stderr/INFO  Converting dense weights: 757it [00:04, 175.78it/s]
[02:53:33] stderr/INFO  Converting dense weights: 782it [00:04, 180.03it/s]
[02:53:33] stderr/INFO  Converting dense weights: 805it [00:04, 189.73it/s]
[02:53:33] stderr/INFO  Converting dense weights: 826it [00:04, 184.96it/s]
[02:53:34] stderr/INFO  Converting dense weights: 847it [00:04, 169.96it/s]
[02:53:34] stderr/INFO  Converting dense weights: 892it [00:05, 184.43it/s]
[02:53:34] stderr/INFO  Converting dense weights: 911it [00:05, 173.31it/s]
[02:53:34] stderr/INFO  Converting dense weights: 929it [00:05, 149.11it/s]
[02:53:34] stderr/INFO  Converting dense weights: 945it [00:05, 133.12it/s]
[02:53:34] stderr/INFO  Converting dense weights: 962it [00:05, 133.57it/s]
[02:53:35] stderr/INFO  Converting dense weights: 986it [00:05, 158.80it/s]
[02:53:35] stderr/INFO  Loading GLM-5.3 dense weights: 100%|██████████| 45/45 [00:05<00:00,  7.71it/s]
[02:53:37] stdout/INFO  [2026-09-06|02:53:37|core|rank=0] INFO     expert banks: slow path (parallel build)
[02:53:37] stdout/WARN  [2026-09-06|02:53:37|core|rank=0] WARNING  parallel reader unavailable (freetoken.models.glm5_next provides no load_nvfp4_expert_sources_parallel); falling back to serial build
[02:53:38] stderr/INFO  Loading GLM-5.3 NVFP4 experts (compressed-tensors):   0%|          | 0/10 [00:00<?, ?it/s]
[02:53:39] stderr/INFO  Converting expert banks: 0.00B [00:00, ?B/s]
[02:53:41] stderr/INFO  Converting expert banks: 3.80GB [00:00, 5.63GB/s]
[02:53:42] stderr/INFO  Converting expert banks: 7.60GB [00:02, 3.03GB/s]
[02:53:45] stderr/INFO  Loading GLM-5.3 NVFP4 experts (compressed-tensors):  10%|█         | 1/10 [00:06<00:54,  6.09s/it]
[02:53:46] stderr/INFO  Converting expert banks: 15.2GB [00:06, 2.06GB/s]
[02:53:48] stderr/INFO  Converting expert banks: 19.0GB [00:08, 2.34GB/s]
[02:53:50] stderr/INFO  Converting expert banks: 22.8GB [00:09, 2.36GB/s]
[02:53:52] stderr/INFO  Converting expert banks: 26.6GB [00:11, 2.33GB/s]
[02:53:54] stderr/INFO  Loading GLM-5.3 NVFP4 experts (compressed-tensors):  20%|██        | 2/10 [00:15<01:02,  7.83s/it]
[02:53:57] stderr/INFO  Converting expert banks: 34.2GB [00:16, 2.07GB/s]
[02:54:00] stderr/INFO  Converting expert banks: 38.0GB [00:19, 1.74GB/s]
[02:54:03] stderr/INFO  Converting expert banks: 41.8GB [00:22, 1.60GB/s]
[02:54:07] stderr/INFO  Loading GLM-5.3 NVFP4 experts (compressed-tensors):  30%|███       | 3/10 [00:28<01:13, 10.50s/it]
[02:54:10] stderr/INFO  Converting expert banks: 49.4GB [00:29, 1.34GB/s]
[02:54:14] stderr/INFO  Converting expert banks: 53.2GB [00:32, 1.33GB/s]
[02:54:17] stderr/INFO  Converting expert banks: 57.0GB [00:35, 1.28GB/s]
[02:54:20] stderr/INFO  Converting expert banks: 60.8GB [00:39, 1.24GB/s]
[02:54:24] stderr/INFO  Loading GLM-5.3 NVFP4 experts (compressed-tensors):  40%|████      | 4/10 [00:44<01:15, 12.61s/it]
[02:54:28] stderr/INFO  Converting expert banks: 68.4GB [00:45, 1.23GB/s]
[02:54:31] stderr/INFO  Converting expert banks: 72.2GB [00:49, 1.18GB/s]
[02:54:35] stderr/INFO  Converting expert banks: 76.0GB [00:53, 1.19GB/s]
[02:54:39] stderr/INFO  Loading GLM-5.3 NVFP4 experts (compressed-tensors):  50%|█████     | 5/10 [01:00<01:09, 13.89s/it]
[02:54:43] stderr/INFO  Converting expert banks: 83.6GB [01:01, 1.09GB/s]
[02:54:47] stderr/INFO  Converting expert banks: 87.4GB [01:04, 1.09GB/s]
[02:54:50] stderr/INFO  Converting expert banks: 91.2GB [01:08, 1.08GB/s]
[02:54:54] stderr/INFO  Converting expert banks: 95.0GB [01:11, 1.13GB/s]
[02:54:58] stderr/INFO  Loading GLM-5.3 NVFP4 experts (compressed-tensors):  60%|██████    | 6/10 [01:18<01:00, 15.10s/it]
[02:55:02] stderr/INFO  Converting expert banks: 103GB [01:19, 1.10GB/s] 
[02:55:06] stderr/INFO  Converting expert banks: 106GB [01:23, 1.05GB/s]
[02:55:10] stderr/INFO  Converting expert banks: 110GB [01:27, 1.06GB/s]
[02:55:15] stderr/INFO  Loading GLM-5.3 NVFP4 experts (compressed-tensors):  70%|███████   | 7/10 [01:36<00:48, 16.21s/it]
[02:55:17] stderr/INFO  Converting expert banks: 118GB [01:36, 938MB/s] 
[02:55:22] stderr/INFO  Converting expert banks: 122GB [01:39, 1.09GB/s]
[02:55:26] stderr/INFO  Converting expert banks: 125GB [01:43, 1.05GB/s]
[02:55:30] stderr/INFO  Converting expert banks: 129GB [01:47, 1.02GB/s]
[02:55:34] stderr/INFO  Loading GLM-5.3 NVFP4 experts (compressed-tensors):  80%|████████  | 8/10 [01:54<00:33, 16.80s/it]
[02:55:38] stderr/INFO  Converting expert banks: 137GB [01:55, 1.03GB/s]
[02:55:42] stderr/INFO  Converting expert banks: 141GB [01:59, 1.02GB/s]
[02:55:46] stderr/INFO  Converting expert banks: 144GB [02:03, 1.02GB/s]
[02:55:51] stderr/INFO  Loading GLM-5.3 NVFP4 experts (compressed-tensors):  90%|█████████ | 9/10 [02:12<00:16, 16.98s/it]
[02:55:54] stderr/INFO  Converting expert banks: 152GB [02:12, 951MB/s] 
[02:55:58] stderr/INFO  Converting expert banks: 156GB [02:15, 1.04GB/s]
[02:55:58] stderr/INFO  Loading GLM-5.3 NVFP4 experts (compressed-tensors): 100%|██████████| 10/10 [02:21<00:00, 14.12s/it]
[02:55:59] stdout/INFO  [2026-09-06|02:55:59|core|rank=0] INFO     NVFP4 expert backend: triton
[02:55:59] stdout/INFO  wrote FTW checkpoint -> /media/storage_zone/Virtual Machines/wsl/home/models/.convert-GLM-5.3-Flash-NVFP4
[02:55:59] stdout/INFO    tensors: 1008 weight + 252 experts_bank
[02:55:59] stdout/INFO    FTW: 176.34 GiB across 24 shard(s) (<= 8.0 GiB each)
[02:55:59] stdout/INFO    quant_format: nvfp4  fingerprint=b7f7ae36125c5f78
[02:55:59] stdout/INFO    converted in 150.1s
[02:56:09] stderr/INFO  kernel warmup skipped for GLM-5.3-Flash-NVFP4: engine busy or daemon unavailable
[02:56:15] cmd/INFO  ft serve --model /media/storage_zone/Virtual Machines/wsl/home/models/GLM-5.3-Flash-NVFP4 --port 1919 --moe-backend auto --max-running-requests 4 --memory-ratio 0.88 --host 192.168.2.100 --cors-origins tauri://localhost,http://tauri.localhost,http://localhost:1420 --gpu GPU-bbc4d898-b6b0-e369-23c5-3bf5dc3047de
[02:56:18] stdout/INFO  [2026-09-06|02:56:18] INFO     Parsed arguments:
[02:56:18] stdout/INFO  ServerArgs(model_path='/media/storage_zone/Virtual Machines/wsl/home/models/GLM-5.3-Flash-NVFP4', tp_info=DistributedInfo(rank=0, size=1), dtype=torch.bfloat16, max_running_req=4, attention_backend='auto', moe_backend='auto', nvfp4_backend='triton', ple_backend='disk', expert_load='auto', moe_cache_size=0, moe_cache_rate=None, moe_cache_auto=False, kv_reserve_tokens=8192, moe_cache_policy='lru', moe_prefill_overlap=True, moe_prefill_hit_d2d=False, moe_collect_stats=False, moe_cpu_threads=0, moe_cpu_layers=None, moe_hybrid_max_fetch=-1, cuda_graph_bs=None, cuda_graph_max_bs=None, page_size=1, memory_ratio=0.88, linear_state_cache_ratio=2.0, swa_full_tokens_ratio=0.2, swa_num_pages_override=None, distributed_timeout=60.0, use_dummy_weight=False, use_pynccl=True, max_seq_len_override=None, num_page_override=None, num_token_override=None, max_extend_tokens=8192, cache_type='radix', offline_mode=False, decode_log_interval=40, special_token_ckpt=False, _unique_suffix='.pid=8332', server_host='192.168.2.100', server_port=1919, num_tokenizer=0, silent_output=False, shell_mode=False, served_model_name='GLM-5.3-Flash-NVFP4', tool_call_parser='glm47', reasoning_parser='glm', sampling_defaults='model', max_output_tokens=None, enable_cache_report=False, cors_origins='tauri://localhost,http://tauri.localhost,http://localhost:1420', gpu=('GPU-bbc4d898-b6b0-e369-23c5-3bf5dc3047de',), gpu_assigned=None)
[02:56:18] stdout/INFO  [2026-09-06|02:56:18|initializer] INFO     --gpu GPU-bbc4d898-b6b0-e369-23c5-3bf5dc3047de -> GPU-bbc4d898-b6b0-e369-23c5-3bf5dc3047de
[02:56:18] stdout/INFO  [2026-09-06|02:56:18|FrontendAPI] INFO     Default sampling config (source=model): temperature=1.0, top_k=-1, top_p=1.0
[02:56:18] stderr/INFO  INFO:     Started server process [8332]
[02:56:18] stderr/INFO  INFO:     Waiting for application startup.
[02:56:18] stderr/INFO  INFO:     Application startup complete.
[02:56:18] stderr/INFO  INFO:     Uvicorn running on http://192.168.2.100:1919 (Press CTRL+C to quit)
[02:56:20] stderr/INFO  /home/zanry/.freetoken/venv/lib/python3.12/site-packages/freetoken/engine/engine.py:1034: FutureWarning: torch.cuda._set_allocator_settings is deprecated. Use torch._C._accelerator_setAllocatorSettings instead.
[02:56:20] stderr/INFO    torch.cuda.memory._set_allocator_settings("expandable_segments:True")
[02:56:20] stdout/INFO  [2026-09-06|02:56:20|core|rank=0] INFO     Enabled expandable_segments (override via PYTORCH_ALLOC_CONF)
[02:56:20] stdout/INFO  [2026-09-06|02:56:20|core|rank=0] INFO     Auto-selected attention backend: dsa
[02:56:20] stdout/WARN  [2026-09-06|02:56:20|core|rank=0] WARNING  Page size 1 is auto-adjusted to 64 for latent-KV attention.
[02:56:20] stdout/INFO  [2026-09-06|02:56:20|core|rank=0] INFO     Auto-selected MoE backend: offload
[02:56:20] stdout/INFO  [2026-09-06|02:56:20|core|rank=0] INFO     No MoE cache sizing flag given; defaulting to --moe-cache-auto for auto-selected backend 'offload'
[02:56:20] stdout/INFO  [2026-09-06|02:56:20|core|rank=0] INFO     Resolved config: moe_backend='offload', attention_backend='dsa', cache_type='hybrid_radix', page_size=64
[02:56:20] stdout/INFO  [2026-09-06|02:56:20|core|rank=0] INFO     Free memory before loading model: 30.23 GiB
[02:57:09] stdout/INFO  [2026-09-06|02:57:09|core|rank=0] INFO     expert banks: FTW fast path (FTW checkpoint /media/storage_zone/Virtual Machines/wsl/home/models/GLM-5.3-Flash-NVFP4)
[02:57:09] stderr/INFO  Process freetoken-TP0-scheduler:
[02:57:09] stdout/ERROR  [2026-09-06|02:57:09|FrontendAPI] ERROR    Backend supervisor: AssertionError: cache budget too small: minimum plan (moe=576 slots, kv=128 pages) needs 8261206016 B > budget 6944361021 B (raise memory_ratio, lower kv_reserve_tokens, or free GPU memory)
[02:57:09] stderr/INFO  Traceback (most recent call last):
[02:57:09] stderr/INFO    File "/home/zanry/.local/share/uv/python/cpython-3.12.14-linux-x86_64-gnu/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
[02:57:09] stderr/INFO      self.run()
[02:57:09] stderr/INFO    File "/home/zanry/.local/share/uv/python/cpython-3.12.14-linux-x86_64-gnu/lib/python3.12/multiprocessing/process.py", line 108, in run
[02:57:09] stderr/INFO      self._target(*self._args, **self._kwargs)
[02:57:09] stderr/INFO    File "/home/zanry/.freetoken/venv/lib/python3.12/site-packages/freetoken/server/launch.py", line 81, in _run_scheduler
[02:57:09] stderr/INFO      scheduler = Scheduler(args)
[02:57:09] stderr/INFO                  ^^^^^^^^^^^^^^^
[02:57:09] stderr/INFO    File "/home/zanry/.freetoken/venv/lib/python3.12/site-packages/freetoken/scheduler/scheduler.py", line 65, in __init__
[02:57:09] stderr/INFO      self.engine = Engine(config)
[02:57:09] stderr/INFO                    ^^^^^^^^^^^^^^
[02:57:09] stderr/INFO    File "/home/zanry/.freetoken/venv/lib/python3.12/site-packages/freetoken/engine/engine.py", line 343, in __init__
[02:57:09] stderr/INFO      self._init_offload_moe_cache(config)
[02:57:09] stderr/INFO    File "/home/zanry/.freetoken/venv/lib/python3.12/site-packages/freetoken/engine/engine.py", line 595, in _init_offload_moe_cache
[02:57:09] stderr/INFO      size, pages, overlap = self._resolve_auto_moe_cache_size(config, banks)
[02:57:09] stderr/INFO                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[02:57:09] stderr/INFO    File "/home/zanry/.freetoken/venv/lib/python3.12/site-packages/freetoken/engine/engine.py", line 490, in _resolve_auto_moe_cache_size
[02:57:09] stderr/INFO      return resolve_moe_cache_auto(
[02:57:09] stderr/INFO             ^^^^^^^^^^^^^^^^^^^^^^^
[02:57:09] stderr/INFO    File "/home/zanry/.freetoken/venv/lib/python3.12/site-packages/freetoken/engine/cache_budget.py", line 122, in resolve_moe_cache_auto
[02:57:09] stderr/INFO      return plan_cache_budget(
[02:57:09] stderr/INFO             ^^^^^^^^^^^^^^^^^^
[02:57:09] stderr/INFO    File "/home/zanry/.freetoken/venv/lib/python3.12/site-packages/freetoken/engine/cache_budget.py", line 89, in plan_cache_budget
[02:57:09] stderr/INFO      assert total <= budget_bytes, (
[02:57:09] stderr/INFO             ^^^^^^^^^^^^^^^^^^^^^
[02:57:09] stderr/INFO  AssertionError: cache budget too small: minimum plan (moe=576 slots, kv=128 pages) needs 8261206016 B > budget 6944361021 B (raise memory_ratio, lower kv_reserve_tokens, or free GPU memory)
[02:57:18] health/ERROR  AssertionError: cache budget too small: minimum plan (moe=576 slots, kv=128 pages) needs 8261206016 B > budget 6944361021 B (raise memory_ratio, lower kv_reserve_tokens, or free GPU memory)  (×5)
[02:57:23] stdout/ERROR  [2026-09-06|02:57:19|FrontendAPI] ERROR    Backend worker is gone and cannot be restarted; stopping the API server
[02:57:23] stderr/INFO  INFO:     Shutting down
[02:57:23] stderr/INFO  INFO:     Waiting for application shutdown.
[02:57:23] stderr/INFO  INFO:     Application shutdown complete.
[02:57:23] stderr/INFO  INFO:     Finished server process [8332]
[02:57:23] stderr/INFO  /home/zanry/.local/share/uv/python/cpython-3.12.14-linux-x86_64-gnu/lib/python3.12/multiprocessing/resource_tracker.py:279: UserWarning: resource_tracker: There appear to be 3 leaked semaphore objects to clean up at shutdown
[02:57:23] stderr/INFO    warnings.warn('resource_tracker: There appear to be %d '
[03:01:58] stderr/INFO  Converting dense weights: 0it [00:00, ?it/s]
[03:01:59] stderr/INFO  Converting dense weights: 1it [00:00,  1.02it/s]In file included from /home/zanry/.local/share/uv/python/cpython-3.12.14-linux-x86_64-gnu/include/python3.12/Python.h:12,
[03:01:59] stderr/INFO                   from /tmp/tmpqb8l3jsw/__triton_launcher.c:7:
[03:01:59] stderr/INFO  /home/zanry/.local/share/uv/python/cpython-3.12.14-linux-x86_64-gnu/include/python3.12/pyconfig.h:1877:9: warning: ‘_POSIX_C_SOURCE’ redefined
[03:01:59] stderr/INFO   1877 | #define _POSIX_C_SOURCE 200809L
[03:01:59] stderr/INFO        |         ^~~~~~~~~~~~~~~
[03:01:59] stderr/INFO  In file included from /usr/include/x86_64-linux-gnu/bits/libc-header-start.h:33,
[03:01:59] stderr/INFO                   from /usr/include/stdlib.h:26,
[03:01:59] stderr/INFO                   from /home/zanry/.freetoken/venv/lib/python3.12/site-packages/triton/backends/nvidia/include/cuda.h:56,
[03:01:59] stderr/INFO                   from /tmp/tmpqb8l3jsw/__triton_launcher.c:2:
[03:01:59] stderr/INFO  /usr/include/features.h:319:10: note: this is the location of the previous definition
[03:01:59] stderr/INFO    319 | # define _POSIX_C_SOURCE        202405L
[03:01:59] stderr/INFO        |          ^~~~~~~~~~~~~~~
[03:02:05] stderr/INFO  Converting dense weights: 404it [00:07, 63.65it/s]
[03:02:13] stderr/INFO  Converting dense weights: 867it [00:15, 74.74it/s]
[03:02:13] stderr/INFO  Loading weights: 100%|██████████| 3/3 [00:15<00:00,  5.11s/it]
[03:02:13] stdout/INFO  wrote FTW checkpoint -> /media/storage_zone/Virtual Machines/wsl/home/models/.convert-Qwen3.8-27B-NVFP4-RTX5090
[03:02:13] stdout/INFO    tensors: 867 weight + 0 experts_bank
[03:02:13] stdout/INFO    FTW: 27.19 GiB across 4 shard(s) (<= 8.0 GiB each)
[03:02:13] stdout/INFO    quant_format: None  fingerprint=c6c80cdb7ee304be
[03:02:13] stdout/INFO    converted in 15.4s
[03:02:14] stderr/INFO  kernel warmup skipped for Qwen3.8-27B-NVFP4-RTX5090: engine busy or daemon unavailable
[03:02:26] cmd/INFO  ft serve --model /media/storage_zone/Virtual Machines/wsl/home/models/Qwen3.8-27B-NVFP4-RTX5090 --port 1919 --max-running-requests 4 --memory-ratio 0.88 --host 192.168.2.100 --cors-origins tauri://localhost,http://tauri.localhost,http://localhost:1420 --gpu GPU-bbc4d898-b6b0-e369-23c5-3bf5dc3047de
[03:02:28] stdout/INFO  [2026-09-06|03:02:28] INFO     Parsed arguments:
[03:02:28] stdout/INFO  ServerArgs(model_path='/media/storage_zone/Virtual Machines/wsl/home/models/Qwen3.8-27B-NVFP4-RTX5090', tp_info=DistributedInfo(rank=0, size=1), dtype=torch.bfloat16, max_running_req=4, attention_backend='auto', moe_backend='auto', nvfp4_backend='triton', ple_backend='disk', expert_load='auto', moe_cache_size=0, moe_cache_rate=None, moe_cache_auto=False, kv_reserve_tokens=8192, moe_cache_policy='lru', moe_prefill_overlap=True, moe_prefill_hit_d2d=False, moe_collect_stats=False, moe_cpu_threads=0, moe_cpu_layers=None, moe_hybrid_max_fetch=-1, cuda_graph_bs=None, cuda_graph_max_bs=None, page_size=1, memory_ratio=0.88, linear_state_cache_ratio=2.0, swa_full_tokens_ratio=0.2, swa_num_pages_override=None, distributed_timeout=60.0, use_dummy_weight=False, use_pynccl=True, max_seq_len_override=None, num_page_override=None, num_token_override=None, max_extend_tokens=8192, cache_type='radix', offline_mode=False, decode_log_interval=40, special_token_ckpt=False, _unique_suffix='.pid=12913', server_host='192.168.2.100', server_port=1919, num_tokenizer=0, silent_output=False, shell_mode=False, served_model_name='Qwen3.8-27B-NVFP4-RTX5090', tool_call_parser='qwen3_coder', reasoning_parser='qwen3', sampling_defaults='model', max_output_tokens=None, enable_cache_report=False, cors_origins='tauri://localhost,http://tauri.localhost,http://localhost:1420', gpu=('GPU-bbc4d898-b6b0-e369-23c5-3bf5dc3047de',), gpu_assigned=None)
[03:02:28] stdout/INFO  [2026-09-06|03:02:28|initializer] INFO     --gpu GPU-bbc4d898-b6b0-e369-23c5-3bf5dc3047de -> GPU-bbc4d898-b6b0-e369-23c5-3bf5dc3047de
[03:02:28] stdout/INFO  [2026-09-06|03:02:28|FrontendAPI] INFO     Default sampling config (source=model): temperature=1.0, top_k=20, top_p=0.95
[03:02:28] stderr/INFO  INFO:     Started server process [12913]
[03:02:28] stderr/INFO  INFO:     Waiting for application startup.
[03:02:28] stderr/INFO  INFO:     Application startup complete.
[03:02:28] stderr/INFO  INFO:     Uvicorn running on http://192.168.2.100:1919 (Press CTRL+C to quit)
[03:02:30] stderr/INFO  /home/zanry/.freetoken/venv/lib/python3.12/site-packages/freetoken/engine/engine.py:1034: FutureWarning: torch.cuda._set_allocator_settings is deprecated. Use torch._C._accelerator_setAllocatorSettings instead.
[03:02:30] stderr/INFO    torch.cuda.memory._set_allocator_settings("expandable_segments:True")
[03:02:30] stdout/INFO  [2026-09-06|03:02:30|core|rank=0] INFO     Enabled expandable_segments (override via PYTORCH_ALLOC_CONF)
[03:02:30] stdout/INFO  [2026-09-06|03:02:30|core|rank=0] INFO     Auto-selected attention backend: fi
[03:02:30] stdout/INFO  [2026-09-06|03:02:30|core|rank=0] INFO     Resolved config: attention_backend='fi', cache_type='hybrid_radix', page_size=1
[03:02:30] stdout/INFO  [2026-09-06|03:02:30|core|rank=0] INFO     Free memory before loading model: 29.94 GiB
[03:02:40] stderr/INFO  Process freetoken-TP0-scheduler:
[03:02:40] stdout/ERROR  [2026-09-06|03:02:40|FrontendAPI] ERROR    Backend supervisor: AssertionError: Not enough memory for KV cache, try reducing --num-pages
[03:02:40] stderr/INFO  Traceback (most recent call last):
[03:02:40] stderr/INFO    File "/home/zanry/.local/share/uv/python/cpython-3.12.14-linux-x86_64-gnu/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
[03:02:40] stderr/INFO      self.run()
[03:02:40] stderr/INFO    File "/home/zanry/.local/share/uv/python/cpython-3.12.14-linux-x86_64-gnu/lib/python3.12/multiprocessing/process.py", line 108, in run
[03:02:40] stderr/INFO      self._target(*self._args, **self._kwargs)
[03:02:40] stderr/INFO    File "/home/zanry/.freetoken/venv/lib/python3.12/site-packages/freetoken/server/launch.py", line 81, in _run_scheduler
[03:02:40] stderr/INFO      scheduler = Scheduler(args)
[03:02:40] stderr/INFO                  ^^^^^^^^^^^^^^^
[03:02:40] stderr/INFO    File "/home/zanry/.freetoken/venv/lib/python3.12/site-packages/freetoken/scheduler/scheduler.py", line 65, in __init__
[03:02:40] stderr/INFO      self.engine = Engine(config)
[03:02:40] stderr/INFO                    ^^^^^^^^^^^^^^
[03:02:40] stderr/INFO    File "/home/zanry/.freetoken/venv/lib/python3.12/site-packages/freetoken/engine/engine.py", line 353, in __init__
[03:02:40] stderr/INFO      self.num_pages = self._pool_cls.solve_num_pages(config, available_memory)
[03:02:40] stderr/INFO                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[03:02:40] stderr/INFO    File "/home/zanry/.freetoken/venv/lib/python3.12/site-packages/freetoken/kvcache/base.py", line 73, in solve_num_pages
[03:02:40] stderr/INFO      assert num_pages > 1, "Not enough memory for KV cache, try reducing --num-pages"
[03:02:40] stderr/INFO             ^^^^^^^^^^^^^
[03:02:40] stderr/INFO  AssertionError: Not enough memory for KV cache, try reducing --num-pages
[03:02:49] health/ERROR  AssertionError: Not enough memory for KV cache, try reducing --num-pages  (×5)
[03:02:50] stdout/ERROR  [2026-09-06|03:02:50|FrontendAPI] ERROR    Backend worker is gone and cannot be restarted; stopping the API server
[03:02:50] stderr/INFO  INFO:     Shutting down
[03:02:50] stderr/INFO  INFO:     Waiting for application shutdown.
[03:02:50] stderr/INFO  INFO:     Application shutdown complete.
[03:02:50] stderr/INFO  INFO:     Finished server process [12913]
[03:02:50] stderr/INFO  /home/zanry/.local/share/uv/python/cpython-3.12.14-linux-x86_64-gnu/lib/python3.12/multiprocessing/resource_tracker.py:279: UserWarning: resource_tracker: There appear to be 3 leaked semaphore objects to clean up at shutdown
[03:02:50] stderr/INFO    warnings.warn('resource_tracker: There appear to be %d '
[03:04:02] cmd/INFO  ft serve --model /media/storage_zone/Virtual Machines/wsl/home/models/GLM-5.3-Flash-NVFP4 --port 1919 --moe-backend auto --max-running-requests 4 --memory-ratio 0.85 --host 192.168.2.100 --cors-origins tauri://localhost,http://tauri.localhost,http://localhost:1420 --gpu GPU-bbc4d898-b6b0-e369-23c5-3bf5dc3047de
[03:04:04] stdout/INFO  [2026-09-06|03:04:04] INFO     Parsed arguments:
[03:04:04] stdout/INFO  ServerArgs(model_path='/media/storage_zone/Virtual Machines/wsl/home/models/GLM-5.3-Flash-NVFP4', tp_info=DistributedInfo(rank=0, size=1), dtype=torch.bfloat16, max_running_req=4, attention_backend='auto', moe_backend='auto', nvfp4_backend='triton', ple_backend='disk', expert_load='auto', moe_cache_size=0, moe_cache_rate=None, moe_cache_auto=False, kv_reserve_tokens=8192, moe_cache_policy='lru', moe_prefill_overlap=True, moe_prefill_hit_d2d=False, moe_collect_stats=False, moe_cpu_threads=0, moe_cpu_layers=None, moe_hybrid_max_fetch=-1, cuda_graph_bs=None, cuda_graph_max_bs=None, page_size=1, memory_ratio=0.85, linear_state_cache_ratio=2.0, swa_full_tokens_ratio=0.2, swa_num_pages_override=None, distributed_timeout=60.0, use_dummy_weight=False, use_pynccl=True, max_seq_len_override=None, num_page_override=None, num_token_override=None, max_extend_tokens=8192, cache_type='radix', offline_mode=False, decode_log_interval=40, special_token_ckpt=False, _unique_suffix='.pid=15390', server_host='192.168.2.100', server_port=1919, num_tokenizer=0, silent_output=False, shell_mode=False, served_model_name='GLM-5.3-Flash-NVFP4', tool_call_parser='glm47', reasoning_parser='glm', sampling_defaults='model', max_output_tokens=None, enable_cache_report=False, cors_origins='tauri://localhost,http://tauri.localhost,http://localhost:1420', gpu=('GPU-bbc4d898-b6b0-e369-23c5-3bf5dc3047de',), gpu_assigned=None)
[03:04:04] stdout/INFO  [2026-09-06|03:04:04|initializer] INFO     --gpu GPU-bbc4d898-b6b0-e369-23c5-3bf5dc3047de -> GPU-bbc4d898-b6b0-e369-23c5-3bf5dc3047de
[03:04:04] stdout/INFO  [2026-09-06|03:04:04|FrontendAPI] INFO     Default sampling config (source=model): temperature=1.0, top_k=-1, top_p=1.0
[03:04:04] stderr/INFO  INFO:     Started server process [15390]
[03:04:04] stderr/INFO  INFO:     Waiting for application startup.
[03:04:04] stderr/INFO  INFO:     Application startup complete.
[03:04:04] stderr/INFO  INFO:     Uvicorn running on http://192.168.2.100:1919 (Press CTRL+C to quit)
[03:04:06] stderr/INFO  /home/zanry/.freetoken/venv/lib/python3.12/site-packages/freetoken/engine/engine.py:1034: FutureWarning: torch.cuda._set_allocator_settings is deprecated. Use torch._C._accelerator_setAllocatorSettings instead.
[03:04:06] stderr/INFO    torch.cuda.memory._set_allocator_settings("expandable_segments:True")
[03:04:06] stdout/INFO  [2026-09-06|03:04:06|core|rank=0] INFO     Enabled expandable_segments (override via PYTORCH_ALLOC_CONF)
[03:04:06] stdout/INFO  [2026-09-06|03:04:06|core|rank=0] INFO     Auto-selected attention backend: dsa
[03:04:06] stdout/WARN  [2026-09-06|03:04:06|core|rank=0] WARNING  Page size 1 is auto-adjusted to 64 for latent-KV attention.
[03:04:06] stdout/INFO  [2026-09-06|03:04:06|core|rank=0] INFO     Auto-selected MoE backend: offload
[03:04:06] stdout/INFO  [2026-09-06|03:04:06|core|rank=0] INFO     No MoE cache sizing flag given; defaulting to --moe-cache-auto for auto-selected backend 'offload'
[03:04:06] stdout/INFO  [2026-09-06|03:04:06|core|rank=0] INFO     Resolved config: moe_backend='offload', attention_backend='dsa', cache_type='hybrid_radix', page_size=64
[03:04:06] stdout/INFO  [2026-09-06|03:04:06|core|rank=0] INFO     Free memory before loading model: 30.05 GiB
[03:04:48] stdout/INFO  [2026-09-06|03:04:48|core|rank=0] INFO     expert banks: FTW fast path (FTW checkpoint /media/storage_zone/Virtual Machines/wsl/home/models/GLM-5.3-Flash-NVFP4)
[03:04:48] stderr/INFO  Process freetoken-TP0-scheduler:
[03:04:48] stdout/ERROR  [2026-09-06|03:04:48|FrontendAPI] ERROR    Backend supervisor: AssertionError: cache budget too small: minimum plan (moe=576 slots, kv=128 pages) needs 8261206016 B > budget 5840260300 B (raise memory_ratio, lower kv_reserve_tokens, or free GPU memory)
[03:04:48] stderr/INFO  Traceback (most recent call last):
[03:04:48] stderr/INFO    File "/home/zanry/.local/share/uv/python/cpython-3.12.14-linux-x86_64-gnu/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
[03:04:48] stderr/INFO      self.run()
[03:04:48] stderr/INFO    File "/home/zanry/.local/share/uv/python/cpython-3.12.14-linux-x86_64-gnu/lib/python3.12/multiprocessing/process.py", line 108, in run
[03:04:48] stderr/INFO      self._target(*self._args, **self._kwargs)
[03:04:48] stderr/INFO    File "/home/zanry/.freetoken/venv/lib/python3.12/site-packages/freetoken/server/launch.py", line 81, in _run_scheduler
[03:04:48] stderr/INFO      scheduler = Scheduler(args)
[03:04:48] stderr/INFO                  ^^^^^^^^^^^^^^^
[03:04:48] stderr/INFO    File "/home/zanry/.freetoken/venv/lib/python3.12/site-packages/freetoken/scheduler/scheduler.py", line 65, in __init__
[03:04:48] stderr/INFO      self.engine = Engine(config)
[03:04:48] stderr/INFO                    ^^^^^^^^^^^^^^
[03:04:48] stderr/INFO    File "/home/zanry/.freetoken/venv/lib/python3.12/site-packages/freetoken/engine/engine.py", line 343, in __init__
[03:04:48] stderr/INFO      self._init_offload_moe_cache(config)
[03:04:48] stderr/INFO    File "/home/zanry/.freetoken/venv/lib/python3.12/site-packages/freetoken/engine/engine.py", line 595, in _init_offload_moe_cache
[03:04:48] stderr/INFO      size, pages, overlap = self._resolve_auto_moe_cache_size(config, banks)
[03:04:48] stderr/INFO                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[03:04:48] stderr/INFO    File "/home/zanry/.freetoken/venv/lib/python3.12/site-packages/freetoken/engine/engine.py", line 490, in _resolve_auto_moe_cache_size
[03:04:48] stderr/INFO      return resolve_moe_cache_auto(
[03:04:48] stderr/INFO             ^^^^^^^^^^^^^^^^^^^^^^^
[03:04:48] stderr/INFO    File "/home/zanry/.freetoken/venv/lib/python3.12/site-packages/freetoken/engine/cache_budget.py", line 122, in resolve_moe_cache_auto
[03:04:48] stderr/INFO      return plan_cache_budget(
[03:04:48] stderr/INFO             ^^^^^^^^^^^^^^^^^^
[03:04:48] stderr/INFO    File "/home/zanry/.freetoken/venv/lib/python3.12/site-packages/freetoken/engine/cache_budget.py", line 89, in plan_cache_budget
[03:04:48] stderr/INFO      assert total <= budget_bytes, (
[03:04:48] stderr/INFO             ^^^^^^^^^^^^^^^^^^^^^
[03:04:48] stderr/INFO  AssertionError: cache budget too small: minimum plan (moe=576 slots, kv=128 pages) needs 8261206016 B > budget 5840260300 B (raise memory_ratio, lower kv_reserve_tokens, or free GPU memory)
[03:04:57] health/ERROR  AssertionError: cache budget too small: minimum plan (moe=576 slots, kv=128 pages) needs 8261206016 B > budget 5840260300 B (raise memory_ratio, lower kv_reserve_tokens, or free GPU memory)  (×5)
[03:05:01] stdout/ERROR  [2026-09-06|03:04:58|FrontendAPI] ERROR    Backend worker is gone and cannot be restarted; stopping the API server
[03:05:01] stderr/INFO  INFO:     Shutting down
[03:05:01] stderr/INFO  INFO:     Waiting for application shutdown.
[03:05:01] stderr/INFO  INFO:     Application shutdown complete.
[03:05:01] stderr/INFO  INFO:     Finished server process [15390]
[03:05:01] stderr/INFO  /home/zanry/.local/share/uv/python/cpython-3.12.14-linux-x86_64-gnu/lib/python3.12/multiprocessing/resource_tracker.py:279: UserWarning: resource_tracker: There appear to be 3 leaked semaphore objects to clean up at shutdown
[03:05:01] stderr/INFO    warnings.warn('resource_tracker: There appear to be %d '
[03:10:15] cmd/INFO  ft serve --model /media/storage_zone/Virtual Machines/wsl/home/models/GLM-5.3-Flash-NVFP4 --port 1919 --moe-backend hybrid --max-running-requests 4 --memory-ratio 0.88 --host 192.168.2.100 --cors-origins tauri://localhost,http://tauri.localhost,http://localhost:1420 --gpu GPU-bbc4d898-b6b0-e369-23c5-3bf5dc3047de
[03:10:17] stdout/INFO  [2026-09-06|03:10:17] INFO     Parsed arguments:
[03:10:17] stdout/INFO  ServerArgs(model_path='/media/storage_zone/Virtual Machines/wsl/home/models/GLM-5.3-Flash-NVFP4', tp_info=DistributedInfo(rank=0, size=1), dtype=torch.bfloat16, max_running_req=4, attention_backend='auto', moe_backend='hybrid', nvfp4_backend='triton', ple_backend='disk', expert_load='auto', moe_cache_size=0, moe_cache_rate=None, moe_cache_auto=True, kv_reserve_tokens=8192, moe_cache_policy='lru', moe_prefill_overlap=True, moe_prefill_hit_d2d=False, moe_collect_stats=False, moe_cpu_threads=0, moe_cpu_layers=None, moe_hybrid_max_fetch=-1, cuda_graph_bs=None, cuda_graph_max_bs=None, page_size=1, memory_ratio=0.88, linear_state_cache_ratio=2.0, swa_full_tokens_ratio=0.2, swa_num_pages_override=None, distributed_timeout=60.0, use_dummy_weight=False, use_pynccl=True, max_seq_len_override=None, num_page_override=None, num_token_override=None, max_extend_tokens=8192, cache_type='radix', offline_mode=False, decode_log_interval=40, special_token_ckpt=False, _unique_suffix='.pid=18890', server_host='192.168.2.100', server_port=1919, num_tokenizer=0, silent_output=False, shell_mode=False, served_model_name='GLM-5.3-Flash-NVFP4', tool_call_parser='glm47', reasoning_parser='glm', sampling_defaults='model', max_output_tokens=None, enable_cache_report=False, cors_origins='tauri://localhost,http://tauri.localhost,http://localhost:1420', gpu=('GPU-bbc4d898-b6b0-e369-23c5-3bf5dc3047de',), gpu_assigned=None)
[03:10:17] stdout/INFO  [2026-09-06|03:10:17|initializer] INFO     --gpu GPU-bbc4d898-b6b0-e369-23c5-3bf5dc3047de -> GPU-bbc4d898-b6b0-e369-23c5-3bf5dc3047de
[03:10:17] stdout/INFO  [2026-09-06|03:10:17|FrontendAPI] INFO     Default sampling config (source=model): temperature=1.0, top_k=-1, top_p=1.0
[03:10:17] stderr/INFO  INFO:     Started server process [18890]
[03:10:17] stderr/INFO  INFO:     Waiting for application startup.
[03:10:17] stderr/INFO  INFO:     Application startup complete.
[03:10:17] stderr/INFO  INFO:     Uvicorn running on http://192.168.2.100:1919 (Press CTRL+C to quit)
[03:10:18] stderr/INFO  /home/zanry/.freetoken/venv/lib/python3.12/site-packages/freetoken/engine/engine.py:1034: FutureWarning: torch.cuda._set_allocator_settings is deprecated. Use torch._C._accelerator_setAllocatorSettings instead.
[03:10:18] stderr/INFO    torch.cuda.memory._set_allocator_settings("expandable_segments:True")
[03:10:18] stdout/INFO  [2026-09-06|03:10:18|core|rank=0] INFO     Enabled expandable_segments (override via PYTORCH_ALLOC_CONF)
[03:10:19] stdout/INFO  [2026-09-06|03:10:19|core|rank=0] INFO     Auto-selected attention backend: dsa
[03:10:19] stdout/WARN  [2026-09-06|03:10:19|core|rank=0] WARNING  Page size 1 is auto-adjusted to 64 for latent-KV attention.
[03:10:19] stdout/INFO  [2026-09-06|03:10:19|core|rank=0] INFO     Resolved config: moe_backend='hybrid', attention_backend='dsa', cache_type='hybrid_radix', page_size=64
[03:10:19] stdout/INFO  [2026-09-06|03:10:19|core|rank=0] INFO     Free memory before loading model: 30.04 GiB
[03:11:07] stdout/INFO  [2026-09-06|03:11:07|core|rank=0] INFO     expert banks: FTW fast path (FTW checkpoint /media/storage_zone/Virtual Machines/wsl/home/models/GLM-5.3-Flash-NVFP4)
[03:11:07] stderr/INFO  Process freetoken-TP0-scheduler:
[03:11:07] stdout/ERROR  [2026-09-06|03:11:07|FrontendAPI] ERROR    Backend supervisor: AssertionError: cache budget too small: minimum plan (moe=576 slots, kv=128 pages) needs 8261206016 B > budget 6630325616 B (raise memory_ratio, lower kv_reserve_tokens, or free GPU memory)
[03:11:07] stderr/INFO  Traceback (most recent call last):
[03:11:07] stderr/INFO    File "/home/zanry/.local/share/uv/python/cpython-3.12.14-linux-x86_64-gnu/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
[03:11:07] stderr/INFO      self.run()
[03:11:07] stderr/INFO    File "/home/zanry/.local/share/uv/python/cpython-3.12.14-linux-x86_64-gnu/lib/python3.12/multiprocessing/process.py", line 108, in run
[03:11:07] stderr/INFO      self._target(*self._args, **self._kwargs)
[03:11:07] stderr/INFO    File "/home/zanry/.freetoken/venv/lib/python3.12/site-packages/freetoken/server/launch.py", line 81, in _run_scheduler
[03:11:07] stderr/INFO      scheduler = Scheduler(args)
[03:11:07] stderr/INFO                  ^^^^^^^^^^^^^^^
[03:11:07] stderr/INFO    File "/home/zanry/.freetoken/venv/lib/python3.12/site-packages/freetoken/scheduler/scheduler.py", line 65, in __init__
[03:11:07] stderr/INFO      self.engine = Engine(config)
[03:11:07] stderr/INFO                    ^^^^^^^^^^^^^^
[03:11:07] stderr/INFO    File "/home/zanry/.freetoken/venv/lib/python3.12/site-packages/freetoken/engine/engine.py", line 343, in __init__
[03:11:07] stderr/INFO      self._init_offload_moe_cache(config)
[03:11:07] stderr/INFO    File "/home/zanry/.freetoken/venv/lib/python3.12/site-packages/freetoken/engine/engine.py", line 595, in _init_offload_moe_cache
[03:11:07] stderr/INFO      size, pages, overlap = self._resolve_auto_moe_cache_size(config, banks)
[03:11:07] stderr/INFO                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[03:11:07] stderr/INFO    File "/home/zanry/.freetoken/venv/lib/python3.12/site-packages/freetoken/engine/engine.py", line 490, in _resolve_auto_moe_cache_size
[03:11:07] stderr/INFO      return resolve_moe_cache_auto(
[03:11:07] stderr/INFO             ^^^^^^^^^^^^^^^^^^^^^^^
[03:11:07] stderr/INFO    File "/home/zanry/.freetoken/venv/lib/python3.12/site-packages/freetoken/engine/cache_budget.py", line 122, in resolve_moe_cache_auto
[03:11:07] stderr/INFO      return plan_cache_budget(
[03:11:07] stderr/INFO             ^^^^^^^^^^^^^^^^^^
[03:11:07] stderr/INFO    File "/home/zanry/.freetoken/venv/lib/python3.12/site-packages/freetoken/engine/cache_budget.py", line 89, in plan_cache_budget
[03:11:07] stderr/INFO      assert total <= budget_bytes, (
[03:11:07] stderr/INFO             ^^^^^^^^^^^^^^^^^^^^^
[03:11:07] stderr/INFO  AssertionError: cache budget too small: minimum plan (moe=576 slots, kv=128 pages) needs 8261206016 B > budget 6630325616 B (raise memory_ratio, lower kv_reserve_tokens, or free GPU memory)
[03:11:12] health/ERROR  AssertionError: cache budget too small: minimum plan (moe=576 slots, kv=128 pages) needs 8261206016 B > budget 6630325616 B (raise memory_ratio, lower kv_reserve_tokens, or free GPU memory)  (×3)
[03:11:12] stderr/INFO  INFO:     Shutting down
[03:11:13] stderr/INFO  INFO:     Waiting for application shutdown.
[03:11:13] stderr/INFO  INFO:     Application shutdown complete.
[03:11:13] stderr/INFO  INFO:     Finished server process [18890]
[03:11:13] stderr/INFO  /home/zanry/.local/share/uv/python/cpython-3.12.14-linux-x86_64-gnu/lib/python3.12/multiprocessing/resource_tracker.py:279: UserWarning: resource_tracker: There appear to be 3 leaked semaphore objects to clean up at shutdown
[03:11:13] stderr/INFO    warnings.warn('resource_tracker: There appear to be %d '
[03:12:18] cmd/INFO  ft serve --model /media/storage_zone/Virtual Machines/wsl/home/models/GLM-5.3-Flash-NVFP4 --port 1919 --moe-backend offload --max-running-requests 4 --memory-ratio 0.88 --host 192.168.2.100 --cors-origins tauri://localhost,http://tauri.localhost,http://localhost:1420 --gpu GPU-bbc4d898-b6b0-e369-23c5-3bf5dc3047de
[03:12:20] stdout/INFO  [2026-09-06|03:12:20] INFO     Parsed arguments:
[03:12:20] stdout/INFO  ServerArgs(model_path='/media/storage_zone/Virtual Machines/wsl/home/models/GLM-5.3-Flash-NVFP4', tp_info=DistributedInfo(rank=0, size=1), dtype=torch.bfloat16, max_running_req=4, attention_backend='auto', moe_backend='offload', nvfp4_backend='triton', ple_backend='disk', expert_load='auto', moe_cache_size=0, moe_cache_rate=None, moe_cache_auto=True, kv_reserve_tokens=8192, moe_cache_policy='lru', moe_prefill_overlap=True, moe_prefill_hit_d2d=False, moe_collect_stats=False, moe_cpu_threads=0, moe_cpu_layers=None, moe_hybrid_max_fetch=-1, cuda_graph_bs=None, cuda_graph_max_bs=None, page_size=1, memory_ratio=0.88, linear_state_cache_ratio=2.0, swa_full_tokens_ratio=0.2, swa_num_pages_override=None, distributed_timeout=60.0, use_dummy_weight=False, use_pynccl=True, max_seq_len_override=None, num_page_override=None, num_token_override=None, max_extend_tokens=8192, cache_type='radix', offline_mode=False, decode_log_interval=40, special_token_ckpt=False, _unique_suffix='.pid=21893', server_host='192.168.2.100', server_port=1919, num_tokenizer=0, silent_output=False, shell_mode=False, served_model_name='GLM-5.3-Flash-NVFP4', tool_call_parser='glm47', reasoning_parser='glm', sampling_defaults='model', max_output_tokens=None, enable_cache_report=False, cors_origins='tauri://localhost,http://tauri.localhost,http://localhost:1420', gpu=('GPU-bbc4d898-b6b0-e369-23c5-3bf5dc3047de',), gpu_assigned=None)
[03:12:20] stdout/INFO  [2026-09-06|03:12:20|initializer] INFO     --gpu GPU-bbc4d898-b6b0-e369-23c5-3bf5dc3047de -> GPU-bbc4d898-b6b0-e369-23c5-3bf5dc3047de
[03:12:20] stdout/INFO  [2026-09-06|03:12:20|FrontendAPI] INFO     Default sampling config (source=model): temperature=1.0, top_k=-1, top_p=1.0
[03:12:21] stderr/INFO  INFO:     Started server process [21893]
[03:12:21] stderr/INFO  INFO:     Waiting for application startup.
[03:12:21] stderr/INFO  INFO:     Application startup complete.
[03:12:21] stderr/INFO  INFO:     Uvicorn running on http://192.168.2.100:1919 (Press CTRL+C to quit)
[03:12:22] stderr/INFO  /home/zanry/.freetoken/venv/lib/python3.12/site-packages/freetoken/engine/engine.py:1034: FutureWarning: torch.cuda._set_allocator_settings is deprecated. Use torch._C._accelerator_setAllocatorSettings instead.
[03:12:22] stderr/INFO    torch.cuda.memory._set_allocator_settings("expandable_segments:True")
[03:12:22] stdout/INFO  [2026-09-06|03:12:22|core|rank=0] INFO     Enabled expandable_segments (override via PYTORCH_ALLOC_CONF)
[03:12:22] stdout/INFO  [2026-09-06|03:12:22|core|rank=0] INFO     Auto-selected attention backend: dsa
[03:12:22] stdout/WARN  [2026-09-06|03:12:22|core|rank=0] WARNING  Page size 1 is auto-adjusted to 64 for latent-KV attention.
[03:12:22] stdout/INFO  [2026-09-06|03:12:22|core|rank=0] INFO     Resolved config: moe_backend='offload', attention_backend='dsa', cache_type='hybrid_radix', page_size=64
[03:12:22] stdout/INFO  [2026-09-06|03:12:22|core|rank=0] INFO     Free memory before loading model: 29.98 GiB
[03:13:05] stdout/INFO  [2026-09-06|03:13:05|core|rank=0] INFO     expert banks: FTW fast path (FTW checkpoint /media/storage_zone/Virtual Machines/wsl/home/models/GLM-5.3-Flash-NVFP4)
[03:13:05] stderr/INFO  Process freetoken-TP0-scheduler:
[03:13:05] stdout/ERROR  [2026-09-06|03:13:05|FrontendAPI] ERROR    Backend supervisor: AssertionError: cache budget too small: minimum plan (moe=576 slots, kv=128 pages) needs 8261206016 B > budget 6734265712 B (raise memory_ratio, lower kv_reserve_tokens, or free GPU memory)
[03:13:05] stderr/INFO  Traceback (most recent call last):
[03:13:05] stderr/INFO    File "/home/zanry/.local/share/uv/python/cpython-3.12.14-linux-x86_64-gnu/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
[03:13:05] stderr/INFO      self.run()
[03:13:05] stderr/INFO    File "/home/zanry/.local/share/uv/python/cpython-3.12.14-linux-x86_64-gnu/lib/python3.12/multiprocessing/process.py", line 108, in run
[03:13:05] stderr/INFO      self._target(*self._args, **self._kwargs)
[03:13:05] stderr/INFO    File "/home/zanry/.freetoken/venv/lib/python3.12/site-packages/freetoken/server/launch.py", line 81, in _run_scheduler
[03:13:05] stderr/INFO      scheduler = Scheduler(args)
[03:13:05] stderr/INFO                  ^^^^^^^^^^^^^^^
[03:13:05] stderr/INFO    File "/home/zanry/.freetoken/venv/lib/python3.12/site-packages/freetoken/scheduler/scheduler.py", line 65, in __init__
[03:13:05] stderr/INFO      self.engine = Engine(config)
[03:13:05] stderr/INFO                    ^^^^^^^^^^^^^^
[03:13:05] stderr/INFO    File "/home/zanry/.freetoken/venv/lib/python3.12/site-packages/freetoken/engine/engine.py", line 343, in __init__
[03:13:05] stderr/INFO      self._init_offload_moe_cache(config)
[03:13:05] stderr/INFO    File "/home/zanry/.freetoken/venv/lib/python3.12/site-packages/freetoken/engine/engine.py", line 595, in _init_offload_moe_cache
[03:13:05] stderr/INFO      size, pages, overlap = self._resolve_auto_moe_cache_size(config, banks)
[03:13:05] stderr/INFO                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[03:13:05] stderr/INFO    File "/home/zanry/.freetoken/venv/lib/python3.12/site-packages/freetoken/engine/engine.py", line 490, in _resolve_auto_moe_cache_size
[03:13:05] stderr/INFO      return resolve_moe_cache_auto(
[03:13:05] stderr/INFO             ^^^^^^^^^^^^^^^^^^^^^^^
[03:13:05] stderr/INFO    File "/home/zanry/.freetoken/venv/lib/python3.12/site-packages/freetoken/engine/cache_budget.py", line 122, in resolve_moe_cache_auto
[03:13:05] stderr/INFO      return plan_cache_budget(
[03:13:05] stderr/INFO             ^^^^^^^^^^^^^^^^^^
[03:13:05] stderr/INFO    File "/home/zanry/.freetoken/venv/lib/python3.12/site-packages/freetoken/engine/cache_budget.py", line 89, in plan_cache_budget
[03:13:05] stderr/INFO      assert total <= budget_bytes, (
[03:13:05] stderr/INFO             ^^^^^^^^^^^^^^^^^^^^^
[03:13:05] stderr/INFO  AssertionError: cache budget too small: minimum plan (moe=576 slots, kv=128 pages) needs 8261206016 B > budget 6734265712 B (raise memory_ratio, lower kv_reserve_tokens, or free GPU memory)
[03:13:15] health/ERROR  AssertionError: cache budget too small: minimum plan (moe=576 slots, kv=128 pages) needs 8261206016 B > budget 6734265712 B (raise memory_ratio, lower kv_reserve_tokens, or free GPU memory)  (×5)
[03:13:15] stdout/ERROR  [2026-09-06|03:13:15|FrontendAPI] ERROR    Backend worker is gone and cannot be restarted; stopping the API server
[03:13:15] stderr/INFO  INFO:     Shutting down
[03:13:15] stderr/INFO  INFO:     Waiting for application shutdown.
[03:13:15] stderr/INFO  INFO:     Application shutdown complete.
[03:13:15] stderr/INFO  INFO:     Finished server process [21893]
[03:13:15] stderr/INFO  /home/zanry/.local/share/uv/python/cpython-3.12.14-linux-x86_64-gnu/lib/python3.12/multiprocessing/resource_tracker.py:279: UserWarning: resource_tracker: There appear to be 3 leaked semaphore objects to clean up at shutdown
[03:13:15] stderr/INFO    warnings.warn('resource_tracker: There appear to be %d '

Anything else

No response

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Desktopproblem related to FreeToken DesktopbugSomething isn't workinglinux

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions