Feature Request: Native GGUF Model Support
Currently, FreeToken relies heavily on Hugging Face model formats such as Safetensors and specific quantization formats (e.g., FP8 and MXFP4). While this works well for datacenter-class models, many local and edge AI users rely heavily on the .gguf format popularized by the llama.cpp ecosystem.
Without native GGUF support, users may need to maintain duplicate model files or rely on conversion scripts, which can introduce additional complexity and may not map cleanly to FreeToken's edge-native MoE optimizations.
Solution
I propose adding a native GGUF loader/bridge in Python that would allow FreeToken to load .gguf models directly.
This could involve:
- Parsing GGUF file headers and metadata using the
gguf Python library.
- Extracting tensor weights from the GGUF file.
- Mapping GGUF tensor names and structures to FreeToken's existing internal tensor layouts.
- Updating the CLI to accept
.gguf model paths natively, for example:
ft serve --model my-model.gguf
The goal would be to allow users to serve GGUF models directly without requiring an intermediate conversion step.
Alternatives considered
An alternative approach would be to provide a standalone conversion utility, such as:
However, native GGUF loading would provide a significantly better developer and user experience. It would eliminate the need for duplicate model files, reduce storage requirements, simplify the setup process, and make FreeToken more accessible to users already familiar with the llama.cpp ecosystem.
Additional context
Since FreeToken is positioned as an edge-native serving engine, supporting one of the most widely adopted model formats in the local and edge AI ecosystem seems like a natural addition.
If the maintainers agree that this feature is valuable and aligns with the project's roadmap, I'd be happy to take this on and submit a Pull Request implementing native GGUF support.
Feature Request: Native GGUF Model Support
Currently, FreeToken relies heavily on Hugging Face model formats such as Safetensors and specific quantization formats (e.g., FP8 and MXFP4). While this works well for datacenter-class models, many local and edge AI users rely heavily on the
.ggufformat popularized by thellama.cppecosystem.Without native GGUF support, users may need to maintain duplicate model files or rely on conversion scripts, which can introduce additional complexity and may not map cleanly to FreeToken's edge-native MoE optimizations.
Solution
I propose adding a native GGUF loader/bridge in Python that would allow FreeToken to load
.ggufmodels directly.This could involve:
ggufPython library..ggufmodel paths natively, for example:The goal would be to allow users to serve GGUF models directly without requiring an intermediate conversion step.
Alternatives considered
An alternative approach would be to provide a standalone conversion utility, such as:
However, native GGUF loading would provide a significantly better developer and user experience. It would eliminate the need for duplicate model files, reduce storage requirements, simplify the setup process, and make FreeToken more accessible to users already familiar with the
llama.cppecosystem.Additional context
Since FreeToken is positioned as an edge-native serving engine, supporting one of the most widely adopted model formats in the local and edge AI ecosystem seems like a natural addition.
If the maintainers agree that this feature is valuable and aligns with the project's roadmap, I'd be happy to take this on and submit a Pull Request implementing native GGUF support.