AIThis post was created with the assistance of artificial intelligence (AI).

🔍 Read the full analysis: Transformers Expands Its Support To Llama.cpp Quants on ThorstenMeyerAI.com

Before you orderOffer from Amazon

Get the latest gadgets delivered free with Prime

  • Fast, free delivery on millions of items
  • Prime Video, Amazon Music and more included
  • Member-only deals all year
Start your free Prime trial Free trial for eligible customers · Cancel anytime
As an affiliate, we earn on qualifying purchases.

TL;DR

Hugging Face has added support for loading GGUF-quantized checkpoints in Transformers through the familiar from_pretrained API. The current implementation targets Apple Silicon and Qwen3.5, is available on the main branch, and uses llama.cpp’s ggml kernels when compatible kernels are available.

Hugging Face has added GGUF model support to its Transformers library, allowing users to load quantized checkpoints through the from_pretrained API, as described in the original analysis. The initial implementation targets Apple Silicon Macs and the Qwen3.5 architecture, bringing a format widely used with llama.cpp into a PyTorch-based library familiar to many model developers.

Users can select a GGUF checkpoint from the Hugging Face Hub and pass its filename through the gguf_file argument to from_pretrained, then generate text through Transformers. Hugging Face says the implementation reuses llama.cpp’s ggml kernels to keep inference performance close to llama.cpp. The feature is currently on the library’s main branch, rather than in a stable release.

On supported Apple Silicon systems, when weights remain packed on Metal, Transformers can load compatible ggml/Metal layer kernels and use ggml-org/ggml-attn for attention. If that attention kernel cannot be fetched, the loader falls back to PyTorch’s standard sdpa implementation and issues a warning. Users can also select sdpa directly with attn_implementation=”sdpa”.

The setup depends on an Apple Silicon Mac, a supported PyTorch version, the latest Transformers code, and a compatible version of the kernels library. Hugging Face says kernel builds generally support the two latest PyTorch releases. Without a compatible quantization kernel, the model can still load through dequantization, but that uses more memory. GGUF checkpoints can also be served using transformers serve, which provides a local OpenAI-compatible API for clients configured to use that endpoint.

At a glance
announcementWhen: Available on the Transformers main bran…
The developmentHugging Face has added native GGUF checkpoint loading to Transformers, initially for Qwen3.5 models on Apple Silicon.
At a glance
announcementWhen: announced April 2026; available via tra…
The developmentHugging Face announced that the transformers library can now run llama.cpp-style GGUF quantized models natively, using ggml kernels for near-llama.cpp performance on Apple Silicon.

GGUF Enters the Transformers Workflow

The change gives developers already using Transformers and PyTorch a way to work with GGUF checkpoints without switching their loading code to a llama.cpp-derived application. GGUF models are distributed by publishers and communities including Unsloth, LM Studio Community, bartowski and ggml-org. The new path may make those checkpoints easier to incorporate into local development workflows, although the current hardware and architecture limits narrow who can use it.

Quantization can reduce the memory needed to store model weights, making local inference possible on machines that cannot accommodate a full-precision checkpoint. Hugging Face lists its Qwen3.5-4B example at 8.42 GB in BF16, compared with 2.74 GB in Q4_K_M. That is a file-size comparison for this particular model; it does not establish a general performance or quality result. Actual memory use and output quality can vary with the model, runtime and task.

Hugging Face describes llama.cpp as its reference for local inference performance and says its announcement includes comparisons across three GGUF checkpoints: a small dense model, a larger dense model and a mixture-of-experts model. Those comparisons are relevant to users weighing a familiar Transformers workflow against llama.cpp, but performance conclusions depend on the tested hardware and models. The release does not establish that the two implementations perform identically across other systems or workloads.

Products Worth Considering

Logitech MX Keys S Combo Wireless Keyboard and Mouse, Graphite

Logitech MX Keys S Combo Wireless Keyboard and Mouse, Graphite

Amazon Kindle Scribe (32GB) with Premium Pen, Tungsten (Like-New)

Amazon Kindle Scribe (32GB) with Premium Pen, Tungsten (Like-New)

As an affiliate, we earn on qualifying purchases.

From GGUF Files to Local Inference

GGUF is a file format developed by the llama.cpp project for model weights and related information. A file can include metadata such as tokenizer details and an optional chat template. GGUF checkpoints are available at different quantization levels, which reduce storage and memory needs by representing many weights with fewer bits. Some formats, including Q4_K_M, use mixed precision, retaining higher precision for selected tensors.

For Unsloth’s Qwen3.5-4B, Hugging Face lists Q6_K at 3.53 GB and Q5_K_M at 3.14 GB, alongside the BF16 and Q4_K_M sizes. The company suggests starting with Q4_K_M and moving to higher-bit variants if more memory is available. It cautions that quality changes from quantization depend on the model and task, so those file sizes alone do not show how well a checkpoint will perform on a user’s work.

The announcement comes amid interest in running capable language models locally. Hugging Face co-founder Julien Chaumond recently described running Qwen3.6 27B inside the Pi coding agent through llama.cpp on a MacBook Pro. That demonstration is a separate example of local inference; it does not show that the new Transformers implementation supports Qwen3.6 or delivers the same results.

“We’re adding support for running GGUF models efficiently in transformers, so you can use checkpoints sized for your laptop’s memory through the familiar transformers APIs.”

— Hugging Face announcement

Hardware and Model Support Remain Narrow

The announced implementation is focused on Apple Silicon and Qwen3.5. Hugging Face has not stated when it will support CUDA, Linux or Windows, or which additional model architectures will follow. The announcement also places the feature on the main branch without giving a date for its inclusion in a stable Transformers release.

The announcement references benchmark comparisons against llama.cpp, but results depend on the specific hardware, checkpoints and settings; the source material does not provide enough detail to make a general performance claim. It is also unclear how quickly compatible kernels will be available for other PyTorch versions or how broad architecture coverage will become. Users should assess quantized output on their own tasks, as Hugging Face advises.

Stable Release and Wider Compatibility

The next concrete milestone is a stable Transformers release that includes GGUF support; no release date has been announced. Until then, users seeking the feature need to use the main branch and meet the stated platform, PyTorch and kernel requirements.

Further development could add more architectures or hardware backends, but Hugging Face has not provided a schedule or confirmed specific additions. Users can follow the Transformers and GGUF documentation and the kernels library for compatibility details. Broader availability will depend on what Hugging Face ships in later releases.

Key Questions

What did Hugging Face add?

Transformers can load GGUF-quantized checkpoints through from_pretrained using the gguf_file argument.

Which systems and models are supported initially?

The initial implementation targets Apple Silicon Macs and the Qwen3.5 architecture. Hugging Face has not announced a timeline for other platforms or model families.

Is the feature in a stable Transformers release?

It is available on the Transformers main branch. A stable release date has not been announced.

Will a GGUF model always use quantized kernels?

No. The stated setup requires compatible kernels. Without them, the loader can dequantize the model, which uses more memory.

Does the announcement show that Transformers matches llama.cpp performance?

Hugging Face says it uses llama.cpp’s ggml kernels and references comparisons across three checkpoints. Results depend on the tested hardware and models; the announcement does not establish matching performance for every setup.

Primary source: Hugging Face · via ThorstenMeyerAI.com

You May Also Like

The Future Of Desktop Automation: Watch-Once Spoken Commands

New watch-once spoken commands enable users to automate desktop workflows with a single recording, promising increased efficiency for power users.

Elixir v1.20 released: now a gradually typed language

Elixir v1.20 now features a gradually typed system with type inference, enabling bug detection without annotations, enhancing safety and developer experience.

Postgres 19 Is Late Because So Many More Bugs Are Being Found Now

A PostgreSQL developer says a surge in reported bugs and fixes has pushed version 19 past its usual release window, with November now likely.

The $725 Billion Question: Hyperscaler Capex Q1 2026 and What the Earnings Don’t Answer

The Big Four hyperscalers announced a combined $725 billion in AI-related capital expenditure for 2026, raising questions about future revenue growth and industry sustainability.