Skip to content

MACaQ Quantizes a 122B Model in 64 Minutes on 32.5 GB

An Apache-2.0 toolkit from the huawei-csl GitHub account brings model-agnostic quantization to llama.cpp, vLLM, Diffusers and PyTorch.

Draftship Editorial3 min read

The huawei-csl GitHub account published MACaQ on 2026-11-10, and by 2026-11-15 the repository had picked up 301 stars and 21 forks in five days. The project pitches MACaQ GGUF quantization as model-agnostic: the same toolkit is aimed at text, image, video and audio models rather than text alone, with separate workflows for llama.cpp, vLLM, Diffusers and PyTorch. It is written in C++ and released under Apache-2.0.

The headline claim is speed and memory. According to the project, MACaQ can quantize a 122B-parameter model in 64 minutes using 32.5 GB of device memory. The authors also say you can get lower perplexity than MR-GPTQ and DuQuant++, or match their best Llama-3-8B perplexity with 11 times less quantization time. The paper behind it is cited as arXiv 2611.10649, with Lorenz K. Muller, Philippe Bich, Chiara Boretti, Hyun-Min Chang, Jiawei Zhuang and Lukas Cavigelli listed as authors. The full setup is documented in the MACaQ repository.

Four backends, four different jobs

Each backend gets its own environment, and the project asks for Python 3.12 with a separate environment per backend.

  • Diffusers handles image, video and audio generation at W4A4. The documented examples are black-forest-labs/FLUX.2-klein-4B for image generation, Wan-AI/Wan2.2-TI2V-5B-Diffusers for video and image-to-video, and ylacombe/stable-audio-1.0 for audio. Image-to-video uses conditioning images during calibration.
  • llama.cpp covers text generation and writes 2-, 3- and 4-bit GGUF files. Documented models run from Qwen3-4B up to Qwen3-Next-80B-A3B-Instruct, and include the MoE workflow for Qwen3-Coder-30B-A3B-Instruct. CPU execution is supported here.
  • vLLM targets text generation and serving, either W4A4 or weight-only.
  • PyTorch is used for speech generation at W4A4, with Qwen3-TTS and Chatterbox Turbo as the listed examples. Saved checkpoints load without repeating calibration.

The numbers the project publishes

For Qwen3.8-27B on vLLM, the README compares MACaQ W4A4 against BF16. BF16 records perplexity 2.8463 at 1.00x prefill and 1.00x decode. MACaQ W4A4 records perplexity 2.8994 with 2.33x prefill and 2.58x decode.

A separate table compares MACaQ-R, the refinement tool, against an existing published quant of Qwen3.6-35B-A3B. Both files are 22.36 GB. The Unsloth UD-Q4_K_XL file shows KL divergence to FP16 of 0.015731, while MACaQ-R Q4_K_XL shows 0.013367, where lower is better.

Refining a GGUF you already have

MACaQ-R is the part that matters if you already run quantized models. It rewrites the weight codes inside a quantized GGUF while preserving the native formats and block scales, so the resulting file still runs on a standard llama.cpp install. The project says a MACaQ-R GGUF can be started with llama cli -m models/model-macaq-r.gguf or served with llama serve -m models/model-macaq-r.gguf.

The refinement command takes a quantized model, its matching original weights from Hugging Face, an FP16 teacher file and a calibration text file. One MACaQ-R GGUF for Qwen3.6-35B-A3B is published on Hugging Face for people who would rather download than build.

Limits and open questions

The Ascend work is not turnkey. MACaQ W4A4 dense and grouped matmul kernels appear in two CANN pull requests, ops-nn #12568 and ops-tilelang #44, and the project states that full-model execution requires integration with an Ascend inference runtime.

The published quality comparisons in the source are for text models. No perplexity or quality figures are given for the image, video or audio workflows, so how much those lose at W4A4 is not stated. The 32.5 GB figure is described only as device memory, without naming the hardware. And MACaQ-R needs the original source weights plus a teacher file and calibration data, so it is not a standalone repair for a GGUF whose originals you cannot obtain.

What this means for AI app builders

Quantization at this level is about serving weights yourself. If you are building an app on a hosted builder, the model is a configuration choice rather than a pipeline you maintain, and Draftship's paid plans include a choice of AI model alongside the built-in database, sign-in and hosting. The guide to the AI chat and choosing a model covers that side. MACaQ is worth a look when the plan is to run open weights on your own hardware and the file format needs to stay compatible with standard llama.cpp. Draftship is not affiliated with the MACaQ project.

Primary sources

This article was drafted with AI assistance from the primary sources listed below and checked automatically against them. An editor hasn't reviewed it yet.