Skip to content

smolmaxxing runs 80B models on a phone from flash

A C-based patch set for llama.cpp reports 16B mixture-of-experts models on roughly 1 GB of RAM and an 80B model on an Android phone, with per-device token speeds.

Draftship Editorial4 min read

On 2026-11-14 the GitHub repository hyperionaidev/smolmaxxing showed 650 stars and 23 forks, gathered in the ten days since it was created on 2026-11-04. Written mainly in C and released under the MIT license, it is a set of patches and tooling built to run LLMs on small devices that would normally be written off: a streaming box, a 2013 tablet and a mid-range phone.

What the repository actually is

Everything in smolmaxxing is a patch on llama.cpp. The README describes four strands of work: kernels written for specific chips, memory savings inside each model, a pager that reads mixture-of-experts weights from flash, and a streaming path for dense models larger than RAM. The author states that every number was measured on the device itself, and the repo carries a lab notebook file with 165 experiments, failures included.

Three devices are covered. A Xiaomi TV Box S (3rd Gen) with an Amlogic S7D, four Cortex-A55 cores, 1.9 GB of RAM of which about 1 GB is usable, on Android 14 without root. An iPad Air from 2013 with an Apple A7, 1 GB of RAM and about 700 MB usable, on jailbroken iOS 12.5.7. And a OnePlus Nord CE 2 from 2022 with a MediaTek Dimensity 900, 7.4 GB of RAM and about 3 GB free, on Android 13 without root.

The measured results: the TV box runs Qwen3.5-2B at 7.7 tokens per second as its everyday model, and Ling-mini-2.0, a 16B mixture-of-experts model, at 5.1 tokens per second. The iPad runs Bonsai-1.7B in a 1-bit format at 24 to 26 tokens per second, and the same 16B Ling model at 2.6 to 4.5 tokens per second. The phone runs Qwen3.6-35B-A3B at 4.9 to 5.7 tokens per second and Qwen3-Next-80B-A3B at 2.6 to 3.1 tokens per second.

How a 16B model fits in about a gigabyte

The README names the techniques. A 1-bit lookup-table kernel and repacked 2-bit and ternary formats are tuned for the Cortex-A55, with separate versions for the A7, which has no dot-product instructions. Rarely used vocabulary output rows are dropped from the head, cutting the memory each model needs. On the TV box, a headless mode stops the launcher and Google services while a model runs.

The large models depend on paging. A mixture-of-experts model touches only a few experts per token, so the popular ones stay in RAM, the rest are read from flash on demand, and the next layer's experts are fetched ahead of time. The author credits this mechanism for 8B and 16B models on about 1 GB, and for 35B and 80B models on a phone. Dense models bigger than RAM are streamed at the flash's speed.

What it takes to try it

Model files are on Hugging Face. The phone files load in stock llama.cpp, but the TV box and iPad files do not, because stock llama.cpp cannot load their pruned heads, and the published speeds assume the patched expert pager. The phone route calls for an Android device with 8 GB of RAM and points at Qwen3.6-35B-A3B at 12.3 GB and Qwen3-Next-80B-A3B at 27.6 GB across two files. The README asks owners of other small devices to run a model and submit their numbers, and says single-board computers, routers and more phones are coming. The 1-bit kernel needs only basic NEON, and the streaming approach is described as working on any Linux or Android device where the model is bigger than RAM.

Licensing splits by part: the code is MIT, the llama.cpp patches are derivatives of llama.cpp, model weights keep their own licenses, and the evaluation texts are Wikipedia excerpts under CC BY-SA 4.0.

Limits and open questions

The results cover three specific devices, each with its own patch and setup, so they are not a general benchmark of phones or set-top boxes. The iPad path requires a jailbreak on iOS 12.5.7. "Biggest model" here means the biggest the author tried and measured, not a hardware ceiling. The source does not report sustained performance under heat, power draw, or behaviour on long contexts, and it does not say how many of these tricks are needed before a model becomes usable rather than merely loadable. Chat benchmark results and raw logs sit in the bench directory for anyone who wants to check them.

What this suggests for builders

This suggests that for local inference on cheap hardware, memory capacity, not raw compute, is the binding constraint, and that reading weights from flash can stretch a device well past its RAM. For anyone building an app with AI, the numbers are a useful calibration of the size and speed tradeoff: a small model can feel responsive while a 16B model on 1 GB is a demonstration rather than a product. Draftship does not run models on-device and has no connection to this project; it builds and hosts apps from a plain-language description. If you are choosing a model for a chat feature in an app you are making, Draftship's guidance on using the AI chat and choosing a model covers that decision, and the original smolmaxxing announcement has the full device-by-device detail.

Primary sources

This article was drafted with AI assistance from the primary sources listed below and checked automatically against them. An editor hasn't reviewed it yet.