Qwen3.8-Flash-Next-GGUF

How to Run Qwen3.8-Flash-Next Locally

Built from Qwen's original weights with our own importance matrix. The calibration corpora behind our builds are public.

Qwen3.8-Flash-Next is the first open-weight release of the architecture behind Qwen4.
These GGUFs are self-quantized from Qwen's original weights with our own importance matrix, published alongside the quants.
The quants are still uploading and need a llama.cpp build with Qwen3.8-Flash-Next support; Atomic Chat runs it as support ships.

Running a 176B model on a 64 GB MacBook

Qwen3.8-Flash-Next has 177B parameters. Our 85 GB quant runs on an M5 Max with
64 GB of memory, with vision, at 36 tok/s. That is not a typo: the file is
larger than the machine's entire RAM.

It works because 39 GB of that file never enters memory at all.

memory breakdown [MiB]  |  total    free     self   model  context  compute
  - MTL0 (Apple M5 Max) |  57344 = 13217 + (44125 = 43720 +   256 +    148)
  - Host                |                   37279 = 37265 +     0 +     14

Why this is possible

51B of the model's 177B parameters are not weights in the usual sense. They are
an n-gram lookup table. The model hashes the last three tokens, and that
hash points at 16 rows of 160 values each. Roughly 2.7 KB per token, read once
per forward pass, out of a 39 GB table.

That is a 1-in-13-million read ratio, at a deterministic address. At 36 tok/s it
comes to about 3 MB/s of random reads, and NVMe answers in under 100 µs against
a 28 ms per-token budget. Common n-grams stay in page cache anyway.

Compare that with the experts: they touch about 6B parameters per token,
gigabytes of traffic, and would be hopeless from disk. That is why ordinary
offloading fails when you run out of memory, and why this table is different.

> [!IMPORTANT]
> On Apple Silicon this only works if the table sits in its own GGUF shard.
> llama.cpp hands Metal the…