Ternary-Bonsai-2-27B-gguf

Prism ML Website  | 
Whitepaper  | 
Demo & Examples  | 
Discord

Bonsai 2 27B — GGUF

Full 27B-class reasoning in ternary transformer weights, for llama.cpp (CUDA, Metal, CPU)

> \~9.3x smaller than FP16 (ideal) | 98.2% of FP16 intelligence retained | \~47 tok/s on an Apple M5 Max laptop

Highlights

• \~5.9 GB language model (down from \~54 GB FP16) — full 27B-class reasoning on a standard laptop or a single GPU
• 98.2% of FP16 intelligence retained: 84.78 average across 14 thinking-mode benchmarks — far above the conventional IQ2_XXS build (72.59) at about 82% of its footprint, and within 0.4 points of UD-Q4_K_XL at three times the footprint
• Retains thinking, reasoning, and agentic behavior deep in the sub-4-bit regime, where conventional low-bit representations collapse: math within half a point of full precision (96.57), coding level with the baseline (89.42), agentic tool calling at 74.92
• End-to-end ternary language weights across embeddings, attention projections, MLP projections, and LM head, at a *true* 1.72 bits per weight — no high-precision escape hatches behind a low-bit label; the vision tower ships as a separate Q8_0 mmproj pack
• 262K-token context on-device, kept practical by the Qwen3.8-27B hybrid-attention backbone (\~75% linear attention)
• Two GGUF packings with custom ternary hybrid-attention kernels for llama.cpp (CUDA, Metal) — PTQ1_0 packs trits densely (1.75 bits/weight, 5.95 GB), PQ2_0 stores each trit in a 2-bit slot (2.13 bits/weight, 7.21 GB); packed weights are consumed directly, never expanded back to FP16
• MLX companion: also available as Ternary-Bonsai-2-27B-mlx-2bit for native Apple Silicon inference

Resources

• **[Whitepaper](https://github.com/PrismML-Eng/Bonsai-demo/blob/main/bonsai-2-27b-whitepape…

Source: https://huggingface.co/prism-ml/Ternary-Bonsai-2-27B-gguf