majentik/gemma-4-31B-it-RotorQuant-MLX-2bit

KV-cache quantization without any fork (recommended, 2026): upstream llama.cpp/Ollama now cover this natively — use -ctk q8_0 -ctv q8_0 (half KV memory, negligible quality loss: perplexity +0.002–0.05) or -ctk q4_0 -ctv q4_0 (quarter memory, ≈7.6% perplexity increase). In Ollama: OLLAMA_KV_CACHE_TYPE=q8_0 with OLLAMA_FLASH_ATTENTION=1. Keep K and V types symmetric to stay on the fast fused Flash-Attention path. Since April 2026, mainline llama.cpp also applies Hadamard rotation to KV activations (PR #21038), which greatly improves low-bit KV quality (opt-out: LLAMA_ATTN_ROT_DISABLE=1).

The RotorQuant/TurboQuant fork flow below is experimental/legacy: the TurboQuant llama.cpp PR was closed without merging (June 2026) and the fork is unmaintained relative to mainline. It is NOT required to use this model.

Gemma 4 31B-it - RotorQuant MLX 2-bit

2-bit weight-quantized MLX version of google/gemma-4-31B-it with the legacy RotorQuant KV-cache fork (superseded by upstream llama.cpp KV options). Optimized for Apple Silicon inference via the MLX framework. The most aggressive quantization, fitting the full model in the smallest possible footprint.

Approximate model size: ~9 GB

Model Specifications

Property	Value
Base Model	google/gemma-4-31B-it
Parameters	31 billion
Architecture	Dense transformer
Modality	Multimodal: image + text input, text output
License	Apache 2.0
Weight Quantization	2-bit (~9 GB)
KV-Cache Quantization	RotorQuant
Framework	MLX (Apple Silicon)

Quickstart

import mlx.core as mx
from mlx_lm import load, generate

model, tokenizer = load("majentik/gemma-4-31B-it-RotorQuant-MLX-2bit")

prompt = "Describe this image in detail."
response = generate(model, tokenizer, prompt=prompt, max_tokens=512)
print(response)

For multimodal usage with images:

from mlx_vlm import load, generate

model, processor = load("majentik/gemma-4-31B-it-RotorQuant-MLX-2bit")

prompt = "What do you see in this image?"
output = generate(model, processor, prompt=prompt, image="path/to/image.jpg", max_tokens=512)
print(output)

About the RotorQuant / TurboQuant labels

RotorQuant and TurboQuant are this project's release labels, not distinct quantization algorithms — for any given tier, both brand repos carry byte-identical weights produced with the standard MLX / llama.cpp quantizers. No brand-specific speedup is claimed or measured. The KV-cache fork these labels originally referred to is legacy; for KV-cache memory savings use the upstream options described above (-ctk/-ctv q8_0, OLLAMA_KV_CACHE_TYPE).

KV-Cache Quantization Comparison

Method	Prefill Speed	Decode Speed	Memory Savings	Reference
TurboQuant	1x (baseline)	1x (baseline)	High	arXiv: 2504.19874

Memory Estimates (Gemma 4 31B-it)

Precision	Approximate Size	MLX Variant
FP16 (original)	~62 GB	--
8-bit quantized	~31 GB	RotorQuant-MLX-8bit
4-bit quantized	~17 GB	RotorQuant-MLX-4bit
2-bit quantized	~9 GB	This model

Hardware Requirements

This model requires approximately 9 GB of unified memory. Recommended hardware:

• Apple M1 (16 GB+)
• Apple M2 (16 GB+)
• Apple M3 (16 GB+)
• Apple M4 (16 GB+)
• Any Apple Silicon Mac with 16 GB+ unified memory

See Also

• google/gemma-4-31B-it -- Base model
• majentik/gemma-4-31B-it-RotorQuant-MLX-8bit -- MLX 8-bit variant
• majentik/gemma-4-31B-it-RotorQuant-MLX-4bit -- MLX 4-bit variant
• majentik/gemma-4-31B-it-TurboQuant-MLX-2bit -- TurboQuant MLX 2-bit variant
• RotorQuant GitHub
• MLX Framework

Quant trade-off (MLX lane)

Bits	Approx size	Use case	Recommendation
2-bit	~8.1 GB	Aggressive quantization	Very low-RAM Macs
3-bit	~11 GB	Lossy but small	Low-RAM Macs
4-bit	~13 GB	Balanced default	Recommended for most Macs
5-bit	~16 GB	Higher fidelity	Quality-sensitive
6-bit	~19 GB	Approaching FP16 quality	High-fidelity
8-bit	~24 GB	Near-lossless reference	Fidelity-critical work

(Current variant — 2bit — is bolded.)

Variants in this family

(Showing 14 sibling variants under majentik/gemma-4-31b-it-*. The current variant — RotorQuant-MLX-2bit — is bolded.)

Variant	Runtime	Approx size	Use case
RotorQuant-GGUF-IQ4_XS	llama.cpp	~27 GB	Lossy 4-bit, low-RAM CPU/edge
RotorQuant-GGUF-Q2_K	llama.cpp	~19 GB	Lossy, low-RAM CPU/edge
RotorQuant-GGUF-Q3_K_M	llama.cpp	~24 GB	Smaller 3-bit, CPU-friendly
RotorQuant-GGUF-Q4_K_M	llama.cpp	~34 GB	Balanced default
RotorQuant-GGUF-Q5_K_M	llama.cpp	~41 GB	Higher fidelity, more RAM
RotorQuant-GGUF-Q8_0	llama.cpp	~65 GB	Near-lossless reference
RotorQuant-MLX-2bit	mlx-lm	~9.9 GB	Apple Silicon, smallest
RotorQuant-MLX-4bit	mlx-lm	~19 GB	Apple Silicon balanced
RotorQuant-MLX-8bit	mlx-lm	~37 GB	Apple Silicon reference
TurboQuant-MLX-2bit	mlx-lm	~9.9 GB	Apple Silicon, smallest
TurboQuant-MLX-4bit	mlx-lm	~19 GB	Apple Silicon balanced
TurboQuant-MLX-8bit	mlx-lm	~37 GB	Apple Silicon reference