Google released the Gemma 3 QAT family, quantized versions of their open-weight Gemma 3 language models. The models use Quantization-Aware Training (QAT) to maintain high accuracy when the weights are quantized from 16 to 4 bits.
All four Gemma 3 model sizes are now available in QAT versions: 1B, 4B, 12B, and 27B parameters. The quantized versions require as little as 25% of the VRAM needed by the 16 bit models. Google claims that the 27B model can run on a desktop NVIDIA RTX 3090 GPU with 24GB VRAM, while the 12B model can run on a laptop NVIDIA RTX 4060 GPU with 8GB VRAM. The smaller models can run on mobile phones or other edge devices. By using Quantization-Aware Training, Google was able to reduce the accuracy loss from quantization, as much as 54%. According to Google,
While top performance on high-end hardware is great for cloud deployments and research, we heard you loud and clear: you want the power of Gemma 3 on the hardware you already own. We’re committed to making powerful AI accessible, and that means enabling efficient performance on the consumer-grade GPUs found in desktops, laptops, and even phones…Bringing state-of-the-art AI performance to accessible hardware is a key step in democratizing AI development…We can’t wait to see what you build with Gemma 3 running locally!
InfoQ covered Google’s initial launch of the Gemma series in 2024, which was quickly followed by Gemma 2. The open-source models achieved performance competitive with models 2x larger by incorporating design elements from Google’s flagship Gemini LLMs. The latest iteration, Gemma 3, has performance improvements that make it the “top open compact model,” according to Google. Gemma 3 also added vision capabilities, except in the 1B size.
While the unquantized Gemma 3 models exhibit impressive performance for their size, they still require substantial GPU resources. For example, the unquantized 12B model requires an RTX 5090 with 32GB of VRAM. To allow the quantization of model weights without sacrificing performance, Google used QAT. This technique simulates inference-time quantization during training, instead of simply quantizing the model after it’s trained.
Google dev Omar Sanseviero wrote about using the QAT models in a thread on X and suggested there was still room for improvement:
We still recommend playing with the models (e.g. we didn’t quantize the embeddings, some people even did 3-bit quantization and it was working better than naive 4 bits)
Users praised the QAT models’ performance in a discussion on Hacker News:
I have a few private “vibe check” questions and the 4 bit QAT 27B model got them all correctly. I’m kind of shocked at the information density locked in just 13 GB of weights. If anyone at Deepmind is reading this — Gemma 3 27B is the single most impressive open source model I have ever used. Well done!
Django Web Framework co-creator Simon Willison wrote about his experiments with the models and said:
Having spent a while putting it through its paces via Open WebUI and Tailscale to access my laptop from my phone I think this may be my new favorite general-purpose local model. Ollama appears to use 22GB of RAM while the model is running, which leaves plenty on my 64GB machine for other applications.
The Gemma 3 QAT model weights are available on HuggingFace and in several popular LLM frameworks, including Ollama, LM Studio, Gemma.cpp, and llama.cpp.
