Google’s open-source generative artificial intelligence (AI) model Gemma 3 supports vision-language understanding, long context handling, and improved multi-linguality. In a recent blog post, Google DeepMind and AI Studio teams discussed the new features in Gemma 3. The model also highlights KV-cache memory reduction, a new tokenizer and offers better performance and higher resolution vision encoders.
Gemma 3 Technical Report summarizes these new features and capabilities. The new vision-language understanding capability includes the models (4B, 12B and 27B parameters) using a custom Sigmoid loss for Language-Image Pre-training (SigLIP) vision encoder, which enables the models to interpret visual input. The encoder operates on fixed 896×896 square images and to handle the images with different aspect ratios or high resolutions, a “Pan & Scan” algorithm is employed. This involves adaptively cropping the image, resizing each crop to 896×896, and then encoding it. The Pan & Scan method further improves performance on tasks involving non-square aspect ratios, high-resolution images, and text reading in images. The new model also treats images as a sequence of compact “soft tokens” produced by MultiModalProjector. This technique cuts down on the inference resources needed for image processing by representing visual data with a fixed number of 256 vectors.
The vision encoder processing in Gemma 3 uses bi-directional attention with image inputs. Bidirectional attention is a good approach for understanding tasks (as opposed to prediction tasks) where we have the entire text and need to deeply understand it (like in models such as BERT).
Architectural changes for memory efficiency include modifications to reduce KV-cache memory usage, which tends to increase with long context. These changes reduce the memory overhead during inference with long context compared to global-only attention mechanisms used in Gemma 1 and the 1:1 local/global ratio used in Gemma 2. This allows for the analysis of longer documents and conversations without losing context. Specifically, it can handle 32k tokens for the 1B model and 128k tokens for larger models.
Gemma 3 also introduces an improved tokenizer. The vocabulary size has been changed to 262k, but uses the same SentencePiece tokenizer. To avoid errors, they recomend to use the new tokenizer with Gemma 3. This is the same tokenizer as Gemini which is more balanced for non-English languages. Gemma 3 has improved multilingual capabilities due to a revisited data mixture with an increased amount of multilingual data (both monolingual and parallel). The team also revised the pre-training data mixture and post-training process to enhance its multilingual capabilities.
Gemma 3 models showed better performance compared to Gemma 2 on both pre-trained instruction-tuned versions across various benchmarks. It is a better model that fits in a single consumer GPU or TPU host. The Gemma 27B IT model ranks among the top 10 models in LM Arena as of Apr 12, 2025, outperforming much larger open models and showing a significantly higher Elo score than Gemma 2.
Gemma 3 models’ longer context handling can generalize to 128k context length after Rotary Position Embedding (RoPE) rescaling during pre-training. They increased RoPE base frequency from 10k to 1M on global self-attention layers, and kept the frequency of local layers at 10k.
For more information on Gemma 3 model, check out the developer guide, model card, meme generator, and Gemmaverse to explore Gemma models developed by the community.
