Gemma 4 Deep Dive — Cassidy Hardin, Researcher, Google DeepMind

Cassidy Hardin, researcher at Google DeepMind, presents a technical deep dive into the Gemma 4 model family at AI Engineer Europe 2026, covering architectural innovations that enable state-of-the-art performance at practical parameter counts.

Gemma 4 launches with four model sizes organized into two tiers. The larger tier includes a 31B dense model and a 26B mixture-of-experts (MoE) model; the smaller tier consists of two on-device "effective" models at 2B and 4B scale. The 31B dense model ranked third on the LM Arena global leaderboard, outperforming models more than 20 times its size, and supports a 256K context length with native function calling, structured JSON output, and thinking capabilities designed for autonomous agentic workflows. The 26B MoE model is the first MoE in the Gemma family; it uses 128 total experts with 8 activated per forward pass plus one shared expert that is always active and is three times the size of individual experts, yielding only 3.8–3.9 billion active parameters during inference. Both 31B and 26B models rank in the top six of all open-source models on LM Arena. A key licensing change accompanies the launch: all Gemma 4 models move to Apache 2.0, replacing the prior custom license to improve developer accessibility across the full development lifecycle.

On the architecture side, Gemma 4 introduces interleaved local and global attention layers at a 5:1 ratio (4:1 for the 2B effective model). Local layers use a sliding window of 512 tokens (1,024 in larger models), while global layers attend to all preceding tokens. The final layer is always a global layer. To control memory cost, grouped query attention (GQA) is applied differently by layer type: local layers group 2 queries per key-value head; global layers group 8 queries per key-value head, with key-value head dimension doubled to 512 (versus 256 for local) to offset the performance impact of reduced heads.

The on-device effective models (E2B and E4B) introduce Per-Layer Embeddings (PLE) as the mechanism enabling large representational depth at low active-parameter cost. The E2B has 5.1 billion representational parameters but only 2.3 billion effective parameters. PLE gives each of the model's layers (35 layers for E2B, 42 for E4B) its own dedicated embedding table stored in flash memory rather than VRAM. Each PLE table uses a compact 256-dimensional embedding that is projected up to the full model embedding size at runtime, avoiding the VRAM bottleneck that constrains on-device inference on phones and laptops.

For multimodality, Gemma 4 integrates vision and audio natively from training. The 31B and 26B use a 550M-parameter vision encoder; the effective models use a lighter 150M-parameter encoder. A major improvement over Gemma 3 is support for variable aspect ratios and variable resolutions. In Gemma 3, images were processed via "pan and scan" — splitting variable-ratio images into multiple fixed squares passed sequentially. Gemma 4 instead encodes images as variable-sized grids of 16×16-pixel patches with explicit spatial positional encodings, then pools 3×3 grids of patches into single soft tokens. Developers can select from five soft-token budgets; higher budgets (560 or 1120 tokens) are recommended for OCR and spatial object recognition.

Audio support is added to the E2B and E4B effective models for translation and speech recognition, using a 305M-parameter conformer audio encoder. Raw audio is processed through a mel spectrogram, split into chunks, downsampled via two convolutional layers, and output as N/4 soft tokens passed into the conformer.

Deployment options include self-hosting via Hugging Face, Kaggle, or Ollama, and cloud-hosted access for the 31B and 26B models through Google AI Studio and Vertex AI.