A closer look at Gemma 4 with Baseten and NVIDIA
Summary
This session from Google Cloud Next '26 brings together Jason Davenport (Google Cloud), Jay Rodge (NVIDIA), and Philip Kiely (Baseten) to discuss the full inference stack powering modern AI applications, framed around the concept of a "Full Stack Seating Chart" spanning silicon, frameworks, and application layers. The conversation is anchored by the launch of Gemma 4 and the deep partnership between Google, NVIDIA, and Baseten.
Jay Rodge opens by announcing two major hardware milestones: Google Cloud will be among the first providers to offer NVIDIA's next-generation Vera Rubin GPUs (arriving in the second half of 2026), and Google Cloud is now adding Blackwell GPUs (RTX Pro 6000) featuring 96GB of VRAM — enough to run multiple large models on a single chip simultaneously.
Philip Kiely of Baseten frames inference not as a single technology but as an entire discipline: covering CUDA optimization, distributed systems, infrastructure orchestration, and latency SLAs all at once. He describes Baseten as an inference platform built for hypergrowth — serving billions of inferences with a focus on low latency, high reliability, and elastic auto-scaling. Baseten runs on Google Cloud infrastructure, using GKE (Google Kubernetes Engine) to pool GPUs across multiple regions globally and route requests to minimize latency. He also highlights being an early production adopter of NVIDIA Dynamo for inference and significant investment in NVFP4 quantization and Blackwell migration work.
On Gemma 4 specifically, Kiely notes Baseten was a day-zero support partner for the launch. He praises the Gemma family for its multimodality (native image input), its range of sizes (2B to 30B parameters), and its suitability as a fine-tuning base for task-specific enterprise intelligence. The new Gemma 4 26B A4B Mixture-of-Experts (MoE) model activates only 4 billion parameters during inference, delivering performance comparable to much larger models at dramatically higher speeds.
Rodge recommends TensorRT-LLM as the go-to inference optimization SDK for NVIDIA hardware, describing it as requiring minimal code changes to unlock best-in-class performance. He also points to NVFP4 as the optimal precision format when combined with Blackwell GPUs, and highlights NVIDIA Dynamo (also open-source) as a companion scaling framework.
The demo portion shows Baseten's platform deploying Gemma 4's E2B model on an L4 GPU via one-click deployment. A live inference call with a dog image correctly identifies a black Labrador puppy. The demo then shifts to a simulated GPT OSS 120B deployment showing auto-scaling behavior: as inference volume rises, replica count increases automatically to maintain consistent response times, and scales back down when traffic drops. Kiely emphasizes that the platform abstracts all of this away from developers.
On GPU selection, Kiely offers a key insight: total cost of ownership (TCO) at scale often favors more expensive but more efficient GPUs. A GPU that costs twice as much but handles three times the volume reduces the total number of units needed and lowers overall cost. GKE's low inter-model latency is cited as particularly valuable for agentic and multi-model compound AI systems, saving dozens of milliseconds per turn across multi-step agent workflows.
Kiely also presents his new book, "Inference Engineering," written to fill a gap in centralized learning resources covering the full inference stack from CUDA to distributed systems. The session closes with both guests planning to attend the NVIDIA booth and engage developers about agentic workflows and pain points.
Key Takeaways
- •Inference is a full-stack discipline — not just an inference engine. It spans CUDA, infrastructure, distributed systems, quantization, and latency SLAs all at once.
- •NVIDIA Blackwell (RTX Pro 6000) with 96GB VRAM allows multiple large models to run on a single GPU, significantly changing deployment economics.
- •Gemma 4's MoE architecture activates only 4B parameters during inference despite 26B total, delivering near-27B-class intelligence at much lower compute cost.
- •Use TensorRT-LLM + NVFP4 + Blackwell GPUs together for maximum inference performance — this is NVIDIA's recommended stack for Gemma 4.
- •TCO at scale often favors higher-cost, higher-efficiency GPUs: a GPU that costs 2x but handles 3x the volume lowers total fleet size and overall cost.
- •Auto-scaling inference (replica count tracking request volume) is essential for hypergrowth AI apps — platforms like Baseten abstract this from developers.
- •Multi-region GPU pooling via GKE allows unified compute access across geographies, reducing latency by routing requests to optimal nodes.
- •For agentic and multi-model compound AI systems, low inter-model network latency (solved by GKE) compounds into meaningful end-to-end latency savings.
- •Gemma models (2B–30B) are well-suited as fine-tuning bases for task-specific enterprise use cases — a better fit than very large models like 120B for many workloads.
- •Developers can deploy Gemma 4 models from Hugging Face on Baseten with a single click, getting an OpenAI-compatible API endpoint with inference optimizations pre-applied.
- •NVIDIA Dynamo is open-source and available for production inference scaling — Baseten was an early adopter running it in production shortly after its GTC announcement.
Transcript
for getting us back on track so quickly. So, thank you so much for that. All right, I am gathered here this afternoon with two of my favorite people. First one, Jay Raj with Nvidia and Philip Kylie with Base 10. So great to have you guys here on the show to talk about cool things with Cloud. Yeah, thank you for having us. All right. One of the things that we've been hearing you starting off with Thomas's keynote today is really about inference. >> Mhm. And so, you know, there's so much going on ...
