Next '26: The Future of AI Infrastructure
Summary
Amin Vahdat, SVP and Google Chief Technologist for AI and Infrastructure, delivers a pre-announcement keynote at Google Cloud Next '26, joined by Ben Gilbert and David Rosenthal of the Acquired podcast for an on-stage interview. Vahdat frames Google's AI infrastructure ambitions through the lens of the company's 25-year mission to organize the world's information, arguing that just as web search required inventing new infrastructure from scratch in the early 2000s, solving intelligence today demands a similarly foundational reinvention.
The centerpiece announcement is Google's eighth generation of TPUs — notably, the first time Google has released two distinct chips in a single year: TPU 8T (optimized for training) and TPU 8I (optimized for inference). TPU 8T features 9,600 chips per pod with nearly 3x the floating-point performance of the prior generation, twice the scale-out network bandwidth, and four times the scale-up bandwidth. TPU 8I, described as the "reasoning engine" for the agentic era, quadruples the pod size to 1,152 chips, delivers 10x the floating-point exaflops per pod, and offers 7x larger HBM capacity. Vahdat emphasizes that these chips were designed independently from the ground up — not as derivatives of one another — to serve their distinct workloads.
A key architectural innovation in TPU 8I is a novel network topology called "Board Fly," developed in close collaboration with DeepMind. Unlike the prior topology optimized for throughput and bandwidth, Board Fly dramatically reduces the network diameter — minimizing the distance between any two chips — to achieve the low-latency response times demanded by AI agents operating in real time. Vahdat notes this kind of insight only emerges from deep integration between hardware teams and model researchers.
The interview portion revisits the origin of TPUs in 2013, when Google realized that supporting voice recognition for all Google users would require building "two or three additional Googles" worth of CPU infrastructure. Custom silicon offered 100x efficiency gains that no CPU roadmap could match. From a 30–40 person team and a low tens-of-millions-of-dollars investment, TPUs have grown into the backbone of Google Search, Ads, YouTube, Gmail, and Gemini — as well as third-party workloads like Citadel's trading systems, which achieved 2–4x efficiency gains and 30% cost reduction using TPUs for non-ML workloads.
Vahdat discusses reliability at scale as one of the industry's hardest unsolved problems: when tens of thousands of chips coordinate at nanosecond scale, even a single chip failure halts computation. Google targets over 97% "good put" (forward computational progress, not just raw throughput) on 10,000-chip clusters. Silent data corruption — where a chip silently produces wrong results — is identified as the most dangerous failure mode. He also predicts that the agentic era will trigger a comeback for general-purpose CPUs, as agent orchestration, sandboxing, and code execution require substantial general-purpose compute alongside specialized accelerators. He closes by suggesting the industry may move beyond two chip variants to additional specialized chips for emerging workloads.
Key Takeaways
- •Google is releasing two specialized TPU chips simultaneously for the first time: TPU 8T for training (3x FP performance, 4x scale-up bandwidth) and TPU 8I for inference (10x exaflops, 7x HBM, 4x larger pod).
- •The new 'Board Fly' network topology in TPU 8I reduces inter-chip network diameter to minimize latency — a direct response to agentic AI workloads where response time matters more than throughput.
- •Hardware roadmaps must be co-designed with model researchers 2+ years in advance; Google's bet on agents and low-latency inference was made internally before the topic was publicly mainstream.
- •Silent data corruption — a chip occasionally producing wrong results silently — is a more dangerous failure mode than fail-stop errors, and remains an active research challenge at scale.
- •Google measures 'good put' (actual forward computational progress) rather than raw throughput; achieving 97%+ good put on 10,000-chip clusters requires automated failure detection and recovery in under minutes.
- •TPUs are economically viable beyond AI/ML: Citadel uses them for trading infrastructure, achieving 2–4x efficiency gains and 30% cost reduction.
- •The agentic era will drive a resurgence in general-purpose CPU demand, as agent orchestration, sandboxing, and code execution require substantial non-accelerated compute alongside GPUs/TPUs.
- •Google's vertical integration — from energy and data centers through silicon, software, and frontier models — enables optimizations that no single-layer vendor can achieve.
- •The web search analogy holds: training a model is like building the index (upfront, hard, one-time), but serving is where value is created and where infrastructure investment must ultimately scale.
- •Google Search has changed more fundamentally in the last 2 years than in the prior 25, powered entirely by TPU-backed AI inference at a scale that makes AI overviews economically viable even on non-monetizable queries.
Transcript
[music] [applause and cheering] >> Wow, fantastic to see everyone. We're really, really thrilled to have you all out here tonight. It's a big night for us leading up to a big few days at Google Next. Infrastructure is foundational to everything that we do. We're going to be talking a lot about AI. We're going to be talking a lot about agents. In the end, it is the infrastructure that powers it all. And as a reminder, you all are getting to see this before literally anyone else. So, please do kee...
