AI Infrastructure & FinOps Intelligence Brief | Google Cloud Next 2026 — ConferenceDigest
AI Infrastructure & FinOps Intelligence Brief
For: Cloud infrastructure buyers, FinOps leaders, and AI capacity planners
AI Infrastructure & FinOps Intelligence Brief
**Google Cloud Next 2026 — Strategic Analysis for AI Infrastructure Buyers and FinOps Leaders**
Executive Summary
Google Cloud Next 2026 marked a structural inflection point in AI infrastructure procurement: the era of homogeneous accelerator fleets is over. The central announcement — two purpose-built eighth-generation TPU variants released in a single year for the first time in Google's history — signals that the industry has moved past a unified chip-for-all model into a world where training and inference workloads demand fundamentally different silicon. For infrastructure buyers and FinOps leaders, this bifurcation reshapes every capacity decision made in the next two to four quarters, from reserved instance commitments to on-demand GPU spot strategies.
The most strategically significant signal from Next '26 is not any single chip announcement but the inference-eclipses-training thesis articulated by Amin Vahdat (SVP and Chief Technologist, AI and Infrastructure) and reinforced by Acquired podcast hosts Ben Gilbert and David Rosenthal in their live session. Inference compute loads have now overtaken training as the dominant AI workload on Google's infrastructure. Vahdat used a precise analogy: training is the one-time web crawl, inference is the ongoing query-serving that eventually consumed all infrastructure investment. That transition took 15 years on the web. In AI it is happening in approximately four years. The implication for FinOps teams is fundamental: your cost models, reserved capacity strategies, and GPU-vs-TPU tradeoff analyses must be reoriented from training-centric to inference-centric planning horizons.
The Citadel Securities case study — 2x to 4x faster workloads at 30% lower cost using Ironwood TPUs — provides the clearest vendor-neutral benchmark yet for evaluating TPU adoption beyond pure ML workloads. Critically, Citadel is a market maker, not an AI-native company. Their quantitative research workloads are compute-intensive but not transformer training jobs. The result validates that TPU economics can outperform GPU alternatives even for non-ML parallel computation, a finding that expands the total addressable use case for FinOps-driven TPU evaluation. Combined with Philip Kiely's (Baseten) TCO framework — a GPU that costs 2x but handles 3x the inference volume reduces fleet size and lowers total cost — the economic case for rethinking accelerator procurement strategy is now grounded in production evidence, not theoretical benchmarks.
The arrival of NVIDIA Blackwell (RTX Pro 6000, 96GB VRAM) and forthcoming Vera Rubin NVL72 on Google Cloud, announced by Jay Rodge (NVIDIA) in the Gemma 4 session, further complicates the procurement landscape. Buyers now face a genuine three-way decision surface: Google TPUs for deep vertical integration with GKE and Gemini, Blackwell GPUs for NVIDIA-ecosystem workloads and the TensorRT-LLM + NVFP4 optimization stack, and Vera Rubin for next-generation capacity commitments in H2 2026. Each path carries distinct TCO profiles, software ecosystem dependencies, and FinOps risk surfaces that demand structured evaluation frameworks rather than default reordering of incumbent fleet capacity.
Key Findings
Finding 1: The TPU v8 Split — Two Chips, Two Radically Different Value Propositions
For the first time in Google's history, a single generation produced two independent chip designs rather than one. TPU v8T (training) and TPU v8I (inference) were developed from the ground up as distinct architectures, not derivatives of each other, according to Vahdat in "Next '26: The Future of AI Infrastructure."
TPU v8T (Training): 9,600 chips per pod, 2.8x improvement over Ironwood in FP performance, 121 exaflops FP4 per pod, twice the scale-out network bandwidth, four times the scale-up bandwidth, two petabytes of shared bandwidth memory — enough, per the Opening Keynote, to hold the Library of Congress 100 times over. This chip is built for pre-training and large-scale fine-tuning workloads where aggregate throughput and memory bandwidth dominate the performance equation.
TPU v8I (Inference): 1,152 chips per pod (4x larger than prior inference configurations), 10x floating-point exaflops per pod, 7x larger HBM capacity, and a novel "Board Fly" network topology co-designed with DeepMind. The 5x speedup over prior inference silicon (cited by David Rosenthal in the Acquired live session) is projected to shift a meaningful subset of AI tasks from asynchronous response patterns to synchronous, real-time user-experience patterns. Vahdat described this topology specifically: unlike prior networks optimized for throughput, Board Fly minimizes network diameter — the distance between any two chips — to achieve the sub-millisecond inter-chip latency demanded by agent orchestration workloads.
FinOps implication: Organizations currently pooling training and inference workloads on the same accelerator fleet are leaving efficiency on the table. Procurement teams should evaluate workload decomposition — separating training reservations (v8T) from inference capacity (v8I) — rather than purchasing a single chip class for both.
Finding 2: The "Good Put" Metric — Why 97% Is the New SLA Floor
Vahdat introduced a term that every infrastructure buyer should understand before signing an SLA: "good put." This is not throughput. Good put measures actual forward computational progress — the fraction of cycles on a chip cluster that advance the computation rather than recovering from failures, retrying corrupted operations, or idling during fault detection. On 10,000-chip clusters, even a 3% good put loss compounds into significant wasted capacity and cost.
Google targets over 97% good put on its largest clusters. Achieving that number requires automated failure detection and recovery in under minutes, and active defense against what Vahdat identified as the most dangerous failure mode: silent data corruption, where a chip produces wrong results without triggering an error flag. For buyers evaluating multi-thousand-chip commitments with any vendor, good put — not peak theoretical FLOPS — should be the primary contractual performance metric requested.
FinOps implication: Request good put SLAs in accelerator capacity contracts, not raw throughput specifications. A cluster operating at 85% good put wastes roughly 15% of your accelerator spend on non-productive cycles. The delta between a 90% and 97% good put guarantee on a $10M annual TPU commitment is approximately $700K in effective wasted capacity.
Finding 3: Ironwood TCO — The Citadel Benchmark Is the Market Reference Point
The Citadel Securities session ("Citadel Securities runs workloads up to 4x faster with Ironwood") is the most important vendor-supplied benchmark at Next '26 for FinOps evaluation. Citadel — one of the world's largest market makers, trading over half a trillion dollars per day — migrated quantitative research workloads to Ironwood TPUs and achieved 2x to 4x faster execution at 30% lower cost compared to prior infrastructure. Critically, these are not transformer training workloads. They are massively parallel computational research jobs. Ironwood's thousands of parallel chips per single workload compressed research timelines from weeks or days to hours or minutes.
The strategic implication is that TPU economic advantages are not confined to the ML training use case. Buyers with parallel numerical computation workloads — financial modeling, simulation, scientific computing, large-scale data processing — should include TPU in their accelerator evaluation even if their primary AI ambitions are secondary. The Citadel result also validates the "steel sharpens steel" co-optimization pattern: deep collaboration between Google Cloud and Citadel on TPU-level tuning produced gains above what standard deployments achieve. Buyers who treat cloud accelerators as commodity procurements rather than joint engineering partnerships systematically underperform against peers who engage vendor optimization programs.
Finding 4: Blackwell and Vera Rubin — NVIDIA's Dual-Horizon Expansion on Google Cloud
Jay Rodge's announcement in "A closer look at Gemma 4 with Baseten and NVIDIA" introduced two NVIDIA hardware milestones with distinct planning horizons. First, Blackwell RTX Pro 6000 GPUs (96GB VRAM) are now available on Google Cloud, enabling multiple large models to run simultaneously on a single GPU — a material change to inference deployment density and per-GPU revenue potential. Second, NVIDIA Vera Rubin NVL72 will arrive on Google Cloud in H2 2026, making Google among the first providers to offer it.
The Opening Keynote added a critical infrastructure detail: a new network called Virgo links 134,000 chips with 47 petabits per second of bandwidth — a scale that makes multi-node inference serving at previously cost-prohibitive model sizes economically viable. For buyers currently planning 2026 H2 and 2027 capacity, Vera Rubin introduces a new performance tier that may make current Blackwell reservations less optimal if workloads can wait six to nine months.
Procurement decision framework: For immediate inference needs (Q2–Q3 2026), Blackwell is the correct choice. For capacity planning extending into Q4 2026 and beyond, buyers should evaluate whether Vera Rubin access timelines on Google Cloud justify either deferring commitments or negotiating Vera Rubin upgrade provisions into current Blackwell contracts.
Philip Kiely (Baseten) described in the Gemma 4 session that the 26B A4B Mixture-of-Experts model activates only 4 billion parameters during any single inference pass, despite containing 26 billion total parameters. This architecture delivers near-27B-class intelligence output while incurring inference compute costs closer to a 4B dense model. The 40 million downloads in three weeks post-launch (cited in the "From smartphones to Raspberry Pi" session) confirms that developer demand for MoE-class open weights is real and accelerating.
For FinOps teams pricing inference workloads: the activated-parameter count, not the total parameter count, drives actual compute and memory bandwidth consumption. A fleet sized for 26B dense model inference will be materially over-provisioned for Gemma 4 MoE workloads. Conversely, capacity sized for 4B dense inference may be under-provisioned for the quality requirements Gemma 4 MoE actually satisfies. Inference cost modeling must now account for sparse activation ratios as a first-class variable alongside context length and request concurrency.
The hybrid inference pattern described by Omar Sanseviero using Cactus Compute (a YC startup cited in the session) is equally significant: routing approximately 70–80% of daily inference volume to a local or edge-hosted Gemma instance, with only complex queries escalating to cloud Gemini. This architecture is not a cost-saving hack but a design pattern that fundamentally alters the unit economics of per-query inference at scale.
Finding 6: GKE as the Inference Orchestration Layer — Multi-Region Pooling and Agentic Latency
Philip Kiely described Baseten's architecture explicitly: Baseten runs on Google Cloud infrastructure, using Google Kubernetes Engine (GKE) to pool GPUs across multiple regions globally and route inference requests to the node with the lowest latency. This is not a Google-proprietary pattern — it is an open Kubernetes capability — but GKE's inter-node latency characteristics are cited as specifically advantageous for agentic multi-model compound AI systems.
In agentic workflows where a single user request triggers 10 to 20 sequential or parallel model calls, network latency between inference steps compounds across the full orchestration chain. GKE's low inter-model latency saves, per Kiely, "dozens of milliseconds per turn" — which across a multi-step agent workflow translates to seconds of end-user latency improvement. For buyers building agent-first products where response time is a core UX metric, infrastructure topology at the GPU-pool level is a legitimate product decision, not merely an ops concern.
Kiely also reinforced the TCO argument for higher-cost accelerators: a GPU that costs twice as much but handles three times the inference volume reduces total fleet size and lowers aggregate cost. The relevant FinOps metric is not GPU unit cost but cost per query at target latency SLA. Buyers optimizing solely on sticker price per GPU-hour will systematically overprovision fleet size relative to peers using higher-efficiency accelerators.
Finding 7: The Capex Narrative — Google's $175–185B Signal and What It Means for Capacity Availability
Sundar Pichai disclosed in the Opening Keynote that Google's capex has grown nearly 6x in four years — from $31 billion in 2022 to a planned $175–185 billion in 2026, with just over half of ML compute directed at the cloud business. Ben Gilbert and David Rosenthal specifically flagged this number in the Acquired live session as a structural signal: the organization that invented transformers, employs leading AI researchers, and is making the largest infrastructure bet in its history is explicitly directing that bet at cloud customers.
For enterprise procurement teams, the capex signal has a direct practical implication: TPU capacity availability from Google Cloud is less likely to be supply-constrained in the near term than capacity from providers without equivalent infrastructure investment. Logan Kilpatrick acknowledged in his AI Studio interview that "token supply constraints are real even at Google's scale," but the directional trajectory of $175–185B annual capex represents a different availability risk profile than a provider spending $15–25B annually. Buyers with large-scale capacity requirements should evaluate Google Cloud's infrastructure investment trajectory as a supply-risk variable alongside pricing in long-term capacity negotiations.
Finding 8: The Inference-Eclipses-Training Thesis — Structural Implications for Long-Term Capacity Strategy
The most analytically rigorous framing at Next '26 came from David Rosenthal in the Acquired live session: the transition from training-dominant to inference-dominant AI compute mirrors the web's evolution from crawl infrastructure to query-serving infrastructure, compressed from 15 years to approximately 4 years. Rosenthal's insight is that this transition is not visible in raw compute volumes alone — it requires reading the architectural signal of Google releasing a dedicated inference chip (TPU v8I) independently from its training chip for the first time.
For FinOps leaders building 12-to-24-month capacity plans: the implication is that inference capacity will face stronger demand growth and potentially tighter supply than training capacity in the near term. Training jobs are still large, but they are batch-schedulable, location-flexible, and interruptible. Inference serving is always-on, latency-sensitive, and geographically constrained by user proximity. Organizations that lock in inference-optimized capacity at current pricing before the Vera Rubin demand wave hits in H2 2026 are likely to have better unit economics than those who wait.
Strategic Implications
Bifurcate your accelerator strategy. The era of a single GPU SKU for all AI workloads is over. Organizations should maintain separate procurement tracks for training (batch, throughput-optimized, preemptible-eligible) and inference (latency-optimized, always-on, SLA-bound). This is not a hardware preference — it is a cost structure decision. Continuing to serve inference on training-optimized chips wastes margin at the infrastructure layer.
Re-price inference with MoE activation ratios. Legacy cost models that price inference by total parameter count are structurally wrong for MoE architectures like Gemma 4 26B/4B. FinOps teams must update their cost-per-inference models to use activated parameter count as the primary compute variable. Organizations that reprice correctly will find their inference budgets go further — and will be able to run larger MoE models within the same cost envelope previously allocated to smaller dense models.
Evaluate TPU for non-ML parallel workloads. The Citadel result establishes a production benchmark: 2–4x speedup at 30% lower cost for quantitative research workloads. Any organization running large-scale parallel numerical computation — financial modeling, simulation, scientific computing — should include Ironwood TPU in its next accelerator evaluation cycle, not just GPU options. The economics may be more favorable than assumed.
Treat good put as a contractual floor. Before signing any large accelerator commitment, request good put metrics for the specific cluster configuration being purchased. A vendor promising 500 TFLOPS at 85% good put delivers less effective compute than one promising 400 TFLOPS at 97% good put. Build good put minimums into SLA language the same way you would latency percentiles.
Model the hybrid inference architecture. The Cactus Compute pattern — 70–80% of queries served locally or at edge via Gemma, 20–30% escalated to cloud — is not just a cost-saving pattern for startups. At enterprise scale, routing even 50% of inference volume to cheaper local or edge infrastructure can reduce cloud inference spend by 30–40% while maintaining quality for complex queries. FinOps teams should model this architecture against their current per-query cloud costs before the next capacity renewal.
Plan Vera Rubin timing into H2 2026 commitments. If your workloads can absorb a 6-to-9-month deployment delay, negotiate Vera Rubin upgrade options or access timelines into current Blackwell contracts rather than locking into multi-year Blackwell commitments at H1 2026 pricing.
Action Items
Q2 2026 (Immediate — Next 90 Days)
Audit current accelerator fleet for training vs. inference workload mix. Instrument your GPU/TPU clusters to measure actual workload type distribution. If more than 40% of cycles are serving inference rather than training, you are likely on the wrong silicon for that portion of the fleet. Quantify the cost delta before the next contract renewal.
Run a Gemma 4 MoE cost model against current inference spend. Pull your current inference workload specifications (tokens per day, latency SLA, model size used). Re-model using Gemma 4 26B A4B MoE's effective 4B activated parameter footprint. The gap between current spend and MoE-equivalent spend is your efficiency opportunity. This analysis takes less than a week and quantifies the migration business case.
Request a TPU Ironwood benchmark for your parallel compute workloads. If your organization runs any large-scale numerical computation outside of ML training — quantitative research, financial simulation, scientific workloads — contact Google Cloud to arrange a Citadel-pattern benchmark. Use the Citadel result (2–4x speedup, 30% lower cost) as your negotiating baseline for what an acceptable result looks like before committing engineering time.
Initiate Blackwell RTX Pro 6000 capacity negotiation. 96GB VRAM per GPU changes inference density calculations meaningfully. Model your current inference fleet utilization against consolidated Blackwell deployment. If your models fit within 96GB, you may be able to reduce total GPU count materially while maintaining or improving throughput.
Add good put minimums to all outstanding and upcoming SLA negotiations. Establish 97% as the internal target floor. Accept nothing below 95% without a compensating cost reduction. Get this language into contracts before renewal windows close.
Q3 2026 (Planning Horizon — 90–180 Days)
Evaluate GKE multi-region inference pooling for agent workloads. If you are building or scaling agent-first products, audit your current inference topology for inter-model latency. Consolidating onto GKE with multi-region GPU pooling is a proven architecture (Baseten's production implementation) that reduces agent workflow end-to-end latency by measurable seconds. Prioritize this for any product where agent response time is a user-facing SLA.
Model the hybrid inference architecture against Q4 2026 capacity plan. Build a scenario model: what does your inference cost structure look like if 60% of volume is routed to Gemma 4 (edge or local) and 40% remains on cloud? This model should inform whether your Q4 2026 cloud inference capacity reservation needs to grow proportionally with AI usage growth — or whether hybrid routing allows you to grow AI usage without proportionally growing cloud inference spend.
Negotiate Vera Rubin upgrade provisions into H2 2026 Blackwell contracts. If your capacity commitments extend past Q3 2026, push for contractual language that gives you early access to Vera Rubin NVL72 when it becomes available on Google Cloud. Frame this as a capacity continuity provision rather than a hardware upgrade negotiation — it avoids the perception of spec-shopping and focuses on SLA continuity.
Sessions to Watch
1. "Next '26: The Future of AI Infrastructure"
Speakers: Amin Vahdat, Ben Gilbert, David Rosenthal
The anchor session for infrastructure buyers. Vahdat's disclosure of Board Fly topology, the 97% good put target, TPU v8T vs. v8I architectural rationale, and the silent data corruption problem are all must-understand concepts before any large accelerator procurement discussion. Gilbert and Rosenthal provide the best independent analytical frame for the training-to-inference transition thesis. 33-minute runtime, highest signal density of any session at Next '26 for this audience.
2. Google Cloud Next '26 Opening Keynote
Speakers: Thomas Kurian, Sundar Pichai, Amin Vahdat, and others
The canonical source for TPU v8T/v8I specs (121 exaflops FP4 per pod for v8T; 9.8x performance over Ironwood for v8I), the Virgo network announcement (134,000 chips, 47 petabits/second bandwidth), Vera Rubin availability confirmation, and the Citadel Securities customer reference. Also contains the capex disclosure ($175–185B) and the 75% of Google Cloud customers in production AI stat that contextualizes the infrastructure demand trajectory.
3. "Citadel Securities runs workloads up to 4x faster with Ironwood"
The shortest and most economically important session for FinOps evaluation. The Citadel benchmark (2–4x faster, 30% lower cost) is the production reference point for TPU v8 Ironwood economics on non-ML parallel workloads. Its brevity (89 seconds) belies its significance: this is the first major publicly disclosed benchmark from a non-AI company demonstrating TPU economic advantages over traditional HPC infrastructure.
4. "A closer look at Gemma 4 with Baseten and NVIDIA"
Speakers: Jay Rodge (NVIDIA), Philip Kiely (Baseten)
The primary source for Blackwell RTX Pro 6000 and Vera Rubin NVL72 announcements, the MoE 26B/4B activated-parameter economics, Kiely's TCO framework (2x cost, 3x throughput = lower fleet size), GKE multi-region pooling architecture, and the TensorRT-LLM + NVFP4 + Blackwell optimization stack recommendation. Essential for any buyer evaluating GPU vs. TPU or planning NVIDIA hardware procurement.
5. "Acquired's Ben Gilbert and David Rosenthal live from Google Cloud Next"
Speakers: Ben Gilbert, David Rosenthal
The most analytically independent perspective at Next '26. Rosenthal's web-crawl-to-query analogy for the training-to-inference transition is the most precise framing of the structural market shift available from any session. His observation that the 5x TPU v8I speedup will move AI tasks from asynchronous to synchronous UX — with unknown downstream effects — is the most consequential forward-looking claim for product and infrastructure strategy. Not a product announcement session; a strategic interpretation session.
6. "Google Cloud Next '26 Opening Keynote: Building the Agentic Enterprise"
Speaker: Thomas Kurian
The procurement-relevant version of the full keynote for buyers who need the compressed announcement set. Contains the explicit TPU8T/TPU8I specs (two petabytes shared bandwidth memory for training; inference variant optimized for RL and inference with near-zero latency), the Citadel reference, and the Agentic Data Cloud performance claims (BigQuery 35% faster, 40% cost reduction; 230x token reduction in AI inferencing). Best used as the executive briefing document for C-suite infrastructure decision makers.
7. "From Prototype to Production: Building with Gemini Enterprise Agent Platform"
Speakers: Dave Elliott, Addy Osmani
For buyers planning infrastructure to support agentic AI workloads rather than model training, this session establishes the demand context. The 10–20 API calls per single human click estimate is the key load planning figure: if you are modeling agentic AI traffic, your infrastructure must be sized for agent-generated API volume, not human-generated click volume. The gap is an order of magnitude. FinOps teams need this number before pricing inference capacity for agent-first products.
8. "From systems of intelligence to systems of action: Yasmeen Ahmad on the agentic data cloud"
Speaker: Yasmeen Ahmad
Contains the most directly FinOps-relevant performance benchmarks for the data layer: BigQuery 35% speed improvement and 40% cost reduction year-over-year, 230x token reduction in AI inferencing through stack optimization, Apache Spark Lightning Engine 5x faster than vanilla Spark at 2x better price-performance than the previous market leader. These are the cost reduction claims that infrastructure buyers should use as negotiating benchmarks in their data platform and inference cost discussions with Google Cloud sales teams.