Google AI Infrastructure Architecture Evolution

2015-2026 | 2026-09-28

01 Scope and executive synthesis

Coverage: TPU and ICI are deep; Axion, offload and datacenter networking are full at the publicly disclosed level; video acceleration is proportional. Period: 2015-28 September 2026. The latest architecture disclosures reviewed include Cloud Next 2026, the 2026 training-TPU retrospective, and the public Hot Chips 2026 program. Pixel Tensor and edge Coral are excluded. Availability refers to internal deployment or cloud service availability, explicitly distinguished below.

Google’s central architectural asset is continuity of the compiler-controlled tensor machine inside a changing system. The progression is from a latency-constrained inference coprocessor, to a distributed training computer, to workload-specific training and serving systems. Axion and Titanium extend that control to the host and infrastructure path. The important comparison with a GPU vendor is therefore the complete path from graph compilation to job placement, memory movement, collective communication and recovery, rather than matrix throughput alone.

02 Reading the specifications

All compute figures are per physical TPU package and dense unless stated otherwise; TFLOPS counts a multiply-add as two operations. Cloud names such as v3-8 count TensorCores, not eight physical chips. GB and GiB retain source units. HBM bandwidth is not bidirectional ICI bandwidth. Full pod footprint, usable allocation and maximum single-job slice can differ. A source’s first deployment date is not automatically public cloud GA. n/d means not established in the reviewed primary documents; n/a means not applicable.

03 Axion: bringing the host into the platform

The first Axion C4A service became generally available in October 2024 and uses Neoverse V2. N4A, generally available on 27 January 2026, uses the N3 core and targets flexible general-purpose VM shapes. These are distinct workload positions, not proof that every N-series core supersedes every C-series core. Published VM limits describe sold resources; they do not establish physical die core counts, memory channels or socket topology.

Architectural implication: a host can bottleneck AI without executing the main matrix workload. Tokenization, input preparation, dispatch and checkpoint movement consume CPU and I/O resources; infrastructure processing competes with those tasks unless offloaded. Titanium removes portions of networking and storage work from the application CPU. The eighth-generation TPU disclosure explicitly includes Axion hosts. This creates a co-design opportunity, but it does not establish coherent CPU-TPU shared physical memory or uniform access latency.

Host CPU comparison

04 TPU v1 and the separate inference branch

TPU v1 entered internal service in 2015. Its integer systolic array, software-managed buffers and deterministic execution addressed inference under strict response-time limits. The ISCA 2017 paper is valuable because it connects the machine to production workload measurements, including utilization and memory constraints. A large arithmetic array cannot compensate for insufficient weight reuse; the 34 GB/s off-chip interface made that limitation concrete. Its DDR-based memory and lack of a training pod distinguish it sharply from later HBM systems.

TPU v4i is a distinct inference design, deployed in 2020 and documented at ISCA 2021. Its 7 nm implementation and enlarged on-chip storage reflect an effort to retain useful working sets while controlling serving power. It should not be folded into the training TPU v4 row. More generally, the i, e and p branches represent different optimization objectives; a later number is insufficient to infer which device is best for a particular latency, capacity or scaling requirement.

05 TPU v2-v4: from chip to training computer

TPU v2 introduced BF16 training, HBM and a two-dimensional ICI torus. TPU v3 expanded matrix resources and memory capacity and moved to liquid cooling. Hot Chips 2020 explains the architectural organization; the later training retrospective connects it to larger systems and resilience. The important implementation consequence is that more compute in the same generation of manufacturing did not remain a chip-only change: power delivery, cooling, network size and the compiler’s partitioning strategy had to evolve together.

TPU v4 combines stronger compute with SparseCore acceleration for embeddings and an optically reconfigurable interconnect. Optical circuit switches change physical connectivity at a coarse timescale; they are not packet switches executing each collective. Reconfiguration helps assemble healthy slices, vary topology and accommodate unavailable components. The 4,096-chip system is therefore also an availability architecture: at this scale, recovering usable capacity matters alongside reducing collective time. The ISCA 2023 paper supplies the explicit product link.

06 v5e, v5p and Trillium: two scaling objectives

TPU v5e is a cost-oriented training and inference platform with a smaller physical domain; v5p targets much larger training systems and more HBM per chip. Trillium (v6e) advances the efficiency branch with more compute and memory bandwidth. The e branch is not merely a partially disabled p chip: its domain size, memory budget and deployment economics should be evaluated on their own. For a model that fits in a small slice, paying for a much larger physical scale-up capability may yield little benefit.

The design question is the bottleneck after partitioning. Dense training can reuse weights enough to benefit from a larger MXU budget, whereas embedding traffic, small batches and decode expose memory and communication. A fair comparison holds model, precision, batch, latency objective and usable chip count constant. Comparing a complete v5p pod with a small Trillium slice answers a procurement-capacity question, not a per-chip architectural efficiency question.

07 Ironwood and the eighth-generation split

Ironwood/TPU7x introduces two compute chiplets with distinct local memory spaces. The package contains two TensorCores, four SparseCores and 192 GiB of HBM according to the cloud specification table. The compiler-visible locality differs from the earlier MegaCore abstraction. A larger package peak does not erase cross-chiplet movement: sharding and kernel placement now have an additional physical boundary to respect. The cloud documentation lists JAX and PyTorch support and explicitly excludes TensorFlow on TPU7x.

The April 2026 TPU 8 disclosure makes workload specialization explicit. TPU 8t retains a large torus for training; TPU 8i uses Boardfly for a smaller-diameter serving domain and adds a collective acceleration engine. The published 8i footprint is 1,152 chips, while the topology description separately states up to 1,024 active chips. These numbers must remain separate. Both products are recorded here as announced; the reviewed architectural announcement does not establish general availability.

Architectural interpretation: the split is evidence that domain size and network diameter have become independent product variables. A training system can tolerate more hops when long collective transfers amortize startup cost. Autoregressive inference repeatedly pays synchronization latency and may prefer fewer hops even with a smaller domain. More SRAM helps only the portion of the working set that actually fits; it does not imply that an arbitrary long-context KV cache resides entirely on chip.

08 Accelerator generation comparison

TPU family

09 ICI, Jupiter, Virgo and infrastructure offload

ICI is the accelerator fabric; Jupiter is the datacenter network. The Jupiter Rising and Jupiter Evolving papers explain the progression from centrally controlled Clos fabrics to optical reconfiguration in the datacenter. These disclosures should not be conflated with the TPU’s local ICI topology. Virgo adds a training-oriented scale-out fabric in the TPU 8 era. A million-chip distributed training claim describes aggregation across systems, not a million-chip coherent memory domain or a single ICI pod.

Titanium is an infrastructure offload system, not a publicly documented merchant DPU SKU with a complete per-chip datasheet. Falcon describes a reliable hardware transport; a transport specification is not itself a Google-branded NIC product. The appropriate report boundary is to map the function and deployment path, retaining n/d for undisclosed silicon implementation. Argos video acceleration belongs alongside this portfolio because it shows the same workload-driven specialization at warehouse scale, but its video engines are not TPU tensor cores.

Networking and offload roles

ICI evolution

10 Software and integrated systems

XLA preserves a relatively stable programming contract while hardware changes. JAX expresses distributed computation; Pallas and Mosaic expose lower-level control for kernels whose data movement dominates. SparseCore and collective engines have value only when the software places and schedules work for them. Therefore portability and peak utilization are separate milestones: executing an unchanged model does not mean that its partitioning, fusion and scratchpad schedule exploit a new generation.

Software evolution

2017-2020: the training pod

flowchart TD
  host["CPU host"]
  gpu["TPU v2 / v3 + HBM"]
  pod["2D ICI torus"]
  sw["TensorFlow / XLA"]
  net["Datacenter network"]
  host ---|"PCIe"| gpu
  gpu ---|"ICI"| pod
  sw ---|"partition"| pod
  gpu ---|"host path"| net

Logical roles, not a physical wiring diagram.

2021-2025: reconfigurable scale

flowchart TD
  host["CPU host + offload"]
  gpu["v4 / v5p / Ironwood"]
  pod["ICI + optical circuits"]
  sw["JAX / XLA / kernels"]
  net["Jupiter / storage"]
  host ---|"PCIe"| gpu
  gpu ---|"scale-up"| pod
  sw ---|"placement"| pod
  host ---|"DCN"| net

OCS changes connectivity; computation remains in the TPU.

2026: separate training and serving systems

flowchart TD
  host["Axion host"]
  gpu["TPU 8t / 8i + HBM"]
  pod["Torus / Boardfly"]
  sw["XLA / Pallas / Pathways"]
  net["Virgo / Jupiter / storage"]
  host ---|"host attach"| gpu
  gpu ---|"ICI"| pod
  sw ---|"schedule"| pod
  gpu ---|"DCN / direct I/O"| net

Announced architecture; public GA is not inferred from this diagram.

System domains

11 Bandwidth balance and architectural trade-offs

Derived ratios below use memory bandwidth divided by dense compute, in bytes/FLOP. Capacity divided by compute has units of byte-seconds/FLOP and is deliberately reported separately in the canonical data. v5p uses 95 GiB of usable cloud memory, while the retrospective describes 96 GiB physical capacity. Ironwood bandwidth is 7,380 GB/s in the current cloud table; the retrospective rounds it differently. These are source-basis differences, not additional hardware generations.

Derived balance metrics

Architectural interpretation: from v2 to Ironwood, BF16 compute grows about 50 times while HBM bandwidth grows about 10.5 times. The required reuse rises accordingly. Larger arrays shift responsibility toward compiler tiling, fusion and local storage. At system scale, optical reconfiguration trades a more complex control plane for improved slice assembly and fault accommodation. At the host boundary, direct I/O reduces copies but makes end-to-end scheduling and access protection more consequential. The practical evaluation should therefore include sustained throughput during communication, fault recovery time and the cost of recompiling or retuning models, not only isolated GEMM.

Capacity and host balance

12 Conference-to-product index

Verified disclosure map

ISCA, ASPLOS, Hot Chips and IEEE Micro provide direct product disclosures in the selected corpus. MICRO and HPCA research should be included only when a documented product connection exists; none is asserted here. An ISSCC/JSSC circuit-paper-to-TPU mapping was not established in this pass, so circuit parameters are drawn from the identified product papers rather than attributed to ISSCC. Cloud Next and OCP are useful for launches and systems; MWC, Computex, GTC and other cloud events do not automatically add independent evidence for Google-owned silicon.

ISSCC context and implementation topics

ISSCC contributes both the domain-specialization rationale and a power-delivery implementation perspective. The 2026 forum entry establishes Google’s participation in that engineering discussion, but does not by itself reveal a particular TPU voltage-regulator circuit. These records complement the ISCA product papers and Hot Chips architecture disclosures rather than replacing them with inferred circuit specifications.

13 Naming crosswalk and remaining disclosure gaps

Names and boundaries

The largest unresolved implementation fields are the physical Axion package organization, exact TDP and per-die process information for recent TPUs, and complete link electrical parameters. These gaps limit package-level power and area comparisons but do not prevent analysis of the disclosed memory and network hierarchy. The next useful evidence would be a circuit-level implementation paper, a complete TPU 8 service specification, and measured software behavior across Ironwood’s chiplet boundary.

Source map

01 Scope and executive synthesis — C01, C03, C05, C09, P08, P10

02 Reading the specifications — C02, P07, P09

03 Axion: bringing the host into the platform — E01, P08, P10, P11

04 TPU v1 and the separate inference branch — C01, C02

05 TPU v2-v4: from chip to training computer — C03, C04, C05

06 v5e, v5p and Trillium: two scaling objectives — P04, P05, P06

07 Ironwood and the eighth-generation split — E03, P07, P08, P13

08 Accelerator generation comparison — C01, C02, C03, C04, C05, E02, E03, P04, P05, P06, P07, P08

09 ICI, Jupiter, Virgo and infrastructure offload — C03, C04, C05, C06, C07, C08, E05, P07, P08, P10, P14

10 Software and integrated systems — C03, C04, C05, P07, P08, P13

11 Bandwidth balance and architectural trade-offs — C03, C04, C05, P05, P07, P08, P10

12 Conference-to-product index — C01, C02, C03, C04, C05, C06, C07, C08, C09, C10, C11, C12, P08

13 Naming crosswalk and remaining disclosure gaps — C02, P06, P07, P08, P10

References

Product and technical documentation

[P04] Cloud TPU v5e. https://docs.cloud.google.com/tpu/docs/v5e. Indexed record or searchable official page; product details cross-checked with the associated technical document.

[P05] Cloud TPU v5p. https://docs.cloud.google.com/tpu/docs/v5p. Indexed record or searchable official page; product details cross-checked with the associated technical document.

[P06] Cloud TPU v6e (Trillium). https://docs.cloud.google.com/tpu/docs/v6e. Indexed record or searchable official page; product details cross-checked with the associated technical document.

[P07] TPU7x (Ironwood). https://docs.cloud.google.com/tpu/docs/tpu7x. Indexed record or searchable official page; product details cross-checked with the associated technical document.

[P08] Inside the eighth-generation TPU: An architecture deep dive. https://cloud.google.com/blog/products/compute/tpu-8t-and-tpu-8i-technical-deep-dive. Indexed record or searchable official page; product details cross-checked with the associated technical document.

[P09] TPU architecture. https://docs.cloud.google.com/tpu/docs/system-architecture-tpu-vm. Indexed record or searchable official page; product details cross-checked with the associated technical document.

[P10] Axion N4A generally available. https://cloud.google.com/blog/products/compute/axion-based-n4a-vms-now-in-preview. Indexed record or searchable official page; product details cross-checked with the associated technical document.

[P11] Arm VMs on Compute. https://docs.cloud.google.com/compute/docs/instances/arm-on-compute. Indexed record or searchable official page; product details cross-checked with the associated technical document.

[P13] Inside the Ironwood codesigned AI stack. https://cloud.google.com/blog/products/compute/inside-the-ironwood-tpu-codesigned-ai-stack. Indexed record or searchable official page; product details cross-checked with the associated technical document.

[P14] Falcon RDMA network profiles. https://docs.cloud.google.com/vpc/docs/rdma-network-profiles. Indexed record or searchable official page; product details cross-checked with the associated technical document.

Conference records and implementation disclosures

[C01] In-Datacenter Performance Analysis of a Tensor Processing Unit, ISCA 2017. https://arxiv.org/pdf/1704.04760. Primary source; accessed by 2026-09-28

[C02] Ten Lessons From Three Generations Shaped Google’s TPUv4i, ISCA 2021. https://markgottscho.com/cv/papers/2021_NJouppi_ISCA.pdf. Primary source; accessed by 2026-09-28

[C03] TPU v4, ISCA 2023. https://arxiv.org/pdf/2304.01433. Primary source; accessed by 2026-09-28

[C04] Google training chips TPUv2 and TPUv3, Hot Chips 2020. https://www.hc32.hotchips.org/assets/program/conference/day2/HotChips2020_ML_Training_Google_Norrie_Patil.v01.pdf. Primary source; accessed by 2026-09-28

[C05] Google’s Training Supercomputers from TPU v2 to Ironwood, IEEE Micro 2026. https://arxiv.org/pdf/2606.15870. Primary source; accessed by 2026-09-28

[C06] Jupiter Rising, SIGCOMM 2015. https://research.google/pubs/jupiter-rising-a-decade-of-clos-topologies-and-centralized-control-in-googles-datacenter-network/. Primary source; accessed by 2026-09-28

[C07] Jupiter Evolving, SIGCOMM 2022. https://research.google/pubs/jupiter-evolving-transforming-googles-datacenter-network-via-optical-circuit-switches-and-software-defined-networking/. Primary source; accessed by 2026-09-28

[C08] Warehouse-Scale Video Acceleration, ASPLOS 2021. https://research.google/pubs/warehouse-scale-video-acceleration-co-design-and-deployment-in-the-wild/. Primary source; accessed by 2026-09-28

[C09] Hot Chips 2026 conference program. https://hc2026.hotchips.org/program/conference/. Official indexed record or related announcement; full document download unavailable.

[C10] ISSCC plenary archive: Patterson2018 / Dean2020. https://www.isscc.org/isscc-plenary-videos. Indexed record or searchable official page; product details cross-checked with the associated technical document.

[C11] Dean ISSCC2020 companion paper. https://arxiv.org/abs/1911.05289. Indexed record or searchable official page; product details cross-checked with the associated technical document.

[C12] ISSCC2026 Forum3: Google power delivery. https://submissions.mirasmart.com/ISSCC2026/PDF/ISSCC2026AdvanceProgram.pdf. Indexed record or searchable official page; product details cross-checked with the associated technical document.

Launches, ecosystem and deployment

[E01] First Axion C4A generally available. https://cloud.google.com/blog/products/compute/first-google-axion-processor-c4a-now-ga-with-titanium-ssd/. Indexed record or searchable official page; product details cross-checked with the associated technical document.

[E02] Ironwood and Axion portfolio, November 2025. https://cloud.google.com/blog/products/compute/ironwood-tpus-and-new-axion-based-vms-for-your-ai-workloads. Indexed record or searchable official page; product details cross-checked with the associated technical document.

[E03] Two chips for the agentic era, Cloud Next 2026. https://blog.google/innovation-and-ai/infrastructure-and-cloud/google-cloud/eighth-generation-tpu-agentic-era/. Primary source; accessed by 2026-09-28

[E05] Data center and global networks built for the AI era. https://cloud.google.com/blog/products/networking/data-center-and-global-networks-built-for-ai-era. Indexed record or searchable official page; product details cross-checked with the associated technical document.