AWS AI Infrastructure Architecture Evolution

2015-2026 | 2026-09-28

01 Scope and executive synthesis

Deep coverage: Graviton CPUs and Inferentia/Trainium accelerators. Full coverage: NeuronLink, UltraServer, Nitro, EFA and the Neuron software model. Period: 2015-28 September 2026; the CPU product sequence starts in 2018. FPGA instances and third-party GPUs are complementary context, not AWS-designed silicon. Reviewed milestones include re:Invent 2024/2025 materials, Graviton5 general availability in June 2026, and current Neuron architecture documentation.

AWS’s architectural strategy is to make specialization deployable through a stable cloud service boundary. Graviton supplies general compute; Trainium and Inferentia supply model compute; Nitro supplies isolation and infrastructure offload; EFA supplies distributed communication. The economic unit is an instance or a tightly connected system with a supported software stack. The principal engineering question is how much specialized silicon can be exposed without forcing customers to operate a bespoke machine.

02 Numerical and availability conventions

CPU counts are per package, not the largest multi-socket EC2 instance. A Graviton vCPU represents a physical core rather than an SMT thread. Accelerator figures are per physical device and dense; cFP8, FP8 and MXFP8 are not interchangeable formats. HBM GiB and GB are retained as published. NeuronLink totals whose direction is unspecified remain quoted totals rather than invented one-way bandwidth. Service GA, preview and future roadmap are separate statuses. n/d means not established; n/a means not applicable.

03 Graviton1 and Graviton2: establishing the platform

The first Graviton A1 instances established an Arm64 deployment path in 2018. Graviton2 made that path a broad server platform through 64 Neoverse N1 cores, private L2 caches and an eight-channel memory interface. The move was more than an ISA substitution: software builds, native libraries, runtime optimization and instance sizing had to become routine. A single-thread-per-core model also changes the meaning of a vCPU when comparing against an SMT-based x86 instance.

Architectural interpretation: the migration succeeds when an application’s dependency graph is portable and its working set fits the memory and cache envelope. The useful experiments are fleet-representative throughput and tail latency at a fixed service objective. Compiler-generated Arm code is necessary but insufficient: synchronization primitives, cryptography, compression and managed runtimes can determine whether silicon resources translate into service capacity.

04 Graviton3-5: vector growth and a new chiplet boundary

Graviton3 kept 64 cores while moving to Neoverse V1 and DDR5; Graviton3E is a related HPC-oriented variant rather than a separate ISA generation. Graviton4 increased the package to 96 Neoverse V2 cores and twelve DDR5 channels, with two-socket configurations exposed in selected large instances. Wider aggregate vector capability should not be confused with a fixed architectural SVE vector length: the V1 and V2 implementations organize their datapaths differently.

Graviton5 changes the partitioning: four chiplets each integrate 48 cores, memory controllers and PCIe controllers. The package has 192 cores and 192 MB L3, supports DDR5-8800 and PCIe 6, and uses Neoverse V3. The previous separate I/O and memory-controller dies disappear from this organization. The vendor quotes up to 420 GB/s between chiplets without fully specifying the directional basis; it is retained as a quoted link figure. NUMA and cache partitioning must be interpreted at the selected VM size, not inferred from the package total.

Architectural implication: placing memory controllers beside compute can reduce traffic that crosses a dedicated I/O boundary, but distributed controllers make locality and allocation policy more important. Doubling core count with twelve channels also means that per-core peak DRAM bandwidth need not rise with package bandwidth. Larger L3 can reduce traffic, but it cannot remove the streaming bandwidth limit. These two effects should be measured separately before attributing an application gain to core IPC.

Graviton generations

05 Inferentia and Trainium1: the NeuronCore foundation

Inferentia1 combines four NeuronCore-v1 units with external DRAM. It targets inference and benefits from pipelining work through cores. Inferentia2 and Trainium1 use NeuronCore-v2, HBM, separate tensor/vector/scalar engines and programmable GPSIMD resources. This shared core lineage does not make the cloud products identical: Inf2 and Trn1 expose different chip counts, networking and intended workload envelopes. The move to HBM makes larger models practical while increasing the importance of explicit data movement.

Software-managed SBUF and accumulation storage are architectural resources, not transparent caches. A kernel must arrange tensors to fit the array’s contraction dimension and the scratchpad’s partitioning. DMA overlap is therefore part of the algorithm. This is a useful distinction from a superficial “GPU alternative” description: the compiler and kernel author carry substantial responsibility for locality, even when a framework hides that responsibility from the model author.

06 Trainium2, Trainium3 and the Trainium4 roadmap

Trainium2 expands to eight NeuronCore-v3 units, larger on-chip storage and a 96 GiB HBM system. Logical NeuronCore configuration can combine physical resources behind a larger logical core. Dedicated collective-control cores support communication alongside computation. Trn2 UltraServers extend the directly connected resource pool beyond one conventional server, so both the physical core count and the logical execution mode matter when interpreting utilization.

Trainium3 retains eight physical cores but changes the low-precision datapath. The NKI guide explains quad-packed microscaling inputs on a physical 128×128 array, presenting a larger logical contraction dimension. The overview gives 2,517 TFLOPS for MXFP8/MXFP4 and 671 TFLOPS for dense BF16. Thus the headline low-precision improvement is not a universal BF16 improvement. Trn3 UltraServers reach 144 chips and use NeuronSwitch-v1; their aggregate improvement also includes a larger chip count.

Trainium4 remains a roadmap item in the reviewed disclosure. AWS states intended NVLink Fusion support alongside Graviton and EFA in MGX-based infrastructure. This is a future integration direction, not a reason to relabel existing NeuronLink systems as NVLink systems. Qualification must ultimately cover memory semantics, collective software, fault handling and topology as well as physical link compatibility.

07 Accelerator comparison

Inferentia and Trainium

Source reconciliation: the current Trainium3 overview reports 4.9 TB/s HBM and 16 collective cores; the NKI guide gives 4.7 TB/s and 20 CC-Cores in its introductory list. The overview is used consistently for package comparison, while the guide supports datapath discussion. Likewise, Trainium1 is rounded to 190 TFLOPS in its own guide and 191 in a later comparison. These discrepancies are retained in the evidence record rather than silently combined.

08 Nitro and EFA: infrastructure versus AI traffic

Nitro separates much of the networking, storage and management work from customer CPU execution. Its security chip, cards and minimal hypervisor form a system, not a single AI NIC. Successive Nitro versions appear across EC2 CPU and accelerator families; their network limits depend on the instance. Nitro v6 is publicly associated with the 2025 generation of network-optimized instances. The 2026 Nitro Isolation Engine adds a formally verified isolation component, which is a specific security property rather than proof that every part of the system is formally verified.

EFA presents an application communication interface backed by AWS’s scalable reliable datagram transport. Multipath delivery and fast recovery address the variability of a shared datacenter network. The accelerator collective library, provider interface, EFA device and underlying fabric are separate layers. Their combined behavior determines job performance; a port line rate alone says little about small-message latency, tail behavior or communication-compute overlap. ENA front-end traffic and EFA collective traffic should also remain distinct in the system map.

Offload and network families

09 Systems, software and bandwidth hierarchy

Neuron provides graph compilation, runtime, libraries and framework integration. NKI exposes the hardware memory hierarchy to specialist kernel authors; logical-core configuration changes how physical resources are presented. A production migration should track supported operators, compilation cost, numerical behavior, profiling and collective overlap together. AWS FPGA instances remain a programmable acceleration option, while GPUs remain part of the same cloud portfolio; neither is evidence that AWS designed the underlying third-party silicon.

System progression

Interconnect lineage

2019-2022 integrated stack

flowchart TD
  host["EC2 CPU host"]
  gpu["Inferentia1 / Trainium1"]
  domain["NeuronLink / instance"]
  soft["Neuron / frameworks"]
  net["Nitro + EFA"]
  host ---|"host attach"| gpu
  gpu ---|"scale-up"| domain
  soft ---|"compile / map"| domain
  gpu ---|"scale-out"| net

Logical data and control paths. Graviton is not assumed to host every historical accelerator instance.

2024-2026 integrated stack

flowchart TD
  host["EC2 CPU host"]
  gpu["Trainium2 / Trainium3"]
  domain["UltraServer / NeuronSwitch"]
  soft["Neuron / NKI / LNC"]
  net["Nitro + EFA / UltraCluster"]
  host ---|"host attach"| gpu
  gpu ---|"scale-up"| domain
  soft ---|"compile / map"| domain
  gpu ---|"scale-out"| net

Logical data and control paths. Graviton is not assumed to host every historical accelerator instance.

10 Derived balance and engineering consequences

Per-device derived ratios

Derived CPU balance: Graviton4 provides 537.6/96 = 5.6 GB/s per core; Graviton5 provides 844.8/192 = 4.4 GB/s per core, a 21.4% reduction in this theoretical streaming metric. Package L3 per core rises from 0.375 MB to 1 MB, before VM partitioning. The architectural bet is that improved cores, more cache and locality compensate for less bandwidth per core on enough real workloads. A streaming workload and a large instruction-footprint service can therefore move in opposite directions.

For accelerators, Trainium3 improves memory bandwidth per BF16 FLOP substantially because BF16 compute remains nearly flat while memory bandwidth rises. Against its faster MX datapath, the bandwidth balance is tighter. This is why a procurement model should carry separate BF16 and microscaling rooflines, and why a token-throughput claim needs batch size and latency limits. System-level gains also depend on whether the model can exploit the enlarged scale-up domain without adding excessive communication or idle capacity.

11 Conference map, software timeline and naming

Verified disclosure channels

The reviewed corpus supports stronger implementation coverage through AWS architecture guides than through a verified ISCA/MICRO/HPCA/ASPLOS or ISSCC product-paper sequence. No such sequence is invented. Hot Chips was searched, but no specific AWS generation talk is used as an implementation source in this edition. This is a disclosure-channel difference from Google, not evidence of less architectural sophistication. re:Invent materials establish deployment and system context; Neuron and Graviton guides supply the detailed programming-visible architecture.

Hardware and software alignment

Product crosswalk

Open implementation fields include package power, precise per-die process and packaging information for several accelerator generations, and fully directional NeuronLink electrical specifications. The system tables keep rack power and per-accelerator EFA bandwidth n/d when the selected configuration is not fixed. The next decision-relevant evidence is a matched workload comparison with compilation and communication costs included, rather than an extrapolation from chip peak to fleet efficiency.

Source map

01 Scope and executive synthesis — C01, E01, E02, E03, P01, P02, P03, P04, P07, P08

02 Numerical and availability conventions — P01, P02, P03, P04, P06

03 Graviton1 and Graviton2: establishing the platform — C01, P01

04 Graviton3-5: vector growth and a new chiplet boundary — E01, E03, E05, E06, E07, P01

05 Inferentia and Trainium1: the NeuronCore foundation — P02, P05, P06, P09

06 Trainium2, Trainium3 and the Trainium4 roadmap — C01, E02, P03, P04, P09

07 Accelerator comparison — E02, E04, E08, E09, P02, P03, P04, P05, P06, P09

08 Nitro and EFA: infrastructure versus AI traffic — E01, E02, P07, P08, P10, P11

09 Systems, software and bandwidth hierarchy — C01, E02, P02, P03, P04, P09

10 Derived balance and engineering consequences — E01, E02, E05, P01, P02, P03, P04, P09

11 Conference map, software timeline and naming — C01, C02, E01, E02, E03, E04, P01, P02, P03, P04, P05, P06, P08, P09

References

Product and technical documentation

[P01] AWS Graviton technical guide. https://raw.githubusercontent.com/aws/aws-graviton-getting-started/main/README.md. Primary source; accessed by 2026-09-28

[P02] Trainium architecture. https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/arch/neuron-hardware/trainium.html. Primary source; accessed by 2026-09-28

[P03] Trainium2 architecture. https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/arch/neuron-hardware/trainium2.html. Primary source; accessed by 2026-09-28

[P04] Trainium3 architecture. https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/arch/neuron-hardware/trainium3.html. Primary source; accessed by 2026-09-28

[P05] Inferentia architecture. https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/arch/neuron-hardware/inferentia.html. Primary source; accessed by 2026-09-28

[P06] Inferentia2 architecture. https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/arch/neuron-hardware/inferentia2.html. Primary source; accessed by 2026-09-28

[P07] Security design of the AWS Nitro System. https://docs.aws.amazon.com/whitepapers/latest/security-design-of-aws-nitro-system/security-design-of-aws-nitro-system.html. Indexed record or searchable official page; product details cross-checked with the associated technical document.

[P08] Elastic Fabric Adapter. https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/efa.html. Indexed record or searchable official page; product details cross-checked with the associated technical document.

[P09] Trainium3 NKI architecture guide. https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/guides/architecture/trainium3_arch.html. Primary source; accessed by 2026-09-28

[P10] EC2 Nitro versions. https://docs.aws.amazon.com/ec2/latest/instancetypes/ec2-nitro-instances.html. Indexed record or searchable official page; product details cross-checked with the associated technical document.

[P11] SRD transport. https://docs.aws.amazon.com/wellarchitected/2023-10-03/framework/perf_networking_choose_network_protocols_improve_performance.html. Indexed record or searchable official page; product details cross-checked with the associated technical document.

Conference records and implementation disclosures

[C01] AWS accelerated computing, re:Invent 2024 CMP207. https://d1.awsstatic.com/onedam/marketing-channels/website/aws/en_US/events/approved/reinvent-2025/reinvent/2024/slides/cmp/CMP207_AWS-accelerated-computing-enables-customer-success-with-generative-AI.pdf. Primary source; accessed by 2026-09-28

[C02] AWS Summit2022 silicon innovation. https://d1.awsstatic.com/events/Summits/aws-summits/The_journey_of_silicon_innovation_at_AWS_CMP201.pdf. Primary source; accessed by 2026-09-28

Launches, ecosystem and deployment

[E01] Graviton5 chiplet architecture. https://www.amazon.science/blog/graviton5s-improved-design-increases-speed-and-energy-efficiency-beyond-moores-law. Primary source; accessed by 2026-09-28

[E02] Trainium3 UltraServers and Trainium4 outlook. https://www.aboutamazon.com/news/aws/trainium-3-ultraserver-faster-ai-training-lower-cost. Primary source; accessed by 2026-09-28

[E03] Graviton5 M9g general availability. https://aws.amazon.com/blogs/aws/now-available-amazon-ec2-m9g-and-m9gd-instances-powered-by-new-aws-graviton5-processors/. Indexed record or searchable official page; product details cross-checked with the associated technical document.

[E04] Trainium3 general availability. https://aws.amazon.com/about-aws/whats-new/2025/12/amazon-ec2-trn3-ultraservers/. Indexed record or searchable official page; product details cross-checked with the associated technical document.

[E05] Graviton4 memory architecture. https://aws.amazon.com/blogs/aws/join-the-preview-for-new-memory-optimized-aws-graviton4-powered-amazon-ec2-instances-r8g/. Indexed record or searchable official page; product details cross-checked with the associated technical document.

[E06] Graviton3 C7g GA. https://aws.amazon.com/blogs/aws/new-amazon-ec2-c7g-instances-powered-by-aws-graviton3-processors/. Indexed record or searchable official page; product details cross-checked with the associated technical document.

[E07] Graviton A1 launch. https://aws.amazon.com/blogs/aws/new-ec2-instances-a1-powered-by-arm-based-aws-graviton-processors/. Indexed record or searchable official page; product details cross-checked with the associated technical document.

[E08] Inferentia Inf1 launch. https://aws.amazon.com/blogs/aws/amazon-ec2-inf1-instances-with-aws-inferentia-chips-for-high-performance-cost-effective-machine-learning-inference/. Indexed record or searchable official page; product details cross-checked with the associated technical document.

[E09] Inf2 instances. https://aws.amazon.com/ec2/instance-types/inf2/. Indexed record or searchable official page; product details cross-checked with the associated technical document.