Hot Chips 36 / 2024 / Recording Q&A
Memory topology, runtime and cooling
Memory topology, runtime support and cooling qualification belong in accelerator purchase criteria.
Updated 2 October 2026 · Analyst interpretation of historical disclosures
Where audience questions concentrated
161 eligible audience asks in the reviewed caption material. CPU sessions led by question volume; power and cooling led among themes. Memory and software questions recurred across CPU, accelerator and interconnect talks. Sparse-compute questioning was concentrated in the accelerator session.
Original Executive Brief figure. Darker cells indicate more audience questions in reviewed session blocks; both axes run from hottest to coolest. Moderator prompts are excluded. Q&A time is not normalized.
Counts, coverage and session key
The figure is reproduced from the existing Executive Brief, without rebuilding its session matrix. Exact annual counts for the twelve common cross-year themes are available in the ten-year data; they do not cover the full annual taxonomy.
Conference coverage is partial. Moderator prompts are excluded from audience-question rankings. Caption errors and uneven session coverage limit comparisons; these counts describe reviewed evidence, not a complete conference census.
Based on the existing executive brief and its review of YouTube automatic captions. Conference coverage is partial; moderator prompts are excluded from audience-question rankings. Caption errors and uneven session coverage limit comparisons. Rankings describe the reviewed material, not the full audience or industry.
Two-page Executive Brief · Exact ten-year theme counts (CSV)
Patterns in the reviewed Q&A
Memory placement and coherence drew sustained probing.
AMD confirmed directed probes but left protocol details unresolved. Questions also tested HBM access efficiency, PIM interfaces and memory overhead.
Runtime flexibility needed practical detail.
Questions examined partitioning, debugging, context switching and concurrent models. Some configuration features were explained while their overhead remained unquantified.
Sparse gains needed full cost accounting.
Onyx questions exposed conversion and metadata costs. Meta separated evaluated or analytical benefits from deployed improvements.
Cooling was examined at system level.
Questions covered rack limits, maintenance, capex and prototype evidence. Chip heat removal alone did not settle the facility decision.
Optical integration still needed qualification.
Intel called OCI a technology demonstration. Laser history from pluggable modules did not establish reliability in the proposed CPU package.
Implications and concrete vendor asks
Analyst inference: these present-day acceptance tests follow from historical uncertainties. Historical statements and roadmap promises do not establish later delivery.
| Decision and timing | Vendor ask and decision test |
|---|---|
| Head nodes Before shortlist | AMD and Intel: expose Venice and DMR cross-die access paths, coherence behaviour and loaded latency. Test CPU work alongside accelerator traffic. Prefer the topology with better delivered host service and memory access under contention. |
| MI400 Before prototype | AMD: specify supported partitions, runtime reconfiguration, isolation and context-switch cost. Demonstrate concurrent models and compare sparse/quantized gains after conversion, metadata and quality costs. Require results from the intended software release. |
| KV-cache offload Before architecture lock | Memory and accelerator suppliers: show dynamic KV allocation, eviction, synchronization and placement across memory tiers. Test long contexts and shared prefixes. Change the design only when capacity savings survive transfer and scheduling overhead. |
| Scale-up and scale-out Before design freeze | Optics and network suppliers: provide loaded latency, error recovery, thermal-transient qualification and field-replaceable boundaries. Separate technology demonstrations from qualified product results before choosing package-integrated optics. |
| Cooling Before facility commitment | Server and cooling suppliers: price the full retrofit, residual air cooling and maintenance burden. Measure sustained workload energy and rack flow limits. Accept a design on facility economics and availability, not a chip-cooling result alone. |
Supporting architectural interpretation
Recurring themes within the reviewed questions and answers. The implications below are architectural interpretation, not a ranking of industry demand.
1. Resource flexibility exposes control costs
Memory placement, directed probes, partitioning, context switching and concurrent models drew questions across the reviewed sessions.
Architectural implication: A configurable resource is useful only when ownership, isolation and switching overhead fit the workload.
2. Efficiency gains need their overheads restored
Sparse-compute questions exposed conversion and metadata costs. Cooling questions expanded the boundary to racks, facilities and maintenance.
Architectural implication: Test the net gain after representation changes and infrastructure costs, rather than carrying a component ratio into a system business case.
3. Demonstration and deployment remain different milestones
Optical I/O was described as a technology demonstration; historical laser reliability did not establish reliability in the proposed package. Cooling also required prototype and operational evidence.
Architectural implication: Qualify the proposed configuration itself, including thermal conditions, repair and software release.
These are present-day evaluation recommendations drawn from the historical evidence, rather than claims that later products were discussed in this conference year.
Memory placement and ownership
Request cross-die paths, coherence behaviour and loaded latency. Test dynamic allocation, eviction and synchronization across memory tiers, including shared prefixes.
Runtime and useful acceleration
Measure partitioning, reconfiguration, isolation and context-switch cost. Compare sparse and quantized gains after conversion, metadata and quality costs.
Facility and optical qualification
Distinguish a technology demonstration from qualified product evidence. Price retrofit work, residual air cooling and maintenance; measure sustained energy and rack flow limits.
Sources and scope
Based on the existing executive brief and its review of YouTube automatic captions. Conference coverage is partial; moderator prompts are excluded from audience-question rankings. Caption errors and uneven session coverage limit comparisons. Rankings describe the reviewed material, not the full audience or industry.
Research source: Hot Chips 2024 Executive Brief. Public edition omits raw question logs, private correspondence, personal attribution and attendee-only links.
Official Hot Chips 2024 site ↗ · Ten-year comparison →
Annual heatmap reproduced from Hot Chips 2024 Executive Brief. The format update preserves its source population and does not add new transcript or recording verification.