ARCHITECTURE / 2015—2026
AI 基础设施研究文库
← Architecture NotesMarkdown ↓

E06 · Inference & placement · 2026-09-28 · 406 words

When Separating Prefill and Decode Creates More Work

Separated inference phases repeatedly move conversation state.
Separated inference phases repeatedly move conversation state. Conceptual illustration; not a measurement.

Separating prefill and decode is a sensible response to incompatible resource demands. Prefill benefits from substantial parallel arithmetic; decode repeatedly revisits weights and growing state while serving tight token deadlines. Isolation lets each phase choose its own batching, hardware, and scheduling policy. The inherited assumption is more ambitious: that independently improving two phases improves the service connecting them. That conclusion requires the movement of state and the loss of placement flexibility to remain smaller than the interference removed.

Consider an illustrative request for which separation removes 40 ms of interference but exposes 60 ms of state transfer and handoff queueing. With other terms unchanged, the request becomes 20 ms slower. A subsequent conversational turn may enlarge the state that must cross the partition again. This is not evidence against specialized pools. It identifies the resource missing from a phase centered comparison: continuity of a conversation at a particular location over time.

Let I denote colocated interference avoided, H exposed transfer and restoration, ΔQ the change in queueing, and ΔC the change in execution time under specialized placement. Separation improves the request's critical path when I > H + ΔQ + ΔC. Count only exposed communication, and compare equivalent outputs. The inequality does not establish a fleet throughput result: separation can improve capacity while making some requests slower. Both first token and subsequent token constraints must therefore be checked against aggregate admission and the resulting queue distributions.

Placement should operate on sessions as well as phases. A scheduler can retain short returning conversations, separate large fresh prefills, or move execution toward resident state when the destination queue permits. That requires a contract exposing state location, transfer size, restoration cost, pool occupancy, and feasible parallelism. The choice must include the interference exported to other requests by a transfer. A locally attractive handoff can congest a shared link and invalidate several other schedulers' decisions. Reserved transfer capacity or coordinated admission can make the choice predictable.

The decisive experiment replays the same conversation distribution across colocation, fixed separation, and adaptive placement. Preserve arrival timing, model quality, context lengths, and hardware budgets. Trace exposed movement rather than charging theoretical bytes twice, and test bursty follow-up turns instead of only isolated prompts. Separation remains compelling when interference dominates and communication is well provisioned. Its advantage shrinks when state continuity dominates. Does your deployment separate phases because their resource profiles differ, or because measured service capacity still improves after paying to reconnect them?

Evidence check: keep the transfer boundary explicit

Web addendum · 12 November 2026 · Historical evidence from OSDI 2024. The original 406-word essay and dated Markdown download are unchanged.

DistServe, §§4.2–4.3 and 6.1 (printed pp. 199–201), provides a concrete placement mechanism. For slow inter-node links, corresponding prefill and decode pipeline stages use separate GPUs within the same NVLink-connected node. Decode pulls KV state from a prefill-side buffer. Disaggregation therefore need not expose every transfer to the slower inter-node network.

The reported testbed uses four nodes, each with eight A100 80GB GPUs, and 25 Gbps inter-node links. Its evaluated models are FP16 OPT; arrivals use a synthetic Poisson process. Section 4.3 leaves preemption and fault tolerance unimplemented. These boundaries matter before transferring the result to bursty conversational traffic or cross-rack service.

Editorial consequence: compare placement and admission policies, not merely the labels “colocated” and “disaggregated.” Measure the exposed transfer path and recovery behavior. This historical paper motivates the experiment above; it does not validate the essay's illustrative 40/60 ms numbers. Original paper and author record · Source and scope record.