When Separating Prefill and Decode Creates More Work
Separating prefill and decode is a sensible response to incompatible resource demands. Prefill benefits from substantial parallel arithmetic; decode repeatedly revisits weights and growing state while serving tight token deadlines. Isolation lets each phase choose its own batching, hardware, and scheduling policy. The inherited assumption is more ambitious: that independently improving two phases improves the service connecting them. That conclusion requires the movement of state and the loss of placement flexibility to remain smaller than the interference removed.
Consider an illustrative request for which separation removes 40 ms of interference but exposes 60 ms of state transfer and handoff queueing. With other terms unchanged, the request becomes 20 ms slower. A subsequent conversational turn may enlarge the state that must cross the partition again. This is not evidence against specialized pools. It identifies the resource missing from a phase centered comparison: continuity of a conversation at a particular location over time.
Let I denote colocated interference avoided, H exposed transfer and restoration, ΔQ the change in queueing, and ΔC the change in execution time under specialized placement. Separation improves the request's critical path when I > H + ΔQ + ΔC. Count only exposed communication, and compare equivalent outputs. The inequality does not establish a fleet throughput result: separation can improve capacity while making some requests slower. Both first token and subsequent token constraints must therefore be checked against aggregate admission and the resulting queue distributions.
Placement should operate on sessions as well as phases. A scheduler can retain short returning conversations, separate large fresh prefills, or move execution toward resident state when the destination queue permits. That requires a contract exposing state location, transfer size, restoration cost, pool occupancy, and feasible parallelism. The choice must include the interference exported to other requests by a transfer. A locally attractive handoff can congest a shared link and invalidate several other schedulers' decisions. Reserved transfer capacity or coordinated admission can make the choice predictable.
The decisive experiment replays the same conversation distribution across colocation, fixed separation, and adaptive placement. Preserve arrival timing, model quality, context lengths, and hardware budgets. Trace exposed movement rather than charging theoretical bytes twice, and test bursty follow-up turns instead of only isolated prompts. Separation remains compelling when interference dominates and communication is well provisioned. Its advantage shrinks when state continuity dominates. Does your deployment separate phases because their resource profiles differ, or because measured service capacity still improves after paying to reconnect them?