# Speculative Decoding Can Spend More Than It Saves

Speculative decoding trades additional computation for fewer sequential target model steps. A draft proposes tokens, and verification determines which can be accepted under the algorithm's output contract. The approach is attractive when target execution has unused parallel capacity and accepted tokens amortize an expensive sequential operation. The inherited shortcut is to promote an isolated request's latency improvement into a fleet efficiency claim. Drafting and verification consume resources that may already be the limiting factor under production concurrency.

Consider illustrative measurements: a baseline completes one accepted token using 10 units of constrained accelerator time. A speculative round consumes eight units of draft work and twelve of verification while accepting an average of three tokens. Its constrained work is 6.7 units per token. If acceptance falls to one token, that figure becomes twenty. The crossover is straightforward, but only if both work terms describe the same constrained resource. Parallel wall time cannot be added as though it were accelerator service demand.

Let d and v be draft and verification demand on a shared bottleneck, a expected accepted tokens per round, and b baseline demand per accepted token. Speculation reduces that demand when (d+v)/a < b. Other resources require their own constraints: draft memory, target memory traffic, network transfers, and scheduling slots can change the winning configuration. Acceptance depends on the request distribution and draft policy, not merely model names. Output quality and sampling semantics must remain equivalent wherever exactness is claimed; otherwise the comparison is purchasing a different service.

Admission should therefore select speculation depth and draft placement using marginal accepted tokens per constrained resource. Lightly loaded execution may rationally spend spare work to shorten latency. Near saturation, the same policy may need to reduce depth or disable speculation to protect capacity. This requires coordination across requests. Every stream cannot independently assume that idle target cycles will remain available after all streams begin drafting. A controller also needs hysteresis, because noisy acceptance estimates can create oscillation between modes and consume the savings through repeated rescheduling.

The evaluation must sweep concurrency and prompt classes while preserving the quality contract. Record accepted tokens, rejected draft work, verification demand, cache occupancy, queueing, and deadline compliance. Report latency and capacity separately, then connect them through the actual service objective. A technique that improves interactive latency can remain valuable even when it spends more compute; the mistake is calling both improvements the same result. Does your speculative engine use spare capacity to buy responsiveness, or continue spending that capacity after other requests have already claimed it?