# Sparsity Saves FLOPs and Buys a Barrier

Sparsity promises to avoid operations whose inputs or weights contribute nothing useful. The arithmetic saving is real when the execution machinery can identify omitted work cheaply and redistribute the remaining work efficiently. Dense execution, however, buys regular scheduling, predictable memory access, and balanced collective progress. The inherited shortcut compares retained operations with original operations and treats that fraction as a runtime forecast. In distributed execution, the slowest participant and the movement required to reach it can dominate the saved arithmetic.

Consider an illustrative eight worker layer with 8 ms of dense compute per worker. Sparse routing reduces average useful arithmetic to the equivalent of 4 ms, but the busiest worker still needs 7 ms and exposed routing plus synchronization adds 2 ms. The layer now takes 9 ms before other unchanged costs. Half the average arithmetic has disappeared while elapsed time has increased. This is a mechanism example, not a benchmark prediction; a balanced partition with inexpensive communication could produce the opposite result.

Write the synchronous stage time as max_i(C_i + M_i) + S, where C_i is exposed compute, M_i exposed data movement on worker i, and S remaining synchronization overhead. Avoid charging communication again when already included in M_i. Sparsity wins only when the new critical path beats the dense stage, including dispatch, metadata, packing, and any extra barriers. Average nonzero density cannot establish that condition. Nor should numerical sparsity, structured hardware support, and routed expert sparsity be merged: they remove work through different mechanisms and create different scheduling constraints.

The architecture needs a balance contract alongside its sparse arithmetic contract. Expose block granularity, supported patterns, dispatch cost, and the conditions under which work can move between execution units. At system scale, admission and routing should consider queue depth, memory residency, and collective timing together. Replication can reduce hot destinations but consumes capacity and introduces placement decisions. Padding can recover regularity while surrendering part of the FLOP saving. Neither is a failure; both are explicit purchases of predictability that belong in the comparison against dense execution.

Evaluate the distribution of worker completion times, not only achieved sparse throughput. Trace useful arithmetic, padded work, metadata traffic, dispatch queues, and waiting at every synchronization point under realistic input skew. Preserve accuracy and workload distribution across comparisons, since pruning or routing changes may alter the task itself. A sparse engine is valuable when eliminated operations shorten the service's constrained path or free a resource that can be productively reused. Does your sparsity metric count arithmetic removed, or the barrier time that removal actually releases?