# When Remote Cache Is Slower Than Memory: MI Is Correctness, ESF&OD Is Economics

Cache coherence determines which value a requester may observe. It does not require every valid copy to be an economical supplier. Forwarding from a peer cache was attractive when that path beat memory in latency or conserved scarce memory bandwidth. The inherited shortcut was to treat the existence of a cache copy as sufficient reason to use it. Once packages contain distinct latency domains, validity and source selection become separate architectural decisions with different objectives.

Consider an illustrative requester that reaches home DRAM in 100 ns but a remote owner in 200 ns. A dirty owner remains authoritative: memory cannot supply stale data simply because it is closer. However, retaining dirty ownership across that slow boundary commits later readers to the expensive path. The relevant question is when to make memory authoritative again, including the writeback, ordering, and directory work required to establish that fact safely.

For a serialized comparison, let N readers each incur owner latency L_o under retention. Cleaning after the first remote read costs L_o + W + (N−1)L_m, where W is the additional exposed cleaning cost and L_m is subsequent memory latency. Cleaning wins when (N−1)(L_o−L_m) > W. With W = 150 ns and the illustrative latencies above, three readers cross that boundary. This compares accumulated service cost, not necessarily elapsed application time. Concurrent requests, multicast, overlap, and owner reaccess change the result and must enter the actual model.

The architectural consequence is scoped ownership policy, not weakened coherence. Keep dirty ownership where sharing is cheap; evaluate cleaning when reads cross an expensive domain and the expected reader population justifies it. After cleaning, the home can choose among valid clean suppliers according to topology and congestion. A designated forwarder remains useful only while its service cost warrants that role. Write intensive sharing can favor retention because cleaning may buy little before the next invalidation. Prediction errors should change performance, never which data is permitted to escape.

The experiment therefore needs more than average cache to cache latency. Measure source and destination domains, dirty sharing duration, reader fanout, memory queueing, cleaning completion, and subsequent writer locality. Compare policies at matched memory pressure and include bandwidth consumed by writebacks. If memory is saturated, a slower peer may still protect aggregate throughput. MSI names the core correctness states; extra states and forwarding roles encode implementation choices around them. Does your directory optimize the cheapest legal supplier, or preserve an ownership decision whose original latency assumption no longer holds?