A Checkpoint Is Only as Useful as Its Restore Path
Checkpointing is often optimized as a periodic write problem: reduce pause time, overlap serialization, and move bytes away from expensive compute. That focus makes sense when failures are rare and restoration is a predictable inverse of saving. The inherited shortcut is to equate a successfully written checkpoint with a bounded recovery time. The recovery path may need different bandwidth, additional memory, replacement workers, metadata reconstruction, and a topology no longer available after the failure that made the checkpoint necessary.
Imagine an illustrative 4 TB checkpoint whose effective restore path sustains 100 GB/s. Reading alone takes at least forty seconds in decimal units. If eight recovery jobs share that same aggregate path equally, the floor becomes 320 seconds per job before rebuilding execution state or obtaining capacity. A fast asynchronous save says nothing about this contention. Worse, the copies easiest to write may sit in the same failure domain, while the durable remote copy requires a slower path precisely during a widespread incident.
A simple periodic model makes the missing term visible. With checkpoint interval T, exposed save cost c, failure rate λ, and restore time r, approximate lost time per unit time is c/T + λ(T/2 + r). This assumes independent rare failures, uniformly distributed lost progress, and no overlapping incidents. If r is constant, reducing it improves availability without changing the model's optimal T. If checkpoint placement or frequency changes r through contention, the decisions become coupled. Neither case permits restore time to disappear from the architecture merely because save overhead is small.
Specify a recovery contract before choosing checkpoint placement. Define which failures a copy survives, how freshness is established, what replacement topology is acceptable, and which resources restoration may reserve. Separate durable completion from background transfer still in flight. Include model state, optimizer state where needed, random state, and coordination metadata according to the application's consistency requirements. Prioritize restores against background saves deliberately: the same storage network cannot promise unrestricted bandwidth to both. Recovery admission may need to stagger jobs to preserve useful progress rather than letting every job saturate the path simultaneously.
Test restoration from the surviving copy into replacement capacity under realistic concurrent recovery, not only by loading a local file into idle machines. Measure time to resumed useful work, including validation and warmup, and verify that the restored computation satisfies its consistency contract. A checkpoint is valuable because it limits lost work and restores service, not because it produced a high write bandwidth figure. Does your reliability budget price the entire route back to productive execution, or stop accounting when the checkpoint write returns successfully?