Can I read Universal Checkpointing: Efficient and Flexible Checkpointing for Large Scale Distributed Training on EtoBox?

Universal Checkpointing: Efficient and Flexible Checkpointing for Large Scale Distributed Training by Lian, Xinyu; Jacobs, Sam Ade; Kurilenko, Lev; Tanaka, Masahiro; Bekman, Stas; Ruwase, Olatunji; Zhang, Minjia is a scholarly article available to read on EtoBox.

What is Universal Checkpointing: Efficient and Flexible Checkpointing for Large Scale Distributed Training about?

Existing checkpointing approaches seem ill-suited for distributed training even though hardware limitations make model parallelism, i.e., sharding model state across multiple accelerators, a requirement for model scaling. Consolidating distributed model state into a single checkpoint unacceptably slows down training, and is impractical at extreme scales. Distributed checkpoints, in contrast, are tightly coupled to the model parallelism and hardware configurations of the training run, and thus unusable on different configurations. To address this problem, we propose Universal Checkpointing, a technique that enables efficient checkpoint creation while providing the flexibility of resuming on arbitrary parallelism strategy and hardware configurations. Universal Checkpointing unlocks unprecedented capabilities for large-scale training such as improved resilience to hardware failures through continued training on remaining healthy hardware, and reduced training time through opportunistic exploitation of elastic capacity. The key insight of Universal Checkpointing is the selection of the optimal representation in each phase of the checkpointing life cycle: distributed representation for

Author

Lian, Xinyu; Jacobs, Sam Ade; Kurilenko, Lev; Tanaka, Masahiro; Bekman, Stas; Ruwase, Olatunji; Zhang, Minjia

Published

2024

Language

EN