About this document
Sparse Allreduce for Deep Learning by dabazeng520 is a document available to read on EtoBox.
The document presents O𝑘-Top𝑘, a novel sparse allreduce algorithm designed to reduce communication overhead in distributed deep learning by achieving less than 6𝑘 communication volume, which is asymptotically optimal. It integrates with a decentralized parallel Stochastic Gradient Descent optimizer and demonstrates significant improvements in training throughput and model accuracy compared to existing methods. Empirical evaluations on the Piz Daint supercomputer show O𝑘-Top𝑘
- Author
- dabazeng520
- Language
- EN