Can I read Layer-wise Adaptive Gradient Sparsification for Distributed Deep Learning with Convergence Guarantees on EtoBox?

Layer-wise Adaptive Gradient Sparsification for Distributed Deep Learning with Convergence Guarantees by Shi, Shaohuai; Tang, Zhenheng; Wang, Qiang; Zhao, Kaiyong; Chu, Xiaowen is a scholarly article available to read on EtoBox.

What is Layer-wise Adaptive Gradient Sparsification for Distributed Deep Learning with Convergence Guarantees about?

To reduce the long training time of large deep neural network (DNN) models, distributed synchronous stochastic gradient descent (S-SGD) is commonly used on a cluster of workers. However, the speedup brought by multiple workers is limited by the communication overhead. Two approaches, namely pipelining and gradient sparsification, have been separately proposed to alleviate the impact of communication overheads. Yet, the gradient sparsification methods can only initiate the communication after the backpropagation, and hence miss the pipelining opportunity. In this paper, we propose a new distributed optimization method named LAGS-SGD, which combines S-SGD with a novel layer-wise adaptive gradient sparsification (LAGS) scheme. In LAGS-SGD, every worker selects a small set of "significant" gradients from each layer independently whose size can be adaptive to the communication-to-computation ratio of that layer. The layer-wise nature of LAGS-SGD opens the opportunity of overlapping communications with computations, while the adaptive nature of LAGS-SGD makes it flexible to control the communication time. We prove that LAGS-SGD has convergence guarantees and it has the same order of conv

Author

Shi, Shaohuai; Tang, Zhenheng; Wang, Qiang; Zhao, Kaiyong; Chu, Xiaowen

Published

2019

Language

EN