Can I read rTop- k : A Statistical Estimation Approach to Distributed SGD on EtoBox?

rTop- k : A Statistical Estimation Approach to Distributed SGD by Barnes, Leighton Pate ;Inan, Huseyin A. ;Isik, Berivan ;Ozgur, Ayfer is a Computer Science article available to read on EtoBox.

What is rTop- k : A Statistical Estimation Approach to Distributed SGD about?

The large communication cost for exchanging gradients between different nodes significantly limits the scalability of distributed training for large-scale learning models. Motivated by this observation, there has been significant recent interest in techniques that reduce the communication cost of distributed Stochastic Gradient Descent (SGD), with gradient sparsification techniques such as top-k and random-k shown to be particularly effective. The same observation has also motivated a separate line of work in distributed statistical estimation theory focusing on the impact of communication constraints on the estimation efficiency of different statistical models. The primary goal of this paper is to connect these two research lines and demonstrate how statistical estimation models and their analysis can lead to new insights in the design of communication-efficient training techniques. We propose a simple statistical estimation model for the stochastic gradients which captures the sparsity and skewness of their distribution. The statistically optimal communication scheme arising from the analysis of this model leads to a new sparsification technique for SGD, which concatenates random

Who reads rTop- k : A Statistical Estimation Approach to Distributed SGD?

Author

Barnes, Leighton Pate ;Inan, Huseyin A. ;Isik, Berivan ;Ozgur, Ayfer

Publisher

Institute of Electrical and Electronics Engineers (IEEE)

Published

2020

Language

EN

Field

Computer Science (Physical Sciences)