Can I read Distributed Stochastic Gradient Descent with Staleness: A Stochastic Delay Differential Equation Based Framework on EtoBox?

Distributed Stochastic Gradient Descent with Staleness: A Stochastic Delay Differential Equation Based Framework by Yu, Siyuan; Chen, Wei; Poor, H. Vincent is a scholarly article available to read on EtoBox.

What is Distributed Stochastic Gradient Descent with Staleness: A Stochastic Delay Differential Equation Based Framework about?

Distributed stochastic gradient descent (SGD) has attracted considerable recent attention due to its potential for scaling computational resources, reducing training time, and helping protect user privacy in machine learning. However, the staggers and limited bandwidth may induce random computational/communication delays, thereby severely hindering the learning process. Therefore, how to accelerate asynchronous SGD by efficiently scheduling multiple workers is an important issue. In this paper, a unified framework is presented to analyze and optimize the convergence of asynchronous SGD based on stochastic delay differential equations (SDDEs) and the Poisson approximation of aggregated gradient arrivals. In particular, we present the run time and staleness of distributed SGD without a memorylessness assumption on the computation times. Given the learning rate, we reveal the relevant SDDE's damping coefficient and its delay statistics, as functions of the number of activated clients, staleness threshold, the eigenvalues of the Hessian matrix of the objective function, and the overall computational/communication delay. The formulated SDDE allows us to present both the distributed SGD'

Author

Yu, Siyuan; Chen, Wei; Poor, H. Vincent

Published

2024

Language

EN