Can I read Tight Time Complexities in Parallel Stochastic Optimization with Arbitrary Computation Dynamics on EtoBox?

Tight Time Complexities in Parallel Stochastic Optimization with Arbitrary Computation Dynamics by Tyurin, Alexander is a scholarly article available to read on EtoBox.

What is Tight Time Complexities in Parallel Stochastic Optimization with Arbitrary Computation Dynamics about?

In distributed stochastic optimization, where parallel and asynchronous methods are employed, we establish optimal time complexities under virtually any computation behavior of workers/devices/CPUs/GPUs, capturing potential disconnections due to hardware and network delays, time-varying computation powers, and any possible fluctuations and trends of computation speeds. These real-world scenarios are formalized by our new universal computation model. Leveraging this model and new proof techniques, we discover tight lower bounds that apply to virtually all synchronous and asynchronous methods, including Minibatch SGD, Asynchronous SGD (Recht et al., 2011), and Picky SGD (Cohen et al., 2021). We show that these lower bounds, up to constant factors, are matched by the optimal Rennala SGD and Malenia SGD methods (Tyurin & Richt\'arik, 2023).

Author

Tyurin, Alexander

Published

2024

Language

EN