Can I read Non-asymptotic estimates for TUSLA algorithm for non-convex learning with applications to neural networks with ReLU activation function on EtoBox?

Non-asymptotic estimates for TUSLA algorithm for non-convex learning with applications to neural networks with ReLU activation function by Lim, Dong-Young; Neufeld, Ariel; Sabanis, Sotirios; Zhang, Ying is a scholarly article available to read on EtoBox.

What is Non-asymptotic estimates for TUSLA algorithm for non-convex learning with applications to neural networks with ReLU activation function about?

We consider non-convex stochastic optimization problems where the objective functions have super-linearly growing and discontinuous stochastic gradients. In such a setting, we provide a non-asymptotic analysis for the tamed unadjusted stochastic Langevin algorithm (TUSLA) introduced in Lovas et al. (2020). In particular, we establish non-asymptotic error bounds for the TUSLA algorithm in Wasserstein-1 and Wasserstein-2 distances. The latter result enables us to further derive non-asymptotic estimates for the expected excess risk. To illustrate the applicability of the main results, we consider an example from transfer learning with ReLU neural networks, which represents a key paradigm in machine learning. Numerical experiments are presented for the aforementioned example which support our theoretical findings. Hence, in this setting, we demonstrate both theoretically and numerically that the TUSLA algorithm can solve the optimization problem involving neural networks with ReLU activation function. Besides, we provide simulation results for synthetic examples where popular algorithms, e.g. ADAM, AMSGrad, RMSProp, and (vanilla) stochastic gradient descent (SGD) algorithm, may fail to

Author

Lim, Dong-Young; Neufeld, Ariel; Sabanis, Sotirios; Zhang, Ying

Published

2021

Language

EN