Can I read Gradient Descent Happens in a Tiny Subspace on EtoBox?

Gradient Descent Happens in a Tiny Subspace by Gur-Ari, Guy; Roberts, Daniel A.; Dyer, Ethan is a scholarly article available to read on EtoBox.

What is Gradient Descent Happens in a Tiny Subspace about?

We show that in a variety of large-scale deep learning scenarios the gradient dynamically converges to a very small subspace after a short period of training. The subspace is spanned by a few top eigenvectors of the Hessian (equal to the number of classes in the dataset), and is mostly preserved over long periods of training. A simple argument then suggests that gradient descent may happen mostly in this subspace. We give an example of this effect in a solvable model of classification, and we comment on possible implications for optimization and learning.

Author

Gur-Ari, Guy; Roberts, Daniel A.; Dyer, Ethan

Published

2018

Language

EN