Can I read Deep Networks Always Grok and Here is Why on EtoBox?

Deep Networks Always Grok and Here is Why by Humayun, Ahmed Imtiaz; Balestriero, Randall; Baraniuk, Richard is a scholarly article available to read on EtoBox.

What is Deep Networks Always Grok and Here is Why about?

Grokking, or delayed generalization, is a phenomenon where generalization in a deep neural network (DNN) occurs long after achieving near zero training error. Previous studies have reported the occurrence of grokking in specific controlled settings, such as DNNs initialized with large-norm parameters or transformers trained on algorithmic datasets. We demonstrate that grokking is actually much more widespread and materializes in a wide range of practical settings, such as training of a convolutional neural network (CNN) on CIFAR10 or a Resnet on Imagenette. We introduce the new concept of delayed robustness, whereby a DNN groks adversarial examples and becomes robust, long after interpolation and/or generalization. We develop an analytical explanation for the emergence of both delayed generalization and delayed robustness based on the local complexity of a DNN's input-output mapping. Our local complexity measures the density of so-called linear regions (aka, spline partition regions) that tile the DNN input space and serves as a utile progress measure for training. We provide the first evidence that, for classification problems, the linear regions undergo a phase transition during

Author

Humayun, Ahmed Imtiaz; Balestriero, Randall; Baraniuk, Richard

Published

2024

Language

EN