About this document
Understanding the Vanishing Gradient Problem by Hanan Rosenthal is a document available to read on EtoBox.
The document discusses the vanishing gradient problem in deep neural networks using sigmoid activation functions, highlighting that the derivatives are always less than 1, which leads to exponentially shrinking gradients as layers increase. A numerical example illustrates that with 10 layers, the gradient approaches zero, halting learning. It concludes that modern deep networks avoid sigmoid activations in favor of ReLU, which maintains gradient flow.
- Author
- Hanan Rosenthal
- Language
- EN