Can I read Random ReLU Neural Networks as Non-Gaussian Processes on EtoBox?

Random ReLU Neural Networks as Non-Gaussian Processes by Parhi, Rahul; Bohra, Pakshal; Biari, Ayoub El; Pourya, Mehrsa; Unser, Michael is a scholarly article available to read on EtoBox.

What is Random ReLU Neural Networks as Non-Gaussian Processes about?

We consider a large class of shallow neural networks with randomly initialized parameters and rectified linear unit activation functions. We prove that these random neural networks are well-defined non-Gaussian processes. As a by-product, we demonstrate that these networks are solutions to stochastic differential equations driven by impulsive white noise (combinations of random Dirac measures). These processes are parameterized by the law of the weights and biases as well as the density of activation thresholds in each bounded region of the input domain. We prove that these processes are isotropic and wide-sense self-similar with Hurst exponent 3/2. We also derive a remarkably simple closed-form expression for their autocovariance function. Our results are fundamentally different from prior work in that we consider a non-asymptotic viewpoint: The number of neurons in each bounded region of the input domain (i.e., the width) is itself a random variable with a Poisson law with mean proportional to the density parameter. Finally, we show that, under suitable hypotheses, as the expected width tends to infinity, these processes can converge in law not only to Gaussian processes, but als

Author

Parhi, Rahul; Bohra, Pakshal; Biari, Ayoub El; Pourya, Mehrsa; Unser, Michael

Published

2024

Language

EN