Can I read Statistical physics of deep neural networks: Initialization toward optimal channels on EtoBox?

Statistical physics of deep neural networks: Initialization toward optimal channels by Kangyu Weng; Aohua Cheng; Ziyang Zhang; Pei Sun; Yang Tian is a Physics and Astronomy article available to read on EtoBox.

What is Statistical physics of deep neural networks: Initialization toward optimal channels about?

In deep learning, neural networks serve as noisy channels between input data and its latent representation. This perspective naturally relates deep learning with the pursuit of constructing channels with optimal performance in information transmission and representation. While considerable efforts are concentrated on realizing optimal channel properties during network optimization, we study a frequently overlooked possibility that neural networks can be initialized toward optimal channels. Our theory, consistent with experimental validation, identifies primary mechanics underlying this unknown possibility and suggests intrinsic connections between statistical physics and deep learning. Unlike the conventional theories that characterize neural networks applying the classic mean-field approximation, we offer analytic proof that this extensively applied simplification scheme is not appropriate in studying neural networks as information channels. To fill this gap, we develop a restricted mean-field framework applicable for characterizing the limiting behaviors of information propagation in neural networks without strong assumptions on inputs. Based on it, we propose an analytic theory

Who reads Statistical physics of deep neural networks: Initialization toward optimal channels?

Author

Kangyu Weng; Aohua Cheng; Ziyang Zhang; Pei Sun; Yang Tian

Publisher

American Physical Society (APS)

Published

2023

Language

EN

Field

Physics and Astronomy (Physical Sciences)