Can I read MP-GELU Bayesian Neural Networks: Moment Propagation by GELU Nonlinearity on EtoBox?

MP-GELU Bayesian Neural Networks: Moment Propagation by GELU Nonlinearity by Hirayama, Yuki; Takamaeda-Yamazaki, Sinya is a scholarly article available to read on EtoBox.

What is MP-GELU Bayesian Neural Networks: Moment Propagation by GELU Nonlinearity about?

Bayesian neural networks (BNNs) have been an important framework in the study of uncertainty quantification. Deterministic variational inference, one of the inference methods, utilizes moment propagation to compute the predictive distributions and objective functions. Unfortunately, deriving the moments requires computationally expensive Taylor expansion in nonlinear functions, such as a rectified linear unit (ReLU) or a sigmoid function. Therefore, a new nonlinear function that realizes faster moment propagation than conventional functions is required. In this paper, we propose a novel nonlinear function named moment propagating-Gaussian error linear unit (MP-GELU) that enables the fast derivation of first and second moments in BNNs. MP-GELU enables the analytical computation of moments by applying nonlinearity to the input statistics, thereby reducing the computationally expensive calculations required for nonlinear functions. In empirical experiments on regression tasks, we observed that the proposed MP-GELU provides higher prediction accuracy and better quality of uncertainty with faster execution than those of ReLU-based BNNs.

Author

Hirayama, Yuki; Takamaeda-Yamazaki, Sinya

Published

2022

Language

EN