Can I read FedNAR: Federated Optimization with Normalized Annealing Regularization on EtoBox?

FedNAR: Federated Optimization with Normalized Annealing Regularization by Li, Junbo; Li, Ang; Tian, Chong; Ho, Qirong; Xing, Eric P.; Wang, Hongyi is a scholarly article available to read on EtoBox.

What is FedNAR: Federated Optimization with Normalized Annealing Regularization about?

Weight decay is a standard technique to improve generalization performance in modern deep neural network optimization, and is also widely adopted in federated learning (FL) to prevent overfitting in local clients. In this paper, we first explore the choices of weight decay and identify that weight decay value appreciably influences the convergence of existing FL algorithms. While preventing overfitting is crucial, weight decay can introduce a different optimization goal towards the global objective, which is further amplified in FL due to multiple local updates and heterogeneous data distribution. To address this challenge, we develop {\it Federated optimization with Normalized Annealing Regularization} (FedNAR), a simple yet effective and versatile algorithmic plug-in that can be seamlessly integrated into any existing FL algorithms. Essentially, we regulate the magnitude of each update by performing co-clipping of the gradient and weight decay. We provide a comprehensive theoretical analysis of FedNAR's convergence rate and conduct extensive experiments on both vision and language datasets with different backbone federated optimization algorithms. Our experimental results consist

Author

Li, Junbo; Li, Ang; Tian, Chong; Ho, Qirong; Xing, Eric P.; Wang, Hongyi

Published

2023

Language

EN