Can I read mSAM: Micro-Batch-Averaged Sharpness-Aware Minimization on EtoBox?

mSAM: Micro-Batch-Averaged Sharpness-Aware Minimization by Behdin, Kayhan; Song, Qingquan; Gupta, Aman; Keerthi, Sathiya; Acharya, Ayan; Ocejo, Borja; Dexter, Gregory; Khanna, Rajiv; Durfee, David; Mazumder, Rahul is a scholarly article available to read on EtoBox.

What is mSAM: Micro-Batch-Averaged Sharpness-Aware Minimization about?

Modern deep learning models are over-parameterized, where different optima can result in widely varying generalization performance. The Sharpness-Aware Minimization (SAM) technique modifies the fundamental loss function that steers gradient descent methods toward flatter minima, which are believed to exhibit enhanced generalization prowess. Our study delves into a specific variant of SAM known as micro-batch SAM (mSAM). This variation involves aggregating updates derived from adversarial perturbations across multiple shards (micro-batches) of a mini-batch during training. We extend a recently developed and well-studied general framework for flatness analysis to theoretically show that SAM achieves flatter minima than SGD, and mSAM achieves even flatter minima than SAM. We provide a thorough empirical evaluation of various image classification and natural language processing tasks to substantiate this theoretical advancement. We also show that contrary to previous work, mSAM can be implemented in a flexible and parallelizable manner without significantly increasing computational costs. Our implementation of mSAM yields superior generalization performance across a wide range of tasks

Author

Behdin, Kayhan; Song, Qingquan; Gupta, Aman; Keerthi, Sathiya; Acharya, Ayan; Ocejo, Borja; Dexter, Gregory; Khanna, Rajiv; Durfee, David; Mazumder, Rahul

Published

2023

Language

EN