Skip to content

Opening book details…

Can I read On Noisy Evaluation in Federated Hyperparameter Tuning on EtoBox?

On Noisy Evaluation in Federated Hyperparameter Tuning by Kuo, Kevin; Thaker, Pratiksha; Khodak, Mikhail; Nguyen, John; Jiang, Daniel; Talwalkar, Ameet; Smith, Virginia is a scholarly article available to read on EtoBox.

What is On Noisy Evaluation in Federated Hyperparameter Tuning about?

Hyperparameter tuning is critical to the success of federated learning applications. Unfortunately, appropriately selecting hyperparameters is challenging in federated networks. Issues of scale, privacy, and heterogeneity introduce noise in the tuning process and make it difficult to evaluate the performance of various hyperparameters. In this work, we perform the first systematic study on the effect of noisy evaluation in federated hyperparameter tuning. We first identify and rigorously explore key sources of noise, including client subsampling, data and systems heterogeneity, and data privacy. Surprisingly, our results indicate that even small amounts of noise can significantly impact tuning methods-reducing the performance of state-of-the-art approaches to that of naive baselines. To address noisy evaluation in such scenarios, we propose a simple and effective approach that leverages public proxy data to boost the evaluation signal. Our work establishes general challenges, baselines, and best practices for future work in federated hyperparameter tuning.

Author
Kuo, Kevin; Thaker, Pratiksha; Khodak, Mikhail; Nguyen, John; Jiang, Daniel; Talwalkar, Ameet; Smith, Virginia
Published
2022
Language
EN

More by Kuo, Kevin; Thaker, Pratiksha; Khodak, Mikhail; Nguyen, John; Jiang, Daniel; Talwalkar, Ameet; Smith, Virginia

Browse all works by Kuo, Kevin; Thaker, Pratiksha; Khodak, Mikhail; Nguyen, John; Jiang, Daniel; Talwalkar, Ameet; Smith, Virginia