Can I read Augmenting Iterative Trajectory for Bilevel Optimization: Methodology, Analysis and Extensions on EtoBox?

Augmenting Iterative Trajectory for Bilevel Optimization: Methodology, Analysis and Extensions by Liu, Risheng; Liu, Yaohua; Zeng, Shangzhi; Zhang, Jin is a scholarly article available to read on EtoBox.

What is Augmenting Iterative Trajectory for Bilevel Optimization: Methodology, Analysis and Extensions about?

In recent years, there has been a surge of machine learning applications developed with hierarchical structure, which can be approached from Bi-Level Optimization (BLO) perspective. However, most existing gradient-based methods overlook the interdependence between hyper-gradient calculation and Lower-Level (LL) iterative trajectory, focusing solely on the former. Consequently, convergence theory is constructed with restrictive LL assumptions, which are often challenging to satisfy in real-world scenarios. In this work, we thoroughly analyze the constructed iterative trajectory, and highlight two deficiencies, including empirically chosen initialization and default use of entire trajectory for hyper-gradient calculation. To address these issues, we incrementally introduce two augmentation techniques including Initialization Auxiliary (IA) and Pessimistic Trajectory Truncation (PTT), and investigate various extension strategies such as prior regularization, different iterative mapping schemes and acceleration dynamics to construct Augmented Iterative Trajectory (AIT) for corresponding BLO scenarios (e.g., LL convexity and LL non-convexity). Theoretically, we provide convergence analy

Author

Liu, Risheng; Liu, Yaohua; Zeng, Shangzhi; Zhang, Jin

Published

2023

Language

EN