Can I read HUB: Guiding Learned Optimizers with Continuous Prompt Tuning on EtoBox?

HUB: Guiding Learned Optimizers with Continuous Prompt Tuning by Dai, Gaole; Wu, Wei; Wang, Ziyu; Fu, Jie; Zhang, Shanghang; Huang, Tiejun is a scholarly article available to read on EtoBox.

What is HUB: Guiding Learned Optimizers with Continuous Prompt Tuning about?

Learned optimizers are a crucial component of meta-learning. Recent advancements in scalable learned optimizers have demonstrated their superior performance over hand-designed optimizers in various tasks. However, certain characteristics of these models, such as an unstable learning curve, limited ability to handle unseen tasks and network architectures, difficult-to-control behaviours, and poor performance in fine-tuning tasks impede their widespread adoption. To tackle the issue of generalization in scalable learned optimizers, we propose a hybrid-update-based (HUB) optimization strategy inspired by recent advancements in hard prompt tuning and result selection techniques used in large language and vision models. This approach can be easily applied to any task that involves hand-designed or learned optimizer. By incorporating hand-designed optimizers as the second component in our hybrid approach, we are able to retain the benefits of learned optimizers while stabilizing the training process and, more importantly, improving testing performance. We validate our design through a total of 17 tasks, consisting of thirteen training from scratch and four fine-tuning settings. These tas

Author

Dai, Gaole; Wu, Wei; Wang, Ziyu; Fu, Jie; Zhang, Shanghang; Huang, Tiejun

Published

2023

Language

EN