Can I read Clip-Tuning: Towards Derivative-free Prompt Learning with a Mixture of Rewards on EtoBox?

Clip-Tuning: Towards Derivative-free Prompt Learning with a Mixture of Rewards by Chai, Yekun; Wang, Shuohuan; Sun, Yu; Tian, Hao; Wu, Hua; Wang, Haifeng is a scholarly article available to read on EtoBox.

What is Clip-Tuning: Towards Derivative-free Prompt Learning with a Mixture of Rewards about?

Derivative-free prompt learning has emerged as a lightweight alternative to prompt tuning, which only requires model inference to optimize the prompts. However, existing work did not take full advantage of the over-parameterized characteristics of large pre-trained language models (PLMs). In this paper, we propose Clip-Tuning, a simple yet effective method that adopts diverse frozen "thinned" networks of PLMs to obtain a mixture of rewards and thus advance the derivative-free prompt learning. The thinned networks consist of all the hidden units that survive a stationary dropout strategy, whose inference predictions reflect an ensemble of partial views over prompted training samples. Our method outperforms previous gradient-free prompt learning methods and achieves parity with gradient-based counterparts on seven language understanding benchmarks under few-shot settings.

Author

Chai, Yekun; Wang, Shuohuan; Sun, Yu; Tian, Hao; Wu, Hua; Wang, Haifeng

Published

2022

Language

EN