Can I read Rethinking Visual Prompt Learning as Masked Visual Token Modeling on EtoBox?

Rethinking Visual Prompt Learning as Masked Visual Token Modeling by Liao, Ning; Shi, Bowen; Zhang, Xiaopeng; Cao, Min; Yan, Junchi; Tian, Qi is a scholarly article available to read on EtoBox.

What is Rethinking Visual Prompt Learning as Masked Visual Token Modeling about?

Prompt learning has achieved great success in efficiently exploiting large-scale pre-trained models in natural language processing (NLP). It reformulates the downstream tasks as the generative pre-training ones to achieve consistency, thus improving the performance stably. However, when transferring it to the vision area, current visual prompt learning methods are almost designed on discriminative pre-trained models, and there is also a lack of careful design to unify the forms of pre-training and downstream tasks. To explore prompt learning on the generative pre-trained visual model, as well as keeping the task consistency, we propose Visual Prompt learning as masked visual Token Modeling (VPTM) to transform the downstream visual classification into the pre-trained masked visual token prediction. In addition, we develop the prototypical verbalizer for mapping the predicted visual token with implicit semantics to explicit downstream labels. To our best knowledge, VPTM is the first visual prompt method on the generative pre-trained visual model, which achieves consistency between pre-training and downstream visual classification by task reformulation. Experiments show that VPTM outp

Author

Liao, Ning; Shi, Bowen; Zhang, Xiaopeng; Cao, Min; Yan, Junchi; Tian, Qi

Published

2023

Language

EN