Can I read Unsupervised Prototype Adapter for Vision-Language Models on EtoBox?

Unsupervised Prototype Adapter for Vision-Language Models by Zhang, Yi; Zhang, Ce; Hu, Xueting; He, Zhihai is a scholarly article available to read on EtoBox.

What is Unsupervised Prototype Adapter for Vision-Language Models about?

Recently, large-scale pre-trained vision-language models (e.g. CLIP and ALIGN) have demonstrated remarkable effectiveness in acquiring transferable visual representations. To leverage the valuable knowledge encoded within these models for downstream tasks, several fine-tuning approaches, including prompt tuning methods and adapter-based methods, have been developed to adapt vision-language models effectively with supervision. However, these methods rely on the availability of annotated samples, which can be labor-intensive and time-consuming to acquire, thus limiting scalability. To address this issue, in this work, we design an unsupervised fine-tuning approach for vision-language models called Unsupervised Prototype Adapter (UP-Adapter). Specifically, for the unannotated target datasets, we leverage the text-image aligning capability of CLIP to automatically select the most confident samples for each class. Utilizing these selected samples, we generate class prototypes, which serve as the initialization for the learnable prototype model. After fine-tuning, the prototype model prediction is combined with the original CLIP's prediction by a residual connection to perform downstream

Author

Zhang, Yi; Zhang, Ce; Hu, Xueting; He, Zhihai

Published

2023

Language

EN