Can I read Toward Modality Gap: Vision Prototype Learning for Weakly-supervised Semantic Segmentation with CLIP on EtoBox?

Toward Modality Gap: Vision Prototype Learning for Weakly-supervised Semantic Segmentation with CLIP by Xu, Zhongxing; Tang, Feilong; Chen, Zhe; Su, Yingxue; Zhao, Zhiyi; Zhang, Ge; Su, Jionglong; Ge, Zongyuan is a scholarly article available to read on EtoBox.

What is Toward Modality Gap: Vision Prototype Learning for Weakly-supervised Semantic Segmentation with CLIP about?

The application of Contrastive Language-Image Pre-training (CLIP) in Weakly Supervised Semantic Segmentation (WSSS) research powerful cross-modal semantic understanding capabilities. Existing methods attempt to optimize input text prompts for improved alignment of images and text, by finely adjusting text prototypes to facilitate semantic matching. Nevertheless, given the modality gap between text and vision spaces, the text prototypes employed by these methods have not effectively established a close correspondence with pixel-level vision features. In this work, our theoretical analysis indicates that the inherent modality gap results in misalignment of text and region features, and that this gap cannot be sufficiently reduced by minimizing contrast loss in CLIP. To mitigate the impact of the modality gap, we propose a Vision Prototype Learning (VPL) framework, by introducing more representative vision prototypes. The core of this framework is to learn class-specific vision prototypes in vision space with the help of text prototypes, for capturing high-quality localization maps. Moreover, we propose a regional semantic contrast module that contrasts regions embedding with correspo

Author

Xu, Zhongxing; Tang, Feilong; Chen, Zhe; Su, Yingxue; Zhao, Zhiyi; Zhang, Ge; Su, Jionglong; Ge, Zongyuan

Published

2024

Language

EN