Can I read When CLIP meets cross-modal hashing retrieval: A new strong baseline on EtoBox?

When CLIP meets cross-modal hashing retrieval: A new strong baseline by Xinyu Xia; Guohua Dong; Fengling Li; Lei Zhu; Xiaomin Ying is a Computer Science article available to read on EtoBox.

What is When CLIP meets cross-modal hashing retrieval: A new strong baseline about?

Recent days witness significant progress in various multi-modal tasks made by Contrastive Language-Image Pre-training (CLIP), a multi-modal large-scale model that learns visual representations from natural language supervision. However, the potential effects of CLIP on cross-modal hashing retrieval has not been investigated yet. In this paper, we for the first time explore the effects of CLIP on cross-modal hashing retrieval performance and propose a simple but strong baseline Unsupervised Contrastive Multi-modal Fusion Hashing network (UCMFH). We first extract the off-the-shelf visual and linguistic features from the CLIP model, as the input sources for cross-modal hashing functions. To further mitigate the semantic gap between the image and text features, we design an effective contrastive multi-modal learning module that leverages a multimodal fusion transformer encoder supervising by a contrastive loss, to enhance modality interaction while improving the semantic representation of each modality. Furthermore, we design a contrastive hash learning module to produce high-quality modal-correlated hash codes. Experiments show that significant performance improvement can be made by o

Who reads When CLIP meets cross-modal hashing retrieval: A new strong baseline?

Author

Xinyu Xia; Guohua Dong; Fengling Li; Lei Zhu; Xiaomin Ying

Publisher

Elsevier BV

Published

2023

Language

EN

Field

Computer Science (Physical Sciences)