Can I read Pre-trained Vision-Language Models Learn Discoverable Visual Concepts on EtoBox?

Pre-trained Vision-Language Models Learn Discoverable Visual Concepts by Zang, Yuan; Yun, Tian; Tan, Hao; Bui, Trung; Sun, Chen is a scholarly article available to read on EtoBox.

What is Pre-trained Vision-Language Models Learn Discoverable Visual Concepts about?

Do vision-language models (VLMs) pre-trained to caption an image of a "durian" learn visual concepts such as "brown" (color) and "spiky" (texture) at the same time? We aim to answer this question as visual concepts learned "for free" would enable wide applications such as neuro-symbolic reasoning or human-interpretable object classification. We assume that the visual concepts, if captured by pre-trained VLMs, can be extracted by their vision-language interface with text-based concept prompts. We observe that recent works prompting VLMs with concepts often differ in their strategies to define and evaluate the visual concepts, leading to conflicting conclusions. We propose a new concept definition strategy based on two observations: First, certain concept prompts include shortcuts that recognize correct concepts for wrong reasons; Second, multimodal information (e.g. visual discriminativeness, and textual knowledge) should be leveraged when selecting the concepts. Our proposed concept discovery and learning (CDL) framework is thus designed to identify a diverse list of generic visual concepts (e.g. "spiky" as opposed to "spiky durian"), which are ranked and selected based on visual a

Author

Zang, Yuan; Yun, Tian; Tan, Hao; Bui, Trung; Sun, Chen

Published

2024

Language

EN