Can I read Embracing Diversity: Interpretable Zero-shot classification beyond one vector per class on EtoBox?

Embracing Diversity: Interpretable Zero-shot classification beyond one vector per class by Moayeri, Mazda; Rabbat, Michael; Ibrahim, Mark; Bouchacourt, Diane is a scholarly article available to read on EtoBox.

What is Embracing Diversity: Interpretable Zero-shot classification beyond one vector per class about?

Vision-language models enable open-world classification of objects without the need for any retraining. While this zero-shot paradigm marks a significant advance, even today's best models exhibit skewed performance when objects are dissimilar from their typical depiction. Real world objects such as pears appear in a variety of forms -- from diced to whole, on a table or in a bowl -- yet standard VLM classifiers map all instances of a class to a \it{single vector based on the class label}. We argue that to represent this rich diversity within a class, zero-shot classification should move beyond a single vector. We propose a method to encode and account for diversity within a class using inferred attributes, still in the zero-shot setting without retraining. We find our method consistently outperforms standard zero-shot classification over a large suite of datasets encompassing hierarchies, diverse object states, and real-world geographic diversity, as well finer-grained datasets where intra-class diversity may be less prevalent. Importantly, our method is inherently interpretable, offering faithful explanations for each inference to facilitate model debugging and enhance transparenc

Author

Moayeri, Mazda; Rabbat, Michael; Ibrahim, Mark; Bouchacourt, Diane

Published

2024

Language

EN