Can I read Calling a Spade a Heart: Gaslighting Multimodal Large Language Models via Negation on EtoBox?

Calling a Spade a Heart: Gaslighting Multimodal Large Language Models via Negation by Zhu, Bin; Qi, Huiyan; Gui, Yinxuan; Chen, Jingjing; Ngo, Chong-Wah; Lim, Ee-Peng is a scholarly article available to read on EtoBox.

What is Calling a Spade a Heart: Gaslighting Multimodal Large Language Models via Negation about?

Multimodal Large Language Models (MLLMs) have exhibited remarkable advancements in integrating different modalities, excelling in complex understanding and generation tasks. Despite their success, MLLMs remain vulnerable to conversational adversarial inputs, particularly negation arguments. This paper systematically evaluates state-of-the-art MLLMs across diverse benchmarks, revealing significant performance drops when negation arguments are introduced to initially correct responses. Notably, we introduce the first benchmark GaslightingBench, specifically designed to evaluate the vulnerability of MLLMs to negation arguments. GaslightingBench consists of multiple-choice questions curated from existing datasets, along with generated negation prompts across 20 diverse categories. Throughout extensive evaluation, we find that proprietary models such as Gemini-1.5-flash, GPT-4o and Claude-3.5-Sonnet demonstrate better resilience compared to open-source counterparts like Qwen2-VL and LLaVA. However, all evaluated MLLMs struggle to maintain logical consistency under negation arguments during conversation. Our findings provide critical insights for improving the robustness of MLLMs against

Author

Zhu, Bin; Qi, Huiyan; Gui, Yinxuan; Chen, Jingjing; Ngo, Chong-Wah; Lim, Ee-Peng

Published

2025

Language

EN