Can I read MEGL: Multimodal Explanation-Guided Learning on EtoBox?

MEGL: Multimodal Explanation-Guided Learning by Zhang, Yifei; Jiang, Tianxu; Pan, Bo; Wang, Jingyu; Bai, Guangji; Zhao, Liang is a scholarly article available to read on EtoBox.

What is MEGL: Multimodal Explanation-Guided Learning about?

Explaining the decision-making processes of Artificial Intelligence (AI) models is crucial for addressing their "black box" nature, particularly in tasks like image classification. Traditional eXplainable AI (XAI) methods typically rely on unimodal explanations, either visual or textual, each with inherent limitations. Visual explanations highlight key regions but often lack rationale, while textual explanations provide context without spatial grounding. Further, both explanation types can be inconsistent or incomplete, limiting their reliability. To address these challenges, we propose a novel Multimodal Explanation-Guided Learning (MEGL) framework that leverages both visual and textual explanations to enhance model interpretability and improve classification performance. Our Saliency-Driven Textual Grounding (SDTG) approach integrates spatial information from visual explanations into textual rationales, providing spatially grounded and contextually rich explanations. Additionally, we introduce Textual Supervision on Visual Explanations to align visual explanations with textual rationales, even in cases where ground truth visual annotations are missing. A Visual Explanation Distri

Author

Zhang, Yifei; Jiang, Tianxu; Pan, Bo; Wang, Jingyu; Bai, Guangji; Zhao, Liang

Published

2024

Language

EN