Can I read Knowledge Acquisition Disentanglement for Knowledge-based Visual Question Answering with Large Language Models on EtoBox?

Knowledge Acquisition Disentanglement for Knowledge-based Visual Question Answering with Large Language Models by An, Wenbin; Tian, Feng; Nie, Jiahao; Shi, Wenkai; Lin, Haonan; Chen, Yan; Wang, QianYing; Wu, Yaqiang; Dai, Guang; Chen, Ping is a scholarly article available to read on EtoBox.

What is Knowledge Acquisition Disentanglement for Knowledge-based Visual Question Answering with Large Language Models about?

Knowledge-based Visual Question Answering (KVQA) requires both image and world knowledge to answer questions. Current methods first retrieve knowledge from the image and external knowledge base with the original complex question, then generate answers with Large Language Models (LLMs). However, since the original question contains complex elements that require knowledge from different sources, acquiring different kinds of knowledge in a coupled manner may confuse models and hinder them from retrieving precise knowledge. Furthermore, the ``forward-only'' answering process fails to explicitly capture the knowledge needs of LLMs, which can further hurt answering quality. To cope with the above limitations, we propose DKA: Disentangled Knowledge Acquisition from LLM feedback, a training-free framework that disentangles knowledge acquisition to avoid confusion and uses LLM's feedback to specify the required knowledge. Specifically, DKA requires LLMs to specify what knowledge they need to answer the question and decompose the original complex question into two simple sub-questions: Image-based sub-question and Knowledge-based sub-question. Then we use the two sub-questions to retrieve kn

Author

An, Wenbin; Tian, Feng; Nie, Jiahao; Shi, Wenkai; Lin, Haonan; Chen, Yan; Wang, QianYing; Wu, Yaqiang; Dai, Guang; Chen, Ping

Published

2024

Language

EN