Can I read COCO is "ALL'' You Need for Visual Instruction Fine-tuning on EtoBox?

COCO is "ALL'' You Need for Visual Instruction Fine-tuning by Han, Xiaotian; Wang, Yiqi; Zhai, Bohan; You, Quanzeng; Yang, Hongxia is a scholarly article available to read on EtoBox.

What is COCO is "ALL'' You Need for Visual Instruction Fine-tuning about?

Multi-modal Large Language Models (MLLMs) are increasingly prominent in the field of artificial intelligence. Visual instruction fine-tuning (IFT) is a vital process for aligning MLLMs' output with user's intentions. High-quality and diversified instruction following data is the key to this fine-tuning process. Recent studies propose to construct visual IFT datasets through a multifaceted approach: transforming existing datasets with rule-based templates, employing GPT-4 for rewriting annotations, and utilizing GPT-4V for visual dataset pseudo-labeling. LLaVA-1.5 adopted similar approach and construct LLaVA-mix-665k, which is one of the simplest, most widely used, yet most effective IFT datasets today. Notably, when properly fine-tuned with this dataset, MLLMs can achieve state-of-the-art performance on several benchmarks. However, we noticed that models trained with this dataset often struggle to follow user instructions properly in multi-round dialog. In addition, tradition caption and VQA evaluation benchmarks, with their closed-form evaluation structure, are not fully equipped to assess the capabilities of modern open-ended generative MLLMs. This problem is not unique to the LL

Author

Han, Xiaotian; Wang, Yiqi; Zhai, Bohan; You, Quanzeng; Yang, Hongxia

Published

2024

Language

EN