Can I read TaskGalaxy: Scaling Multi-modal Instruction Fine-tuning with Tens of Thousands Vision Task Types on EtoBox?

TaskGalaxy: Scaling Multi-modal Instruction Fine-tuning with Tens of Thousands Vision Task Types by Chen, Jiankang; Zhang, Tianke; Liu, Changyi; Ding, Haojie; Shi, Yaya; Cheng, Feng; Xiao, Huihui; Wen, Bin; Yang, Fan; Gao, Tingting; Zhang, Di is a scholarly article available to read on EtoBox.

What is TaskGalaxy: Scaling Multi-modal Instruction Fine-tuning with Tens of Thousands Vision Task Types about?

Multimodal visual language models are gaining prominence in open-world applications, driven by advancements in model architectures, training techniques, and high-quality data. However, their performance is often limited by insufficient task-specific data, leading to poor generalization and biased outputs. Existing efforts to increase task diversity in fine-tuning datasets are hindered by the labor-intensive process of manual task labeling, which typically produces only a few hundred task types. To address this, we propose TaskGalaxy, a large-scale multimodal instruction fine-tuning dataset comprising 19,227 hierarchical task types and 413,648 samples. TaskGalaxy utilizes GPT-4o to enrich task diversity by expanding from a small set of manually defined tasks, with CLIP and GPT-4o filtering those that best match open-source images, and generating relevant question-answer pairs. Multiple models are employed to ensure sample quality. This automated process enhances both task diversity and data quality, reducing manual intervention. Incorporating TaskGalaxy into LLaVA-v1.5 and InternVL-Chat-v1.0 models shows substantial performance improvements across 16 benchmarks, demonstrating the cr

Author

Chen, Jiankang; Zhang, Tianke; Liu, Changyi; Ding, Haojie; Shi, Yaya; Cheng, Feng; Xiao, Huihui; Wen, Bin; Yang, Fan; Gao, Tingting; Zhang, Di

Published

2025

Language

EN