Can I read JourneyBench: A Challenging One-Stop Vision-Language Understanding Benchmark of Generated Images on EtoBox?

JourneyBench: A Challenging One-Stop Vision-Language Understanding Benchmark of Generated Images by Wang, Zhecan; Liu, Junzhang; Tang, Chia-Wei; Alomari, Hani; Sivakumar, Anushka; Sun, Rui; Li, Wenhao; Atabuzzaman, Md.; Ayyubi, Hammad; You, Haoxuan; Ishmam, Alvi; Chang, Kai-Wei; Chang, Shih-Fu; Thomas, Chris is a scholarly article available to read on EtoBox.

What is JourneyBench: A Challenging One-Stop Vision-Language Understanding Benchmark of Generated Images about?

Existing vision-language understanding benchmarks largely consist of images of objects in their usual contexts. As a consequence, recent multimodal large language models can perform well with only a shallow visual understanding by relying on background language biases. Thus, strong performance on these benchmarks does not necessarily correlate with strong visual understanding. In this paper, we release JourneyBench, a comprehensive human-annotated benchmark of generated images designed to assess the model's fine-grained multimodal reasoning abilities across five tasks: complementary multimodal chain of thought, multi-image VQA, imaginary image captioning, VQA with hallucination triggers, and fine-grained retrieval with sample-specific distractors. Unlike existing benchmarks, JourneyBench explicitly requires fine-grained multimodal reasoning in unusual imaginary scenarios where language bias and holistic image gist are insufficient. We benchmark state-of-the-art models on JourneyBench and analyze performance along a number of fine-grained dimensions. Results across all five tasks show that JourneyBench is exceptionally challenging for even the best models, indicating that models' vi

Author

Wang, Zhecan; Liu, Junzhang; Tang, Chia-Wei; Alomari, Hani; Sivakumar, Anushka; Sun, Rui; Li, Wenhao; Atabuzzaman, Md.; Ayyubi, Hammad; You, Haoxuan; Ishmam, Alvi; Chang, Kai-Wei; Chang, Shih-Fu; Thomas, Chris

Published

2024

Language

EN