Can I read FMBench: Benchmarking Fairness in Multimodal Large Language Models on Medical Tasks on EtoBox?

FMBench: Benchmarking Fairness in Multimodal Large Language Models on Medical Tasks by Wu, Peiran; Liu, Che; Chen, Canyu; Li, Jun; Bercea, Cosmin I.; Arcucci, Rossella is a scholarly article available to read on EtoBox.

What is FMBench: Benchmarking Fairness in Multimodal Large Language Models on Medical Tasks about?

Advancements in Multimodal Large Language Models (MLLMs) have significantly improved medical task performance, such as Visual Question Answering (VQA) and Report Generation (RG). However, the fairness of these models across diverse demographic groups remains underexplored, despite its importance in healthcare. This oversight is partly due to the lack of demographic diversity in existing medical multimodal datasets, which complicates the evaluation of fairness. In response, we propose FMBench, the first benchmark designed to evaluate the fairness of MLLMs performance across diverse demographic attributes. FMBench has the following key features: 1: It includes four demographic attributes: race, ethnicity, language, and gender, across two tasks, VQA and RG, under zero-shot settings. 2: Our VQA task is free-form, enhancing real-world applicability and mitigating the biases associated with predefined choices. 3: We utilize both lexical metrics and LLM-based metrics, aligned with clinical evaluations, to assess models not only for linguistic accuracy but also from a clinical perspective. Furthermore, we introduce a new metric, Fairness-Aware Performance (FAP), to evaluate how fairly MLLM

Author

Wu, Peiran; Liu, Che; Chen, Canyu; Li, Jun; Bercea, Cosmin I.; Arcucci, Rossella

Published

2024

Language

EN