Can I read Vision-Language Modelling For Radiological Imaging and Reports In The Low Data Regime on EtoBox?

Vision-Language Modelling For Radiological Imaging and Reports In The Low Data Regime by Windsor, Rhydian; Jamaludin, Amir; Kadir, Timor; Zisserman, Andrew is a scholarly article available to read on EtoBox.

What is Vision-Language Modelling For Radiological Imaging and Reports In The Low Data Regime about?

This paper explores training medical vision-language models (VLMs) -- where the visual and language inputs are embedded into a common space -- with a particular focus on scenarios where training data is limited, as is often the case in clinical datasets. We explore several candidate methods to improve low-data performance, including: (i) adapting generic pre-trained models to novel image and text domains (i.e. medical imaging and reports) via unimodal self-supervision; (ii) using local (e.g. GLoRIA) & global (e.g. InfoNCE) contrastive loss functions as well as a combination of the two; (iii) extra supervision during VLM training, via: (a) image- and text-only self-supervision, and (b) creating additional positive image-text pairs for training through augmentation and nearest-neighbour search. Using text-to-image retrieval as a benchmark, we evaluate the performance of these methods with variable sized training datasets of paired chest X-rays and radiological reports. Combined, they significantly improve retrieval compared to fine-tuning CLIP, roughly equivalent to training with the data. A similar pattern is found in the downstream task classification of CXR-related conditions with

Author

Windsor, Rhydian; Jamaludin, Amir; Kadir, Timor; Zisserman, Andrew

Published

2023

Language

EN