Can I read Enhancing Medical Vision-language Contrastive Learning via Inter-matching Relation Modelling on EtoBox?

Enhancing Medical Vision-language Contrastive Learning via Inter-matching Relation Modelling by Li, Mingjian; Meng, Mingyuan; Fulham, Michael; Feng, David Dagan; Bi, Lei; Kim, Jinman is a scholarly article available to read on EtoBox.

What is Enhancing Medical Vision-language Contrastive Learning via Inter-matching Relation Modelling about?

Medical image representations can be learned through medical vision-language contrastive learning (mVLCL) where medical imaging reports are used as weak supervision through image-text alignment. These learned image representations can be transferred to and benefit various downstream medical vision tasks such as disease classification and segmentation. Recent mVLCL methods attempt to align image sub-regions and the report keywords as local-matchings. However, these methods aggregate all local-matchings via simple pooling operations while ignoring the inherent relations between them. These methods therefore fail to reason between local-matchings that are semantically related, e.g., local-matchings that correspond to the disease word and the location word (semantic-relations), and also fail to differentiate such clinically important local-matchings from others that correspond to less meaningful words, e.g., conjunction words (importance-relations). Hence, we propose a mVLCL method that models the inter-matching relations between local-matchings via a relation-enhanced contrastive learning framework (RECLF). In RECLF, we introduce a semantic-relation reasoning module (SRM) and an impor

Author

Li, Mingjian; Meng, Mingyuan; Fulham, Michael; Feng, David Dagan; Bi, Lei; Kim, Jinman

Published

2024

Language

EN